Extrem long save time at save items at BE

look at contao-community-alliance/dc-general#201

we found the first bottle necks...

1.) ToolboxFile.php

core/src/MetaModels/Helper/ToolboxFile.php

Line 700 in 08b5022

public static function convertValueToPath($varValue)

    public static function convertValueToPath($varValue)
    {
+        if (empty($varValue)) {
+            return '';
+        }
        $objFiles = FilesModel::findByPk($varValue);

2.) with "MM-1" and attribute select to "MM-2" which also a have relation per select to "MM-3" we found a strong increase of database queries... may be, we can "cache" all existing queries

core/src/MetaModels/MetaModel.php

Line 704 in 08b5022

    
           public function getIdsFromFilter($objFilter, $strSortBy = '', $intOffset = 0, $intLimit = 0, $strSortOrder = 'ASC')

...
   /**
     * The service container.
     *
     * @var IMetaModelsServiceContainer
     */
    protected $serviceContainer;
+
+    private $existingIds = [];

...

    public function getIdsFromFilter($objFilter, $strSortBy = '', $intOffset = 0, $intLimit = 0, $strSortOrder = 'ASC')
    {
        $arrFilteredIds = $this->getMatchingIds($objFilter);

        // If desired, sort the entries.
        if ($arrFilteredIds && $strSortBy != '') {
            if ($objSortAttribute = $this->getAttribute($strSortBy)) {
                $arrFilteredIds = $objSortAttribute->sortIds($arrFilteredIds, $strSortOrder);
            } elseif (in_array($strSortBy, array('id', 'pid', 'tstamp', 'sorting'))) {
+                if (array_intersect($arrFilteredIds, $this->existingIds) == $arrFilteredIds) {
+                    return $arrFilteredIds;
+                }
                // Sort by database values.
                $arrFilteredIds = $this
                    ->getDatabase()
                    ->prepare(
                        sprintf(
                            'SELECT id FROM %s WHERE id IN(%s) ORDER BY %s %s',
                            $this->getTableName(),
                            $this->buildDatabaseParameterList($arrFilteredIds),
                            $strSortBy,
                            $strSortOrder
                        )
                    )
                    ->execute($arrFilteredIds)
                    ->fetchEach('id');
+                $this->existingIds = array_merge($this->existingIds, $arrFilteredIds);
            } elseif ($strSortBy == 'random') {
                shuffle($arrFilteredIds);
            }
        }
...

I have this message :
Fatal error: Cannot redeclare MetaModels\MetaModel::$serviceContainer in /homepages/4/d406509772/htdocs/.../system/modules/metamodels/classes/src/MetaModels/MetaModel.php on line 692

I can't clear this line ?
protected $serviceContainer;

I noted that the save time increases when i have a lot of items in MM (5000 or 10 000 items) before or after the changes.

add only private $existingIds = [];

I have one page that accesses different MetaModels. It uses about 1200 database queries to load the site (Contao 4 / Symfony debug console).
I changed the following in the MetaModel.php and was able to reduce the number of database queries to about 400(!!!).

+   private $cache;

    public function findById($intId, $arrAttrOnly = array())
    {
         if (!$intId) {
             return null;
         }
+
+        if (isset($this->cache[$intId])) {
+            return $this->cache[$intId];
+        }
+
         $objItems = $this->getItemsWithId(array($intId), $arrAttrOnly);
         if ($objItems && $objItems->first()) {
-            return $objItems->getItem();
+            $this->cache[$intId] = $objItems->getItem();
+            return $this->cache[$intId];
         }
         return null;
     }

Maybe there is a way to reach the same result by using doctrine cache / APCu??

may be it´s a good idea to equalize the terms private $cache; to private $existingIds = [];

@richardhj

@richardhj Your change introduces a problem when $attrOnly is not empty, therefore we can not walk down that road. 😿

@discordier

@discordier Did you ever enabled APCu doctrine cache? Is this available for Contao?

As mentioned in #1352:

Fixed in MM 2.2

Read any tip to speed up MM: https://metamodels.readthedocs.io/de/latest/cookbook/tips/speed-up-backend-list.html

@discordier

Currently there is still the problem in the BE, if many attributes are selected to the filter and there are many items...

@discordier has solution proposals for this

Navigation Menu

Repository navigation

Extrem long save time at save items at BE #1153

26 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Extrem long save time at save items at BE #1153

Description

Activity

Damsgit commented on May 29, 2017

zonky2 commented on May 29, 2017

richardhj commented on Aug 3, 2018

zonky2 commented on Aug 3, 2018

discordier commented on Aug 3, 2018

richardhj commented on Aug 3, 2018

26 remaining items

zonky2 commented on Jan 23, 2022

zonky2 commented on Jan 23, 2022

zonky2 commented on Aug 23, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions