Repository navigation
Huge performance problems (OOM + Execution time) when listing large folders #13434
Description
While debugging #12785 I had to create a lot of files in the user's data folder with the following script, to be precise I created 22559 files.
When now going to the file overview as the user where I created the files the Apache log gets sometimes filled with the following fatal errors such as:
[Sat Jan 17 21:59:24.553293 2015] [:error] [pid 20417] [client XXXXX:55157] PHP Fatal error: Out of memory (allocated 168296448) (tried to allocate 78 bytes) in /var/www/html/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Statement.php on line 231, referer: http://XXXXX/owncloud/index.php/apps/files/
[Sat Jan 17 22:00:19.838615 2015] [:error] [pid 20419] [client XXXXX:54882] PHP Fatal error: Out of memory (allocated 340262912) (tried to allocate 72 bytes) in /var/www/html/owncloud/lib/private/json.php on line 132, referer: http://XXXXX/owncloud/index.php/apps/files/
[Sat Jan 17 22:00:20.680055 2015] [:error] [pid 20368] [client XXXXX:54873] PHP Fatal error: Out of memory (allocated 306184192) (tried to allocate 99 bytes) in /var/www/html/owncloud/lib/private/connector/sabre/filesplugin.php on line 73, referer: http://XXXXX/owncloud/index.php/settings/admin
Thus it is effectively not possible to use ownCloud at all… Sometimes it works but then the AJAX request gathering the file list is very slow (apps/files/ajax/list.php requires more than 50 seconds) and requires way too much memory (more than 150MB for just getting the file list!) than it actually should, just take a look at the following profiling overview:
https://blackfire.io/profiles/e2ddd96d-08c1-4afc-8a2e-414b5a4cb5b4/graph
Memory
- We waste 24MB memory for
OC_Helper::mimetypeIcon - We waste 58.6MB memory for
OC_JSON::encode - We waste 62.9MB memory for
PDOStatement::fetchAll
CPU
- We waste more than 30 seconds on sorting and comparing the file names
- We waste more than 10 seconds on a method that should just format a date…
This is on current master on my local development machine (SQLite) but behaves identically as the 7.0.3 Ubuntu machine (MySQL) where I debug the config.php problem (big fun)
@DeepDiver1975 @karlitschek FYI – major performance problem.
Activity
[-]Out-of-memoy with a lot of files in a single folder[/-][+]Huge performance problems (OOM + Execution time) when listing large folders[/+]LukasReschke commentedon Jan 17, 2015
Ewww, Punic is actually really really slow… Let me see if I can find some quick wins in their code base…
LukasReschke commentedon Jan 17, 2015
Got a simple patch that gives us on average 20% performance win on time-manipulation related actions by using a simple array cache for the timezone. This gives us a 2 second performance gain here in this simple example and a lot of other gains in other cases. PR is upstream: punic/punic#30
Still not perfectly happy with the duration of all Punic related stuff … mhm…
Reuse the array key of mimetypes
LukasReschke commentedon Jan 18, 2015
Some patches for the mimetype icon problem: #13438
PVince81 commentedon Jan 19, 2015
Is that formatting the date for the web UI ? I think we can probably drop that completely in favor of timestamps.
But @DeepDiver1975 seemed to think we should rather use the ISO format here: #12639 (comment) so it might not be the right approach anyway.
LukasReschke commentedon Jan 19, 2015
It is. But those 10 seconds on dates are negligible when considering the 30 seconds on sorting stuff :( – Also using more than 50MB here just for JSON encoding is pretty … much … :)
PVince81 commentedon Jan 19, 2015
Some ideas to optimize the web UI workflow:
shift()(if that doesn't cause additional perf issues for PHP...)Ideally we should resort to pagination, but that will cause trouble with WebDAV...
LukasReschke commentedon Jan 22, 2015
Another 4 seconds are saved by optimizing NaturalSort, thanks @Xenopathic: #13511
RobinMcCorkell commentedon Mar 26, 2015
@LukasReschke Closed? Is this no longer a significant issue?
karlitschek commentedon Mar 26, 2015
@Xenopathic the assumption is that the issue is mostly fixed by the two referenced pull requests
LukasReschke commentedon Mar 26, 2015
(and also the referenced issue at #13473 is quite good)