Repository navigation
Consider using custom iterators to reduce memory usage #13473
Description
Following the performance and memory usage issue with 20k raised by @LukasReschke here #13434
I thought we could give a try using iterators instead of storing everything in memory.
The idea would be to wrap DB results in a custom iterator.
For every call to next() (or whatever it is called in PHP), the next entry is read and wrapped in whichever object we want (ex: FileInfo).
The file cache would then return an iterator instead of the complete set of results.
The list.php call for the web UI could then iterate over every element and format them individually into FileInfo and then JSON. This means we will never store the whole list in memory.
One trouble though is about sorting: we need the whole list for sorting. But maybe there's another way around this.
This pattern could be applied for other things like iterating over search results, users, etc.
What do you think ? @butonic @icewind1991 @DeepDiver1975 @bantu @LukasReschke
Activity
[-]Consider using iterators to reduce memory usage[/-][+]Consider using custom iterators to reduce memory usage[/+]butonic commentedon Jan 20, 2015
I'd prefer to expose page and size parameters to the client. With an iterator we would have to keep a database connection open to iterate over the whole result set. Latest changes to the search API started to introduce page and size, but they still need to be fully passed down to the filecache query. Currently only search_elastic fully exploits page & size. It can also be extended to sorting.
We could write an Iterator on the js side of things to encapsulate the subsequent ajax calls.
my 2 cents
butonic commentedon Jan 20, 2015
page,sizeandsortof course should be used consistently all over the place. Aside from files, listing gallery albums and music come to mind first.icewind1991 commentedon Jan 20, 2015
Sorting cant be easily passed all the way down since it's a reasonable complex sort (case insensitive, folders before files, natural sorting for numbers)
PVince81 commentedon Jan 20, 2015
and also NLS compatible sort, which also needs a custom PHP comparator.
butonic commentedon Jan 20, 2015
NLS compatible sort is handled by COLLATE in SQL.
hutchic commentedon Nov 7, 2015
Might be worth considering the CakePHP3 collection library
ownclouders commentedon Jan 13, 2018
Hey, this issue has been closed because the label
status/STALEis set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.(This is an automated comment from GitMate.io.)
lock commentedon Jul 31, 2019
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.