Repository navigation
[AI] Fix multipart memory limit for buffered files - #1522
puneetdixit200 wants to merge 1 commit into
Conversation
Allow multipart parsing to keep the existing per-file memory threshold without using that same threshold as the total parser memory limit. Fixes bottlepy#1470 ai-assisted-by: OpenAI GPT-5 License: Public Domain
eeshsaxena
left a comment
There was a problem hiding this comment.
The goal (not rejecting two legitimate files whose sizes sum above MEMFILE_MAX) is right, but dropping the mem_limit=self.MEMFILE_MAX argument entirely has a side effect worth a second look.
_MultipartParser.__init__ defaults mem_limit=2 ** 20 and enforces it in the parse loop (if part.size + mem_used > self.mem_limit: raise MultipartError("Memory limit reached.")). So after this change the total in-memory bound for a multipart POST is no longer tied to MEMFILE_MAX at all -- it becomes the hardcoded 1 MB default.
Two consequences:
- An app that lowers
MEMFILE_MAX(e.g. a memory-constrained service setting it to a few KB) no longer has that reflected in the total buffered-multipart limit; it silently gets 1 MB instead. memfile_limit(per-part spill-to-disk) andmem_limit(total in-memory) are genuinely different knobs, so coupling both toMEMFILE_MAXwas the actual bug -- but the fix leavesmem_limitunconfigurable rather than giving it its own source.
Would it be cleaner to pass an explicit total limit that scales with the configured value, e.g. mem_limit=max(self.MEMFILE_MAX, <sensible floor>), or expose a separate MEMFILE_MAX_TOTAL/mem_limit setting, so the total bound stays under the app`s control instead of reverting to the library default?
Fixes #1470.
This keeps
MEMFILE_MAXas the per-file in-memory threshold for multipart uploads, but stops reusing that same 100 KiB value as the parser-wide total memory limit. The parser now uses its existing larger total-memory default while each individual file part still spools to disk once it exceedsMEMFILE_MAX.I manually reviewed the diff and verified the regression path with:
python3 -m unittest test.test_environ.TestRequest.test_multipart_many_small_files_above_memfile_totalpython3 -m unittest test.test_environ test.test_multipartpython3 -m unittest discover -t . -s testgit diff --check HEAD~1..HEADAI disclosure: assisted by OpenAI GPT-5. I reviewed the change and test output before submitting.
License: I explicitly license this contribution as public domain.