Keep MemoryFileSystem append writes private until transaction commit

[glaziermag]

Appending to an existing MemoryFileSystem file inside fs.transaction changes the stored bytes immediately, so an exception leaves the appended data behind. _open() returns the committed buffer for ab/a+b; this change appends to a copy instead, which is committed or discarded with the transaction.

Minimal reproduction on current master:

import fsspec

fs = fsspec.filesystem("memory", global_store=False, skip_instance_cache=True)
fs.pipe_file("target", b"original")
try:
    with fs.transaction:
        with fs.open("target", "ab") as stream:
            stream.write(b"-uncommitted")
        raise RuntimeError("abort transaction")
except RuntimeError:
    pass

assert fs.cat_file("target") == b"original"

Before the fix, the assertion fails with b'original-uncommitted'. The transaction guarantee says an uncaught exception discards the files and leaves target locations untouched.

Validation: 63 memory-backend tests and 7 local transaction tests pass. The regression covers commit/rollback, both append modes and shared/independent stores; all eight cases fail before the fix.

Prepared with AI assistance.

[martindurant]

You mean, the same file has been opened more than once within the transaction? I'm not sure that something we particularly account for in other places either.

[glaziermag]

Yes, e.g. two appends to the same file in one transaction. Without the lookup, the second open starts again from the committed bytes, so appending -1 then -2 to orig would commit orig-2 rather than orig-1-2 as on master. 4255a87 extends it to a file first written earlier in the same transaction. You're right that this isn't handled anywhere else, so if you'd rather keep it minimal I can drop the lookup and only copy on append.

[martindurant]

the second open starts again from the committed bytes,

I'm not sure that's wrong. Let's not process this here.

[glaziermag]

OK, dropped the lookup in 9ad4344, so this is now just the copy on append.