Repository navigation
Upload exactly the server-validated package - #1806
Conversation
Classify each selected file into a semantic upload role on the server, then upload exactly what that classification accepts and report the rest, instead of re-deriving the set in the browser.
114930e to
5741db5
Compare
… longer creates the wrong folder type
|
Small Commit to make sure the pendingUploadType is set on server validation even if there are errors. |
|
The There were some locations where I wrote a 'meta.json' file on desktop, that is now a |
…ll references in code, and documentation
bd5f4d6 to
3ad7c69
Compare
|
config is pretty generic sounding to. When i think config, i think its somethign that can be resued acrosss "datasets". what is the file exactly? its a representation of the dataset and its file assotations and state (and settings/preferances/config of course). Perhaps a name that catures that and is not include meta|data|config| or other generic words. |
|
I'ved updated it so there are three things now |
…ames like 234243.23432.234324.png
|
There was an issue when importing regular files that had multiple '.' in them where it would filter out the png images. This fixes it so the legacy version of importing a folder still works. |
|
Reverting a change I made to web/girder where I swapped the image-sequence to using the openDirectory like on desktop instead of the classic multiFile select on web. There may come a time where we may want to have an option in the future for web to open a directory and then it auto populates the several file input field types. I decided this PR wasn't meant for this change so I'll hold off for another PR in the future to do that. |
BryonLewis
left a comment
There was a problem hiding this comment.
Sorry about the changes and the big refactor to rename. I think the rename will help a bit with code confusion and the added documentation that explicitly calls out the different types will help as well.
I've tested this out per the specifications and it worked properly.
I also did several other tests of uploading including videos, mulitcam, stereo on desktop and web to confirm the functionality.
I experimented with making desktop/web aligned where desktop uses an openDirectory approach instead of multiFile. This may have beend one before because of a limitation with older electron file picker versions. The functionality is handy when you have a lot of files that need to be configured you can instead open the directory and attempt to let the system configure the annotation, meta, config, media files in one click instead of selection. Quickly decided against that as being out of scope for this PR. Maybe an option that can be configured in the future.


First of a four-PR stack. Foundation only: no attachment semantics or frame metadata yet.
Summary
Make server validation authoritative for the browser upload package.
Upload validated one interpretation of the selected files, then rebuilt the package in the browser
with separate rules. The two could disagree: an accepted file could be dropped, an ignored file
could still upload, and ordinary and multicamera imports kept different side files.
validate_filesnow returns semanticroles(media, annotations, dataset configuration) plusignoredwith a per-file reason. The browser uploads the selection minus the ignored names — onederivation, no second classifier, same contract for ordinary and multicamera camera-folder uploads.
Files that upload today still upload; a skipped file is now listed with its reason instead of
vanishing.
The per-role slots stay; their contents and meaning change.
processImportis deleted andsuggestUploadSlotsreplaces it as a convenience split that is never trusted — slots arere-validated at Start upload, and the server decides media type, folder name, fan-out, and what
uploads. A rejected selection now yields a row in an error state rather than a discarded pick,
since the slot editor is the only place to fix it without reopening the OS picker.
Four intentional behavior changes ride along:
annotation extensions alone, so
.yml,.yaml,.json, and.csvwere selectable but.txtwas not — the browser silently withheld a file the server would have classified, which is the
drift this PR exists to remove. The filter is now the union of the annotation and metadata
attachment extensions.
media/annotationslists are replaced byroles+ignored. Bothin-tree consumers (browser upload, girder-worker zip path) are updated here; external callers of
this endpoint must adopt the new shape.
mainaccepted and then mishandled viasilent last-write-wins or an ambiguous config pick are now rejected explicitly: two annotation
JSONs, CSV + JSON, YAML + JSON, more than one YAML, two config JSONs. Everything else validates as
before.
UploadGirderemitted a row index whileUploadexpected the row object, sosplice(-1, 1)retired the last queued row instead of the finished one. A pre-existingmainbug, fixed here because per-row error state makes retiring the wrong row destructive.
The zip path is unchanged: validation gates the whole archive and every extracted file uploads. Only
interactive upload drops
ignoredfiles — the same filesmainalready dropped, silently.Reviewer focus
pre-empts it by withholding a selectable side file.
main; the five new rejections are the intended change.side files
main's media-only path dropped.the displaced folder copy is reported as ignored. Re-picking the folder's own file displaces
nothing.
import menu opened until the server confirms it), never uploads into an unnamed folder.
Manual test
Uses
okeanos-upload-authority-ignorefromokeanos-frame-metadata-fixtures.zip, attached to thisPR — CC0 NOAA Okeanos Explorer media with synthetic metadata and annotations. Select all files in
the directory.
okeanos-frame-metadata-fixtures.zip
notes.txtis listed under "Ignored (not uploaded)", below the pickers, with theunsupported-side-file reason.
notes.txtuploaded as neither media, annotation,nor dataset config.