Skip to content

Upload exactly the server-validated package - #1806

Merged
PaulHax merged 8 commits into
mainfrom
fm-stack-00-upload-package-authority
Aug 4, 2026
Merged

PaulHax merged 8 commits into
mainfrom
fm-stack-00-upload-package-authority

Conversation

@PaulHax

@PaulHax PaulHax commented Jul 27, 2026 •

Copy link
Copy Markdown
Collaborator

First of a four-PR stack. Foundation only: no attachment semantics or frame metadata yet.

Summary

Make server validation authoritative for the browser upload package.

image

Upload validated one interpretation of the selected files, then rebuilt the package in the browser
with separate rules. The two could disagree: an accepted file could be dropped, an ignored file
could still upload, and ordinary and multicamera imports kept different side files.

validate_files now returns semantic roles (media, annotations, dataset configuration) plus
ignored with a per-file reason. The browser uploads the selection minus the ignored names — one
derivation, no second classifier, same contract for ordinary and multicamera camera-folder uploads.
Files that upload today still upload; a skipped file is now listed with its reason instead of
vanishing.

The per-role slots stay; their contents and meaning change. processImport is deleted and
suggestUploadSlots replaces it as a convenience split that is never trusted — slots are
re-validated at Start upload, and the server decides media type, folder name, fan-out, and what
uploads. A rejected selection now yields a row in an error state rather than a discarded pick,
since the slot editor is the only place to fix it without reopening the OS picker.

Four intentional behavior changes ride along:

  • The media file dialog offers every side file a slot can hold. Its filter was built from the
    annotation extensions alone, so .yml, .yaml, .json, and .csv were selectable but .txt
    was not — the browser silently withheld a file the server would have classified, which is the
    drift this PR exists to remove. The filter is now the union of the annotation and metadata
    attachment extensions.
  • Response shape. The flat media/annotations lists are replaced by roles + ignored. Both
    in-tree consumers (browser upload, girder-worker zip path) are updated here; external callers of
    this endpoint must adopt the new shape.
  • Stricter annotation multiplicity. Five selections main accepted and then mishandled via
    silent last-write-wins or an ambiguous config pick are now rejected explicitly: two annotation
    JSONs, CSV + JSON, YAML + JSON, more than one YAML, two config JSONs. Everything else validates as
    before.
  • Row retirement. UploadGirder emitted a row index while Upload expected the row object, so
    splice(-1, 1) retired the last queued row instead of the finished one. A pre-existing main
    bug, fixed here because per-row error state makes retiring the wrong row destructive.

The zip path is unchanged: validation gates the whole archive and every extracted file uploads. Only
interactive upload drops ignored files — the same files main already dropped, silently.

Reviewer focus

  • Server and browser classification cannot drift after validation, and the file dialog no longer
    pre-empts it by withholding a selectable side file.
  • Ignored files stay visible below the pickers and never enter the upload request.
  • Accepted packages behave as on main; the five new rejections are the intended change.
  • Multicamera flattening preserves every server-approved file, including the annotation and config
    side files main's media-only path dropped.
  • When a camera-folder file and a separately picked file share a name, the picked file uploads and
    the displaced folder copy is reported as ignored. Re-picking the folder's own file displaces
    nothing.
  • An edited slot changes what uploads: the package is rebuilt and re-validated at Start upload.
  • An error row stays correctable — slots editable, media slot unfiltered (the row's type is only the
    import menu opened until the server confirms it), never uploads into an unnamed folder.

Manual test

Uses okeanos-upload-authority-ignore from okeanos-frame-metadata-fixtures.zip, attached to this
PR — CC0 NOAA Okeanos Explorer media with synthetic metadata and annotations. Select all files in
the directory.

okeanos-frame-metadata-fixtures.zip

  1. Select all 17 files; confirm the 16 JPEGs validate as image-sequence media.
  2. Confirm notes.txt is listed under "Ignored (not uploaded)", below the pickers, with the
    unsupported-side-file reason.
  3. Create and open the dataset: 16 frames, and notes.txt uploaded as neither media, annotation,
    nor dataset config.
  4. Repeat through the folder/multicamera surface — identical ignored result and media set.

Classify each selected file into a semantic upload role on the server, then
upload exactly what that classification accepts and report the rest, instead
of re-deriving the set in the browser.
@BryonLewis
BryonLewis force-pushed the fm-stack-00-upload-package-authority branch from 114930e to 5741db5 Compare August 4, 2026 12:42
@BryonLewis

Copy link
Copy Markdown
Collaborator

Small Commit to make sure the pendingUploadType is set on server validation even if there are errors.

@BryonLewis

Copy link
Copy Markdown
Collaborator

The meta vs metadata naming in the code was confusing. I've renamed a bunch of areas from meta and metadata to config to make it easier to understand the true difference between the config file for configuration and the metadata file for frame-level metadata. It's a large change across a bunch of stuff but I think it makes everything more readable in the future. I've also updated the documentation to indicate the differences.

There were some locations where I wrote a 'meta.json' file on desktop, that is now a config.json but it has a fallback to load the older meta.json for legacy storage reasons on electron deployed versions.

@BryonLewis
BryonLewis force-pushed the fm-stack-00-upload-package-authority branch from bd5f4d6 to 3ad7c69 Compare August 4, 2026 14:29
@PaulHax

PaulHax commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

config is pretty generic sounding to. When i think config, i think its somethign that can be resued acrosss "datasets". what is the file exactly? its a representation of the dataset and its file assotations and state (and settings/preferances/config of course). Perhaps a name that catures that and is not include meta|data|config| or other generic words.

@BryonLewis

Copy link
Copy Markdown
Collaborator

I'ved updated it so there are three things now
Metadata - The new file that contains media or metadata passthrough information for pipelines
dataset.json - the stored file used on desktop for storing information about the file such as default cameras, media locations and other data that is intrinsic to the Dataset
config.json - the portable configuration file that has types, type-colors, confidence pairs and other settings that can be used amongst different DIVE datasets.

@BryonLewis

Copy link
Copy Markdown
Collaborator

There was an issue when importing regular files that had multiple '.' in them where it would filter out the png images. This fixes it so the legacy version of importing a folder still works.

@BryonLewis

BryonLewis commented Aug 4, 2026 •

Copy link
Copy Markdown
Collaborator

small change to padding for the error/warning messages

image image

@BryonLewis

Copy link
Copy Markdown
Collaborator

Reverting a change I made to web/girder where I swapped the image-sequence to using the openDirectory like on desktop instead of the classic multiFile select on web. There may come a time where we may want to have an option in the future for web to open a directory and then it auto populates the several file input field types. I decided this PR wasn't meant for this change so I'll hold off for another PR in the future to do that.

@BryonLewis BryonLewis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the changes and the big refactor to rename. I think the rename will help a bit with code confusion and the added documentation that explicitly calls out the different types will help as well.

I've tested this out per the specifications and it worked properly.
I also did several other tests of uploading including videos, mulitcam, stereo on desktop and web to confirm the functionality.

I experimented with making desktop/web aligned where desktop uses an openDirectory approach instead of multiFile. This may have beend one before because of a limitation with older electron file picker versions. The functionality is handy when you have a lot of files that need to be configured you can instead open the directory and attempt to let the system configure the annotation, meta, config, media files in one click instead of selection. Quickly decided against that as being out of scope for this PR. Maybe an option that can be configured in the future.

@PaulHax
PaulHax merged commit 8529e83 into main Aug 4, 2026
3 checks passed
@PaulHax
PaulHax deleted the fm-stack-00-upload-package-authority branch August 4, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants