Activity

1. csbrown-noaa

   added

   Type: Feature RequestNew feature or requestNew feature or request

   on Jun 4, 2026
2. BryonLewis commented on Jun 14, 2026

   

   BryonLewis

   on Jun 14, 2026

   Collaborator

   More actions

   The quickest path for something like this to work would involve something like the following:

   User Experience

   1. First you import media to something like Collections(Asset Store Import) -> Media (Girder Folder)
   2. Then you import your annotations to Collects(Asset Store Import) -> Annotations (Girder Folder)
   3. Now that you have 2 sibiling folders I would have some tool/REST endpoint where you can point to a folder of annotations and a folder of media and attempt to align them and import the annotations for the media
   4. For multiple annotations for the same Dataset this would require using the Annotation Sets feature. This was a tool for comparison but it allows for the same Dataset to have multiple annotations and to be able to compare them with each other.
   5. Will need to determine what we consider the main/default annotations vs the annotations that are annotation sets. Maybe I create a tool that allows

   Technical Details

   • For this to be possible I would require to know the path when importing. This means that the current import logic should probably record the bucket name/path for the image-sequence/video
     • For image-sequences I'll make the dataset have a bucket-path Metadata item that will point to the folder that contains the images. Each image could have it's own bucket path as well
     • For videos the created video folder will have to have the bucket-path set as well
   • /dive_rpc/import_bucket_annotations/{AnnotationfolderId} - with a parameter of the destination folder Id.
     • This searches the annotationFolderId for Viame.csv, track JSON or COCO JSON files and indexes them
     • Reads in the annotation files and checks for bucket references (gs:// or s3:// or some URL format for the annotation files)
     • Takes a look in the MediaFolderId and looks at the metadata fioeld of bucket-path and tries to match annotations with files
     • Interprets the first file as the main the annotation and then any additional files are made to be annotations sets using the filename or something else as a reference.

   Test Data Generation

   • In the ./samples/scripts/assetStoreImport folder update the script to generate and load into MinIO decoupled data
     • Update the generateSampleData.py script to make it so it could have an option to generate two folders instead of a folder with sibiling annotation data
       • The first folder would be ./media and that would contain the hierarchy of video/image-sequences
       • The next folder would be ./annotations and this would contain annotations for corresponding files in the media folder. The difference would be that for CSV and COCO JSON instead of using relative paths and filenames it will use some sort of minIO, gs, or s3 based reference.
     • For the minIOConfig.py script I would need an option to hose separated media and annotations and instead of creating a single bucket it would create two buckets (prbobably the same access/secret keys to make it easier) and these would be imported into MinIO for testing purposes
   • This should provide me with two buckets where I can import one to media and another into a collections folder

   Modification to AssetStore Import Logic

   • AssetStore importing logic needs to be updated to add the bucket-path to items as they are imported.
   • For images each image will have it's source bucket-path but the DIVE Dataset folder needs to have the bucket-path without the image filename on it so it can be indexed and retrieved easily
   • Also when data is moved (like videos moved into their own folders) they need to have the bucket-path attached

   REST Endpoint /dive_rpc/sync_bucket_annotations/AnnotationFolderId

   • I think this is probably best as an additional worker job because the number of annotations could be large
   • Job starts by searching the Annotation folder for all viameCSV, track JSON and COCO JSON files inside of it and creating a listing of them. Prints to the job log a list of these found annotation files
   • Now starts opening the annotation files and reads them to find gs:, s3:. http: or other paths to media that aren't relative but are absolute
   • Matches found paths to found files in the MediaFolderId as children. I want to make sure I only have to search known areas because searching a whole system could get time consuming.
   • The job log output notes found and especially mission annotations (ones that don't match media)
   • We can start importing data and assigning it to the DIVE Datasets
     • We have a mapping of annotation files to DIVE Dataset Ids
       • In the case of having multiple annotations associated with the same media the first one is imported as the default annotation and then subsequent runs are imported as annotation sets
     • During the import process the system should log what is happening for each dataset

   UI Implementation

   I'm not sure on this yet, but we need some UI for this to work instead of using the Swagger WebUI or REST endpoints. I'm not sure if I want to make this button happen in Girder or in the DIVE Admin interface.
   The main idea is that you have a dialog come up where you navigate to the media folder and you navigat to the annotation folder and hit import and it imports the data.
3. csbrown-noaa commented on Jun 15, 2026

   

   csbrown-noaa

   on Jun 15, 2026

   Author

   More actions

   Sounds like a heavy lift as is. Will discuss with the team. @BryonLewis thanks for the insights here.
4. BryonLewis commented on Jun 15, 2026

   

   BryonLewis

   on Jun 15, 2026

   Collaborator

   More actions

   This was more just some planning information for myself, I just wanted to store it with the issue for referencing.

   👍React with 👍1Reacted by csbrown-noaa
5. csbrown-noaa commented on Jun 15, 2026

   

   csbrown-noaa

   on Jun 15, 2026

   Author

   More actions

   This is super useful info for us. We're always trying to optimize the limited resources we have for development. Thanks for planning this out so thoroughly.