Repository navigation
Stream images/videos directly from source? #1610
Description
Is it possible to have DIVE stream videos directly from a GCP source rather than piping the data through the server? We have lots of videos that we'll be streaming, and it would be much more efficient to just stream them directly from the buckets to the user. Piping them through the server is going to be the heaviest lift for our server by a couple orders of magnitude, and also introduces latency.
Any thoughts on how difficult this might be?
Activity
BryonLewis commentedon May 18, 2026
The data that is stored in GCP is only duplicated and stored locally if it needs to be transcoded. If not it should directly reference a presigned URL from GCP and that is used for the streaming of data. So while DIVE still talks to a girder endpoint first if it is a GCP imported through the asset store it will use the object storage and presigned URL and won't actually download on the server when visualizing the data.
Now if you happent to run a pipeline or do training and python needs access to a 'file' the container will download the video file before performing any algorithms on it.
So I think it already does what you want it to do. The only time it doesn't is if the video needs to be transcoded, or if your are running ML algorithms (pipelines/training) in a separate container.