AV1 Video Extraction Fails with OpenCV's _retrieve_frames_cv2() due to FFmpeg Compatibility

[ConnorKevin]

We encounter the error when using _retrieve frames_cv2() to extract frames from videos, while moviepy doesn't have this problem.
We found it may be caused by the embedded ffmpeg in opencv, which doesn't support AV1, while moviepy uses the ffmpeg in the local environment, which supports AV1.

So solutions could be:

1. Transform the AV1 video to OpenCV-friendly format (e.g., vp6), using local ffmpeg at first and then still process with cv2
2. Use original moviepy
3. Install opencv from source code to include the local ffmpeg https://github.com/opencv/opencv/issues/24430, which may need dependencies support and sudo that may not be easy in a computing cluster

So my questions are:

Which solution is recommended?
Are there any other elegant solutions?

Thanks for your work and time!

Error traceback:

[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                  [648/1739]
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
[av1 @ 0x55aeee1a9c40] Missing Sequence Header.                                                                                                                                                                            
Video size: 3840.0 x 2160.0                                                                                                                                                                                                
VideoCapture backend: 1900.0                                                                                                                                                                                               
                                                      [av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                                                                             
[av1 @ 0x55aeee49b880] Failed to get pixel format.t/s]                                                                                                                                                                     
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Your platform doesn't suppport hardware accelerated AV1 decoding.                     
[av1 @ 0x55aeee49b880] Failed to get pixel format.    
[av1 @ 0x55aeee49b880] Missing Sequence Header.       
Extract frames:   0%|          | 0/40 [00:00<?, ?it/s]                                                       
2025-04-20 20:32:16,102 - ERROR - Error processing video 3JzKxiXTJQo_160026693: vidcap error                 
Processing batch 1: 100%|██████████| 1/1 [00:00<00:00,  3.29it/s]

[jinlinyi]

Hi Kenan,

I'm not sure what's the best solution on your end. But fyi, I tested my code with cv2 4.11.0 and ffmpeg/4.4.2 and I didn't install opencv from source.

Best,
Linyi

[jinlinyi]

Please note that the released .npz annotation files are aligned with the cv2 frame extraction logic we used internally.

Originally, the released code used moviepy because I ran into the same error with OpenCV due to "AV1 incompatibility" when I tested on my school server. However, I later noticed that moviepy occasionally extracts duplicated frames—at least for the demo video—which led me to switch back to cv2. Somehow, the "AV1 incompatibility" issue no longer occurred (possibly because I was using FFmpeg 4.4.2 by then, though I don’t recall the exact fix).

Hope this information helps.

[jinlinyi]

Hi, I tested my current environment with other videos, and I do see some videos still have this error, so please ignore my comments above. Looking into this issue ...

[jinlinyi]

I believe internally we used cv2 that's compatible with av1 format. So installing cv2 compatible with av1 would be the safest solution.

An alternative solution would be using videos with vp9 format or pre-extract the frames using system ffmpeg if you don't want to build cv2 from source.