Repository navigation
GetAllImageDigestsFromRegistryAsync is suddenly extremely slow #1905
Description
In this dotnet-docker-nightly pipeline run it took 1hr15min, causing the pipeline to time out later in the run:
2025-12-19T01:01:15.0763844Z Querying registry for all image digests...
2025-12-19T02:15:43.0489446Z Checking digest for existing annotation: ...
Previously (in this pipeline run), it took less than one minute:
2025-12-10T21:56:14.9556032Z Querying registry for all image digests...
2025-12-10T22:19:31.3493702Z Checking digest for existing annotation: ...
The earliest occurrence of this behavior I could find is in https://dev.azure.com/dnceng/internal/_build/results?buildId=2861673&view=logs&j=739bb7ec-7296-5b40-7dff-ee3297bfdaa4&t=a08adfce-16dd-580a-541c-89ea6031d6c4 where it took 23 minutes. That is dotnet-docker commit db838f02ef8fb99154680b938054748dc856d8b9.
The cause is likely #1879.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
Disable publishEolAnnotations in core-official.yml
lbussell commentedon Dec 23, 2025
I profiled before/after #1879. There was not a meaningful impact to individual manifest queries outside of the P99 manifest query scenario. The P99 is caused by the first request to each registry. It's hard to tell why this got so slow all of a sudden.
mthalman commentedon Jan 5, 2026
Looking at the history of the nightly pipeline, I see there were a number of builds which failed during the
Wait for Annotation Ingestionstep with a 500 error. Here's the first instance of that: https://dev.azure.com/dnceng/internal/_build/results?buildId=2859897. So it's possible that all of these failures produced a big backlog of images that don't have annotations when they should. Because of the number of failed builds, this would be a larger set of images that need to be annotated than we would have normally. And so that may be the reason for hitting the timeout now.Parallelize `GetAllImageDigestsFromRegistryAsync` (dotnet#1909)