Repository navigation
Add lifecycle annotations to images pushed to MCR #1201
Description
A broader set of services are starting to consume image annotations. Specifically Azure container scanning honors the lifecycle annotation to treat EOL images specially. As a result, the images produced with the .NET Docker infrastructure should have these lifecycle annotations added.
Related links:
-
Thread & Vulnerability Management guidance (MSFT internal link)
How can I mark an image in my ACR as end of life or dictate how long it should be scanned and reported on? Attach annotations* to the image called 'lifecycle annotations' which requires: The ACR Metadata Service to be onboarded for your registry (see callout below) [ORAS CLI](https://oras.land/docs/installation/) must be installed ORAS CLI must be authenticated to your registry [(See How to Authenticate ORAS CLI)](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-oci-artifacts#sign-in-to-a-registry) You can use the ORAS CLI to attach lifecycle annotations by running the following command: Variables: $IMAGE refers to a fully qualified image name or digest (e.g. myregistry.azurecr.io/myimage:v1 OR myregistry.azurecr.io/myimage@sha256:fdbd1bc674f77e7a3b8129ecac062a9b7c22de74968e8ea3a46f071f61f3e99f) Lifecycle.end-of-life.date should be in the expected format of yyyy-mm-dd oras attach --artifact-type application/vnd.microsoft.artifact.lifecycle --annotation "vnd.microsoft.artifact.lifecycle.end-of-life.date=2023-05-12T21:15:10.8343521Z" $IMAGE
Work Items:
- Add a new command for annotating image digests for EOL #1334
- EOL annotations pipeline #1340
- Command for generation of EOL annotation data file #1358
- Remove force annotate option #1364
- Add command to wait for EOL annotation ingestion #1367
- Push EOL annotations in publish stage #1413
- EOL annotation race condition #1417
- Roll out
- [dotnet-docker/nightly] Annotate and remove historical images
- [dotnet-docker/nightly] Enable annotations in pipeline: Enable EOL annotation publishing dotnet-docker#5837
- [buildtools-prereqs] Annotate and remove historical images
- [buildtools-prereqs] Enable annotations in pipeline: Enable matrix trimming and EOL annotations dotnet-buildtools-prereqs-docker#1209
- [dotnet-docker/main] Annotate and remove historical images
- [dotnet-docker/main] Enable annotations in pipeline
- [dotnet-framework-docker] Annotate and remove historical images
- [dotnet-framework-docker] Enable annotations in pipeline: Enable EOL annotations in pipeline microsoft/dotnet-framework-docker#1184
- [buildtools-prereqs/imagebuilder] Annotate and remove historical images
- [buildtools-prereqs/imagebuilder] Enable annotations in pipeline: Enable EOL annotations for Image Builder pipeline #1496
- Clean up
_placeholder tags that we pushed to MAR for historical images that were imported from MAR
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
lbussell commentedon Jan 11, 2024
[Triage] This needs some design work done before we start work on this. We need a way to figure out which images to tag as EOL and when without scanning every image in our ACR.
mthalman commentedon May 10, 2024
Principals:
Scenarios
Known Good State
First we need to get to a known good state with some initial setup work. This involves setting EOL date annotation for all out-of-support manifests. We can debate how accurate these dates need to be. For example, all 5.0-based tags could be given an EOL date of 5.0's EOL date. Dangling manifests would probably just be set to an arbitrary date. But the need here is for all of these out-of-support manifests to have an EOL date in the past. Any manifest that is not contained in https://github.com/dotnet/versions/tree/main/build-info/docker is considered to be out-of-support.
Scenarios: Servicing and Rebuilds
To account for the scenarios of servicing and rebuilds, the logic can be integrated into the publishing pipeline. This logic would work by first getting the Dockerfile paths for all the images that are being published. It would then look up those paths in the image info file (prior to updating that file as part of the publish process) to see if that Dockerfile path had been previously published. If that Dockerfile path existed in the image info file, grab the image digest associated with it. That digest should be annotated with an EOL of today's date.
This also needs to check the multi-arch tags. For example, a tag like 6.0.28-alpine3.18 is EOL when 6.0.29 is published. That's a multi-arch tag so it's associated with multiple Dockerfiles. For each Dockerfile that's being published, check whether it's associated with any multi-arch tags. For each of those tags, find all associated images. Check whether all of those Dockerfiles are included in this current publishing run. If they are then the multi-arch tag (manifest list) should be EOL-annotated.
Scenarios: Distro/.NET out of support
To account for distro versions or .NET major versions going out of support, we need to account for Dockerfile removal. If a Dockerfile path is removed from the image info file as part of publishing, the digest associated with that file should be EOL-annotated. Similarly as with servicing, multi-arch tags should be accounted for as well. If a multi-arch tag has been removed from the image info file, the digest for it should be EOL-annotated.
richlander commentedon May 10, 2024
This looks solid.
Perhaps we should delete all
-previewtags (before 9.0) at the same thing. I don't see value in marking those for EOL (and leaving them).MichaelSimons commentedon May 13, 2024
Look good @mthalman. I had a few thoughts/questions as I reviewed your proposal:
mthalman commentedon May 13, 2024
I was thinking it would either be at the very end of the publishing stage or as a separate post-publishing stage.
richlander commentedon May 13, 2024
We are certain to get the adornment wrong at least once. We need to ensure we can run this stage on its own after quickly altering some data file.
richlander commentedon May 13, 2024
FYI @tmds @baronfel
17 remaining items
mthalman commentedon Jun 5, 2024
I've been thinking about another scenario that we need to account for. In some cases, the pipeline doesn't have a successful run. For example, it may timeout while waiting for images to be ingested by MAR. In that scenario, images will have been published but, because the timeout occurred, it never executed the step which updated the image info file. A retry would potentially resolve this. However, if the build was originally executed by AutoBuilder, and not followed up with a manual retry from a team member, another build will get executed by AutoBuilder later in the day. The images in the second build will supersede the images from the previous one, causing the older ones to be dangling images. Because those previous images were never represented in the image info file, they never get annotated.
One could argue that the publishing of the image info should happen earlier. But that could lead to false data in cases where there is some issue with MAR where it didn't ingest any images at all. Or there may even be a failure in the act of publishing the image info file.
The point is that the image info file doesn't necessarily describe all the images we need to consider. But can we constraint the scenarios that we need to consider for this? Constraining the scenarios is prudent because if you leave it wide open, then you'd essentially have to query every single digest in the registry checking to see if it needs to be annotated. That would be an expensive operation. If we could constrain the number of queries we needed to make, that would be better. One option would be to query for any existing images that are currently tagged with the tags that are being published by the build (this would obviously need to be queried before the current set of images get published). Another option is to query for previously failed builds and grab their image info files to limit what digests should be queried.
mthalman commentedon Jul 10, 2024
Another option that was thought of was to use Log Analytics on the ACR to get definitive data on what images had been pushed to the registry. This wouldn't require our own logging mechanism that would be subject to failure and unreliability. This had been determined to be the way to go. But upon implementation, it's been discovered that pushes of Windows images to the ACR produce "importImage" operations in the log instead of "Push" operations. Ok, that's fine. But the issue is that the Digest field is empty for these importImage operations. That's the key bit of information we need in order to find potential dangling images to be annotated.
mthalman commentedon Jul 11, 2024
Proposing a different direction that should greatly simplify things but has a broader impact on the use of the registry.
The design proposals above have been focused on dealing with the fact that there are thousands of manifests in the registry's repos. Because of that number, it's not feasible to query through all of them to determine what needs to be annotated.
The new proposal is all about cleaning out old images from the registry. Once an image has been EOL-annotated, it can be deleted from the registry (or given a grace period before deletion). This greatly reduces the number of manifests we would need to query to determine what needs to be annotated. Because we can then query all of the manifests without that big of a performance impact, it allows us to simply query for all digests that are not in the list of currently supported digests. Those would then be annotated and subsequently deleted from the registry.
This would then not rely on comparing the new image info versus the old image info. It only needs to take into account the state of the new image info.
richlander commentedon Jul 11, 2024
Deletions is a hard topic. I thought we were not keen on that. I think we'd need to collection some data on how prevalent digest references are. We are certain to break builds. If we did this, we'd need to publish guidance that digest references are only supported as a temporary measure.
In terms of deletions, I'd start with previews. I see that as entirely fair game. I think we should delete all preview images that are > 3 months old. That should remove a lot of digests. In the case of STS, the preview and supported period are close to equal.
mthalman commentedon Jul 11, 2024
This is not about deleting images from MAR. It's only about deleting them from our ACR. That deletion won't propagate to MAR.
richlander commentedon Jul 11, 2024
Oh! I see. If I reverse engineer what you are saying, I think you are positioning ACR as a backup for MAR, and EOL annotated images don't need that?
I still think we should delete all preview images.
lbussell commentedon Jul 11, 2024
This sounds like a good direction to move in - I am struggling to come up with a situation where we'd need an old image from our ACR. It seems like "replace a broken image with an older version" would almost never be preferable to "replace a broken image with a new build from a previous version of the Dockerfile".
The only situation I could think of is: 1. We released a broken image + 2. a distro's package feeds are down at the same time. But in that situation, we could probably just use old images from MAR.
lbussell commentedon Oct 29, 2024
@mthalman is this issue up-to-date? We have started the rollout for .NET Fx right?
mthalman commentedon Oct 29, 2024
Yes, thanks for calling that out. I've updated the checkboxes.
mthalman commentedon Oct 29, 2024
Oh wait, I still need to enable it in the .NET Fx pipeline.
mthalman commentedon Nov 21, 2024
Roll out is all done. Finally closing this one out.
Enable EOL annotations in pipeline