Token expiration error when getting creds after push #1350

Getting the following error when attempting to query for the manifest digest after pushing an image:

ResponseBody: {"error":"invalid_client","error_description":"AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-06-20T21:51:37.0880125Z, assertion valid from 2024-06-20T21:26:46.0000000Z, expiry time of assertion 2024-06-20T21:36:46.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 3e9b8685-f562-4c31-b2a4-0adf04391700 Correlation ID: 8ab2fea3-1a72-4a13-b818-5ca5d0995af7 Timestamp: 2024-06-20 21:51:37Z","error_codes":[700024],"timestamp":"2024-06-20 21:51:37Z","trace_id":"3e9b8685-f562-4c31-b2a4-0adf04391700","correlation_id":"8ab2fea3-1a72-4a13-b818-5ca5d0995af7","error_uri":"[https://login.microsoftonline.com/error?code=700024"}](https://login.microsoftonline.com/error?code=700024%22})

Example build (internal link)

This seems to happen for builds that take longer than 20 minutes roughly. This is basically the same thing that was happening before but that was 60 minutes and fixed by #1321.

The thing that seems to be unique to this scenario is that it's only occurring for build jobs that do not pull base images from the mirrored repo. These are jobs that build images which are only based on images from mcr.microsoft.com. In that case, we never mirror those images. Since we never authenticated to pull those images, we never primed the cache at the beginning of the job. So at the end when it attempted to retrieve the token, it was expired.

Interestingly, this error does not occur when using an old version of Image Builder, prior to #1310.