8dot3name short name creation not persisted

[nbrinks]

Describe the bug
Creating a short name in a layer doesn't appear to persist in subsequent layers, nor is the short name resolvable when starting the container.

To Reproduce

Given a Dockerfile such as the following:

# escape=`
FROM mcr.microsoft.com/windows/server:ltsc2022
SHELL ["cmd", "/S", "/C"]

RUN fsutil file setshortname "C:/Program Files" "PROGRA~1" && `
    fsutil file setshortname "C:/Program Files (x86)" "PROGRA~2" && `
    DIR P* /X
RUN DIR P* /X

Build the image... e.g. docker build -t shortname-example --no-cache .

The build output:

Step 1/4 : FROM mcr.microsoft.com/windows/server:ltsc2022
 ---> 7370f2724f4c
Step 2/4 : SHELL ["cmd", "/S", "/C"]
 ---> Running in b545215c24c2
 ---> Removed intermediate container b545215c24c2
 ---> 9739fd112333
Step 3/4 : RUN fsutil file setshortname "C:/Program Files" "PROGRA~1" &&     fsutil file setshortname "C:/Program Files (x86)" "PROGRA~2" &&     DIR P* /X
 ---> Running in 07e794550b2b
 Volume in drive C has no label.
 Volume Serial Number is BE08-0509

 Directory of C:\

05/10/2024  03:50 PM    <DIR>          PROGRA~1     Program Files
05/10/2024  03:48 PM    <DIR>          PROGRA~2     Program Files (x86)
               0 File(s)              0 bytes
               2 Dir(s)  136,115,924,992 bytes free
 ---> Removed intermediate container 07e794550b2b
 ---> 41ed52b38f1a
Step 4/4 : RUN DIR P* /X
 ---> Running in 5558310de18c
 Volume in drive C has no label.
 Volume Serial Number is BE08-0509

 Directory of C:\

05/10/2024  03:50 PM    <DIR>                       Program Files
05/10/2024  03:48 PM    <DIR>                       Program Files (x86)
               0 File(s)              0 bytes
               2 Dir(s)  136,116,391,936 bytes free
 ---> Removed intermediate container 5558310de18c
 ---> 4df53e83def8
Successfully built 4df53e83def8
Successfully tagged shortname-example:latest

In the second DIR P* /X statement, the short names are gone.

Expected behavior
It is expected that the short names persist so they will resolve in subsequent layers. Also, that the short names can be used in the final image.

Configuration:

• Edition: Windows 11 Enterprise, Version 23H2 (OS Build 22631.3593)
• Base Image being used: Windows Server
• Container engine: docker
• Container Engine version: 25.0.2

Additional context
Strangely enough the short names used to resolve in the containers but suddenly stopped working recently. I haven't been able to pinpoint what exactly changed.

Currently using mklink /D "C:/PROGRA~1" "C:/Program Files" to work around this issue.

[ntrappe-msft]

@nbrinks Hi, thanks for bringing up this Issue. We'd love to learn more about your use case. Can we ask why you're using short names in containers? And do you know when this feature stopped working for you?

[nbrinks]

@ntrappe-msft, We have some legacy build systems that use hardcoded short names. We are attempting to containerize our build environments. Our IT group has been making changes to the short name creation policy. Would changes to that have an impact on containers running with process isolation?

[ntrappe-msft]

@nbrinks Thanks for sharing. I'm waiting to hear back from a few other engineers about whether short names not persisting is expected or not. I'll provide an update as soon as I can.

[ntrappe-msft]

I should be able to provide an answer by the end of this week. I'm waiting on one other team to confirm that they're in agreement.

[ntrappe-msft]

Hi @nbrinks thanks for your patience.

In summary, 8dot3 short names are not designed to be persisted across container layers.

This is why setting the 8dot3 file name in one RUN command cannot be queried in another RUN command, as they are in different layers. Similarly, when you set it in a running container and commit those changes, they are saved into a new layer. When a new container starts from that image, it is in a new layer and cannot be queried.

We'd recommend you do not use 8dot3 short names in containers. This behavior is intentional to prevent naming conflicts across layers. It's clear that we didn't do a good job of making this behavior obvious, and we're working to address this.

[nbrinks]

@ntrappe-msft, thank you for the update. Unfortunately it's not practical for us to not use 8dot3 short names in this particular case. Fortunately the directory symbolic link appears to be a viable workaround. I appreciate the thoughtful investigation.

[sxa]

I've just hit this issue too (different project from nbrinks) - we've got some automated machine setup and I thought setting the default fsutil behavior set disable8dot3 0 or fsutil 8dot3name set 0 wasn't taking effect but it looks like is in some situations and not others as I've got two Azure machines seemingly behaving differently just now (The container that appears to be working just now has not been restarted since it was first created but I don't know if that's relevant).

I'm going to need to consider options here if this isn't a supported use case though - maybe we can ensure that all the shortnames prior to starting the process that relies on them ... Although I quite like @nbrinks' use of mklink too if that'll work in my situation :-)

(EDIT: I've just done a stop/start cycle on a container that has shortnames and they appear to have been retained along with the default behavior of creating shortnames for new directories)

[ntrappe-msft]

@sxa What containers are you running? Also Server Core LTSC 2022? That's odd that 8dot3 short names are persisting for you. But going forward, I would recommend mlink since that's just an alias not a short name as that will have better support in the future.

[sxa]

@sxa What containers are you running? Also Server Core LTSC 2022?
Yes - and the host is also Server 2022.

I've done a couple of of experiments on two server 2022 hosts (10.0.20348.2582). Both have shortname creation enabled on the host.
The machine with docker 27.0.3, build 7d4bcd8 does not show shortnames within container images
The machine with 27.1.1, build 6312585 does show a shortname on C:\Program Files if you run docker run mcr.microsoft.com/windows/servercore:ltsc2022 cmd /c dir /x and seems to work as we need. They have slightly different versions of the container image pulled, but I've tried using the SHAs directly and confirmed it's not related to the image version.

Based on https://download.docker.com/win/static/stable/x86_64/ it looks like 27.1.1 was released on July 23rd so I initially wondered if I'd just set up the two machines before and after a change in the docker code was implemented ... However I've tried pulling the new zip from there on the non-working machine, replacing the files in windows/system32 with the new ones (Not sure what the correct upgrade process should be, but that seems reasonable) and it still isn't working on that machine even after a reboot and re-pull of the image so I'm still not 100% certain what the magic is on the working machine. I will also point out that the "working" machine has a separate docker data directory on an disk mounted as F: instead of in the default C:\ProgramData\docker location. It also has 4 CPUs/16GiB RAM vs 2/8 on the failing one but I wouldn't imagine that's relevant.

I'll look next week at setting up another clean machine to experiment with - maybe installing 27.1.1 on a clean system will work but I'm intrigued to know what the critical difference is - it definitely seems to be possible ...

[sxa]

It looks like the 8dot3 shortname creating in the containers works as long is it's enabled on the file system that hosts the docker data. The 8dot3 setting is 2 by default - setting it to zero to ensure it is enabled on the host seems to allow it to work in the containers. You canm see in the following example that before the fsutil 8dot3name set 0 was run the PROGRA~1 and PROGRA~2 don't show up in the default image:

C:\ProgramData>docker run mcr.microsoft.com/windows/servercore:ltsc2022 cmd /c dir /x
 Volume in drive C has no label.
 Volume Serial Number is B001-3C73

 Directory of C:\

06/08/2023  12:46 PM             5,647              License.txt
07/03/2024  05:08 PM    <DIR>                       Program Files
07/03/2024  09:47 AM    <DIR>                       Program Files (x86)
07/03/2024  10:17 AM    <DIR>                       Users
07/03/2024  10:19 AM    <DIR>                       Windows
               1 File(s)          5,647 bytes
               4 Dir(s)  136,189,595,648 bytes free

C:\ProgramData>fsutil 8dot3name set 0
The registry state is now: 0 (Enable 8dot3 name creation on all volumes).

C:\ProgramData>docker run mcr.microsoft.com/windows/servercore:ltsc2022 cmd /c dir /x
 Volume in drive C has no label.
 Volume Serial Number is B001-3C73

 Directory of C:\

06/08/2023  12:46 PM             5,647              License.txt
07/03/2024  05:08 PM    <DIR>          PROGRA~1     Program Files
07/03/2024  09:47 AM    <DIR>          PROGRA~2     Program Files (x86)
07/03/2024  10:17 AM    <DIR>                       Users
07/03/2024  10:19 AM    <DIR>                       Windows
               1 File(s)          5,647 bytes
               4 Dir(s)  136,188,981,248 bytes free

C:\ProgramData>

@nbrinks Does that make a difference on your Windows 11 system?