Improve devcontainer build speed

[corona10]

@brettcannon
The 35-minute release period is pretty painful. We may need to find out whether we can reduce the build time.
https://github.com/python/cpython-devcontainers/actions/runs/11038928730

[AA-Turner]

The below is extracted from the logs, the main dnf step takes 2,038 seconds (34 minutes).

Notably, no output is produced for ~12 minutes (730 seconds). "Fedora 40 - aarch64 - Updates" then takes another 6 minutes (370 seconds). In the second builddep command, another six minutes (350 seconds) elapse from "enabling fedora-source repository" to the first "Package [...] is already installed.".

cc @hroncok for any advice (as author of python/cpython#103283)

dnf install ...:

[linux/arm64 2/4] RUN dnf -y --nodocs --setopt=install_weak_deps=False install /usr/bin/{blurb,clang,curl,git,ln,tar,xz} 'dnf-command(builddep)' &&     dnf -y --nodocs --setopt=install_weak_deps=False builddep python3 &&     dnf -y clean all
729.3 Fedora 40 openh264 (From Cisco) - aarch64       476  B/s | 2.1 kB     00:04    
785.2 Fedora 40 - aarch64 - Updates                   1.1 MB/s |  38 MB     00:33    
1156.0 Last metadata expiration check: 0:00:23 ago on Wed Sep 25 18:39:54 2024.
1241.3 Package [...] is already installed. [x4]
1250.7 Dependencies resolved.
1251.6 ================================================================================
1251.6  Package                     Arch     Version                    Repo      Size
1251.6 ================================================================================
1251.6 Installing:
...
1251.6 Transaction Summary
1251.6 ================================================================================
1251.6 Install  47 Packages
1251.6 
1251.7 Total download size: 140 M
1251.7 Installed size: 562 M
1251.7 Downloading Packages:
...
1267.9 --------------------------------------------------------------------------------
1267.9 Total                                           8.7 MB/s | 140 MB     00:16     
1293.7 Running transaction check
1295.7 Transaction check succeeded.
1295.7 Running transaction test
1305.3 Transaction test succeeded.
1305.3 Running transaction
1313.6   Preparing        :                                                        1/1 
1315.1   Installing       : systemd-libs-255.12-1.fc40.aarch64                    1/47 
...
1344.5   Running scriptlet: dnf-plugins-core-4.9.0-1.fc40.noarch                 47/47 
...
1348.7 Complete!

dnf builddep python3:

1358.6 enabling fedora-source repository
1358.6 enabling fedora-cisco-openh264-source repository
1358.6 enabling updates-source repository
1370.6 Fedora 40 - Source                              1.2 MB/s | 6.6 MB     00:05    
1433.9 Fedora 40 openh264 (From Cisco) - aarch64 - Sou  30  B/s | 134  B     00:04    
1452.2 Fedora 40 - Updates Source                      359 kB/s | 1.7 MB     00:04    
1544.5 Last metadata expiration check: 0:00:08 ago on Wed Sep 25 18:51:10 2024.
1708.0 Package [...] is already installed. [x12]
1722.1 Dependencies resolved.
1724.3 ================================================================================
1724.3  Package                    Arch    Version                       Repo     Size
1724.3 ================================================================================
1724.3 Installing:
...
1724.3 Transaction Summary
1724.3 ================================================================================
1724.3 Install  233 Packages
1724.3 Upgrade   11 Packages
1724.3 
1724.7 Total download size: 123 M
1724.8 Downloading Packages:
...
1746.9 --------------------------------------------------------------------------------
1746.9 Total                                           5.6 MB/s | 123 MB     00:21     
1861.1 Running transaction check
1867.8 Transaction check succeeded.
1867.8 Running transaction test
1893.0 Transaction test succeeded.
1893.0 Running transaction
1914.6   Preparing        :                                                        1/1 
1917.6   Upgrading        : zlib-ng-compat-2.1.7-2.fc40.aarch64                  1/255 
...
1989.6   Running scriptlet: expat-2.6.0-1.fc40.aarch64                         255/255 
...
2025.9 Complete!

dnf clean:

2036.9 37 files removed
DONE 2037.9s

[hroncok]

Notably, no output is produced for ~12 minutes (730 seconds). "Fedora 40 - aarch64 - Updates" then takes another 6 minutes (370 seconds). In the second builddep command, another six minutes (350 seconds) elapse from "enabling fedora-source repository" to the first "Package [...] is already installed.".

That's pretty bad, but I have no idea why would it take so long. Perhaps it's the emulation/virtualization that is there that is so slow?

[trallard]

I was having a quick nosey here 👋🏽 but looking at the Actions runs, the build is taking < 3mins and the 35 mins + was an outlier.

[corona10]

@trallard

FYI, 3 mins running releases were about Autoconf :)
For the devcontainer it takes a long time, but thanks to your comment, I've written up a new PR for distinguishing which packages were released :)

Please take a look if you are interested :)

#16

[trallard]

Ah that makes sense. Will have a look over there and maybe do some digging on why this is taking so long then.

[corona10]

Last release by Brett took 28 mins.
(36mins -> 28mins!!)
https://github.com/python/cpython-devcontainers/actions/runs/11245912749

The PR by @hroncok enhanced the speed a lot!

[hroncok]

We have a container job at github actions that runs dnf on 4 architectures. The arm64 one (third in the screenshot) is regularly extremely slow:

[efimov-mikhail]

What do you think about splitting actual container creation into two steps? One with big dnf steps, which rarely changes. And another container with all other steps. Unfortunately, we should use some sort of links to actual containers. But regular updates can be much faster with this scheme.

[brettcannon]

I'm personally fine with that if @corona10 and @erlend-aasland are. I have #12 open to split things up in regards to WASI, but I wasn't sure about following through with it due to having to change the CPython repo when new containers go out. We might need to settle on some policy in #11 to know how much work would be expected.

[erlend-aasland]

I'm personally fine with that if @corona10 and @erlend-aasland are.

+1

[corona10]

+1