Skip to content

[BUG] All CI jobs timing out #970

Closed

Description

@invidian

It seems runner image got updated recently and things start to fail. I see this now in docker logs in runner:

Feb 18 14:50:25 runnervmn36qa dockerd[4236]: fatal error: concurrent map writes
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: goroutine 393790 [running]:
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: internal/runtime/maps.fatal({0x5649a38a4931?, 0xc00214cc80?})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /usr/local/go/src/runtime/panic.go:1046 +0x18
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*wildcardCache).Match(0xc0073333f0, {0xc00214cc80, 0xf})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/formatter.go:66 +0xbf
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.match(0xc002814630, {0xc00214cc80, 0xf}, {0x0, 0x0, 0xc001fe9e30?}, 0xc00657c2d0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/matcher.go:56 +0x48f
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*Engine).evaluatePolicy(0xc00657c210, {0x5649a45e89b8, 0xc004333f20}, 0xc0034ec730, 0xc003cf00c0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/engine.go:133 +0x305
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*Engine).evaluatePolicies(0xc00657c210, {0x5649a45e89b8, 0xc004333f20}, 0xc003cf00c0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/engine.go:101 +0x7a
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/sourcepolicy.(*Engine).Evaluate(0xc00657c210, {0x5649a45e89f0, 0xc0034ec6e0}, 0xc003cf00c0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/sourcepolicy/engine.go:84 +0x312
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/solver/llbsolver.(*policyEvaluator).Evaluate(0xc007333180, {0x5649a45e89f0, 0xc0034ec6e0}, 0xc003ef9080)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/llbsolver/policy.go:28 +0x90
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/buildkit/solver/llbsolver.loadLLB.func1()
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/llbsolver/vertex.go:353 +0x6e
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: golang.org/x/sync/errgroup.(*Group).Go.func1()
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 393788
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: goroutine 1 [sync.WaitGroup.Wait, 14 minutes, locked to thread]:
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: sync.runtime_SemacquireWaitGroup(0xc000b05e60?, 0x80?)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /usr/local/go/src/runtime/sema.go:114 +0x2e
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: sync.(*WaitGroup).Wait(0xc000b669e0)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /usr/local/go/src/sync/waitgroup.go:206 +0x85
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.(*daemonCLI).start(0xc000518af0, {0x5649a45e8980, 0x5649a5e42380})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/daemon.go:349 +0x1b25
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.runDaemon(...)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/docker_unix.go:13
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.newDaemonCommand.func1(0xc00048c908, {0xc0006e4210?, 0x7?, 0x5649a385c4a3?})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/docker.go:48 +0xb3
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).execute(0xc00048c908, {0xc0000520d0, 0x3, 0x3})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1015 +0xb02
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00048c908)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1148 +0x465
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).Execute(...)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1071
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/vendor/github.com/spf13/cobra/command.go:1064
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: github.com/moby/moby/v2/daemon/command.daemonRunner.Run({0x5649a45b4df8?}, {0x5649a45e8980, 0x5649a5e42380})
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/daemon/command/docker.go:97 +0x6e
Feb 18 14:50:25 runnervmn36qa dockerd[4236]: main.main()
Feb 18 14:50:25 runnervmn36qa dockerd[4236]:         /root/build-deb/engine/cmd/dockerd/main.go:35 +0x119

Activity

  1. self-assigned this
    on Feb 18, 2026
  2. invidian commented on Feb 19, 2026

    @invidian
    ContributorAuthor

    @cpuguy83 pointed that this is caused by moby/buildkit@f7682fe, so disabling source policy on CI should fix it and it seems to be working. I'll create a PR.

  3. added 2 commits that reference this issue on Feb 19, 2026
    ccc113f
    f7c7801
  4. added a commit that references this issue on Feb 19, 2026
    f551699
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions