Skip to content

seccomp: park slow notification work off the loop - #340

Open
congwang-mk wants to merge 7 commits into
mainfrom
notif-park
Open

congwang-mk wants to merge 7 commits into
mainfrom
notif-park

Conversation

@congwang-mk

Copy link
Copy Markdown
Contributor

Supersedes #337. Its chroot unix socket fix is the first commit here unchanged; the rest replaces its off-loop dispatch with a smaller model.

Problem

Supervised connects, FFI callbacks and chroot/COW filesystem handlers ran on the notification loop, so one slow peer, callback or copy stalled every trapped syscall in the sandbox. The existing escape hatch, NotifAction::Defer, made a terminal decision: its Continue skipped later handlers, it was refused on fork and exec, and it failed with EAGAIN past a fixed cap.

Design

  • Park replaces Defer and resumes where it was returned. A parked handler's Continue runs the next handler; any other result ends the chain. The outcome then takes the same path as an inline decision (pinned opens, fork accounting, policy event, argv freeze), so parking on fork or exec is safe. All parks of a notification share one deadline.
  • No admission failures. Each trapped task has at most one outstanding notification, so parked work is already bounded by the task count. Synchronous work runs as NotifAction::native(lane, closure) in per-sandbox lane slots (Local for filesystem work, External for peers and callbacks) on a process-wide runtime kept apart from the embedder's. A full lane queues; it never returns EAGAIN.
  • Child-memory writes are ordered against exec at the write itself. write_child_mem and write_child_mem_force hold the shared side of a per-sandbox gate for each write made by parked work; the exec freeze holds the exclusive side until the kernel has copied argv. Exec waits microseconds instead of failing, and no call site has to remember a guard.
  • Fork accounting runs once the chain settles on Continue, so no handler can park a fork after it was counted, and a failed response rolls the slot back.
  • Chroot and COW handlers are synchronous functions parked on the local lane. A native closure runs to completion, so a COW copy keeps main's copy-outside-the-lock shape and its bookkeeping always runs. A failed pre-copy now fails the write with EIO instead of continuing onto the lower file.
  • The listener stays open until the last native closure using its fd number returns.
  • Every FFI callback is native External work. sandlock_handler_set_deferred is removed (pre-1.0 hard break). A per-handler run guard waits for running callbacks before the FFI run returns, and callbacks still queued fail with EIO instead of touching freed userdata. Python handlers, sync or async, run off the loop with no flag.

Behavior changes

  • A parked or async handler that returns Continue now hands the syscall to the next handler instead of ending the chain.
  • An FFI run waits for running callbacks before returning; a callback that never returns prevents the run from completing.
  • sandlock_handler_set_deferred is gone from the C ABI and the Python SDK.

Not measured

Every chroot and COW syscall now takes a hop to a worker thread and back to the loop. Hot-path latency for chroot open/stat has not been benchmarked.

Test plan

  • CI run 41534869770 green on all 25 jobs (Rust tests, feature matrix, low-ABI, musl, riscv64 cross-build, cbindgen header, Python, Go)
  • New unit tests: chain resume and termination after a park, park deadline and nesting, lane queueing and abandoned-work slot retention, gate exclusion, FFI run guard drain and queued-callback EIO
  • Python e2e: an async handler's continue_() reaches the next handler on the syscall

🤖 Generated with Claude Code

Chroot mode never moves the child's real root or cwd, yet the named unix
gate checked the raw sun_path lexically and returned Continue, leaving the
kernel to resolve it on the host. An absolute virtual path therefore
reached the host's socket of that name (an escape where Landlock cannot
gate unix connect), and a relative one was checked as if anchored at /
but resolved against the cwd the child was exec'ed in, so ./svc.sock from
a granted cwd was refused while tmp/x.sock from an ungranted one passed.

Serve chroot connect/sendto/sendmsg/sendmmsg on-behalf like non-chroot:
build the virtual path against the tracked virtual cwd, pin the inode
inside the root (mount aware), and check write access on where it really
sits. The trap now stays on under chroot even with Landlock ABI v9.

Signed-off-by: Cong Wang <cwang@multikernel.io>
A deferred handler made a terminal decision: its Continue skipped every
later handler, and the supervisor refused deferral on fork and exec
because answering later would bypass fork accounting and the argv
freeze. Deferrals beyond a fixed cap failed with EAGAIN even though each
trapped task can only have one notification outstanding.

Replace Defer with Park, which resumes where it was returned. A parked
handler's Continue runs the next handler, any other result ends the
chain, and the outcome then takes the same path as an inline decision.
Fork accounting moves out of the chain to run once it settles on
Continue, so no handler can park a fork after it was counted. All parks
of a notification share one deadline, admission no longer fails, a
panicked park still answers its caller, and settled parks are drained
while notifications keep arriving.

Signed-off-by: Cong Wang <cwang@multikernel.io>
Supervised connects ran on the notification loop after approval, so a
slow peer or a full listener backlog stalled every other trapped syscall
in the sandbox.

Let a park carry a synchronous closure tagged with a lane. The closure
runs on a process-wide runtime kept apart from the embedder's, whose
shutdown would otherwise wait for a closure stuck on a peer. Each
sandbox owns its slots per lane, so stalled work only delays its own
sandbox, and a full lane queues instead of failing. External peers get
their own lane so they cannot take the slots local filesystem work
needs. A closure runs to completion and keeps its slot and captures even
after its waiter times out. Both supervised connects now run there.

Signed-off-by: Cong Wang <cwang@multikernel.io>
Parked work keeps running while the loop validates an execve's argv
under the freeze. A write it makes into a CLONE_VM peer's memory can land
on that argv after validation and before the kernel copies it, and the
deferred sendmmsg completion already wrote msg_len that way.

Give each sandbox a gate. Parked futures and native closures run with it
installed, write_child_mem and write_child_mem_force hold its shared side
for each single write, and the exec freeze holds the exclusive side until
the freeze is released. Writes are the only access that can change argv
and each holds the gate for one copy, so an exec waits microseconds and
never fails because unrelated work is still running.

Signed-off-by: Cong Wang <cwang@multikernel.io>
Chroot path resolution and COW copy-ups do synchronous filesystem I/O
while the notification loop waits, so one large copy or slow lookup held
up every trapped syscall in the sandbox.

Make these handlers plain synchronous functions and park them as native
work on the local lane. Peers and callbacks use the other lane, so they
cannot take the slots filesystem work needs. A native closure always
runs to completion, so a copy keeps main's shape of copying outside the
state lock and recording its digest or rolling back its quota after,
even when its waiter timed out. A failed pre-copy now fails the write
with EIO instead of continuing onto the lower file.

Signed-off-by: Cong Wang <cwang@multikernel.io>
A native closure runs to completion after its waiter gives up, so it can
outlive the supervisor and still use the listener's raw fd number for
child-memory access. Once the supervisor closed the listener that number
could be reused by another sandbox's listener.

Share the listener between the supervisor and the native closures it
parks, so its fd stays open until the last closure returns.

Signed-off-by: Cong Wang <cwang@multikernel.io>
A C callback was awaited on the notification loop unless the caller
flagged it deferred, and a deferred callback made a terminal decision
that skipped later handlers and was refused on fork and exec. Callers
had to choose between stalling every trapped syscall and changing what
their Continue meant.

Park every callback on the external lane, where Continue resumes the
chain like any other park, and remove sandlock_handler_set_deferred.
A callback runs to completion after its syscall times out, but the
caller's userdata may only be borrowed until the run returns, so each
run holds a guard per handler: dropping it waits for running callbacks
and releases the container, and callbacks still queued fail with EIO
instead of reaching freed userdata. Python async handlers no longer need
a flag, and a sync handler no longer stalls the supervisor.

Signed-off-by: Cong Wang <cwang@multikernel.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant