fix(tmachine): restore package-installed Podman qualification by elezar · Pull Request #4289 · NVIDIA/OpenShell

elezar · 2026-10-07T11:55:58Z

Summary

Restore package-installed Podman driver qualification from #4107, which was reverted in #4206, on the current tmachine layout. The Podman user-namespace driver suites and the e2e-podman lane now run against the packaged gateway (RPM installer) instead of the binaries installer.

Related Issue

Follow-up to #3454 (already closed by #4107). No new feature scope.

Original changes: #4107, test(tmachine): support package-installed Podman driver suites. Revert: #4206. Product-side follow-up for gateway configuration drop-ins: #5446.

Changes

Add one mechanism for applying settings to a gateway in a tmachine environment: an environment writes TOML tables to /etc/openshell-test/gateway-overrides.toml, and every installer merges that file into the gateway configuration it uses. The merge (merge-gateway-toml.py) replaces keys the file sets, adds missing keys and tables, keeps everything else (including the installed default and its comments), creates a minimal [openshell] version = 2 file when none exists, and refuses to write invalid TOML. The overrides must be valid TOML, and the script fails if any override value is missing from the result instead of silently dropping it. It is idempotent and does not remove keys. Regression tests live next to the script (mise run test:merge-gateway-toml).
- binaries: merged into the generated /etc/openshell/gateway.toml.
- deb/rpm: merged into ~/.config/openshell/gateway.toml of the service owner before the service starts. The RPM seeds from /usr/share/openshell-gateway/gateway.toml.default; the deb gets a minimal file. An environment with no overrides leaves the package configuration untouched.
- k3s: ignores the file.
The Podman userns environments use that file ([openshell.drivers.podman] with userns, uidmap, gidmap), replacing the driver-only podman-driver.toml channel, and run with the rpm installer in CI.
e2e-podman detects the installer: binaries keeps its HTTP tmachine gateway; package installers use /usr/bin/openshell and the registered openshell gateway. It runs on rpm in Branch E2E, Integration Tests and the integration runner.
Document the file, per-installer behavior and merge semantics in TESTING.md, and update CI.md.

Not included: the old gateway-metadata publishing, service restart and rootful socket-group changes, which the current layout does not need. The binaries installer and its templates remain; they are no longer used by CI lanes.

Testing

Local Fedora tmachine guests (QEMU/HVF, SELinux enforcing), RPM installer, before the final rebase onto main and the merge-script hardening (the lanes use simple keys that the script handles the same way):

mise run test:merge-gateway-toml (11 cases, including array-of-tables boundaries, quoted keys and malformed overrides).
Rootful keep-id and rootful default (no overrides) userns lanes passed.
Rootless private userns lane passed, including a temporary override of a key the RPM default already sets to confirm override semantics (reverted).
Rootless auto userns lane passed on the binaries installer.
e2e-podman on rootless Podman with rpm passed.
deb installer, the remaining userns modes, rootful e2e-podman and mise run ci were not run locally.
Branch Checks and Branch E2E on the current head are pending. test:e2e stays enabled; the integration jobs must run and pass before merging. A green workflow with skipped jobs is insufficient.

Checklist

Follows Conventional Commits.
Commits are signed off (DCO).
Contributor testing and CI documentation updated; no public configuration/API changes.

copy-pr-bot · 2026-10-07T11:56:02Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-10-07T11:56:10Z

Label test:e2e applied, but pull-request/4289 does not exist yet. A maintainer needs to comment /ok to test 221cf8a66951dd5e5f1e7d83289f5974825b6a7a to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

elezar · 2026-10-07T12:07:18Z

/ok-to-test 221cf8a

elezar · 2026-10-07T12:15:05Z

/ok-to-test 860aa53

elezar · 2026-10-09T08:29:52Z

/ok-to-test 135ce70

github-actions · 2026-10-28T08:25:51Z

This pull request has had no activity for 14 days and is now marked stale.

elezar · 2026-11-03T14:10:49Z

#5137 (test(e2e): migrate the features and drivers suites to testkit) supersedes the playbook structure this PR edits, so this one will need a rework once that lands.

What changes

test(e2e): migrate the features and drivers suites to testkit #5137 deletes default-userns-baseline.yaml, tests.yaml and userns-profile.yaml and moves the Ansible tree to tests/tmachine/ansible/. The userns coverage becomes a testkit suite (drivers/podman-driver) with eight fedora-podman-{rootful,rootless}-userns-* environments, run through e2e-suites.yml on the binaries installer.
Settings reach the gateway through /etc/openshell-test/podman-driver.toml, appended to gateway.toml by the openshell_gateway role. openshell_packaged_gateway asserts that no such settings are set, so the packaged installers fail instead of testing without the mode.

Plan for this PR

Replace that assert with the XDG gateway.toml mechanism from this PR (openshell_test_gateway: create the minimal versioned TOML if needed, insert the settings, restart the installed service).
Make sure only one mechanism writes [openshell.drivers.podman] on a host, or the table is duplicated.
Move the userns rows from binaries to rpm in e2e-suites.yml.
Keep the rootful e2e-podman socket-access and journal collection changes if test(e2e): migrate the features and drivers suites to testkit #5137 doesn't already cover them.

One thing to carry over: the reference container in #5137's test runs through sudo -n podman (rootful) or as the tmachine user (rootless). If the packaged service runs as a different user, the reference must run as that user so both sides see the same storage and UID ranges. openshell_test_gateway.service_user here would supply it.

…ateway An environment overrides gateway settings by writing the tables to change to /etc/openshell-test/gateway-overrides.toml. Every installer applies the file with merge-gateway-toml.py: the binaries installer to the configuration it generates, and the deb and rpm installers to the service user's gateway.toml, which starts from the package's default configuration instead of replacing it. Each key replaces the same key in the base file or is added to its table, comments are kept, and a result that is not valid TOML fails the setup. Run the Podman user-namespace driver suites with the rpm installer. Signed-off-by: Evan Lezar <elezar@nvidia.com>

Detect the installer in the e2e-podman playbook: the binaries installer keeps its HTTP gateway, and the package installers use /usr/bin/openshell and the registered openshell gateway. Run the lane on rpm in CI. Signed-off-by: Evan Lezar <elezar@nvidia.com>

SDAChess

Two issues in the new TOML merger below. I reproduced both locally; the current namespace fragments do not trigger them. The normal private-mode merge preserves the RPM defaults and is idempotent.

…ents Treat array-of-tables headers as table boundaries so replacing a value before one no longer consumes its header. Require the overrides to be valid TOML, accept single- and double-quoted keys, and fail when any override value is missing from the merged result instead of silently dropping it. Add regression tests and a mise task for them. Signed-off-by: Evan Lezar <elezar@nvidia.com>

github-actions · 2026-11-10T14:21:41Z

🌿 Preview your docs: https://nvidia-preview-pr-4289.docs.buildwithfern.com/openshell

elezar · 2026-11-10T14:25:32Z

Two issues in the new TOML merger below. I reproduced both locally; the current namespace fragments do not trigger them. The normal private-mode merge preserves the RPM defaults and is idempotent.

Thanks @SDAChess. I have addressed the immediate issues, but I want to be careful about writing a general-purpose TOML-merger in python here. We should be covering the immediate use cases that we have in testing.

The follow-up in #5446 which adds "merged config support" would allow us to do away with this entirely once we have first-class merge support.

…#4289) * test(tmachine): apply environment gateway overrides to the packaged gateway An environment overrides gateway settings by writing the tables to change to /etc/openshell-test/gateway-overrides.toml. Every installer applies the file with merge-gateway-toml.py: the binaries installer to the configuration it generates, and the deb and rpm installers to the service user's gateway.toml, which starts from the package's default configuration instead of replacing it. Each key replaces the same key in the base file or is added to its table, comments are kept, and a result that is not valid TOML fails the setup. Run the Podman user-namespace driver suites with the rpm installer. Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(tmachine): run e2e-podman against the packaged gateway Detect the installer in the e2e-podman playbook: the binaries installer keeps its HTTP gateway, and the package installers use /usr/bin/openshell and the registered openshell gateway. Run the lane on rpm in CI. Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(tmachine): harden the gateway TOML merge against unhandled statements Treat array-of-tables headers as table boundaries so replacing a value before one no longer consumes its header. Require the overrides to be valid TOML, accept single- and double-quoted keys, and fail when any override value is missing from the merged result instead of silently dropping it. Add regression tests and a mise task for them. Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar added the test:e2e Requires end-to-end coverage label Oct 7, 2026

elezar marked this pull request as ready for review October 8, 2026 14:55

elezar requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners October 8, 2026 14:55

SDAChess reviewed Oct 8, 2026

View reviewed changes

Comment thread tests/ansible/playbooks/drivers/podman/e2e.yaml Resolved

elezar force-pushed the test/3454-package-driver-validation/elezar branch 3 times, most recently from ba4b773 to a74a797 Compare October 13, 2026 12:50

github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Oct 28, 2026

elezar mentioned this pull request Nov 3, 2026

test(e2e): migrate the features and drivers suites to testkit #5137

Merged

6 tasks done

github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Nov 4, 2026

elezar force-pushed the test/3454-package-driver-validation/elezar branch from a74a797 to 0f95bee Compare November 10, 2026 13:12

elezar added 2 commits November 10, 2026 14:13

elezar force-pushed the test/3454-package-driver-validation/elezar branch from 0f95bee to da7472e Compare November 10, 2026 13:15

elezar mentioned this pull request Nov 10, 2026

test(features): add CDI injection scenarios for Docker and Podman #4461

Merged

6 tasks done

elezar added the area:build Related to CI/CD and builds label Nov 10, 2026

github-actions Bot requested review from TaylorMutch, matthewgrossman and purp November 10, 2026 13:29

elezar requested a review from SDAChess November 10, 2026 13:29

SDAChess reviewed Nov 10, 2026

View reviewed changes

Comment thread tests/tmachine/ansible/roles/openshell_gateway/files/merge-gateway-toml.py Resolved

Comment thread tests/tmachine/ansible/roles/openshell_gateway/files/merge-gateway-toml.py Resolved

SDAChess previously approved these changes Nov 10, 2026

View reviewed changes

elezar dismissed SDAChess’s stale review via 46d7b1a November 10, 2026 14:20

SDAChess approved these changes Nov 10, 2026

View reviewed changes

elezar enabled auto-merge November 10, 2026 14:25

elezar added this pull request to the merge queue Nov 10, 2026

Merged via the queue into main with commit f3d9765 Nov 10, 2026
154 of 156 checks passed

elezar deleted the test/3454-package-driver-validation/elezar branch November 10, 2026 15:25

rhods-devops-app Bot mentioned this pull request Nov 10, 2026

Automated Sync from main to stable red-hat-data-services/openshell#40

Open

Navigation Menu

Repository navigation

fix(tmachine): restore package-installed Podman qualification - #4289

fix(tmachine): restore package-installed Podman qualification#4289
elezar merged 3 commits into
mainfrom
test/3454-package-driver-validation/elezar

elezar commented Oct 7, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

elezar commented Oct 7, 2026

Uh oh!

elezar commented Oct 7, 2026

Uh oh!

Uh oh!

elezar commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

elezar commented Nov 3, 2026

Uh oh!

SDAChess left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

elezar commented Nov 10, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

elezar commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

elezar commented Oct 7, 2026

Uh oh!

elezar commented Oct 7, 2026

Uh oh!

Uh oh!

elezar commented Oct 9, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

elezar commented Nov 3, 2026

Uh oh!

SDAChess left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

elezar commented Nov 10, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

elezar commented Oct 7, 2026 •

edited

Loading