fix(security): clear util-linux CVEs and cut the Go CVE surface in the ingestion images - #31612
Conversation
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
✅ Playwright Results — workflow succeededValidated commit ✅ 110 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 1h 19m 2s ⏱️ Max setup 3m 5s · max shard execution 12m 21s · max shard-job elapsed before upload 18m 34s · reporting 4s 🌐 212.78 requests/attempt · 1.79 app boots/UI scenario · 0.00% common-shard skew Optimization targets still in progress:
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
7c5dcd9 to
cb8e32f
Compare
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
…e ingestion images Two unrelated root causes behind the current Inspector/Snyk findings on the ingestion images. util-linux: the trixie base ships 2.41-5, which carries CVE-2025-14104, CVE-2026-13595 and CVE-2026-27456. trixie-security has 2.41.5-0+deb13u1, so this is a straight upgrade -- added to the late root layer in both ingestion-base Dockerfiles rather than the top-of-file apt RUN, whose cache key never changes and would freeze the Debian index with it. The package set is computed from dpkg rather than hand-listed. One source package produces many binaries -- util-linux, bsdutils, login, mount, liblastlog2-2, libblkid1, libmount1, libsmartcols1, libuuid1 -- and scanners report each separately, so a hand-written list silently leaves behind whichever binary it forgot. liblastlog2-2 is that binary: new in Debian 13, installed in the base at 2.41-5, and absent from every list you would write from memory. Asking dpkg which installed packages came from the util-linux source cannot miss one. The query is guarded because the failure mode is silent: `apt-get install --only-upgrade` with no package arguments exits 0, so a query that returned nothing would produce a green build that shipped the vulnerable packages anyway. An empty result now fails the build instead. Verified in a real build layer on python:3.12-slim-trixie: all nine land on the fixed versions, and the guard fires with a clear message when the query matches nothing. teradatasql: the wheel ships all ten platform builds it supports -- seven Linux/AIX .so variants, a Windows .dll pair, a macOS .dylib -- 337 MB of Go shared objects, of which exactly one is ever dlopen()ed. Every one of them reports the full go/stdlib set (CVE-2026-39821 Critical, CVE-2026-33818, CVE-2026-46600, CVE-2026-56853, CVE-2026-56859, CVE-2026-56862, CVE-2026-56858). This does not fix those CVEs: 20.0.0.65 is the newest release on PyPI and is built with go1.26.5, and the fix is go1.26.6 -- there is nothing to upgrade to until Teradata rebuilds. It does cut the flagged artifacts from ten to two and the package from 337 MB to ~48 MB, leaving only the code the image can actually load. The strip keeps two files, not one: the driver picks the `fips` variant when the host kernel reports /proc/sys/crypto/fips_enabled == 1, which is a property of the node, not the build. It also refuses to delete anything until every keeper for the platform has been found on disk and dlopen()ed, so a surprise leaves a working driver rather than a broken one. Verified against openmetadata/ingestion 2.0.0-rc2 (linux/arm64): 337 MB -> 48 MB, ten Go artifacts -> two, `import teradatasql` and dlopen of the kept library both still succeed, the script is idempotent, and all three failure paths (keeper missing, keeper corrupt, package absent) delete nothing and exit 0.
/usr/bin/docker is a 42 MB Go executable inherited from the apache/airflow base. It reports the same go/stdlib set as the teradatasql libraries (CVE-2026-39821 Critical, plus five Highs and a Medium), and unlike those it has no upgrade path here: Docker's apt repo is not configured in the image, so --only-upgrade cannot reach a rebuilt package. Removing it is the only remediation available, and it is the one Go artifact in these images that can be eliminated outright rather than merely reduced. Nothing in the image invokes the binary. Airflow's DockerOperator and the ingestion test helpers both drive the daemon through the docker-py SDK over the socket. apt-get -s purge confirms no installed package depends on it. Kept as its own commit because this changes the shipped image surface rather than just its package versions -- anyone who execs into the container and runs `docker` by hand loses that. Revert this commit alone if that matters.
cb8e32f to
643390b
Compare
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Review catch: keepers_are_loadable() required *both* keepers to dlopen before anything was deleted, so a FIPS variant that failed to load took the whole strip with it -- 337 MB and ten flagged artifacts left in place behind nothing but a stderr warning, on a green build. Testing the FIPS variant was never worth that risk. It is only ever selected on a host whose kernel reports fips_enabled=1, which a build machine is not, so loading it here proves nothing about the environment that will actually use it. It is also a second Go c-shared object: dlopening it alongside the non-FIPS runtime spins up a second Go runtime in the same process, which can fail -- or abort the interpreter -- for reasons unrelated to whether the file is good. The gate is now the non-FIPS keeper alone. Safety is unchanged: the FIPS variant is in `keepers`, and the delete loop skips those, so it is never removed either way. Verified on both architectures this time, not just arm64: linux/arm64 337 MB -> 48 MB, 10 files -> 2, import teradatasql OK linux/amd64 337 MB -> 70 MB, 10 files -> 2, import teradatasql OK Gate paths re-tested: primary corrupt and primary missing still delete nothing; FIPS corrupt and FIPS missing now strip normally (both previously blocked); package absent is still a clean no-op. Every path still exits 0.
…measured values Review catch: the script header claimed the package drops to "~72 MB". That was an estimate written from the amd64 file sizes before anything was built, and it did not match the 48 MB the PR reported. Both numbers were right for different architectures and neither said so -- amd64 measures 70 MB, arm64 48 MB, because the x86 libraries are larger. Now stated as both, marked as measured. The same audit caught a second drift the review did not flag: the three Dockerfile comments said "nine of them dead weight" while the strip keeps two and deletes eight. Exactly one library is loaded on any given host, so nine are idle, but the FIPS variant cannot be deleted at build time because the choice is made from the host kernel -- "nine dead weight" read as though nine were removable. Reworded to say only the two this architecture can load are kept. Comments only; no behaviour change. Re-ran the strip against openmetadata/ingestion:2.0.0-rc2 to confirm: strips 8, import teradatasql OK.
Code Review ✅ Approved 1 resolved / 1 findingsUpgrades util-linux to clear critical CVEs and strips unused Go architectures from the teradatasql wheel alongside dropping the docker CLI, addressing the missing FIPS variant dlopen handling. No issues found. ✅ 1 resolved✅ Edge Case: Strip silently no-ops if FIPS variant can't dlopen at build
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
…e ingestion images (#31612) * fix(security): clear util-linux CVEs and cut the Go CVE surface in the ingestion images Two unrelated root causes behind the current Inspector/Snyk findings on the ingestion images. util-linux: the trixie base ships 2.41-5, which carries CVE-2025-14104, CVE-2026-13595 and CVE-2026-27456. trixie-security has 2.41.5-0+deb13u1, so this is a straight upgrade -- added to the late root layer in both ingestion-base Dockerfiles rather than the top-of-file apt RUN, whose cache key never changes and would freeze the Debian index with it. The package set is computed from dpkg rather than hand-listed. One source package produces many binaries -- util-linux, bsdutils, login, mount, liblastlog2-2, libblkid1, libmount1, libsmartcols1, libuuid1 -- and scanners report each separately, so a hand-written list silently leaves behind whichever binary it forgot. liblastlog2-2 is that binary: new in Debian 13, installed in the base at 2.41-5, and absent from every list you would write from memory. Asking dpkg which installed packages came from the util-linux source cannot miss one. The query is guarded because the failure mode is silent: `apt-get install --only-upgrade` with no package arguments exits 0, so a query that returned nothing would produce a green build that shipped the vulnerable packages anyway. An empty result now fails the build instead. Verified in a real build layer on python:3.12-slim-trixie: all nine land on the fixed versions, and the guard fires with a clear message when the query matches nothing. teradatasql: the wheel ships all ten platform builds it supports -- seven Linux/AIX .so variants, a Windows .dll pair, a macOS .dylib -- 337 MB of Go shared objects, of which exactly one is ever dlopen()ed. Every one of them reports the full go/stdlib set (CVE-2026-39821 Critical, CVE-2026-33818, CVE-2026-46600, CVE-2026-56853, CVE-2026-56859, CVE-2026-56862, CVE-2026-56858). This does not fix those CVEs: 20.0.0.65 is the newest release on PyPI and is built with go1.26.5, and the fix is go1.26.6 -- there is nothing to upgrade to until Teradata rebuilds. It does cut the flagged artifacts from ten to two and the package from 337 MB to ~48 MB, leaving only the code the image can actually load. The strip keeps two files, not one: the driver picks the `fips` variant when the host kernel reports /proc/sys/crypto/fips_enabled == 1, which is a property of the node, not the build. It also refuses to delete anything until every keeper for the platform has been found on disk and dlopen()ed, so a surprise leaves a working driver rather than a broken one. Verified against openmetadata/ingestion 2.0.0-rc2 (linux/arm64): 337 MB -> 48 MB, ten Go artifacts -> two, `import teradatasql` and dlopen of the kept library both still succeed, the script is idempotent, and all three failure paths (keeper missing, keeper corrupt, package absent) delete nothing and exit 0. * fix(security): drop the unused docker CLI from the ingestion image /usr/bin/docker is a 42 MB Go executable inherited from the apache/airflow base. It reports the same go/stdlib set as the teradatasql libraries (CVE-2026-39821 Critical, plus five Highs and a Medium), and unlike those it has no upgrade path here: Docker's apt repo is not configured in the image, so --only-upgrade cannot reach a rebuilt package. Removing it is the only remediation available, and it is the one Go artifact in these images that can be eliminated outright rather than merely reduced. Nothing in the image invokes the binary. Airflow's DockerOperator and the ingestion test helpers both drive the daemon through the docker-py SDK over the socket. apt-get -s purge confirms no installed package depends on it. Kept as its own commit because this changes the shipped image surface rather than just its package versions -- anyone who execs into the container and runs `docker` by hand loses that. Revert this commit alone if that matters. * fix(security): gate the teradatasql strip on the non-FIPS library only Review catch: keepers_are_loadable() required *both* keepers to dlopen before anything was deleted, so a FIPS variant that failed to load took the whole strip with it -- 337 MB and ten flagged artifacts left in place behind nothing but a stderr warning, on a green build. Testing the FIPS variant was never worth that risk. It is only ever selected on a host whose kernel reports fips_enabled=1, which a build machine is not, so loading it here proves nothing about the environment that will actually use it. It is also a second Go c-shared object: dlopening it alongside the non-FIPS runtime spins up a second Go runtime in the same process, which can fail -- or abort the interpreter -- for reasons unrelated to whether the file is good. The gate is now the non-FIPS keeper alone. Safety is unchanged: the FIPS variant is in `keepers`, and the delete loop skips those, so it is never removed either way. Verified on both architectures this time, not just arm64: linux/arm64 337 MB -> 48 MB, 10 files -> 2, import teradatasql OK linux/amd64 337 MB -> 70 MB, 10 files -> 2, import teradatasql OK Gate paths re-tested: primary corrupt and primary missing still delete nothing; FIPS corrupt and FIPS missing now strip normally (both previously blocked); package absent is still a clean no-op. Every path still exits 0. * docs(security): correct the teradatasql size and count claims to the measured values Review catch: the script header claimed the package drops to "~72 MB". That was an estimate written from the amd64 file sizes before anything was built, and it did not match the 48 MB the PR reported. Both numbers were right for different architectures and neither said so -- amd64 measures 70 MB, arm64 48 MB, because the x86 libraries are larger. Now stated as both, marked as measured. The same audit caught a second drift the review did not flag: the three Dockerfile comments said "nine of them dead weight" while the strip keeps two and deletes eight. Exactly one library is loaded on any given host, so nine are idle, but the FIPS variant cannot be deleted at build time because the choice is made from the host kernel -- "nine dead weight" read as though nine were removable. Reworded to say only the two this architecture can load are kept. Comments only; no behaviour change. Re-ran the strip against openmetadata/ingestion:2.0.0-rc2 to confirm: strips 8, import teradatasql OK.
…e ingestion images (#31612) * fix(security): clear util-linux CVEs and cut the Go CVE surface in the ingestion images Two unrelated root causes behind the current Inspector/Snyk findings on the ingestion images. util-linux: the trixie base ships 2.41-5, which carries CVE-2025-14104, CVE-2026-13595 and CVE-2026-27456. trixie-security has 2.41.5-0+deb13u1, so this is a straight upgrade -- added to the late root layer in both ingestion-base Dockerfiles rather than the top-of-file apt RUN, whose cache key never changes and would freeze the Debian index with it. The package set is computed from dpkg rather than hand-listed. One source package produces many binaries -- util-linux, bsdutils, login, mount, liblastlog2-2, libblkid1, libmount1, libsmartcols1, libuuid1 -- and scanners report each separately, so a hand-written list silently leaves behind whichever binary it forgot. liblastlog2-2 is that binary: new in Debian 13, installed in the base at 2.41-5, and absent from every list you would write from memory. Asking dpkg which installed packages came from the util-linux source cannot miss one. The query is guarded because the failure mode is silent: `apt-get install --only-upgrade` with no package arguments exits 0, so a query that returned nothing would produce a green build that shipped the vulnerable packages anyway. An empty result now fails the build instead. Verified in a real build layer on python:3.12-slim-trixie: all nine land on the fixed versions, and the guard fires with a clear message when the query matches nothing. teradatasql: the wheel ships all ten platform builds it supports -- seven Linux/AIX .so variants, a Windows .dll pair, a macOS .dylib -- 337 MB of Go shared objects, of which exactly one is ever dlopen()ed. Every one of them reports the full go/stdlib set (CVE-2026-39821 Critical, CVE-2026-33818, CVE-2026-46600, CVE-2026-56853, CVE-2026-56859, CVE-2026-56862, CVE-2026-56858). This does not fix those CVEs: 20.0.0.65 is the newest release on PyPI and is built with go1.26.5, and the fix is go1.26.6 -- there is nothing to upgrade to until Teradata rebuilds. It does cut the flagged artifacts from ten to two and the package from 337 MB to ~48 MB, leaving only the code the image can actually load. The strip keeps two files, not one: the driver picks the `fips` variant when the host kernel reports /proc/sys/crypto/fips_enabled == 1, which is a property of the node, not the build. It also refuses to delete anything until every keeper for the platform has been found on disk and dlopen()ed, so a surprise leaves a working driver rather than a broken one. Verified against openmetadata/ingestion 2.0.0-rc2 (linux/arm64): 337 MB -> 48 MB, ten Go artifacts -> two, `import teradatasql` and dlopen of the kept library both still succeed, the script is idempotent, and all three failure paths (keeper missing, keeper corrupt, package absent) delete nothing and exit 0. * fix(security): drop the unused docker CLI from the ingestion image /usr/bin/docker is a 42 MB Go executable inherited from the apache/airflow base. It reports the same go/stdlib set as the teradatasql libraries (CVE-2026-39821 Critical, plus five Highs and a Medium), and unlike those it has no upgrade path here: Docker's apt repo is not configured in the image, so --only-upgrade cannot reach a rebuilt package. Removing it is the only remediation available, and it is the one Go artifact in these images that can be eliminated outright rather than merely reduced. Nothing in the image invokes the binary. Airflow's DockerOperator and the ingestion test helpers both drive the daemon through the docker-py SDK over the socket. apt-get -s purge confirms no installed package depends on it. Kept as its own commit because this changes the shipped image surface rather than just its package versions -- anyone who execs into the container and runs `docker` by hand loses that. Revert this commit alone if that matters. * fix(security): gate the teradatasql strip on the non-FIPS library only Review catch: keepers_are_loadable() required *both* keepers to dlopen before anything was deleted, so a FIPS variant that failed to load took the whole strip with it -- 337 MB and ten flagged artifacts left in place behind nothing but a stderr warning, on a green build. Testing the FIPS variant was never worth that risk. It is only ever selected on a host whose kernel reports fips_enabled=1, which a build machine is not, so loading it here proves nothing about the environment that will actually use it. It is also a second Go c-shared object: dlopening it alongside the non-FIPS runtime spins up a second Go runtime in the same process, which can fail -- or abort the interpreter -- for reasons unrelated to whether the file is good. The gate is now the non-FIPS keeper alone. Safety is unchanged: the FIPS variant is in `keepers`, and the delete loop skips those, so it is never removed either way. Verified on both architectures this time, not just arm64: linux/arm64 337 MB -> 48 MB, 10 files -> 2, import teradatasql OK linux/amd64 337 MB -> 70 MB, 10 files -> 2, import teradatasql OK Gate paths re-tested: primary corrupt and primary missing still delete nothing; FIPS corrupt and FIPS missing now strip normally (both previously blocked); package absent is still a clean no-op. Every path still exits 0. * docs(security): correct the teradatasql size and count claims to the measured values Review catch: the script header claimed the package drops to "~72 MB". That was an estimate written from the amd64 file sizes before anything was built, and it did not match the 48 MB the PR reported. Both numbers were right for different architectures and neither said so -- amd64 measures 70 MB, arm64 48 MB, because the x86 libraries are larger. Now stated as both, marked as measured. The same audit caught a second drift the review did not flag: the three Dockerfile comments said "nine of them dead weight" while the strip keeps two and deletes eight. Exactly one library is loaded on any given host, so nine are idle, but the FIPS variant cannot be deleted at build time because the choice is made from the host kernel -- "nine dead weight" read as though nine were removable. Reworded to say only the two this architecture can load are kept. Comments only; no behaviour change. Re-ran the strip against openmetadata/ingestion:2.0.0-rc2 to confirm: strips 8, import teradatasql OK.
Fixes the util-linux CVEs in the ingestion images and cuts the go/stdlib surface. Collate side: open-metadata/openmetadata-collate#5825 — the two must land together, since the Collate image inherits a pinned
ingestion-basetag.util-linux — fixed
The trixie base ships 2.41-5;
trixie-securityhas 2.41.5-0+deb13u1, which fixes all three. Added to the late root layer of bothingestion-baseDockerfiles, not the top-of-file aptRUN— that layer's cache key never changes, so an upgrade there freezes its Debian index with it.The package set is computed from dpkg, not hand-listed:
One source package produces many binaries —
util-linux,bsdutils,login,mount,liblastlog2-2,libblkid1,libmount1,libsmartcols1,libuuid1— and scanners report each separately, so a hand-written list silently leaves behind whichever binary it forgot.liblastlog2-2is exactly that binary: new in Debian 13, installed in the base at 2.41-5, and absent from any list written from memory. Asking dpkg which installed packages came from theutil-linuxsource cannot miss one.The query is guarded:
apt-get install --only-upgradewith no package arguments exits 0, so a query that silently returned nothing would give a green build that shipped the vulnerable packages anyway. An empty result fails the build instead.Verified in a real build layer — all nine land on the fix (
2.41.5-0+deb13u1,loginas1:4.16.0-2+really2.41.5-0+deb13u1).go/stdlib — not fixable, surface reduced
Every Go artifact in a published ingestion image, all built with go1.26.5. Inventory taken from
openmetadata/ingestion:2.0.0-rc2— picked simply because it is a public tag that ships theteradataextra, so this is a real build rather than a synthetic one:teradatasql/teradatasql.*.{so,dll,dylib}teradatasqlalchemy→teradatasqladbc_driver_flightsql/libadbc_driver_flightsql.soiometeextra/usr/bin/dockerdocker-ce-cli, from the airflow baseNo version bump fixes these.
teradatasql20.0.0.65 andadbc-driver-flightsql1.12.0 are the newest releases on PyPI and both ship go1.26.5 builds. The fix is go1.26.6 — they clear only when Teradata and Apache Arrow rebuild.What this PR does is cut the surface. The
teradatasqlwheel ships all ten platform builds it supports — 337 MB, of which exactly one file is everdlopen()ed.ingestion/scripts/strip_teradatasql_arch_libs.shremoves the ones this image cannot load: 10 artifacts → 2, and 337 MB → 48 MB on arm64 / 70 MB on amd64 (the x86 libraries are larger).Two files are kept, not one: the driver picks the
fipsvariant when the host kernel reports/proc/sys/crypto/fips_enabled == 1— a property of the node, not the build. The script mirrors the driver's selection branch-for-branch, derives the arch at build time (multi-arch images), and deletes nothing until every keeper has been found on disk anddlopen()ed. Every path exits 0.adbc_driver_flightsqlstays — real runtime dependency, single-arch, nothing to strip. Both extras are in theslimbuild too (filter_requirementstakes an exclusion set), so these counts apply toingestion-base-slim, the image the Snyk target builds.Commits
DockerOperatorand the test helpers both use thedocker-pySDK over the socket, andapt-get -s purgeshows no dependents. Anyone who execs in and runsdockerby hand loses that; revert this commit alone if that matters.Verification
Layer-level builds, not scripts in isolation (
linux/arm64):python:3.12-slim-trixie→ all nine util-linux binaries on the fixed versions above.openmetadata/ingestion:2.0.0-rc2viaCOPY --chown+RUN→ 337 MB → 48 MB (arm64), 10 artifacts → 2,import teradatasqlanddlopenstill succeed.ingestion-basechain is covered separately: same strip under its conditions (useradd -u 1000 openmetadata,HOME=/home/openmetadata, user-site install) — rules out a silent no-op from the interpreter not seeing the install.linux/arm64337 MB → 48 MB andlinux/amd64337 MB → 70 MB, 10 artifacts → 2,import teradatasqlOK on each.docker build --checkclean (the oneWORKDIR ingestion/warning is pre-existing).A full multi-hour image build was not run; each changed layer was built and inspected directly.
Docker/Python only — no Java, no UI. Cherry-picks cleanly to
1.13and2.0(identical Dockerfiles apart fromRI_VERSION).🤖 Generated with Claude Code
Greptile Summary
This PR reduces the CVE surface of ingestion images while preserving the runtime libraries selected for each supported platform.
Confidence Score: 5/5
The PR appears safe to merge because no blocking failure remains.
No blocking failure remains.
Important Files Changed
Reviews (6): Last reviewed commit: "Merge branch 'main' into security/ingest..." | Re-trigger Greptile