Skip to content

fix(operator): detect vMCP volume drift - #6299

Merged
ChrisJBurns merged 4 commits into
stacklok:mainfrom
jstar0:fix/5619-vmcp-volume-drift
Sep 1, 2026
Merged

fix(operator): detect vMCP volume drift#6299
ChrisJBurns merged 4 commits into
stacklok:mainfrom
jstar0:fix/5619-vmcp-volume-drift

Conversation

@jstar0

@jstar0 jstar0 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Changes to Secrets, ConfigMaps, and CA bundles referenced by a vMCP deployment can change the generated pod volumes without changing the existing pod-template hash. The operator therefore leaves a stale deployment running.

This change hashes the complete generated volume and volume-mount state used by the pod template and compares that hash during reconciliation. It also removes the annotation when the desired state no longer contains a volume hash.

Fixes #5619

Type of change

  • Bug fix
  • New feature
  • Refactoring (no behavior change)
  • Dependency update
  • Documentation
  • Other (describe):

Test plan

  • Unit tests (go test ./cmd/thv-operator/controllers -count=1)
  • E2E tests (task test-e2e)
  • Linting (task lint-fix)
  • Manual testing (reviewed generated volume state and annotation cleanup paths)

API Compatibility

  • This PR does not break the v1beta1 API.

Changes

File Change
virtualmcpserver_controller.go Compare generated volume hash and prune stale hash annotations.
virtualmcpserver_deployment.go Hash generated volumes and mounts from the same builder output used by the pod template.
controller tests Cover volume drift detection and annotation cleanup.

Does this introduce a user-facing change?

No. The operator now rolls deployments when referenced volume inputs change.

Special notes for reviewers

The focused controller package tests pass on the rebased branch.

Hash the complete generated volume and volume-mount state so changes to referenced Secrets, ConfigMaps, and CA bundles roll the Deployment.

Signed-off-by: King Star <mcxin.y@gmail.com>
@ChrisJBurns

Copy link
Copy Markdown
Collaborator

@jstar0 I think you'll have to rebase and resolve some conflicts

@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.47170% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.02%. Comparing base (6c14396) to head (f7143a6).
⚠️ Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
...perator/controllers/virtualmcpserver_deployment.go 76.08% 11 Missing ⚠️
...perator/controllers/virtualmcpserver_controller.go 71.42% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6299      +/-   ##
==========================================
- Coverage   78.08%   78.02%   -0.06%     
==========================================
  Files         767      767              
  Lines       74269    74307      +38     
==========================================
- Hits        57992    57980      -12     
- Misses      16272    16322      +50     
  Partials        5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jstar0 added 2 commits August 29, 2026 15:10
…drift

# Conflicts:
#	cmd/thv-operator/controllers/virtualmcpserver_controller.go
#	cmd/thv-operator/controllers/virtualmcpserver_deployment.go
Signed-off-by: King Star <mcxin.y@gmail.com>
@jstar0

jstar0 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed the branch with current main via signed merge commit 2e8b4f05, resolving the mergeDeploymentAnnotations and deployment-metadata conflicts: the prune list now covers all four operator-owned hash/checksum annotations, volume construction stays centralized in buildPodVolumesForVmcp (which already folds in the CA-bundle, telemetry, and embedded auth-server volumes added on main), and the volume-drift regressions adopt the caBundleChecksum parameter added by #6428 (a42ccfe0).

Verification on the merged head: go build ./cmd/thv-operator/..., go test ./cmd/thv-operator/controllers/... (29s, ok), and golangci-lint (v2.12.2, repo config) on ./cmd/thv-operator/controllers/... all pass. A scoped govulncheck run reports only Go 1.26.5 standard-library advisories fixed in go1.26.6, unrelated to this change.

ChrisJBurns
ChrisJBurns previously approved these changes Aug 29, 2026

@ChrisJBurns ChrisJBurns left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-Agent Consensus Review

Agents consulted: Kubernetes reconciliation, Go/code quality, test/intent, Codex cross-review

Consensus Summary

# Finding Consensus Severity Action
1 Add CA-bundle drift and hash-contract regressions 10/10 MEDIUM Fix
2 Isolate the legacy missing-annotation transition 8/10 LOW Fix
3 Make the volume hash a required input 9/10 LOW Fix

Overall

This PR fixes false-negative vMCP volume drift by centralizing generated volume construction and storing a deterministic fingerprint on the Deployment. The approach is sound: the PodSpec and annotation use the same desired snapshot, ordering-only changes are normalized, and operator-owned annotation cleanup preserves unrelated metadata.

The findings are about making the regression guarantees load-bearing across the CA-bundle and upgrade paths, and expressing the hash invariant at compile time. None identifies a current runtime correctness defect, but addressing them would reduce the chance of this drift class returning.


Generated with Codex using the dev:pr-review workflow

Comment thread cmd/thv-operator/controllers/virtualmcpserver_deployment_test.go
Comment thread cmd/thv-operator/controllers/virtualmcpserver_controller_test.go
Comment thread cmd/thv-operator/controllers/virtualmcpserver_deployment.go Outdated
Make the volume hash a required deployment metadata input and pin its ordering and full-field contract. Exercise ConfigMap-backed CA drift and the legacy missing-annotation transition through fake clients.

Signed-off-by: King Star <mcxin.y@gmail.com>
@ChrisJBurns

Copy link
Copy Markdown
Collaborator

/retest

@ChrisJBurns
ChrisJBurns merged commit 8b6d670 into stacklok:main Sep 1, 2026
56 of 57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vMCP drift detection never compares volumes/volumeMounts, so auth-server (and CA-bundle) secret changes don't redeploy

2 participants