Conversation
…stock) Alex's original NVFP4 crash was the AGG worker, which uses the stock nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.1-efa directly. That image ships flashinfer_cubin root-owned, and FLASHINFER_CUBIN_DIR is inert (flashinfer prefers the installed package over the env var), so NVFP4 profile_run crashes on the non-root worker with PermissionError[13]. The golden image (../disagg/dynamo-vllm-efa/Dockerfile) already 'covers every scenario: aggregated, EP16, PP>1' and now carries the build-time cubin chmod fix (merged in aws-samples#43). Point agg/.env at it (public.ecr.aws/hpc-cloud/ dynamo-vllm-efa:1.3.1-patched, overridable), matching how disagg/.env already works, so NVFP4 agg works out of the box with no initContainer / runAsUser. Correct the now-misleading FLASHINFER_CUBIN_DIR comment. BF16 works on either. Signed-off-by: Anton Alexander <dmvevents@gmail.com>
iankouls-aws
pushed a commit
that referenced
this pull request
Aug 8, 2026
Alex asked for the missing agg DGD (we only had disagg/dgd.yaml-template).
Adds agg/dgd.yaml-template: the operator-managed (DynamoGraphDeployment,
nvidia.com/v1alpha1) equivalent of agg/deployment.yaml-template — Frontend +
a single VllmWorker (componentType worker, NO prefill/decode split, NO NIXL /
disaggregation-mode / KV-transfer), TP=${GPU_PER_WORKER}/PP1, single node.
Carries the operator gotchas proven in disagg/dgd.yaml-template: v1alpha1
storage version, worker port named 'system' (operator injects probes by name),
no frontend readinessProbe (operator injects its own), HF_HOME=/tmp (uid 1000
can't write root-owned /shared). Uses the golden image via .env; wire
MANIFEST_TYPE=dgd into run.sh/stop.sh.
NVFP4 note: depends on the golden image (agg/.env → golden, PR #44) for the
cubin fix; on the current stock 1.3.1-efa .env, BF16 works but NVFP4 needs #44
merged first.
Signed-off-by: Anton Alexander <dmvevents@gmail.com>
Contributor
|
will not merge ... suggested changes are already in place |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gap
The NVFP4 cubin fix merged in #43 lives in the golden
dynamo-vllm-efaDockerfile, which the disagg path builds and uses. But the agg path — which is where the original crash happened (nemotron3-ultra-550b-agg-worker) — still pointsagg/.envat the stocknvcr.io/nvidia/ai-dynamo/vllm-runtime:1.3.1-efa. So NVFP4 agg still crashes:FLASHINFER_CUBIN_DIR(set in the agg template) does not fix it — flashinfer's_get_cubin_dir()(jit/env.py) prefers the installedflashinfer_cubinpackage over the env var, and NVFP4 symlinks cubins into that root-owned package dir.Fix
The golden image already states it "covers every scenario: aggregated, EP16, PP>1" and now carries the build-time cubin chmod (from #43). Point
agg/.envat it —public.ecr.aws/hpc-cloud/dynamo-vllm-efa:1.3.1-patched(overridable), exactly howdisagg/.envalready works — so NVFP4 agg works out of the box with the worker still non-root, no initContainer, norunAsUser. Also corrects the now-misleadingFLASHINFER_CUBIN_DIRcomment. BF16 works on either image.Aligns with the maintainer's stated preference ("we don't need to use the nvidia stock image until they fix it").
Verification
agg/.envsources cleanly and resolves topublic.ecr.aws/hpc-cloud/dynamo-vllm-efa:1.3.1-patched; MODEL_VARIANT BF16/NVFP4 toggle intact.deployment.yaml-templatestill parses (3 docs); agg worker unchanged otherwise (non-root, no initContainer — the fix is in the image).Related: #43 (build-time cubin fix in the golden Dockerfile, merged); #42 (initContainer variant, closed as superseded).