ci(executorch): run a coalesced TensorRT + CUDA program in the reference runner gate - #4572
Open
shoumikhin wants to merge 1 commit into
Open
ci(executorch): run a coalesced TensorRT + CUDA program in the reference runner gate#4572shoumikhin wants to merge 1 commit into
shoumikhin wants to merge 1 commit into
Conversation
shoumikhin
force-pushed
the
coalesced-exec-gate
branch
from
August 25, 2026 19:51
9f579dc to
10c8110
Compare
…nce runner gate
The ExecuTorch gate exports one program, x + 1, and TensorRT takes it whole. So
nothing in CI ever runs a program where TensorRT and ExecuTorch's own CUDA
backend each own part of the same graph. That coalesced case is the whole point
of combining the two backends, and it is not covered end to end today. There is
a composition test that checks both delegates land in the file, but it never
loads or runs the program.
This adds the missing run.
A new example, examples/torchtrt_executorch_example/export_coalesced.py, exports
cos(erfinv(tanh(x))). TensorRT has no converter for erfinv, so a CudaPartitioner
catch-all gives that operator to the CUDA backend while TensorRT keeps the rest.
The script fails if the saved .pte does not carry both a TensorRTBackend and a
CudaBackend delegate, so a partitioning change cannot quietly turn this into a
TensorRT-only run that still passes.
The script also writes <model>.expected next to the .pte, holding the output
shape and the eager reference value for an all-ones input. Both reference
runners fill inputs with 1.0 and this model is elementwise, so one number
describes the whole expected output. Reading it from a file, instead of
hard-coding a number in the shell script, keeps the expectation tied to the
model.
verify-executorch-reference-runner.sh now takes an optional third argument, the
coalesced .pte. When given, it runs both the CMake-built runner and the packaged
runner on it and compares every printed value against that reference. TensorRT,
AOTInductor and eager PyTorch use different kernels for the same math, so the
comparison uses a tolerance of 0.001 rather than matching printed digits.
The existing x + 1 assertions keep the same strength. They now go through the
same helper with a zero tolerance, because x + 1 on ones is exact in float32.
Usage:
python examples/torchtrt_executorch_example/export_coalesced.py \
--model_path=coalesced.pte
.github/scripts/verify-executorch-reference-runner.sh \
model.pte kv_cache_decode.pte coalesced.pte
Test plan
On a Linux x86_64 host with an NVIDIA A100 GPU:
- Ran export_coalesced.py. It reported delegates
['TensorRTBackend', 'CudaBackend', 'TensorRTBackend'] and wrote "[64,64]" and
"0.6722" into the .expected file.
- Ran the resulting .pte through the reference runner. It printed
"output[0] shape=[64,64]" and first 8 values of 0.6722, an exact match to the
eager result.
- Deleted the aoti_cuda_blob.ptd that the CUDA backend writes and ran again.
Same output, so this model needs no external weight file.
- Exercised the new shell assertion helper against captured runner output:
correct output passes; one wrong value fails; a wrong shape fails; a missing
values line fails; a value inside the tolerance passes and one outside it
fails.
- shellcheck, bash -n, black and isort are clean on the changed files.
Not yet observed in CI: the ExecuTorch runtime build job currently fails on main
when the packaged reference runner aborts on the existing x + 1 model, and the
test job is skipped while that is true. Both happen before this new code runs.
shoumikhin
force-pushed
the
coalesced-exec-gate
branch
from
August 26, 2026 18:05
10c8110 to
eef38c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The ExecuTorch gate exports one program,
x + 1, and TensorRT takes it whole. Sonothing in CI ever runs a program where TensorRT and ExecuTorch's own CUDA backend
each own part of the same graph. That coalesced case is the whole point of
combining the two backends, and it is not covered end to end today. There is a
composition test that checks both delegates land in the file, but it never loads
or runs the program.
This adds the missing run.
A new example,
examples/torchtrt_executorch_example/export_coalesced.py, exportscos(erfinv(tanh(x))). TensorRT has no converter forerfinv, so aCudaPartitionercatch-all gives that operator to the CUDA backend while TensorRTkeeps the rest. The script fails if the saved
.ptedoes not carry both aTensorRTBackendand aCudaBackenddelegate, so a partitioning change cannotquietly turn this into a TensorRT-only run that still passes.
The script also writes
<model>.expectednext to the.pte, holding the outputshape and the eager reference value for an all-ones input. Both reference runners
fill inputs with 1.0 and this model is elementwise, so one number describes the
whole expected output. Reading it from a file, instead of hard-coding a number in
the shell script, keeps the expectation tied to the model.
verify-executorch-reference-runner.shnow takes an optional third argument, thecoalesced
.pte. When given, it runs both the CMake-built runner and the packagedrunner on it and compares every printed value against that reference. TensorRT,
AOTInductor and eager PyTorch use different kernels for the same math, so the
comparison uses a tolerance of 0.001 rather than matching printed digits.
The existing
x + 1assertions keep the same strength. They now go through thesame helper with a zero tolerance, because
x + 1on ones is exact in float32.Usage:
Type of change
Test plan
On a Linux x86_64 host with an NVIDIA A100 GPU:
export_coalesced.py. It reported delegates['TensorRTBackend', 'CudaBackend', 'TensorRTBackend']and wrote[64,64]and0.6722into the.expectedfile..ptethrough the reference runner. It printedoutput[0] shape=[64,64]and first 8 values of0.6722, an exact match to theeager result.
aoti_cuda_blob.ptdthat the CUDA backend writes and ran again.Same output, so this model needs no external weight file.
correct output passes; one wrong value fails; a wrong shape fails; a missing
values line fails; a value inside the tolerance passes and one outside it fails.
shellcheck,bash -n,blackandisortare clean on the changed files.Not yet observed in CI: the ExecuTorch runtime build job currently fails on main
when the packaged reference runner aborts on the existing
x + 1model, and thetest job is skipped while that is true. Both happen before this new code runs.
Checklist: