Skip to content

ci(executorch): run a coalesced TensorRT + CUDA program in the reference runner gate - #4572

Open
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:coalesced-exec-gate
Open

ci(executorch): run a coalesced TensorRT + CUDA program in the reference runner gate#4572
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:coalesced-exec-gate

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

Description

The ExecuTorch gate exports one program, x + 1, and TensorRT takes it whole. So
nothing in CI ever runs a program where TensorRT and ExecuTorch's own CUDA backend
each own part of the same graph. That coalesced case is the whole point of
combining the two backends, and it is not covered end to end today. There is a
composition test that checks both delegates land in the file, but it never loads
or runs the program.

This adds the missing run.

A new example, examples/torchtrt_executorch_example/export_coalesced.py, exports
cos(erfinv(tanh(x))). TensorRT has no converter for erfinv, so a
CudaPartitioner catch-all gives that operator to the CUDA backend while TensorRT
keeps the rest. The script fails if the saved .pte does not carry both a
TensorRTBackend and a CudaBackend delegate, so a partitioning change cannot
quietly turn this into a TensorRT-only run that still passes.

The script also writes <model>.expected next to the .pte, holding the output
shape and the eager reference value for an all-ones input. Both reference runners
fill inputs with 1.0 and this model is elementwise, so one number describes the
whole expected output. Reading it from a file, instead of hard-coding a number in
the shell script, keeps the expectation tied to the model.

verify-executorch-reference-runner.sh now takes an optional third argument, the
coalesced .pte. When given, it runs both the CMake-built runner and the packaged
runner on it and compares every printed value against that reference. TensorRT,
AOTInductor and eager PyTorch use different kernels for the same math, so the
comparison uses a tolerance of 0.001 rather than matching printed digits.

The existing x + 1 assertions keep the same strength. They now go through the
same helper with a zero tolerance, because x + 1 on ones is exact in float32.

Usage:

python examples/torchtrt_executorch_example/export_coalesced.py \
  --model_path=coalesced.pte

.github/scripts/verify-executorch-reference-runner.sh \
  model.pte kv_cache_decode.pte coalesced.pte

Type of change

  • New feature (non-breaking change which adds functionality)

Test plan

On a Linux x86_64 host with an NVIDIA A100 GPU:

  • Ran export_coalesced.py. It reported delegates
    ['TensorRTBackend', 'CudaBackend', 'TensorRTBackend'] and wrote [64,64] and
    0.6722 into the .expected file.
  • Ran the resulting .pte through the reference runner. It printed
    output[0] shape=[64,64] and first 8 values of 0.6722, an exact match to the
    eager result.
  • Deleted the aoti_cuda_blob.ptd that the CUDA backend writes and ran again.
    Same output, so this model needs no external weight file.
  • Exercised the new shell assertion helper against captured runner output:
    correct output passes; one wrong value fails; a wrong shape fails; a missing
    values line fails; a value inside the tolerance passes and one outside it fails.
  • shellcheck, bash -n, black and isort are clean on the changed files.

Not yet observed in CI: the ExecuTorch runtime build job currently fails on main
when the packaged reference runner aborts on the existing x + 1 model, and the
test job is skipped while that is true. Both happen before this new code runs.

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@meta-cla meta-cla Bot added the cla signed label Aug 24, 2026
@github-actions github-actions Bot added the component: tests Issues re: Tests label Aug 24, 2026
@github-actions
github-actions Bot requested a review from lanluo-nvidia August 24, 2026 22:40
@shoumikhin
shoumikhin force-pushed the coalesced-exec-gate branch from 9f579dc to 10c8110 Compare August 25, 2026 19:51
…nce runner gate

The ExecuTorch gate exports one program, x + 1, and TensorRT takes it whole. So
nothing in CI ever runs a program where TensorRT and ExecuTorch's own CUDA
backend each own part of the same graph. That coalesced case is the whole point
of combining the two backends, and it is not covered end to end today. There is
a composition test that checks both delegates land in the file, but it never
loads or runs the program.

This adds the missing run.

A new example, examples/torchtrt_executorch_example/export_coalesced.py, exports
cos(erfinv(tanh(x))). TensorRT has no converter for erfinv, so a CudaPartitioner
catch-all gives that operator to the CUDA backend while TensorRT keeps the rest.
The script fails if the saved .pte does not carry both a TensorRTBackend and a
CudaBackend delegate, so a partitioning change cannot quietly turn this into a
TensorRT-only run that still passes.

The script also writes <model>.expected next to the .pte, holding the output
shape and the eager reference value for an all-ones input. Both reference
runners fill inputs with 1.0 and this model is elementwise, so one number
describes the whole expected output. Reading it from a file, instead of
hard-coding a number in the shell script, keeps the expectation tied to the
model.

verify-executorch-reference-runner.sh now takes an optional third argument, the
coalesced .pte. When given, it runs both the CMake-built runner and the packaged
runner on it and compares every printed value against that reference. TensorRT,
AOTInductor and eager PyTorch use different kernels for the same math, so the
comparison uses a tolerance of 0.001 rather than matching printed digits.

The existing x + 1 assertions keep the same strength. They now go through the
same helper with a zero tolerance, because x + 1 on ones is exact in float32.

Usage:

    python examples/torchtrt_executorch_example/export_coalesced.py \
      --model_path=coalesced.pte
    .github/scripts/verify-executorch-reference-runner.sh \
      model.pte kv_cache_decode.pte coalesced.pte

Test plan

On a Linux x86_64 host with an NVIDIA A100 GPU:

- Ran export_coalesced.py. It reported delegates
  ['TensorRTBackend', 'CudaBackend', 'TensorRTBackend'] and wrote "[64,64]" and
  "0.6722" into the .expected file.
- Ran the resulting .pte through the reference runner. It printed
  "output[0] shape=[64,64]" and first 8 values of 0.6722, an exact match to the
  eager result.
- Deleted the aoti_cuda_blob.ptd that the CUDA backend writes and ran again.
  Same output, so this model needs no external weight file.
- Exercised the new shell assertion helper against captured runner output:
  correct output passes; one wrong value fails; a wrong shape fails; a missing
  values line fails; a value inside the tolerance passes and one outside it
  fails.
- shellcheck, bash -n, black and isort are clean on the changed files.

Not yet observed in CI: the ExecuTorch runtime build job currently fails on main
when the packaged reference runner aborts on the existing x + 1 model, and the
test job is skipped while that is true. Both happen before this new code runs.
@shoumikhin
shoumikhin force-pushed the coalesced-exec-gate branch from 10c8110 to eef38c7 Compare August 26, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant