Skip to content

fix: engines default to no runtime cache; the module owns the implicit one - #4482

Open
tp5uiuc wants to merge 4 commits into
pytorch:mainfrom
tp5uiuc:fix/python-runtime-cache-on-cpp-build
Open

fix: engines default to no runtime cache; the module owns the implicit one#4482
tp5uiuc wants to merge 4 commits into
pytorch:mainfrom
tp5uiuc:fix/python-runtime-cache-on-cpp-build

Conversation

@tp5uiuc

@tp5uiuc tp5uiuc commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Description

What

An engine built without a TorchTensorRTModule came up with a path-string
runtime_cache and attached a cache nothing owned, freeing it out from under
the engine. Engines now default to no cache; the module owns the implicit one.

Why

RuntimeSettings.runtime_cache defaults to a path string, and
TorchTensorRTModule resolves it into a RuntimeCache it owns, so the compile
path is fine. An engine reached without a module -- built from packed engine
info, or loaded as a graph constant from a saved ExportedProgram -- never
passes through that resolver, so TRTRuntimeConfig wrapped the string in a
local, handed the live IRuntimeCache to the engine's IRuntimeConfig, and let
it be collected on return. The next createExecutionContext read freed memory.

The C++ runtime has no such path: RuntimeSettings::runtime_cache is an
intrusive_ptr defaulting to nullptr, with no string form to mis-own.

The module's post-load resets have to move with the engine, or the two disagree:
a module holding a stale path string lets a later runtime_config(...) block --
a call that need not mention caching -- resolve it and leave an autosaving handle
installed at the shared default path on exit.

How

  • _TRTEngine.py: engines construct TRTRuntimeConfig with
    runtime_cache=None, matching the C++ default. RuntimeSettings()'s own
    default is unchanged -- that one belongs to the module, which pushes the
    resolved handle down in setup_engine.
  • _TorchTensorRTModule.py: the set_extra_state / __setstate__ resets drop
    to runtime_cache=None so the module agrees with the engine it rebuilds; an
    empty-string runtime_cache normalizes to None so no string reaches an
    engine.
  • _runtime_config.py: _apply_settings accepts only None or a
    RuntimeCache and raises TypeError on a string; TRTRuntimeConfig's own
    no-settings default follows suit. The docstring already claimed raw strings
    were not accepted here.
  • Tests: module-less engines default to no cache, execute without one, accept an
    explicitly attached RuntimeCache, and reject strings; the compile path still
    builds and persists its implicit cache; the post-load resets leave no cache,
    and a cuda-graph-only context manager over a loaded module installs none on
    enter and leaves none on exit.

Testing

TensorRT-RTX on A100, tests/py/dynamo/runtime/ plus
models/test_runtime_cache_models.py:

Build before after
Python runtime 4 failed, 170 passed, 60 skipped 174 passed, 60 skipped
C++ runtime 4 failed, 171 passed, 59 skipped 175 passed, 59 skipped

The four failures before are the new tests themselves; each was confirmed
failing on the parent commit and passing here, on both builds. There are no
other failures on either build.

models/test_export_serde.py, test_cross_runtime_serde.py and
test_fallback_data_dependent_ops.py were run before and after on both builds
because the post-load resets sit on the torch.load / load_state_dict paths:
identical results either side, with the only failures (test_save_load_aoti,
test_save_load_ts) pre-existing and unrelated.

Save/load deployment was checked on both runtimes: the loaded engine runs with
no cache attached, and an explicitly attached RuntimeCache persists.

Cost / Gotchas

An engine used without a module now gets no runtime cache instead of an implicit
one, and neither do modules restored by torch.load / load_state_dict. No
configuration loses working behaviour: that path raised on the Python runtime,
already attached nothing on the C++ runtime, and never initializes a runtime
config on standard TensorRT. Callers wanting a cache there attach a
RuntimeCache explicitly.

Supersedes #4541 -- the TestPythonRuntimeAliasedIO failures it skipped are
fixed here rather than skipped.

Followups

runtime_config() / runtime_cache() locate engines via named_modules(), so
neither reaches an engine loaded as a graph constant. Attaching settings to a
saved-and-loaded program currently requires the engine API directly.

Restoring implicit caching to load_state_dict -- by re-applying the setter
after engine construction in set_extra_state, as setup_engine does -- is
deliberately left out; it changes C++-runtime behaviour too and wants its own
testing.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@meta-cla meta-cla Bot added the cla signed label Aug 13, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [Python] Issues re: Python API labels Aug 13, 2026
@github-actions
github-actions Bot requested a review from narendasan August 13, 2026 00:42
@tp5uiuc
tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from c3c4cf7 to 03513c2 Compare August 13, 2026 01:36
@tp5uiuc tp5uiuc changed the title fix: python engine crashed attaching a runtime cache on builds with the C++ runtime fix: path-string runtime cache freed itself out from under the engine Aug 13, 2026
@tp5uiuc
tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from 03513c2 to 5522dcd Compare August 13, 2026 01:48
@tp5uiuc tp5uiuc added this to the v2.15.0 milestone Aug 18, 2026
@tp5uiuc
tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch 2 times, most recently from 3a8a16e to f42577c Compare August 21, 2026 00:50
@tp5uiuc tp5uiuc self-assigned this Aug 21, 2026
@tp5uiuc
tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from f42577c to c31831c Compare August 21, 2026 07:51
@github-actions github-actions Bot added component: core Issues re: The core compiler component: runtime component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Aug 21, 2026
@tp5uiuc tp5uiuc changed the title fix: path-string runtime cache freed itself out from under the engine fix: engines default to no runtime cache; the module owns the implicit one Aug 21, 2026
@tp5uiuc
tp5uiuc marked this pull request as ready for review August 21, 2026 08:03
A TRTEngine built without a module came up with the default
RuntimeSettings, whose runtime_cache is a path string. Nothing owned the
wrapper that string implies, so attaching it handed the live IRuntimeCache
to the engine's IRuntimeConfig and then let it be collected -- a
use-after-free at the next createExecutionContext.

Engines now default to runtime_cache=None, matching the cpp side, where
RuntimeSettings::runtime_cache is an intrusive_ptr defaulting to nullptr
and no string form exists. The implicit cache belongs to the module, which
resolves its path string to a RuntimeCache and pushes it down in
setup_engine; that path is unchanged.

TorchTensorRTModule resets its own RuntimeSettings on the post-load paths
(set_extra_state, __setstate__) and those resets move to runtime_cache=None
too, so the module and the engine it rebuilds agree. Leaving them at the
string default would let a later runtime_config(...) block resolve the
stale path and install an autosaving handle at the shared default location
on exit -- switching caching on via a call that never mentioned it.

An engine reached without a module -- built from packed engine info, or
loaded as a graph constant from a saved ExportedProgram -- now runs with no
cache instead of a dangling one, and a caller can attach a RuntimeCache
explicitly. No configuration loses working behaviour: on the Python runtime
this path raised, on the cpp runtime it already attached nothing, and on
standard TensorRT the runtime config is never initialized.
_apply_settings had three arms, and the str one built a RuntimeCache it did
not outlive. Its own docstring already claimed raw strings were not accepted
here; the code twenty lines below accepted them.

Engines now take only something that owns what it points at -- the Python
equivalent of the cpp intrusive_ptr<RuntimeCacheHandle>. A str raises
TypeError naming the module as the place path strings are resolved. The
class's own default follows: a TRTRuntimeConfig built with no settings would
otherwise start from the string form this commit exists to abolish.

TorchTensorRTModule._resolve_runtime_cache normalizes an empty-string
runtime_cache to None rather than passing it through, so no str can reach an
engine from the module. Also corrects the RuntimeSettings.runtime_cache
docstring, which promised the engine owned the implicit handle and saved it
on __del__, and a reference to a method renamed some time ago.
TestEngineOwnsNoCache pins the contract at the engine: a module-less engine
defaults to no cache, executes without one, accepts an explicitly attached
RuntimeCache, and raises TypeError on a path string.

TestModuleStillOwnsImplicitCache guards the other direction -- the compile
path must keep building, attaching and persisting its implicit cache -- and
covers the empty-string normalization.

TestPostLoadOwnsNoCache pins the same contract on the reset paths, where
module and engine could otherwise disagree: the config's own default, and
what torch.load / load_state_dict leave behind. Its last test is the one
that matters -- a cuda-graph-only context manager over a loaded module must
not install a cache on enter or leave one installed on exit, because
re-applying a path string through the setter creates a handle rather than
restoring one.

@cehongwang cehongwang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall OK

logger.warning(f"Failed to warm-load runtime cache from {rc!r}: {e}")
cache = wrapped.ensure_cache(self._live)
self._live.set_runtime_cache(cache)
self._live.set_runtime_cache(rc.ensure_cache(self._live))
else:
raise TypeError(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to destroy the self._live here because the initialization was not successful?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, done in cb95af3 by resetting the live and propagating the exception up the call stack. Thanks Adrian!

ensure_initialized assigns self._live before _apply_settings runs, so an
exception out of the apply left a half-configured IRuntimeConfig in place --
strategies set, runtime cache never attached. The early-return guard at the
top then made the next call a no-op, so a caller who caught the error and
retried proceeded silently against those partial settings; the error was
only ever raised once.

Reset self._live and re-raise instead, so a retry re-attempts initialization
and fails the same way. Measured before the change: first execute raised
TypeError, second returned normally with _live still populated.
@tp5uiuc
tp5uiuc force-pushed the fix/python-runtime-cache-on-cpp-build branch from 509cfbd to cb95af3 Compare August 26, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants