Repository navigation
Add Rust EPContext data callbacks - #32310
Conversation
Promote named-buffer EPContext callbacks to the stable C and C++ APIs with a required finite read-size policy. Add immutable EP config snapshots, strict participating-EP support negotiation, V28 compatibility shims, hardened sample helpers, example plugin adoption, documentation, and focused API/auto-EP tests.
Mark the strict-read flag unused when ONNX partitioning is excluded, and preserve the V28 EPContext config shims directly in minimal builds where the plugin EP API implementation is intentionally not linked.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ca1d201 to
368d7e5
Compare
Use opaque read options consistently, clarify application/EP size enforcement, update version-31 callback support negotiation, and cover ownership and older-plugin boundaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…untime/contrib_ops/cuda/moe/moe_quantization.cc. (#32932) ### Description <!-- Describe your changes. --> Replace old use of `routing_record` with `routing_snapshot_` in onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fix build error about undeclared identifier `routing_record`. Likely an unlucky merge artifact from two recent PRs #32761 and #32738. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Increase the minimal Android section-size budget by 2 KiB. CI measured 1,598,685 bytes versus the prior 1,597,440-byte budget; the new 1,599,488-byte limit retains 803 bytes of headroom. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement review option 3: remove the proposed read-options APIs and runtime size-limit storage, keep callback/state snapshots and strict support negotiation, and leave allocation/deserialization limits to applications and EPs. Retain an EP-selected bound in the sample helper with boundary coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify the external-data lifecycle query, simplify callback flag documentation, keep allocation-policy guidance in the design document, and require explicit C++ callback state arguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove native read-options dependencies and verify callback-owned limits through the native registration snapshot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy the OrtApi table and env pointer under a short-lived environment guard so ORT callbacks invoked during CreateSession/CreateSessionFromArray can use Environment APIs without deadlocking. Add a regression test for lock release. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Check final current-provider assignments as well as newly accepted capabilities, including resource-accounted singleton survivors omitted from the final capability list. Keep placement and accounting unchanged and cover ONNX/ORT loads, supported callbacks, and excluded claims. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the Android workflow and MoE expert-counting conflicts using the current main versions. Preserve the stable EPContext APIs and retained-NHWC READ support fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use effective provider settings to negotiate callback support before every capability-discovery path, including AOT inlining, NHWC passes, and ORT-format loading. TensorRT external output cannot bypass WRITE negotiation through ORT's embed setting, and QNN capability-time context reads require READ support before backend setup. Preserve post-assignment validation and no-callback behavior. Add real-I/O regressions and control cases. Ten baseline failures reproduce the gaps; all 162 targeted framework tests pass after the fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the output dispatch exhaustive after the latest core merge and verify both new types remain explicitly unsupported by the Rust tensor API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tianlei Wu (tianleiwu)
left a comment
There was a problem hiding this comment.
Reviewed the isolated stacked diff against GopalakrishnanN/stable-epcontext-data-api. The callback trampoline contains panics, enforces the configured size bound before native allocation, maps failures explicitly, and retains callback state safely across session creation and teardown. The registration/clearing and compatibility paths are well covered, and all reported CI checks pass.
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The unsafe cross-language callback and lifetime behavior warrants final human review, especially for untested non-Windows and 32-bit targets.
Review effort: Balanced
Findings: None
What changed in this PR
Adds safe Rust bindings for stable EPContext data-read callbacks, including ownership, error handling, and API compatibility. No actionable issues identified.
Changes:
- Adds bounded, concurrent, panic-contained callback registration.
- Preserves callback lifetime and lock-free session creation.
- Updates bindings generation and unsupported tensor-type handling.
| File | Description |
|---|---|
rust/BUILD.md |
Documents header and version overrides. |
rust/onnxruntime-sys/build.rs |
Improves header discovery and release selection. |
rust/onnxruntime/src/environment.rs |
Validates C API availability safely. |
rust/onnxruntime/src/error.rs |
Adds callback and API-version errors. |
rust/onnxruntime/src/lib.rs |
Exports callback errors and tests FLOAT6 handling. |
rust/onnxruntime/src/memory.rs |
Reorders imports. |
rust/onnxruntime/src/session.rs |
Implements callback registration, trampoline, ownership, and tests. |
rust/onnxruntime/src/tensor/ort_output_tensor.rs |
Handles newly generated unsupported enum variants. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The base branch was changed.
Description
Adds Rust bindings for the stable EPContext data read callback APIs in #32265.
max_data_sizestays in Rust callback state and is enforced before native output allocation; allocations inside the callback remain application-owned policy.97d3907fdc: the environment lock is released before native session creation, so callbacks can reenter environment APIs without deadlocking.e0cc266cdefef4e27104b14e4091a9d61f75a214, including its main-branch update, retained-NHWC READ-support validation, and provider-specific callback preflight before capability-time I/O.E0004build failure from the newly merged FLOAT6 enum variants by adding both to the existing unsupported-output dispatch. Extends unsupported-type conversion tests for both variants; this does not add FLOAT6 tensor support.Motivation and Context
This independent binding PR remains stacked on #32265. Rust applications retain bounded, application-managed EPContext reads and safe callback ownership while following the stable transport ABI. Rust currently has no model-compilation/write-callback surface.
The latest core checks provider-specific callback requirements before capability discovery, including AOT, NHWC, and ORT-format paths, rather than relying solely on checks immediately before
Compile(). Its main-branch merge also requires the explicit FLOAT6 compatibility cases so regenerated Rust bindings compile.Retarget to
mainafter #32265 lands.Testing
Revalidated on Windows with Rust 1.99.0, libclang 18, regenerated bindings from current source headers, and the runtime freshly built from
e0cc266cde.cargo fmt --manifest-path .\rust\Cargo.toml --all -- --check— passed, with existing warnings for nightly-only rustfmt settings.ORT_RUST_STRATEGY=system,ORT_RUST_LIB_LOCATIONpointing to the fresh import-library prefix, andRUST_ONNXRUNTIME_LIBRARY_PATHpointing to the freshly rebuilt DLL:cargo test --manifest-path rust\Cargo.toml -j 4 -p onnxruntime --lib -- --test-threads=1— 33 passed, 0 failed/ignored.git diff --check e0cc266cdefef4e27104b14e4091a9d61f75a214 HEAD— passed for the binding PR delta.Non-Windows, 32-bit execution, and hardware-provider-specific callback/preflight paths were not run. Existing compiler warnings outside this compatibility fix remain unchanged.