Skip to content

Add Rust EPContext data callbacks - #32310

Merged
Tianlei Wu (tianleiwu) merged 26 commits into
mainfrom
gnallasamy-microsoft-rust-epcontext-bindings
Oct 6, 2026
Merged

Tianlei Wu (tianleiwu) merged 26 commits into
mainfrom
gnallasamy-microsoft-rust-epcontext-bindings

Conversation

@GopalakrishnanN

@GopalakrishnanN Gopalakrishnan Nallasamy (GopalakrishnanN) commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds Rust bindings for the stable EPContext data read callback APIs in #32265.

  • Provides transactional registration/clearing, Arc-backed callback ownership, explicit callback errors, and an unwind-contained allocator-safe trampoline.
  • Uses API 31 callback/state-only native registration and retrieval. max_data_size stays in Rust callback state and is enforced before native output allocation; allocations inside the callback remain application-owned policy.
  • Preserves the newer fix from 97d3907fdc: the environment lock is released before native session creation, so callbacks can reenter environment APIs without deadlocking.
  • Merges Promote EPContext data callbacks to stable API #32265 at e0cc266cdefef4e27104b14e4091a9d61f75a214, including its main-branch update, retained-NHWC READ-support validation, and provider-specific callback preflight before capability-time I/O.
  • Fixes the resulting Rust E0004 build failure from the newly merged FLOAT6 enum variants by adding both to the existing unsupported-output dispatch. Extends unsupported-type conversion tests for both variants; this does not add FLOAT6 tensor support.
  • The stable callback signatures remain unchanged; preflight remains core-owned.

Motivation and Context

This independent binding PR remains stacked on #32265. Rust applications retain bounded, application-managed EPContext reads and safe callback ownership while following the stable transport ABI. Rust currently has no model-compilation/write-callback surface.

The latest core checks provider-specific callback requirements before capability discovery, including AOT, NHWC, and ORT-format paths, rather than relying solely on checks immediately before Compile(). Its main-branch merge also requires the explicit FLOAT6 compatibility cases so regenerated Rust bindings compile.

Retarget to main after #32265 lands.

Testing

Revalidated on Windows with Rust 1.99.0, libclang 18, regenerated bindings from current source headers, and the runtime freshly built from e0cc266cde.

  • cargo fmt --manifest-path .\rust\Cargo.toml --all -- --check — passed, with existing warnings for nightly-only rustfmt settings.
  • With ORT_RUST_STRATEGY=system, ORT_RUST_LIB_LOCATION pointing to the fresh import-library prefix, and RUST_ONNXRUNTIME_LIBRARY_PATH pointing to the freshly rebuilt DLL: cargo test --manifest-path rust\Cargo.toml -j 4 -p onnxruntime --lib -- --test-threads=1 — 33 passed, 0 failed/ignored.
  • This includes the environment-lock regression, registration/replacement/clearing, ownership, concurrency, size boundaries, allocator failures, panic/error handling, and both new unsupported FLOAT6 conversion cases.
  • git diff --check e0cc266cdefef4e27104b14e4091a9d61f75a214 HEAD — passed for the binding PR delta.

Non-Windows, 32-bit execution, and hardware-provider-specific callback/preflight paths were not run. Existing compiler warnings outside this compatibility fix remain unchanged.

Gopalakrishnan Nallasamy and others added 12 commits August 25, 2026 14:47
Promote named-buffer EPContext callbacks to the stable C and C++ APIs with a required finite read-size policy. Add immutable EP config snapshots, strict participating-EP support negotiation, V28 compatibility shims, hardened sample helpers, example plugin adoption, documentation, and focused API/auto-EP tests.
Mark the strict-read flag unused when ONNX partitioning is excluded, and preserve the V28 EPContext config shims directly in minimal builds where the plugin EP API implementation is intentionally not linked.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@GopalakrishnanN
Gopalakrishnan Nallasamy (GopalakrishnanN) force-pushed the gnallasamy-microsoft-rust-epcontext-bindings branch from ca1d201 to 368d7e5 Compare August 29, 2026 07:16
Gopalakrishnan Nallasamy and others added 13 commits September 29, 2026 15:43
Use opaque read options consistently, clarify application/EP size enforcement, update version-31 callback support negotiation, and cover ownership and older-plugin boundaries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…untime/contrib_ops/cuda/moe/moe_quantization.cc. (#32932)

### Description
<!-- Describe your changes. -->

Replace old use of `routing_record` with `routing_snapshot_` in
onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

Fix build error about undeclared identifier `routing_record`. Likely an
unlucky merge artifact from two recent PRs #32761 and #32738.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Increase the minimal Android section-size budget by 2 KiB. CI measured 1,598,685 bytes versus the prior 1,597,440-byte budget; the new 1,599,488-byte limit retains 803 bytes of headroom.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement review option 3: remove the proposed read-options APIs and runtime size-limit storage, keep callback/state snapshots and strict support negotiation, and leave allocation/deserialization limits to applications and EPs. Retain an EP-selected bound in the sample helper with boundary coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify the external-data lifecycle query, simplify callback flag documentation, keep allocation-policy guidance in the design document, and require explicit C++ callback state arguments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove native read-options dependencies and verify callback-owned limits through the native registration snapshot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy the OrtApi table and env pointer under a short-lived environment guard so ORT callbacks invoked during CreateSession/CreateSessionFromArray can use Environment APIs without deadlocking. Add a regression test for lock release.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Check final current-provider assignments as well as newly accepted capabilities, including resource-accounted singleton survivors omitted from the final capability list. Keep placement and accounting unchanged and cover ONNX/ORT loads, supported callbacks, and excluded claims.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the Android workflow and MoE expert-counting conflicts using the current main versions. Preserve the stable EPContext APIs and retained-NHWC READ support fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use effective provider settings to negotiate callback support before every capability-discovery path, including AOT inlining, NHWC passes, and ORT-format loading. TensorRT external output cannot bypass WRITE negotiation through ORT's embed setting, and QNN capability-time context reads require READ support before backend setup. Preserve post-assignment validation and no-callback behavior.

Add real-I/O regressions and control cases. Ten baseline failures reproduce the gaps; all 162 targeted framework tests pass after the fix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Synchronize with #32265 at e0cc266 while preserving the environment-lock reentrancy fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the output dispatch exhaustive after the latest core merge and verify both new types remain explicitly unsupported by the Rust tensor API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@tianleiwu Tianlei Wu (tianleiwu) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the isolated stacked diff against GopalakrishnanN/stable-epcontext-data-api. The callback trampoline contains panics, enforces the configured size bound before native allocation, maps failures explicitly, and retains callback state safely across session creation and teardown. The registration/clearing and compatibility paths are well covered, and all reported CI checks pass.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The unsafe cross-language callback and lifetime behavior warrants final human review, especially for untested non-Windows and 32-bit targets.

Review effort: Balanced
Findings: None

What changed in this PR

Adds safe Rust bindings for stable EPContext data-read callbacks, including ownership, error handling, and API compatibility. No actionable issues identified.

Changes:

  • Adds bounded, concurrent, panic-contained callback registration.
  • Preserves callback lifetime and lock-free session creation.
  • Updates bindings generation and unsupported tensor-type handling.
File Description
rust/​BUILD.md Documents header and version overrides.
rust/​onnxruntime-sys/​build.rs Improves header discovery and release selection.
rust/​onnxruntime/​src/​environment.rs Validates C API availability safely.
rust/​onnxruntime/​src/​error.rs Adds callback and API-version errors.
rust/​onnxruntime/​src/​lib.rs Exports callback errors and tests FLOAT6 handling.
rust/​onnxruntime/​src/​memory.rs Reorders imports.
rust/​onnxruntime/​src/​session.rs Implements callback registration, trampoline, ownership, and tests.
rust/​onnxruntime/​src/​tensor/​ort_output_tensor.rs Handles newly generated unsupported enum variants.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Base automatically changed from GopalakrishnanN/stable-epcontext-data-api to main October 5, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants