Skip to content

types: let custom bytes and string types decode their own payloads - #825

Open
vanderbr wants to merge 1 commit into
anthropics:mainfrom
reallyme:decode-hooks
Open

vanderbr wants to merge 1 commit into
anthropics:mainfrom
reallyme:decode-hooks

Conversation

@vanderbr

@vanderbr vanderbr commented Nov 13, 2026 •

Copy link
Copy Markdown

Part of #822.

A custom ProtoBytes or ProtoString representation builds itself from WirePayload in from_wire. Fragmented input first passes through an ordinary temporary Bytes, so a representation that needs to control every copy cannot protect that allocation. The new overridable decode_length_delimited(&mut FieldPayload<impl Buf>) lets it copy directly into its own storage.

FieldPayload implements Buf, including copy_to_bytes and, with std, chunks_vectored. Its underlying buffer and limit are private. The decoder validates the length before invoking the hook, and skips unread field bytes on a successful return. Existing implementations inherit defaults that preserve from_wire, PREFERS_OWNED_PAYLOAD, allocation sharing and the borrowed UTF-8 fast path.

Generated singular, optional, repeated, oneof and map paths use decode_string_to_ctx / decode_bytes_to_ctx. These validate against the innermost enclosing message, preventing a malformed field from exposing the parent's following fields or later stream frames to a hook. Sub-message and map-entry lengths are checked at runtime too. An incomplete prefix is rejected even if bytes outside the frame would complete it as zero; valid empty fields and supported noncanonical zero prefixes wholly inside the frame remain accepted.

The bounded window stays synchronized if a delegated default conversion rejects an owned or fragmented payload. A hook that inspects, retries or recovers from that error still cannot access the next field. Borrowed conversion errors leave their original field bytes unread, so a successful fallback skips exactly those bytes. Propagated errors retain an unspecified cursor position within the field.

DecodeContext records the enclosing message end through the default Message::merge_to_limit. Hand-written overrides and code decoding a different buffer must record the appropriate boundary with with_message_end (zero for a buffer holding exactly that message), or enter through merge / merge_to_limit. These compatibility requirements and the context-free helpers' buffer-only bounds are documented. Downstream generated custom-field code needs regeneration to use the context-aware entry points.

The stronger field-window guarantee applies to custom decode hooks. Ordinary String / Vec<u8> / Bytes fields, unknown fields and MessageSet payload copies retain their existing checks and can consume bytes past an enclosing boundary before the parent reports an error. Extending the bounds to those paths is separate follow-up work; this PR does not claim whole-decoder frame isolation.

Length-delimited entry points record the frame end before invoking a handwritten merge_to_limit override. Context-aware hooks therefore keep the enclosing frame boundary even when that override supplies the tag loop. The manual-decoder documentation distinguishes this context guarantee from an override that deliberately ignores it.

Validation:

  • All-feature runtime tests and red/green override/frame-context regressions.
  • Strict Clippy, rustdoc and formatting; Rust 1.75 library validation.
  • The corrected runtime also passes the combined upstream proposal workspace tests and strict checks.

The external protobuf conformance suite was not rerun for this follow-up.

Contributed by ReallyMe Technologies Limited.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@vanderbr

Copy link
Copy Markdown
Author

recheck

vanderbr added a commit to reallyme/buffa that referenced this pull request Nov 18, 2026
Merge feature/native-adapters at ce3fd6e and port the published corrections
from anthropics#825 (a04ac69) and anthropics#828 (8ffbf8a).

Keep decode-hook windows bounded after fallible conversion, reject length
prefixes that cross message boundaries, and select test maps from the
runtime configuration. Preserve fork cleanup, compiler requirements,
package versions and dependency pins.
Let custom bytes and string representations decode directly from bounded field payloads. Keep hooks within their message and stream frame, including entry points that dispatch to handwritten merge overrides. Preserve borrowed decoding and allocation-sharing defaults.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant