Skip to content

json: add runtime JSON policies for custom bytes and string types - #828

Open
vanderbr wants to merge 1 commit into
anthropics:mainfrom
reallyme:json-policy-runtime
Open

vanderbr wants to merge 1 commit into
anthropics:mainfrom
reallyme:json-policy-runtime

Conversation

@vanderbr

@vanderbr vanderbr commented Nov 13, 2026 •

Copy link
Copy Markdown

Part of #822. Runtime prerequisite for #829; automatic policy selection and generated routing belong to that companion PR.

Problem and change

Custom ProtoBytes and ProtoString representations need explicit control over JSON import storage and export behavior. Add opt-in ProtoBytesJson and ProtoStringJson traits: serialization takes borrowed bytes or text, so owned messages and views can share the same policy, while deserialization constructs the representation.

Runtime-owned helpers cover singular, optional, repeated, oneof, map-key and map-value positions, including custom serde keys, policy keys and string keys represented as bytes. Repeated and map-value helpers use wrappers owned by buffa, so a policy type from another crate needs no consumer-side ProtoElemJson impl. Null handling preserves each field shape's protobuf meaning; null repeated elements and map values are rejected before the policy runs. Map-key policies must round-trip and preserve distinct keys. Generic key-parser errors use a fixed reason instead of forwarding text that might disclose the rejected key.

The standard bytes helper now streams base64 through Serializer::collect_str and chooses the standard or URL-safe alphabet before decoding once. New bytes::decode_into and bytes::max_decoded_len helpers let a policy decode directly into caller-owned storage. Buffer sizing, partial writes on failure, cleanup ownership and possible allocation when constructing a Serde error are documented.

Compatibility and boundaries

Existing generation behavior, accepted base64 inputs and decoded bytes are preserved. Serialized bytes remain identical for serde_json and serializers whose collect_str conforms to serialize_str; invalid-input error wording can change. Dependencies, version constraints, toolchain and Rust 1.75 MSRV are unchanged.

Descriptor-driven reflective JSON does not consult these policies in either direction. Export refusal therefore does not provide a confidentiality boundary. Documentation also explains parser/serializer scratch, Any buffering, and ordinary outer-container allocations: repeated decoding stages values in a Vec before conversion, and element cleanup cannot erase copies left by container relocation. A sensitive policy must establish its cleanup owner before a fallible decode.

bytes::decode_into reports a fixed diagnostic for malformed base64 or insufficient caller storage, without forwarding input symbols or offsets. Accepted base64 remains unchanged. The destination stays caller-owned, may contain partial output on failure, and the caller-supplied Serde error type can still allocate.

Validation:

  • Focused caller-buffer diagnostic regressions plus default/all-feature runtime tests.
  • Strict Clippy, rustdoc and formatting; Rust 1.75 and no_std consumer checks.
  • The corrected runtime also passes the combined upstream proposal workspace tests and strict checks.

The external protobuf conformance suite was not rerun for this follow-up.

Contributed by ReallyMe Technologies Limited.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@vanderbr

Copy link
Copy Markdown
Author

recheck

@vanderbr
vanderbr force-pushed the json-policy-runtime branch from fa1579a to 8ffbf8a Compare November 17, 2026 16:22
@vanderbr vanderbr changed the title json: add JSON policy traits for custom bytes and string types json: add runtime JSON policies for custom bytes and string types Nov 17, 2026
vanderbr added a commit to reallyme/buffa that referenced this pull request Nov 18, 2026
Merge feature/native-adapters at ce3fd6e and port the published corrections
from anthropics#825 (a04ac69) and anthropics#828 (8ffbf8a).

Keep decode-hook windows bounded after fallible conversion, reject length
prefixes that cross message boundaries, and select test maps from the
runtime configuration. Preserve fork cleanup, compiler requirements,
package versions and dependency pins.
Add opt-in import and export policies for custom representations across scalar, collection and map-key positions. Decode base64 into caller-owned storage with fixed diagnostics that never quote input symbols or offsets. Keep existing policy and default JSON behavior compatible.
@vanderbr
vanderbr force-pushed the json-policy-runtime branch from 8ffbf8a to 60fd5b5 Compare November 18, 2026 23:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant