Skip to content

codegen: packed_views option holds packed varint view fields as PackedView - #816

Open
nyurik wants to merge 2 commits into
anthropics:mainfrom
nyurik:packed-views
Open

nyurik wants to merge 2 commits into
anthropics:mainfrom
nyurik:packed-views

Conversation

@nyurik

@nyurik nyurik commented Nov 7, 2026 •

Copy link
Copy Markdown

Codegen follow up to #815, closes #814 (github stack is not available for remote forks). This PR includes #815.


Generated views decode every packed varint field into a Vec (RepeatedView), even when the caller never reads it. That's one heap allocation per field per message. For a Mapbox Vector Tile, that's two per feature:

message Feature {
  repeated uint32 tags = 2 [packed = true];
  repeated uint32 geometry = 4 [packed = true];
}

The new opt-in option makes views hold the fields it selects as borrowed PackedViews, so decoding a view makes no allocation for them:

buffa_build::Config::new().packed_views_in(&[".vector_tile.Tile.Feature"])
// or every field: .packed_views(); plugin: packed_views_in=<path>, packed_views=true

// generated view field (was RepeatedView<'a, u32>)
pub geometry: ::buffa::PackedView<'a, ::buffa::packed_view::Uint32>,

let view = FeatureView::decode_view(&bytes)?;   // geometry: checked and borrowed
for v in &view.geometry { /* u32 */ }           // decoded as iterated

On the 207 real-world tiles of mapbox/mvt-fixtures, a full read goes from 781,975 allocations per pass to 11,846. It's 21% fewer instructions and 12% faster on x86 (c8a.large), and 24% fewer instructions and 22% faster on ARM (c8g.large). Plain buffa views then beat the 745-line hand-written scanner of nyurik/fast-mvt#61. Measurements are in #814.

Why the option is per field
A PackedView has no slice to index or deref to, and decodes its elements again on every iteration. That suits a field read once, in order, such as tile geometry, but not one read by index or more than once. So rules select fields by proto path, as the other per-field options do.

What changes

  • Config, modeled on unbox_oneof / unbox_oneof_in:

    • CodeGenConfig::packed_view_fields: Vec<(String, bool)>: proto path prefixes (a package, a message with its nested messages, or a field; "." is everything), the last matching rule winning, as for preserve_unknown_fields_in;
    • buffa-build: packed_views_in(paths), and packed_views() for ".";
    • plugin: repeatable packed_views_in=<path>, and packed_views=true;
    • a row for each plugin option in the guide's options table, and a changelog fragment.
  • One predicate, packed_view_codec, picks the fields: a rule selects the field, the field is repeated and packed, its type is a varint scalar (int32, int64, uint32, uint64, sint32, sint64, bool), and the view doesn't implement reflection (generate_reflection_vtable). The field path is built next to field_repeated_repr's. The predicate is used for:

    • the view field type;
    • the decode arm (extend_packed for a packed record, push for an unpacked element);
    • ViewEncode;
    • JSON (via to_vec());
    • to_owned_message with a custom repeated_type.

    Eager and lazy views both get it, so a message can mix PackedView and RepeatedView fields.

  • ViewEncode re-encodes the elements canonically, so its output stays identical to the owned message's, as the ViewEncode docs promise. For this, PackedView gains packed_len and write_packed (hidden, for generated code) and FromIterator, so a view can be built from application data.

  • Warnings:

    • a rule that matches no generated field (FieldTypeRuleMatchedNothing, as for repeated_type_in);
    • rules that cannot apply because views are off or implement reflection (PackedViewsIgnored).
  • Off by default: generated code is unchanged without it.

Not included: fixed-width fields, enums (because of closed-enum unknown-value routing) and views that implement reflection keep RepeatedView; each can follow. buffa-build and the plugin only add rules that select fields; excluding a field from a broader rule needs a false rule in CodeGenConfig, and a builder for it can follow.

Source impact for selected fields: code that indexes such a field uses iter() or to_vec() instead.

Tests

  • Codegen:
    • packed varint fields become PackedView; fixed-width fields, [packed = false] fields, and views with vtable reflection keep RepeatedView;
    • field rules, message rules and a later false rule; whole-segment matching (.Msg.id doesn't select .Msg.ids); a mixed message's two decode arms;
    • output is unchanged when no rule matches;
    • the three warnings.
  • buffa-build and plugin option parsing: paths normalized, repeated, and rejected when empty.
  • PackedView encoding: canonical re-encoding of borrowed and decoded fields, and FromIterator.
  • End to end (buffa-test):
    • packed_views.proto with every field selected:
      • every varint type, in eager and lazy views;
      • the round trip against the owned message, including ViewEncode bytes;
      • split and unpacked records;
      • malformed payloads failing exactly as owned decoding does;
      • JSON against the owned message.
    • packed_views_scoped.proto with field and message rules: a PackedView field, a RepeatedView varint field and a nested message's field in one message, round-tripped through eager and lazy views, ViewEncode bytes, to_owned_message and JSON.

A packed field is a run of varints in one length-delimited record:

```proto
repeated uint32 geometry = 4 [packed = true];
// on the wire: 22 03 09 02 04 (tag, length 3, values 9 2 4)
```

buffa can only decode that payload into a `Vec`: `extend_packed_uint32` for owned messages, `RepeatedView` in views. So every packed field of every message costs an allocation, even when it is never read. `PackedView` borrows the payload instead. It checks the payload and counts its elements once, then decodes the elements as they are iterated.

```rust
use buffa::packed_view::{PackedView, Uint32};

let payload = [0x09, 0x02, 0x04]; // the bytes after the tag and length
let geometry = PackedView::<Uint32>::from_packed(&payload)?; // checked, no allocation
assert_eq!(geometry.len(), 3);
assert_eq!(geometry.iter().collect::<Vec<_>>(), [9, 2, 4]);
```

- **Errors:** `from_packed` fails exactly where decoding each element would, with the same `DecodeError`. Iteration never fails.
- **Several records:** `extend_packed` and `push` accept a field split over several records or sent unpacked. The field then holds decoded elements.
- **Codec markers:** the marker (`Int32`, `Int64`, `Uint32`, `Uint64`, `Sint32`, `Sint64`, `Bool`) names the encoding, which the Rust type alone doesn't: `i32` is `int32` or `sint32`.

This is the runtime half of anthropics#814; a follow-up lets generated views use it. Until then it serves hand-written decoders, such as the one nyurik/fast-mvt#61 wrote to avoid these allocations.

Refs anthropics#814

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

…PackedView

Closes anthropics#814. Builds on the `PackedView` PR; only this commit is new.

Generated views decode every packed varint field into a `Vec` (`RepeatedView`), even when the caller never reads it. That's one heap allocation per field per message. For a Mapbox Vector Tile, that's two per feature:

```proto
message Feature {
  repeated uint32 tags = 2 [packed = true];
  repeated uint32 geometry = 4 [packed = true];
}
```

The new opt-in option makes views hold the fields it selects as borrowed `PackedView`s, so decoding a view makes no allocation for them:

```rust
buffa_build::Config::new().packed_views_in(&[".vector_tile.Tile.Feature"])
// or every field: .packed_views(); plugin: packed_views_in=<path>, packed_views=true

// generated view field (was RepeatedView<'a, u32>)
pub geometry: ::buffa::PackedView<'a, ::buffa::packed_view::Uint32>,

let view = FeatureView::decode_view(&bytes)?;   // geometry: checked and borrowed
for v in &view.geometry { /* u32 */ }           // decoded as iterated
```

On the 207 real-world tiles of mapbox/mvt-fixtures, a full read goes from 781,975 allocations per pass to 11,846. It's 21% fewer instructions and 12% faster on x86 (c8a.large), and 24% fewer instructions and 22% faster on ARM (c8g.large). Plain buffa views then beat the 745-line hand-written scanner of nyurik/fast-mvt#61. Measurements are in anthropics#814.

**Why the option is per field**
A `PackedView` has no slice to index or deref to, and decodes its elements again on every iteration. That suits a field read once, in order, such as tile geometry, but not one read by index or more than once. So rules select fields by proto path, as the other per-field options do.

**What changes**
- **Config**, modeled on `unbox_oneof` / `unbox_oneof_in`:
  - `CodeGenConfig::packed_view_fields: Vec<(String, bool)>`: proto path prefixes (a package, a message with its nested messages, or a field; `"."` is everything), the last matching rule winning, as for `preserve_unknown_fields_in`;
  - buffa-build: `packed_views_in(paths)`, and `packed_views()` for `"."`;
  - plugin: repeatable `packed_views_in=<path>`, and `packed_views=true`;
  - a row for each plugin option in the guide's options table, and a changelog fragment.
- **One predicate, `packed_view_codec`,** picks the fields: a rule selects the field, the field is repeated and packed, its type is a varint scalar (`int32`, `int64`, `uint32`, `uint64`, `sint32`, `sint64`, `bool`), and the view doesn't implement reflection (`generate_reflection_vtable`). The field path is built next to `field_repeated_repr`'s. The predicate is used for:
  - the view field type;
  - the decode arm (`extend_packed` for a packed record, `push` for an unpacked element);
  - `ViewEncode`;
  - JSON (via `to_vec()`);
  - `to_owned_message` with a custom `repeated_type`.

  Eager and lazy views both get it, so a message can mix `PackedView` and `RepeatedView` fields.
- **`ViewEncode` re-encodes the elements canonically**, so its output stays identical to the owned message's, as the `ViewEncode` docs promise. For this, `PackedView` gains `packed_len` and `write_packed` (hidden, for generated code) and `FromIterator`, so a view can be built from application data.
- **Warnings:**
  - a rule that matches no generated field (`FieldTypeRuleMatchedNothing`, as for `repeated_type_in`);
  - rules that cannot apply because views are off or implement reflection (`PackedViewsIgnored`).
- **Off by default:** generated code is unchanged without it.

**Not included:** fixed-width fields, enums (because of closed-enum unknown-value routing) and views that implement reflection keep `RepeatedView`; each can follow. buffa-build and the plugin only add rules that select fields; excluding a field from a broader rule needs a `false` rule in `CodeGenConfig`, and a builder for it can follow.

**Source impact for selected fields:** code that indexes such a field uses `iter()` or `to_vec()` instead.

**Tests**
- **Codegen:**
  - packed varint fields become `PackedView`; fixed-width fields, `[packed = false]` fields, and views with vtable reflection keep `RepeatedView`;
  - field rules, message rules and a later `false` rule; whole-segment matching (`.Msg.id` doesn't select `.Msg.ids`); a mixed message's two decode arms;
  - output is unchanged when no rule matches;
  - the three warnings.
- **buffa-build and plugin option parsing:** paths normalized, repeated, and rejected when empty.
- **`PackedView` encoding:** canonical re-encoding of borrowed and decoded fields, and `FromIterator`.
- **End to end** (`buffa-test`):
  - `packed_views.proto` with every field selected:
    - every varint type, in eager and lazy views;
    - the round trip against the owned message, including `ViewEncode` bytes;
    - split and unpacked records;
    - malformed payloads failing exactly as owned decoding does;
    - JSON against the owned message.
  - `packed_views_scoped.proto` with field and message rules: a `PackedView` field, a `RepeatedView` varint field and a nested message's field in one message, round-tripped through eager and lazy views, `ViewEncode` bytes, `to_owned_message` and JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nyurik added a commit to nyurik/osmflow that referenced this pull request Nov 8, 2026
A new crate, crates/osmflow, reads OpenStreetMap PBF files, from a city extract to the planet:

- `PbfReader::blocks`: blocks in file order, on the calling thread.
- `PbfReader::par_unordered`: a callback on a pool of threads, in any order within each run of one object type, with a
  barrier wherever the type changes, so in a sorted file every node is handled before any way and every way before any
  relation.
- `PbfReader::par_ordered`: an iterator in file order, decoded ahead on the pool.
- `Select` skips object types and fields a caller does not need; the bytes in flight are bounded.

Blocks are decoded with buffa views of a projection of the OSM-binary schema; dense node arrays are borrowed
`PackedView`s, and plain nodes, ways and relations are read by osmflow's own field reader, straight into columns. The
generated code is checked in; `just update-generated` regenerates it (needs protoc), and CI checks it is current. buffa
is patched from the branch of anthropics/buffa#816.

Tests read fixtures from libosmium, osmpbf, osmpbfreader, osm-pbf-iter, osm-io and planetiler, each with its source
and license, plus generated edge cases; every reader mode must match the `osmpbf` and `osmpbfreader` crates, and the
output is snapshotted with insta. The `parse` benchmark compares osmflow with both crates, with the file out of the
page cache, and the README shows the results on the planet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nyurik added a commit to nyurik/osmflow that referenced this pull request Nov 8, 2026
A new crate, crates/osmflow, reads OpenStreetMap PBF files, from a city extract to the planet:

- `PbfReader::blocks`: blocks in file order, on the calling thread.
- `PbfReader::par_unordered`: a callback on a pool of threads, in any order within each run of one object type, with a
  barrier wherever the type changes, so in a sorted file every node is handled before any way and every way before any
  relation.
- `PbfReader::par_ordered`: an iterator in file order, decoded ahead on the pool.
- `Select` skips object types and fields a caller does not need; the bytes in flight are bounded.

Blocks are decoded with buffa views of a projection of the OSM-binary schema; dense node arrays are borrowed
`PackedView`s, and plain nodes, ways and relations are read by osmflow's own field reader, straight into columns. The
generated code is checked in; `just update-generated` regenerates it (needs protoc), and CI checks it is current. buffa
is patched from the branch of anthropics/buffa#816.

Tests read fixtures from libosmium, osmpbf, osmpbfreader, osm-pbf-iter, osm-io and planetiler, each with its source
and license, plus generated edge cases; every reader mode must match the `osmpbf` and `osmpbfreader` crates, and the
output is snapshotted with insta. The `parse` benchmark compares osmflow with both crates, with the file out of the
page cache, and the README shows the results on the planet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AnthonyKrupko

Copy link
Copy Markdown

If the second commit of nyurik#1 goes into #815, FromIterator here needs a small change when rebasing, because PackedView no longer has a decoded-elements form:

impl<C: PackedCodec> FromIterator<C::Elem> for PackedView<'_, C> {
    fn from_iter<I: IntoIterator<Item = C::Elem>>(iter: I) -> Self {
        let mut view = Self::default();
        iter.into_iter().for_each(|e| view.push(e));
        view
    }
}

packed_len and write_packed don't change. I measured nyurik#1 on top of this branch: it builds and gives the numbers in the PR.

buffa's benches don't run with packed_views yet, so nothing in benchmarks/ covers this option. I measured with two small additions to benchmarks/buffa: a packed_views feature that sets .packed_views() in build.rs, and decode_view_iter cases for packed_tile, packed_signed and column_batch that sum every packed varint after decoding. I can send them as a PR against this branch if you want them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Views allocate a Vec for every packed field of every message

2 participants