Repository navigation
Conversation
A packed field is a run of varints in one length-delimited record:
```proto
repeated uint32 geometry = 4 [packed = true];
// on the wire: 22 03 09 02 04 (tag, length 3, values 9 2 4)
```
buffa can only decode that payload into a `Vec`: `extend_packed_uint32` for owned messages, `RepeatedView` in views. So every packed field of every message costs an allocation, even when it is never read. `PackedView` borrows the payload instead. It checks the payload and counts its elements once, then decodes the elements as they are iterated.
```rust
use buffa::packed_view::{PackedView, Uint32};
let payload = [0x09, 0x02, 0x04]; // the bytes after the tag and length
let geometry = PackedView::<Uint32>::from_packed(&payload)?; // checked, no allocation
assert_eq!(geometry.len(), 3);
assert_eq!(geometry.iter().collect::<Vec<_>>(), [9, 2, 4]);
```
- **Errors:** `from_packed` fails exactly where decoding each element would, with the same `DecodeError`. Iteration never fails.
- **Several records:** `extend_packed` and `push` accept a field split over several records or sent unpacked. The field then holds decoded elements.
- **Codec markers:** the marker (`Int32`, `Int64`, `Uint32`, `Uint64`, `Sint32`, `Sint64`, `Bool`) names the encoding, which the Rust type alone doesn't: `i32` is `int32` or `sint32`.
This is the runtime half of anthropics#814; a follow-up lets generated views use it. Until then it serves hand-written decoders, such as the one nyurik/fast-mvt#61 wrote to avoid these allocations.
Refs anthropics#814
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
All contributors have signed the CLA ✍️ ✅ |
…PackedView Closes anthropics#814. Builds on the `PackedView` PR; only this commit is new. Generated views decode every packed varint field into a `Vec` (`RepeatedView`), even when the caller never reads it. That's one heap allocation per field per message. For a Mapbox Vector Tile, that's two per feature: ```proto message Feature { repeated uint32 tags = 2 [packed = true]; repeated uint32 geometry = 4 [packed = true]; } ``` The new opt-in option makes views hold the fields it selects as borrowed `PackedView`s, so decoding a view makes no allocation for them: ```rust buffa_build::Config::new().packed_views_in(&[".vector_tile.Tile.Feature"]) // or every field: .packed_views(); plugin: packed_views_in=<path>, packed_views=true // generated view field (was RepeatedView<'a, u32>) pub geometry: ::buffa::PackedView<'a, ::buffa::packed_view::Uint32>, let view = FeatureView::decode_view(&bytes)?; // geometry: checked and borrowed for v in &view.geometry { /* u32 */ } // decoded as iterated ``` On the 207 real-world tiles of mapbox/mvt-fixtures, a full read goes from 781,975 allocations per pass to 11,846. It's 21% fewer instructions and 12% faster on x86 (c8a.large), and 24% fewer instructions and 22% faster on ARM (c8g.large). Plain buffa views then beat the 745-line hand-written scanner of nyurik/fast-mvt#61. Measurements are in anthropics#814. **Why the option is per field** A `PackedView` has no slice to index or deref to, and decodes its elements again on every iteration. That suits a field read once, in order, such as tile geometry, but not one read by index or more than once. So rules select fields by proto path, as the other per-field options do. **What changes** - **Config**, modeled on `unbox_oneof` / `unbox_oneof_in`: - `CodeGenConfig::packed_view_fields: Vec<(String, bool)>`: proto path prefixes (a package, a message with its nested messages, or a field; `"."` is everything), the last matching rule winning, as for `preserve_unknown_fields_in`; - buffa-build: `packed_views_in(paths)`, and `packed_views()` for `"."`; - plugin: repeatable `packed_views_in=<path>`, and `packed_views=true`; - a row for each plugin option in the guide's options table, and a changelog fragment. - **One predicate, `packed_view_codec`,** picks the fields: a rule selects the field, the field is repeated and packed, its type is a varint scalar (`int32`, `int64`, `uint32`, `uint64`, `sint32`, `sint64`, `bool`), and the view doesn't implement reflection (`generate_reflection_vtable`). The field path is built next to `field_repeated_repr`'s. The predicate is used for: - the view field type; - the decode arm (`extend_packed` for a packed record, `push` for an unpacked element); - `ViewEncode`; - JSON (via `to_vec()`); - `to_owned_message` with a custom `repeated_type`. Eager and lazy views both get it, so a message can mix `PackedView` and `RepeatedView` fields. - **`ViewEncode` re-encodes the elements canonically**, so its output stays identical to the owned message's, as the `ViewEncode` docs promise. For this, `PackedView` gains `packed_len` and `write_packed` (hidden, for generated code) and `FromIterator`, so a view can be built from application data. - **Warnings:** - a rule that matches no generated field (`FieldTypeRuleMatchedNothing`, as for `repeated_type_in`); - rules that cannot apply because views are off or implement reflection (`PackedViewsIgnored`). - **Off by default:** generated code is unchanged without it. **Not included:** fixed-width fields, enums (because of closed-enum unknown-value routing) and views that implement reflection keep `RepeatedView`; each can follow. buffa-build and the plugin only add rules that select fields; excluding a field from a broader rule needs a `false` rule in `CodeGenConfig`, and a builder for it can follow. **Source impact for selected fields:** code that indexes such a field uses `iter()` or `to_vec()` instead. **Tests** - **Codegen:** - packed varint fields become `PackedView`; fixed-width fields, `[packed = false]` fields, and views with vtable reflection keep `RepeatedView`; - field rules, message rules and a later `false` rule; whole-segment matching (`.Msg.id` doesn't select `.Msg.ids`); a mixed message's two decode arms; - output is unchanged when no rule matches; - the three warnings. - **buffa-build and plugin option parsing:** paths normalized, repeated, and rejected when empty. - **`PackedView` encoding:** canonical re-encoding of borrowed and decoded fields, and `FromIterator`. - **End to end** (`buffa-test`): - `packed_views.proto` with every field selected: - every varint type, in eager and lazy views; - the round trip against the owned message, including `ViewEncode` bytes; - split and unpacked records; - malformed payloads failing exactly as owned decoding does; - JSON against the owned message. - `packed_views_scoped.proto` with field and message rules: a `PackedView` field, a `RepeatedView` varint field and a nested message's field in one message, round-tripped through eager and lazy views, `ViewEncode` bytes, `to_owned_message` and JSON. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new crate, crates/osmflow, reads OpenStreetMap PBF files, from a city extract to the planet: - `PbfReader::blocks`: blocks in file order, on the calling thread. - `PbfReader::par_unordered`: a callback on a pool of threads, in any order within each run of one object type, with a barrier wherever the type changes, so in a sorted file every node is handled before any way and every way before any relation. - `PbfReader::par_ordered`: an iterator in file order, decoded ahead on the pool. - `Select` skips object types and fields a caller does not need; the bytes in flight are bounded. Blocks are decoded with buffa views of a projection of the OSM-binary schema; dense node arrays are borrowed `PackedView`s, and plain nodes, ways and relations are read by osmflow's own field reader, straight into columns. The generated code is checked in; `just update-generated` regenerates it (needs protoc), and CI checks it is current. buffa is patched from the branch of anthropics/buffa#816. Tests read fixtures from libosmium, osmpbf, osmpbfreader, osm-pbf-iter, osm-io and planetiler, each with its source and license, plus generated edge cases; every reader mode must match the `osmpbf` and `osmpbfreader` crates, and the output is snapshotted with insta. The `parse` benchmark compares osmflow with both crates, with the file out of the page cache, and the README shows the results on the planet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new crate, crates/osmflow, reads OpenStreetMap PBF files, from a city extract to the planet: - `PbfReader::blocks`: blocks in file order, on the calling thread. - `PbfReader::par_unordered`: a callback on a pool of threads, in any order within each run of one object type, with a barrier wherever the type changes, so in a sorted file every node is handled before any way and every way before any relation. - `PbfReader::par_ordered`: an iterator in file order, decoded ahead on the pool. - `Select` skips object types and fields a caller does not need; the bytes in flight are bounded. Blocks are decoded with buffa views of a projection of the OSM-binary schema; dense node arrays are borrowed `PackedView`s, and plain nodes, ways and relations are read by osmflow's own field reader, straight into columns. The generated code is checked in; `just update-generated` regenerates it (needs protoc), and CI checks it is current. buffa is patched from the branch of anthropics/buffa#816. Tests read fixtures from libosmium, osmpbf, osmpbfreader, osm-pbf-iter, osm-io and planetiler, each with its source and license, plus generated edge cases; every reader mode must match the `osmpbf` and `osmpbfreader` crates, and the output is snapshotted with insta. The `parse` benchmark compares osmflow with both crates, with the file out of the page cache, and the README shows the results on the planet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
If the second commit of nyurik#1 goes into #815, impl<C: PackedCodec> FromIterator<C::Elem> for PackedView<'_, C> {
fn from_iter<I: IntoIterator<Item = C::Elem>>(iter: I) -> Self {
let mut view = Self::default();
iter.into_iter().for_each(|e| view.push(e));
view
}
}
buffa's benches don't run with |
Codegen follow up to #815, closes #814 (github stack is not available for remote forks). This PR includes #815.
Generated views decode every packed varint field into a
Vec(RepeatedView), even when the caller never reads it. That's one heap allocation per field per message. For a Mapbox Vector Tile, that's two per feature:The new opt-in option makes views hold the fields it selects as borrowed
PackedViews, so decoding a view makes no allocation for them:On the 207 real-world tiles of mapbox/mvt-fixtures, a full read goes from 781,975 allocations per pass to 11,846. It's 21% fewer instructions and 12% faster on x86 (c8a.large), and 24% fewer instructions and 22% faster on ARM (c8g.large). Plain buffa views then beat the 745-line hand-written scanner of nyurik/fast-mvt#61. Measurements are in #814.
Why the option is per field
A
PackedViewhas no slice to index or deref to, and decodes its elements again on every iteration. That suits a field read once, in order, such as tile geometry, but not one read by index or more than once. So rules select fields by proto path, as the other per-field options do.What changes
Config, modeled on
unbox_oneof/unbox_oneof_in:CodeGenConfig::packed_view_fields: Vec<(String, bool)>: proto path prefixes (a package, a message with its nested messages, or a field;"."is everything), the last matching rule winning, as forpreserve_unknown_fields_in;packed_views_in(paths), andpacked_views()for".";packed_views_in=<path>, andpacked_views=true;One predicate,
packed_view_codec, picks the fields: a rule selects the field, the field is repeated and packed, its type is a varint scalar (int32,int64,uint32,uint64,sint32,sint64,bool), and the view doesn't implement reflection (generate_reflection_vtable). The field path is built next tofield_repeated_repr's. The predicate is used for:extend_packedfor a packed record,pushfor an unpacked element);ViewEncode;to_vec());to_owned_messagewith a customrepeated_type.Eager and lazy views both get it, so a message can mix
PackedViewandRepeatedViewfields.ViewEncodere-encodes the elements canonically, so its output stays identical to the owned message's, as theViewEncodedocs promise. For this,PackedViewgainspacked_lenandwrite_packed(hidden, for generated code) andFromIterator, so a view can be built from application data.Warnings:
FieldTypeRuleMatchedNothing, as forrepeated_type_in);PackedViewsIgnored).Off by default: generated code is unchanged without it.
Not included: fixed-width fields, enums (because of closed-enum unknown-value routing) and views that implement reflection keep
RepeatedView; each can follow. buffa-build and the plugin only add rules that select fields; excluding a field from a broader rule needs afalserule inCodeGenConfig, and a builder for it can follow.Source impact for selected fields: code that indexes such a field uses
iter()orto_vec()instead.Tests
PackedView; fixed-width fields,[packed = false]fields, and views with vtable reflection keepRepeatedView;falserule; whole-segment matching (.Msg.iddoesn't select.Msg.ids); a mixed message's two decode arms;PackedViewencoding: canonical re-encoding of borrowed and decoded fields, andFromIterator.buffa-test):packed_views.protowith every field selected:ViewEncodebytes;packed_views_scoped.protowith field and message rules: aPackedViewfield, aRepeatedViewvarint field and a nested message's field in one message, round-tripped through eager and lazy views,ViewEncodebytes,to_owned_messageand JSON.