Skip to content

test(normalization): Cover ordering regressions and controls - #6519

Draft
vjeranc wants to merge 7 commits into
masterfrom
test/normalization-ordering-regressions
Draft

vjeranc wants to merge 7 commits into
masterfrom
test/normalization-ordering-regressions

Conversation

@vjeranc

@vjeranc vjeranc commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Add regression tests for normalization ordering, fallbacks, trimming, and measurement name/unit length limits. Cases cover computations reading incomplete inputs, values inserted after validation, and inconsistent size accounting or enforcement of the per-event custom-measurement limit. Keep unsettled behavioral expectations under proposed_contracts.

Verified at f8a43f50ae: 70 intentional failures in normalization_staleness and 43 passing controls in normalization_ordering_controls:

cargo test -p relay-event-normalization --test normalization_staleness --test normalization_ordering_controls --no-fail-fast

Tests call normalize_event directly to isolate normalization from validate_event. The ordering failures can also be reproduced with events that pass validate_event.

make test stops at the intentional failures. Earlier checks passed Rust docs, formatting, and Clippy; Python lint was blocked by missing .venv/bin/pre-commit.

Codex assisted with test preparation and verification.

Refs RELAY-274

Co-authored-by: Codex <noreply@openai.com>
@linear-code

linear-code Bot commented Oct 16, 2026

Copy link
Copy Markdown

RELAY-274

Co-authored-by: Codex <noreply@openai.com>
@61465

61465 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

@vjeranc — spent some time with the staleness file. The 33 cases look like one systemic pattern. Writing it up in case it's useful before you plan the fix PR.

Pattern. Every failing case has two paths producing the same output field. The direct path flows through a validator/enricher correctly; the indirect path (migration, inference, fallback, late normalization) writes to the same field but bypasses whatever ran before it.

A few from the file to show the same shape:

test direct path indirect path that bypasses
environment_from_legacy_tag_is_validated event.environment → validator (event.rs:309) tags.environment → migration in normalize_event_tags runs after validator (event.rs:644)
sdk_from_client_header_is_used_to_scrub_transaction_names sdk.name → transaction scrubbing client header → SDK fallback runs after scrubbing
errno_name_uses_os_parsed_from_description explicit os.name → errno enrichment os.raw_description → OS inference runs after errno enrichment
device_class_uses_device_inferred_from_user_agent explicit device → device.class tag user-agent → device inference runs after device.class derivation
breakdowns_use_normalized_app_start_ops canonical op spelling → breakdown legacy spelling → op normalization runs after breakdown
lcp_tag_backfill_rejects_empty_values explicit lcp.element tag → empty-value rejection trace data → tag backfill runs after rejection

If this reading is right, the structural fix is a dependency-ordering commitment: every writer to field F must run before every reader of F, so derived values pass through the same post-conditions as direct ones. In practice that probably means either (a) a second normalization pass over derived writes until a fixed point, or (b) an explicit dependency DAG the pipeline respects.

Point-fixing each case individually works but risks the same class resurfacing whenever a new derivation is added. The proposed_contracts module name in the file suggests you're already thinking in that direction.

Happy to look at specifics if useful — or stand clear if you've got a shape in mind for the follow-up PR.

vjeranc commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

Yes, I assume there are many indirect paths I haven't covered, since I focused on specific fields and reassignments.

For a complete classification, I imagine you'd need something like an SCC analyzer over Rust's MIR to track which fields can mutate which other fields, and identify early/late mutations that bypass Some(fallback) when they shouldn't. This gives you more test cases to fix but no signal on how to write the code.

Or, as with scoring and derived stats, values might be computed first, only for their source fields to be removed later once invalidated, leaving an event object with derived stats that seemingly came from nowhere.

Multiple passes are a bit dangerous when mutations are involved. And "explicit dependency DAG" is doing a lot of handwaving here. 😀

@61465

61465 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

Fair hits — the "DAG" line really was handwaving, and multiple-passes-over-mutable-state is the exact trap I was pointing at without saying how to avoid it. Agree.

The derived-stats case you raised is a different animal than the field-migration cases I listed, and that distinction is probably the useful cut:

  • Shape A — field-migration / fallback (environment_from_legacy_tag, partial_sdk_uses_name_from_client_header, errno_name_uses_os_parsed_from_description, …): one output field, two writers. The writers are independent; the problem is write-after-validator rather than mutual recursion. A single pass with each writer carrying "runs-before-X" metadata and the orchestrator refusing to call out-of-order writers covers this without a second pass. The invariant is static per writer, so no re-visit, no fixed-point.

  • Shape B — derived-from-mutable (rejected_frame_count_does_not_produce_a_rate, score_output_respects_measurement_config): the derivation runs, then its input gets removed by validation elsewhere, leaving an orphaned derived value. Here the DAG framing legitimately doesn't help — nothing was mis-ordered, the problem is that derivations outlive their sources. The fix is either (a) make derivation observe the source's lifecycle (delete-cascade), or (b) move derivation to the end so it only ever sees post-validation state.

So probably two patterns, not one, and the overlap is smaller than my first comment implied. SCC-over-MIR gives you Shape A at best (because MIR sees the writes but not which validator "owns" which field); Shape B wants runtime cascade semantics, which MIR can't give you.

If any of this is useful for #6519's follow-up, happy to help carve out the Shape-A subset and leave Shape B to a separate design pass. Otherwise, ignore — your plan already covers whatever cut makes sense for the fix PR.

vjeranc and others added 3 commits October 17, 2026 08:08
Add 23 failing regression tests for stale derivations, missing fallbacks, validation order, and precursor loss. Add four passing controls and characterizations to the dedicated controls suite.

Validation: all 42 ordering controls pass and all 61 staleness tests fail as intended on both normalizer implementations.

Co-authored-by: Codex <noreply@openai.com>
Follow a longest Event-only data-SCC condensation path:
request -> user -> {contexts, environment, spans, tags} -> breakdowns.

A REMOTE_ADDR value of {{auto}} shadows a concrete client IP. The unresolved
placeholder then propagates through user.ip_address and user.sentry_user
into child span user/IP tags. The IP lineage spans the first three
components; the final breakdown edge reads span operations and timestamps.

Add one table regression with absent, invalid, and concrete remote-address
controls. Both normalizer implementations reproduce the placeholder case.
Validation: 62 intended staleness failures and 42 passing ordering controls
per implementation.

Co-authored-by: Codex <noreply@openai.com>
Scope the expected print_stdout lint to the two structured-observation helpers consumed by the normalization audit harness. Keep the workspace warning policy active for the rest of both test files.

Validation: rustfmt and Clippy with all features and -D warnings pass. Both normalizers retain 42 passing controls and 62 intentional staleness failures.

Co-authored-by: Codex <noreply@openai.com>
@61465

61465 commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Nice — SCC audit + longest cross-SCC path lands the Shape A/B split cleaner than my framing did. Ping me if any specific regression needs another set of eyes. Otherwise 👍

vjeranc and others added 2 commits October 18, 2026 18:16
Add regressions for metadata-dependent Windows build recovery and stacktrace source context, model-independent AI token rates, UTF-8 byte-budget trimming, and destructive app-start alias collisions. Add a passing control for independent Device fields after local parse errors.

Validate both normalizer variants: 67 intentional regression failures and 43 passing controls per variant, with matching observations.

Co-authored-by: Codex <noreply@openai.com>
Add regressions for Device property-name byte accounting, unusable custom
measurements consuming quota, and measurement name/unit budget underflow.
These audit cases intentionally fail on the current normalizer.

Co-authored-by: Codex <noreply@openai.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants