perf(atomic-memory): stop duplicating source text in extraction input to reduce token consumption by lizining1231 · Pull Request #2139 · oceanbase/powercontext

lizining1231 · 2026-10-20T17:22:10Z

Closes #2137.

Relationship to related work

The same body duplication is being addressed in three places, at two different layers.

Layer	Work	Mechanism	Status
evidence projection (this PR)	#2139	removes carriers that repeat the body before the input is built	open
evidence projection	#2163	same function and same insertion point, different carrier set	open
input composition	#1989 / #1991	shares repeated fields at send time behind a default-off switch	open

This PR and #2163

Both compact source_metadata inside project_atomic_memory_evidence(), so they are alternatives for the same slot rather than increments: they replace the same source_metadata=... expression and add a helper in the same region, which means whichever merges second will conflict in that block.

The two boundary sets differ as follows in now:

case	this PR	#2163

This PR's rule is "remove a carrier only when it repeats the evidence body"; the other set trades a small additional saving for those two unconditional removals. Both token tables measure the same deletion set on different benchmarks (four 255-token Sources here, one 996-token Source in #2163), so the percentage difference comes from the body-to-provenance ratio rather than from a different mechanism.

This PR and #1989 / #1991

No code overlap. #1991 deduplicates at input composition (input_encoding.py) and does not modify project_atomic_memory_evidence(), so the two changes compose in either order. One interaction worth recording: the shadow-only _would_noop() guard reads source_metadata["content"], a field this PR removes, so that diagnostic candidate would always evaluate false. It never affects inference, publication, or cursors, and the lookup can read item.content["content"] instead. #1989 records that the extraction input gate is not implemented by #1991, so this PR does not claim that scope.

Rationale for this change

project_atomic_memory_evidence() copied the stored Source dump verbatim into source_metadata while content already carried the evidential body. One extraction request therefore serialized the same text 2–3 times per Source, spending input tokens and reducing how many Sources fit the 24,000-token extraction budget.

What changes are included in this PR?

src/powercontext/builtin/artifacts/atomic_memory/extraction.py: adds _compact_source_metadata() and its two predicates, and routes the validated evidence body through it instead of forwarding the full Source dump.
Native ContentSource: the metadata copy loses content, and loses wire_content / wire_content_present only when the wire payload equals the body.
Projected SourceObservation: the metadata copy loses payload.content only when it equals the projected body, and loses the consumed text-evidence projection entry.
Unprojected SourceObservation: the metadata copy loses payload when the payload is the body itself.
Preserved fields: name, source_type, metadata, payload.metadata, and projections other than the text-evidence key.
tests/builtin/artifacts/atomic_memory/test_extraction_evidence_projection.py: adds the seven tests listed below.
Compatibility: no schema, API, or prompt-contract change. AtomicMemoryEvidence is not persisted, so there is no migration or backfill. content still carries the body byte-for-byte, and the only test reader of source_metadata (tests/e2e/test_pydantic_ai_chain.py:53-55) reads ["metadata"]["event"], which is preserved.

Are there any user-facing changes?

No. Extraction output, public APIs, and persisted formats are unchanged. The model receives the same evidence body once instead of 2–3 times.

How was this change tested?

Added tests

test_native_source_body_is_serialized_once: the native path serializes the body once and keeps its provenance fields.
test_projected_observation_body_is_serialized_once: a projected observation serializes the body once and keeps payload.metadata.
test_native_wire_content_duplicate_is_serialized_once: a wire payload equal to the body is dropped together with wire_content_present.
test_native_wire_content_with_other_shape_is_kept: a wire payload that differs from the body is retained.
test_observation_without_text_projection_is_serialized_once: an unprojected observation drops payload because the payload is the body itself.
test_observation_keeps_projections_other_than_text_evidence: projections other than the consumed text-evidence key are retained.
test_observation_keeps_payload_content_that_differs_from_projection: payload.content is retained when it differs from the projected body.

Input token reduction

Token counts use the Runtime's character_token_estimator() on the serialized AtomicMemoryExtractionInput, four Sources per case. (Before values were measured on caf2a35a, after values on this branch)

case	before	after	reduction
native, 4 sources, ASCII	2,891	1,858	↓ 35.7%
native, 4 sources, CJK	4,851	2,838	↓ 41.5%
observation, 4 sources, ASCII	4,257	1,929	↓ 54.7%
observation, 4 sources, CJK	7,197	2,909	↓ 59.6%

AI usage statement

AI-assisted with Trae (DeepSeek-V4.1-Flash) for tests, and documentation

The stored Source dump repeated the evidential body 2-3 times in one extraction request. Drop only the carriers whose value equals the body and keep the remaining provenance fields. Input tokens fall 35.7%-41.5% on the native path and 54.7%-59.6% on the projected observation path. Refs oceanbase#2137

Cover native, wire, projected, unprojected, and divergent-payload evidence: each serializes the body once and keeps its provenance fields. Refs oceanbase#2137

Fengzdadi

Reviewed at 0aad06e5. The direction looks right: the duplicated body is real on master (2 copies on the hook capture path, 3 on create_source / observation inputs), and the extraction budget is checked against this same serialized input, so the savings reach _require_budget. Two notes below.

Fengzdadi · 2026-10-21T04:38:35Z

+    if _duplicates_body(dumped.get("wire_content"), body_text):
+        dumped.pop("wire_content", None)
+        dumped.pop("wire_content_present", None)


Structured wire_content is still sent twice. create_source (records.py:212) and subject writes (subject_sources.py:70) store content = _canonical_source_text(wire_content), so for JSON input content is the RFC 8785 text of wire_content, and the equality check never matches. With wire_content={"text": MARKER + " structured"}, the serialized extraction input contains the marker 2 times. test_native_wire_content_with_other_shape_is_kept asserts this duplicate is kept. Since content is always derived from wire_content for ContentSource, could the compaction drop the wire copy by type rather than by value equality?

Thanks review! Fixed in 257b506e, tests in 7d44342e.

Root cause: content stores the canonical text of wire_content. A value comparison can never match a structured payload, so the body also survived in source_metadata.

Change: _compact_source_metadata drops content, wire_content and wire_content_present together once content carries the body. All other provenance fields are untouched.

Impact (probe from perf(atomic-memory): stop duplicating source text in extraction input to reduce token consumption #2137): structured inputs go 2 → 1 copies (dict) and 4 → 2 (list), i.e. a further 41.8%–47.5% of input tokens on top of this PR.

Tests: test_native_structured_wire_content_is_serialized_once replaces test_native_wire_content_with_other_shape_is_kept, and the fixture now builds content the way the writer does. atomic_memory suite: 74 passed, 3 skipped; ruff and ty clean.

Fengzdadi · 2026-10-21T04:38:35Z

+    return isinstance(projection, dict) and projection.get("key") == _TEXT_EVIDENCE_KEY
+
+
+def _compact_source_metadata(value: Source, *, body: JsonValue, body_from_text_projection: bool) -> JsonValue:


Design note (non-blocking): this overlaps the input gate in #1989 / draft #1991, which deduplicates the same Atomic content / source_metadata fields generically behind a default-off switch. Its _would_noop() also requires source_metadata["content"] == content["content"], so after this change that shadow check would always return false. Linking #1989 and noting which layer should own this deduplication would help keep the two from diverging. cc @hidb4ai

补充实证（head 7d44342e）。

上一轮 4333088534 的 P1 已闭合：_duplicates_body 走 _canonical_source_text 语义比较，结构化 wire 时
content 已承载同一份规范化 JSON，因此 wire_content 被正确丢弃。同一探针
（oceanbase-powercontext-pr-2163/probes/probe_structured_wire.py，Windows / Python 3.14.3，PYTHONPATH 指向镜像 src）
对同一场景并排跑三个树：

master 88899876：2 次（content + wire_content 同一份 JSON）；

本 PR 7d44342e：1 次 / 586 bytes；

perf(atomic-memory): send each Source body once to extraction #2163 1c57cf4a：2 次 / 738 bytes。

tests/builtin/artifacts/atomic_memory/test_extraction_evidence_projection.py 本地 7 passed。

这条 design note 现在有具体对象：#2163（今日 08:37 开）对同一个 project_atomic_memory_evidence 做了同一件事，
边界不同——结构化 wire_content 本 PR 删、#2163 保留（重复仍在）；payload.content 本 PR 仅在等于投影正文时删、
#2163 无条件删；其他 projection 的 value.content 本 PR 完整保留、#2163 一律剥离。两个 PR 改的是同一插入点，
后合入者必然冲突。建议只推进其中一个，并在被放弃的一方写明原因；若保留本 PR，请把与 #1989 / #1991 / #2137 / #2163
的关系写进 PR 正文，避免三份实现各自声称同一收益。

Conclusion: Recommend proceeding with this PR#2139 only

This proposal already had an implementation before #2163 was opened

Current state of #2139:

The duplicated body is removed for both representations (native and observation), and the structured wire_content case is fixed and pinned by a test. On the same probe: master 88899876 sends the body 2 times, perf(atomic-memory): stop duplicating source text in extraction input to reduce token consumption #2139 once / 586 bytes, perf(atomic-memory): send each Source body once to extraction #2163 (1c57cf4a) still twice / 738 bytes.

a carrier is removed only when it repeats the evidence body.

I deliberately did not widen that boundary, because may the two wider removals carry evidence-loss risk:

1. Unconditional payload.content removal : when a payload's content differs from the projected body (a summary rather than the full text, or another serialization), that distinct text never reaches the model. No test covers the differing case.
2. Removing value.content from every projection : projections are an extension point, and content is a common key there (for example a ticket-summary projection). Removing it deletes evidence that is not a duplicate and leaves only the sibling keys.
3. Value-comparing wire_content : a structured payload is never equal to its canonical text, so the duplicate survives; this is the inverse failure. #2139 replaces that comparison with a semantic one and pins the result with a test.

If the maintainers are willing to accept that risk, I can widen #2139's boundary to match #2163 (unconditional payload.content, stripping value.content from other projections) and add tests for the differing-content cases.

The relationship with #1989 / #1991 / #2137 / #2163 has been documented in the PR description.

_compact_source_metadata only dropped a carrier whose value equalled the body, which never holds for a structured wire_content. A matching content marks the payload as a repeat, so drop content, wire_content and wire_content_present together. Structured inputs go from 2 to 1 copies (dict) and 4 to 2 (list), saving a further 41.8%-47.5% of input tokens on top of the parent commit. Refs oceanbase#2137

Replace the expectation that a structured wire_content stays beside the body, and build the fixture the way the writer does: content is the canonical text of wire_content. Refs oceanbase#2137

lizining1231 · 2026-10-21T11:42:19Z

Could someone please help re-run the CI? The failure looks flaky

lizining1231 · 2026-10-21T11:57:24Z

There were two flaky CI failures in this PR.
I've located and fixed the first one; here's the related PR #2153

hidb4ai

Reviewed at 7d44342e. Two remaining issues below, reproduced with real SQLite and a controlled extraction pipeline.

hidb4ai · 2026-10-21T12:29:17Z

+    if _duplicates_body(dumped.get("content"), body_text):
+        # ``content`` carries the body, so the structured payload repeats it.
+        dumped.pop("content", None)
+        dumped.pop("wire_content", None)
+        dumped.pop("wire_content_present", None)


[P2] Preserve independent wire evidence for custom Sources

value can be any registered Source, so content == body["content"] does not establish that wire_content repeats it. A valid custom Source with content="The deploy uses canary." and wire_content={"constraint": "Pause when errors rise."}, whose adapter returns TextEvidence(content=source.content, ...), is accepted, persists and reads back correctly. Through a real SQLite Sources.add -> process_memory run, the independent wire evidence appears in the base extraction input but disappears at this head, while flush succeeds. Please only remove the wire fields when their equivalence to the retained body is established; the built-in writer's convention cannot be assumed for every Source Definition.

Thanks review! Here are the change details:

Reproduction
A custom Source with content="The deploy uses canary." and wire_content={"constraint": "Pause when errors rise."}, whose adapter returns TextEvidence(content=source.content, ...):

before: wire_content and wire_content_present disappeared from the extraction input

after: both are preserved

Fix

Added _wire_repeats_body(value, body_text): a wire carrier is dropped only when its canonical text equals the retained body text. The presence of content no longer implies that the wire repeats it.

The same predicate drives the Source dump and the observation payload through _drop_body_carriers, so the two paths cannot diverge.

content and the wire carriers are decided independently; carriers that do reproduce the body still collapse to one copy.

Verification

test_sqlite_native_wire_evidence_survives_process_memory: real SQLite Sources.add -> process_memory for the custom Source above, asserting the independent wire survives.

test_native_keeps_wire_evidence_that_differs_from_body: unit-level counterpart.

Against the pre-fix code, the SQLite test fails with KeyError: 'wire_content'.

hidb4ai · 2026-10-21T12:29:18Z

+            if isinstance(payload, dict) and _duplicates_body(payload.get("content"), body_text):
+                dumped["payload"] = {key: item for key, item in payload.items() if key != "content"}


[P2] Remove duplicate wire content inside observation payloads

This branch removes only payload.content; payload.wire_content remains even when it contains exactly the same body. A remote Source Definition with a standard text-evidence projection and identical content/wire_content values reproduces this after successful manifest registration and observation submission. With a roughly 50,000-character ASCII body, the runtime character estimator reports 25,747 input tokens, and process_memory(..., 1) raises atomic-memory-input-budget before calling the extractor. Removing the residual duplicate reduces the estimate to 13,238, below the default 24,000 limit. Please apply equivalent compaction to the nested wire carriers while preserving distinct wire evidence, and cover observation inputs with both text and structured wire values.

Thanks review! Here are the change details:

Reproduction
An observation with a standard text-evidence projection and content == wire_content, with a 50,000-character ASCII body:

before: process_memory(..., 1) raised atomic-memory-input-budget before the extractor was called

after: the observation is processed, and the serialized extraction input contains the body once, with the estimate below the 24,000 limit

Fix

Observation payloads now apply the same compaction to wire_content and wire_content_present that already applied to payload.content.

Each carrier is dropped independently, only when its canonical text equals the projected body. Distinct structured wire evidence is retained.

Both paths share _drop_body_carriers, so a future change to the rule applies to the Source dump and the observation payload together.

Verification

test_sqlite_observation_duplicate_wire_stays_within_input_budget: real SQLite observation submission followed by process_memory(..., 1), asserting one body occurrence and an input estimate under 24,000.

test_observation_payload_drops_wire_content_equal_to_body and test_observation_payload_keeps_wire_evidence_that_differs_from_body: text and structured wire values, as requested.

Against the pre-fix code, the SQLite observation test fails with atomic-memory-input-budget.

content == body["content"] does not prove wire_content repeats the body, so the Source dump dropped independent wire evidence. Observation payloads had the opposite gap: payload.content was removed while an identical payload.wire_content stayed in place. Compare each wire carrier's canonical text with the evidential body before dropping it, for both the Source dump and the observation payload. Carriers that reproduce the body still collapse to one copy. Refs oceanbase#2137

A native Source keeps a structured wire payload whose canonical text is not the body, and an observation payload keeps independent structured wire evidence. An observation wire that repeats the projected body is removed together with the duplicate content. The SQLite Sources.add -> process_memory path keeps independent wire evidence for a custom Source, and a 50,000-character observation no longer exceeds the extraction input budget. Refs oceanbase#2137

lizining1231 added 2 commits October 21, 2026 01:14

test(atomic-memory): cover evidence body deduplication

0aad06e

Cover native, wire, projected, unprojected, and divergent-payload evidence: each serializes the body once and keeps its provenance fields. Refs oceanbase#2137

lizining1231 requested review from AlexStocks, PsiACE and frostming as code owners October 20, 2026 17:22

Fengzdadi reviewed Oct 21, 2026

View reviewed changes

lizining1231 added 2 commits October 21, 2026 14:16

test(atomic-memory): assert a structured wire is serialized once

7d44342

Replace the expectation that a structured wire_content stays beside the body, and build the fixture the way the writer does: content is the canonical text of wire_content. Refs oceanbase#2137

AlexStocks mentioned this pull request Oct 21, 2026

perf(atomic-memory): send each Source body once to extraction #2163

Open

hidb4ai reviewed Oct 21, 2026

View reviewed changes

lizining1231 added 2 commits October 21, 2026 23:28

AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 21, 2026

		return isinstance(projection, dict) and projection.get("key") == _TEXT_EVIDENCE_KEY


		def _compact_source_metadata(value: Source, *, body: JsonValue, body_from_text_projection: bool) -> JsonValue:

		if isinstance(payload, dict) and _duplicates_body(payload.get("content"), body_text):
		dumped["payload"] = {key: item for key, item in payload.items() if key != "content"}

Conversation

lizining1231 commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relationship to related work

This PR and #2163

This PR and #1989 / #1991

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

How was this change tested?

Added tests

Input token reduction

AI usage statement

Uh oh!

Fengzdadi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lizining1231 commented Oct 21, 2026

Uh oh!

lizining1231 commented Oct 21, 2026

Uh oh!

hidb4ai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lizining1231 commented Oct 20, 2026 •

edited

Loading