Tree optimization for Flash (merge & expand) - #373
Merged
Conversation
# Conflicts: # run_pageindex.py
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| """page -> [heading, ...] from a per-page detection pass, or None.""" | ||
| if not path or not os.path.exists(path): | ||
| return None | ||
| data = json.load(open(path)) |
| page text. Returns the run summary; the refined tree is doc["structure"]. | ||
| """ | ||
| if isinstance(doc, str): | ||
| doc = json.load(open(doc)) |
| and not os.getenv("OPENAI_API_KEY"): | ||
| sys.exit(f"OPENAI_API_KEY is not set (expand model: {model}).") | ||
|
|
||
| original = json.load(open(args.structure)) |
|
|
||
| refined = dict(original) | ||
| refined["structure"] = structure | ||
| json.dump(refined, open(out_path, "w"), indent=2, ensure_ascii=False) |
| "min_gain_ratio": args.min_gain_ratio, "model": model, | ||
| "before": result["before"], "after": result["after"], | ||
| "id_map": result["id_map"], "events": result["log"]}, | ||
| open(args.log, "w"), indent=2, ensure_ascii=False) |
rejojer
force-pushed
the
feat/pageindex-flash
branch
from
July 31, 2026 12:11
00c5640 to
e55f92d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Union end_index semantics
A node's
end_indexnow covers its whole section, subsections included (it previously stopped where the first child begins). The pages between a parent's heading and its first child stay derivable asstart_index… first child'sstart_index. This is the convention retrieval, summaries, and the optimizer below all rely on.Tree optimization (preview)
--optimizerefines a Flash tree to minimize worst-case search cost: pages read per lookup, counting one page per routing step.key_items.python3 run_pageindex.py --flash --optimize --pdf_path doc.pdf python3 run_pageindex.py --flash --optimize-merge-only --pdf_path doc.pdf # no LLM-backed expansion; summaries unchangedThe output gains an
optimizekey with merge/expand counts and before/after search-cost metrics. Example (222-page annual report): worst-case search cost 37 → 10 pages (normalized 0.167 → 0.045).Recursive node summaries
Flash summaries are now composed bottom-up: leaves are summarized from their own pages; a parent's summary is composed from its children's summaries plus the pages no child covers, so it describes the whole section without re-reading the subtree. Leaves under 200 tokens use their raw text as the summary with no model call.
On the annual report this cuts summary input tokens 418K → 221K (1.9x); the saving grows with tree depth. Every raw page now enters exactly one summary prompt.
Optimization stays opt-in; without
--optimizethe Flash pipeline behaves as before, apart from theend_indexsemantics and the summary composition above.