refactor(data): migrate data layout to raw/processed/canonical (path-only) - #13
Merged
Conversation
added 7 commits
August 20, 2026 03:22
…only, no content/logic change) - processed/<ds>/: all.json + train/val/test.json + split_report.json (preprocessing output) - canonical/<ds>/: all.json + resolution_report.json (frozen contract consumed by SFT/RL) - raw/ (placeholder), legacy/ (finance legacy corpus archive); deleted duplicate all_shougang.json - canonical_dataset/targets: data_dir split into semantic processed_dir/canonical_dir - canonical/cli + alignment CLI: read processed, legacy; sft/rl export help updated - tests updated to the new layout (real-data integration runs in-process)
… raw/processed/canonical layout
…ut; analysis docstring; restore skipped data tests
…et->train/eval) + cross-link from new-dataset guide
… docs (plan B forward-fix) - remove one-off standards generator with a private absolute path from tracking (+ gitignore) - replace server absolute paths (/root/..., autodl-tmp) with generic placeholders - drop exact GPU/driver/VRAM details; generalize internal source filenames (finance-content.xlsx, infras.xlsx) - replace internal role reference (学长)
…w notes - discover_corpora: finance.corpus.json registers as corpus:finance (not corpus:finance.corpus), matching the hardcoded build_schema_issues key; verified by running the analysis CLI end-to-end (was KeyError before) - add unit tests guarding the .corpus.json double-suffix key - data/README: drop 'data/raw/.gitkeep placeholder' claim (gitignored); note that gitignored data files are NOT auto-migrated by this PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Data layout 迁移(只改目录结构与路径引用,不改数据/逻辑)
目标布局:
data/raw→data/processed/<ds>/→data/canonical/<ds>/→data/{sft,rl}/(parquet),data/knowledge/不变。迁移
data/<ds>/all.json + {train,val,test}.json + split_report.json→data/processed/<ds>/data/<ds>/canonical/{all.json,resolution_report.json}→data/canonical/<ds>/data/raw/(gitignored,不入库)、data/legacy/finance.corpus.json(archive)shougang/all.json逐字节重复的all_shougang.jsondata/sft/**、data/rl/**视为可再生成产物,未搬移(已用同一管道重新 export 验证一致)data/<ds>/移至新 processed/canonical 层(见data/README.md§「如何使用 data/processed」)。代码(路径语义化,零逻辑改动)
agent.task.canonical_dataset:data_dir→ 语义化processed_dir/canonical_dirscript/canonical/targets.py:CLI--processed-dir/--canonical-dirscript/canonical/cli.py、script/analysis/...:读 processed + legacy corpusscript/verl/{sft,rl}/export.pyhelp、src/agent/training/sft/dataset.pydocstringdata/README.md:分层语义 + 「如何使用 data/processed」runbook(已按文档命令验证)data_contract.md、SFT_BASELINE.md、stage3c迁移注记、新数据集运行说明交叉引用test_canonical_dataset/test_label_whitespace_variants/test_sft_canonical_e2e/test_rl_canonical_e2e路径更新(修复了迁移后被静默 skip 的 real-data 测试)generate_standards_map.py;各分支 tip 已做敏感词清理(master 已有同款 scrub 提交)Review fix(
8316dbc)discover_corpora()现在把finance.corpus.json注册为corpus:finance(原先Path.stem产生corpus:finance.corpus,与build_schema_issues硬编码 key 不符 → 运行必 KeyError)。已用真实数据端到端跑通analyze_dataset_corpus_alignment.py(修复前崩溃、修复后 exit 0)。tests/analysis/test_alignment_legacy_corpus.py,防止.corpus.json双后缀 key 回归。data/raw/.gitkeep:未入库(gitignored),已从 PR 描述与data/README.md中移除「仓库保留该占位」的表述。data/README.md迁移段。一致性验证
script.canonical.targets重新生成的 canonicalall.json与迁移前字节级相同(4 数据集)data/processed+ 仓库代码(去掉 knowledge/canonical/sft/rl)即可全自动重建 canonical + parquet,且与主仓库逐字节一致测试
pytest tests/→ 242 passed, 2 skipped(skip = verl 未安装)