Skip to content

refactor(data): migrate data layout to raw/processed/canonical (path-only) - #13

Merged
xianyu9n merged 7 commits into
masterfrom
refactor/data-layout-migration
Aug 19, 2026
Merged

refactor(data): migrate data layout to raw/processed/canonical (path-only)#13
xianyu9n merged 7 commits into
masterfrom
refactor/data-layout-migration

Conversation

@xianyu9n

@xianyu9n xianyu9n commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Data layout 迁移(只改目录结构与路径引用,不改数据/逻辑)

目标布局data/rawdata/processed/<ds>/data/canonical/<ds>/data/{sft,rl}/(parquet),data/knowledge/ 不变。

迁移

  • data/<ds>/all.json + {train,val,test}.json + split_report.jsondata/processed/<ds>/
  • data/<ds>/canonical/{all.json,resolution_report.json}data/canonical/<ds>/
  • 新增 data/raw/(gitignored,不入库)、data/legacy/finance.corpus.json(archive)
  • 删除与 shougang/all.json 逐字节重复的 all_shougang.json
  • data/sft/**data/rl/** 视为可再生成产物,未搬移(已用同一管道重新 export 验证一致)
  • 注意:data 数据文件均为 gitignored,本 PR 不会通过 git 自动迁移现有本地数据。 已有旧布局环境需手动将 data/<ds>/ 移至新 processed/canonical 层(见 data/README.md §「如何使用 data/processed」)。

代码(路径语义化,零逻辑改动)

  • agent.task.canonical_datasetdata_dir → 语义化 processed_dir / canonical_dir
  • script/canonical/targets.py:CLI --processed-dir / --canonical-dir
  • script/canonical/cli.pyscript/analysis/...:读 processed + legacy corpus
  • script/verl/{sft,rl}/export.py help、src/agent/training/sft/dataset.py docstring
  • data/README.md:分层语义 + 「如何使用 data/processed」runbook(已按文档命令验证)
  • docs:data_contract.mdSFT_BASELINE.mdstage3c 迁移注记、新数据集运行说明 交叉引用
  • tests:test_canonical_dataset / test_label_whitespace_variants / test_sft_canonical_e2e / test_rl_canonical_e2e 路径更新(修复了迁移后被静默 skip 的 real-data 测试)
  • 安全:移除含本机绝对路径的一次性脚本 generate_standards_map.py;各分支 tip 已做敏感词清理(master 已有同款 scrub 提交)

Review fix(8316dbc

  1. legacy corpus key 修复discover_corpora() 现在把 finance.corpus.json 注册为 corpus:finance(原先 Path.stem 产生 corpus:finance.corpus,与 build_schema_issues 硬编码 key 不符 → 运行必 KeyError)。已用真实数据端到端跑通 analyze_dataset_corpus_alignment.py(修复前崩溃、修复后 exit 0)
  2. 单测:新增 tests/analysis/test_alignment_legacy_corpus.py,防止 .corpus.json 双后缀 key 回归。
  3. data/raw/.gitkeep:未入库(gitignored),已从 PR 描述与 data/README.md 中移除「仓库保留该占位」的表述。
  4. gitignored 数据不自动迁移说明已补入 data/README.md 迁移段。

一致性验证

  • 28 个 moved JSON 文件 SHA256 逐字节一致(迁移前后)
  • script.canonical.targets 重新生成的 canonical all.json 与迁移前字节级相同(4 数据集)
  • SFT/RL export 重导出后 24 个 parquet split 行数与迁移前全等;双次导出 parquet 逐字节相同(确定性)
  • 「交接模拟」:仅 data/processed + 仓库代码(去掉 knowledge/canonical/sft/rl)即可全自动重建 canonical + parquet,且与主仓库逐字节一致
  • 无 GPU 环境:canonical build / SFT|RL export / validate 均可 torch-free 本地跑通;smoke 训练未验证(需 verl+GPU)

测试

pytest tests/242 passed, 2 skipped(skip = verl 未安装)

曾立宏 added 7 commits August 20, 2026 03:22
…only, no content/logic change)

- processed/<ds>/: all.json + train/val/test.json + split_report.json (preprocessing output)
- canonical/<ds>/: all.json + resolution_report.json (frozen contract consumed by SFT/RL)
- raw/ (placeholder), legacy/ (finance legacy corpus archive); deleted duplicate all_shougang.json
- canonical_dataset/targets: data_dir split into semantic processed_dir/canonical_dir
- canonical/cli + alignment CLI: read processed, legacy; sft/rl export help updated
- tests updated to the new layout (real-data integration runs in-process)
…ut; analysis docstring; restore skipped data tests
…et->train/eval) + cross-link from new-dataset guide
… docs (plan B forward-fix)

- remove one-off standards generator with a private absolute path from tracking (+ gitignore)
- replace server absolute paths (/root/..., autodl-tmp) with generic placeholders
- drop exact GPU/driver/VRAM details; generalize internal source filenames (finance-content.xlsx, infras.xlsx)
- replace internal role reference (学长)
…w notes

- discover_corpora: finance.corpus.json registers as corpus:finance (not
  corpus:finance.corpus), matching the hardcoded build_schema_issues key;
  verified by running the analysis CLI end-to-end (was KeyError before)
- add unit tests guarding the .corpus.json double-suffix key
- data/README: drop 'data/raw/.gitkeep placeholder' claim (gitignored);
  note that gitignored data files are NOT auto-migrated by this PR
@xianyu9n
xianyu9n merged commit 99a82b6 into master Aug 19, 2026
3 checks passed
@xianyu9n
xianyu9n deleted the refactor/data-layout-migration branch August 19, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant