Repository navigation

Convert hybrid HDF5+Parquet data into wds - #57

Draft

thompsonmj wants to merge 10 commits into

mainfrom

Draft

Convert hybrid HDF5+Parquet data into wds#57
thompsonmj wants to merge 10 commits into
mainfrom
hybrid2wds

Commits on Jun 30, 2026
- Convert the hybrid HDF5+Parquet dataset into WebDataset training shards
  
  thompsonmj
  and
  claude
  committed
  5009164
- Document WebDataset creation from the hybrid HDF5+Parquet format
  
  thompsonmj
  and
  claude
  committed
  b6ce6f3
- Exit the Spark filter process once filtering completes
  
  thompsonmj
  and
  claude
  committed
  a0dfec5
- Rename the tool to tol_hybrid_to_wds and drop empty-taxonomy entries during the build
  
  thompsonmj
  and
  claude
  committed
  e3ee9d1
- Relocate the one-off taxonomy-scrub scripts out of scripts/
  
  thompsonmj
  and
  claude
  committed
  0ac8c2e
Commits on Jul 1, 2026
- Un-force-wrap comments and markdown.
  thompsonmj
  committed
  a32dd03
- Move the WDS companion scripts under the tool
  thompsonmj
  committed
  e9adbec
- Relocate the standalone single-node converter out of the repo root
  thompsonmj
  committed
  45e7447
- Relocate the one-off taxon_com audit out of the repo root
  thompsonmj
  committed
  cf99406
- Update the WDS docs for the relocated scripts
  thompsonmj
  committed
  f5154fb