Image simulations workflow#766
Merged
Merged
Conversation
added 25 commits
May 17, 2026 19:39
Closed
6 tasks
added 2 commits
June 30, 2026 17:26
This was referenced Jul 10, 2026
) Add an opt-in SEED_FROM_POSITION knob to the ngmix module (default False). When on, the object loop replaces the tile-level RNG with a per-object RandomState seeded from the object's sky position, and rebuilds the prior from that same RNG, so every stochastic step of the fit is a deterministic function of (ra, dec, ccd). Why: image-sim m-bias uses the Pujol estimator — the same scene under different applied shears, with shear response read from the branch difference of the SAME objects. Metacal's fixnoise adds a counter-noise realisation from an RNG seeded per tile, so an object gets different added noise in different branches (detection order differs) and the noise fails to cancel in the difference, inflating sigma_m. Seeding per object from position makes the same object draw the same added noise AND the same fit guesses in every branch, so both cancel and the m-bias error shrinks. Design and box math are Fabian's issue #796. The prior must be rebuilt per object, not just the RNG: the ngmix guesser draws its initial guess via prior.sample(), which consumes the RNG the prior was CONSTRUCTED with (ngmix 2.4.0 guessers.py / joint_prior.py). A per-object RNG alone would leave the guess drawing from the shared tile stream and break cancellation. The fixnoise counter-noise itself rides on the observation's attached noise image (use_noise_image=True), which prepare_ngmix_weights draws from the per-object RNG — so no fresh draw from the bootstrapper RNG is needed. Deviation from #796: Fabian combines the 3-arcsec position boxes as box_x + box_y, which collides along anti-diagonals — boxes (10,20) and (11,19) both give seed 30, so two distinct objects share a noise stream. This replaces the sum with a Cantor pairing (a bijection on the non-negative integers) after a zig-zag fold that maps signed box indices (Dec can be negative) onto non-negative ones, reduced mod 2**32 into the valid RandomState seed range. The box math is kept exactly as his: floor(ra*3600/3)+(ccd+1), floor(dec*3600/3)+(ccd+2), first-epoch coordinates (coord_list[0] convention), so all epochs of one object share the stream. Off by default => production path is byte-identical to develop (verified: metacal g for all five types matches unmodified develop bit-for-bit; the full fast suite passes, 326 tests). New property tests cover the seed's range, sub-box jitter invariance, adjacent-box distinctness, and the resolved anti-diagonal collision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9qEVRCWEekT5ACNB3rjkq
… run Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
compile_results filled nfev_fit with np.nan on failed fits, the only int-like column with a float fallback. With SAVE_BATCH enabled, save_results locks column dtypes from batch 1; if that batch has no failures the column locks to int64, and the first later batch with a failure feeds NaN into it, raising "cannot convert float NaN to integer" on append. Use ngmix's own -1 failed/absent-nfev sentinel instead of NaN, keeping the column int64 across every batch. Fixes #795
Companion to the coverage rule in the Snakefile: scans input weight
maps (subsampled), writes tile_coverage_grid_{num}.yaml with an
exclude list read at DAG-build time. Weight-map location is taken
from the get-images ini of the 1p2z variant. Plus two felt fibers
documenting the blank-tile and thread-exhaustion findings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ling) into im_sims Resolutions: - ngmix.py: Martin's zero-noise edge-stamp guard kept, generalized to np.all(sig_noise > 0) so it stays valid on the per-pixel bkg_rms path, composed with the #803 per-object rng draws (his branch predated #803). - ngmix_runner.py: both imports (os + SqliteDict). - scripts/image_sims_pipeline/*, docs, auxdir tile lists, init_run .sh->.py: Martin's side wholesale — his live operational tooling on candide. The shapepipe-produces/sp_validation-orchestrates consolidation is deferred to the #766 develop-merge review, not imposed on his working branch. - fake_psf.py / vignetmaker.py auto-merges reviewed: robustness/logging only, no PSF content change. - His .felt debug fibers ride along (grid_4 blank tiles, snakemake OMP thread exhaustion) — both operationally important. Verified: tests/module/test_ngmix.py 22/22 in the runtime env (srun). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UvQvyJTEwYcLCwDEgpWMJ6
The image_sims_pipeline Snakefile invokes all container commands through this wrapper; it strips SLURM/PMI/PMIX/OMPI env vars that crash OpenMPI inside apptainer on SLURM compute nodes. No replacement exists in run_scratch_local.sh or run_job_sp_canfar_v2.0.bash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The image-simulation pipeline's orchestration now lives in sp_validation (companion PR #225), which drives ShapePipe via `run_job -t image_sims --config_dir`. The in-repo Snakemake driver under scripts/image_sims_pipeline/ (Snakefile, info.py, monitor_mbias.py, config.yaml.template) is superseded and removed to keep a single source of orchestration truth. Its coverage-scan helper, check_tile_coverage.py, is the one piece of reusable logic with no equivalent elsewhere; it moves to scripts/python/ alongside the other run-directly helpers (create_final_cat.py, etc.) and gains the executable bit to match that directory's convention and its own shebang. docs/source/image_sims_calibration.md documented only this retired orchestrator (its Snakemake targets, info.py, monitor_mbias.py) and was an orphan page not wired into any Sphinx toctree; it is removed with the code it described. History of the old pipeline is preserved in the .felt/ notes, which are left untouched as a chronological record. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
The runner self-computes config_dir from --type (example/cfis for data, example/cfis_image_sims for image_sims) and threads it through the cfis symlinks, SP_CONFIG, and job_sp_canfar's own -c. That hardcodes the .ini tree to the in-repo example dirs. Add an optional -c/--config_dir that, when given, overrides the computed config_dir uniformly for any --type. The override is applied after the type branch, so when the flag is absent config_dir is byte-for-byte the historical value and every existing caller (test_tile_det.py, functions.sh, the sp_validation im_pipeline rule) is unaffected. This lets the sp_validation workflow point every stage at an explicit .ini tree, per the image-sims boundary rule (workflow selects and parameterizes the tree). The flag name matches job_sp_canfar_v2.0.bash's own -c/--config_dir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
The cfis_image_sims fake_psf config hardcoded a collaborator-specific absolute path to the SKiLLS PSF dictionary, coupling the in-repo config tree to one machine's layout. Replace it with PSF_DICT_PATH = $PSF_DICT, which the config layer's getexpanded (os.path.expandvars) resolves from the environment. The orchestrating workflow now exports PSF_DICT from its psf_dict config key, so the campaign fact lives upstream in the manifest, not in this repo -- matching the module-config/workflow boundary rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
Add a README to example/cfis_image_sims/ giving the accurate, per-bit mapping of the image-sims chain: for each job bit (1..2048), the ShapePipe module, the .ini file job_sp_canfar_v2.0.bash selects under sim settings, and the special-casing run_job_sp_canfar_v2.0.bash applies. Every row is derived by reading the two bash scripts, so the table stays a truthful reference for the sims path rather than a guess. Captures the parts that surprise: the faked weight-uncompress at bit 2, the placeholder bit 64 with fake_psf actually running inside bit 512, the dashed tile-id convention, forced tile_det=sx, and symlink retrieval. Adds the parallelism model (per-tile fan-out, in-job SMP, no MPI) and the $PSF_DICT env requirement for fake_psf. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Z3fwStiamMsEswbJ66BeF
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H2wXqsx5RprRXVh4MQsn6h
Contributor
Author
|
@cailmdaley is this ready to be merged? |
Contributor
|
not yet, tests are failing and i have a big cleanup/redesign un-pushed. i was waiting for your comments or approval on the redesign, which i sketched out in the description of this PR and in the description of sp_validation PR 225 (the latter contains most of the important design logic). please tell me what you think or just give me the green light to go ahead, i didn't want to step on your toes again :) |
Contributor
Author
|
as fas as I can see the new design looks good, you have green light to go ahead |
Brings in the latest im_sims tip: the develop merge (ngmix v2.0 metacal kernel selection, mask/sextractor/setools fixes, Renovate migration, science guardrail suite) plus two new image-sims commits — info.py's improved merged-HDF5 reading (8294fa5) and the Pujol citation comment on config_tile_Ng_batch_psfex_sx.ini (3bc4729, which merged cleanly). Two modify/delete conflicts, both resolved by keeping our deletions: this branch retired the in-repo Snakemake orchestrator in f01344a, so origin's edits to scripts/image_sims_pipeline/Snakefile (569ffe9) and info.py (8294fa5) are dropped here (git rm). The Snakefile's operational lore — SLURM runtime units, cpus_per_task>=12 vs `ulimit -u 1200` — is ported to sp_validation's workflow, handled separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The [tool.setuptools] script-files list referenced scripts/sh/init_run_v2.0.sh, which was removed in b0804ce when the im_sims retarget moved orchestration to sp_validation. setuptools' build_editable errors on the missing file, breaking every Docker build in CI (`uv pip install --no-deps -e .` → "file '...init_run_v2.0.sh' does not exist"). The other three listed scripts (update_runs_log_file.py, run_job_sp_canfar_v2.0.bash, job_sp_canfar_v2.0.bash) still exist, so this removes only the stale entry. Verified with an editable-install dry-run in the im_sims runtime container: resolves to shapepipe-1.1.0 with no build_editable error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cstring Polish over the image-sims files this PR touches: - check_tile_coverage.py: correct the module docstring's stale "the Snakefile reads the exclude list" wording — orchestration moved to sp_validation (f01344a), so the caller is now the sp_validation workflow, not an in-repo Snakefile. - bin/shapepipe_run.py: drop the "MKDEBUG start/end" print statements that wrapped main(). - ngmix.py: drop two "# MKDEBUG TODO this needs to be checked!" tags appended to the zero-sigma / zero-noise weight guards; the preceding comments already explain why the guards exist. Only strips MKDEBUG introduced by the im_sims merges; the pre-existing marker in create_final_cat.py (also on develop) is left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two private fiber notes (blank-input-tiles-grid-4, snakemake-crash- runtimeerror) were committed on origin/im_sims in 9783a2e, before the develop-side .felt untrack (3d56ec6) landed there, so they survived the merge into this branch. The repo gitignores /.felt/ and treats it as a machine-local symlink into the private loom store (see CLAUDE.md); these tracked files bypass that ignore. Untrack them (kept on disk) so no fiber content rides into the shared CosmoStat repo. Their operational lore is preserved in sp_validation's workflow docs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
i merged in my redesign and reran the whole pipeline, with no changes to the m-bias that comes out, so i am going to merge this and the corresponding sp_validation PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of [Tracking] Image simulations (UNIONS-WL/MultiBand_ImSim#1).
Big picture
To measure the shear bias (m, c) of the ShapePipe chain, we run it end-to-end on simulated images with known injected shear and compare output to input. Each distinct injected-shear value defines a branch — the same sky simulated under a different applied shear. The work splits along the dependency arrow:
Boundary rule: ShapePipe owns its module configs (the
.initree lives here, next to the code whose options it names, so code and config change in the same PR); the workflow selects and parameterizes that tree explicitly. Campaign facts — injected shear, branch naming — live upstream in the workflow's manifest and never appear in this repo.What's here
Sim-specific modules and paths:
fake_psfmodule — per-galaxy PSF stamps from the sims' PSF dictionary, drop-in forpsfex_interp's downstream format. The dictionary path is supplied by the workflow (environment-expanded in the config), not hardcoded.BKG_SUBngmix option — skip background subtraction for background-free sims; composes withngmix_v2.0's vignette and centroid options.SEED_FROM_POSITION, Seed fix based on position for Pujol noise cancellation for image simulations #796) — metacal's counter-noise RNG is seeded from each object's sky position, so the same object gets identical fixnoise across shear branches and the noise cancels in the branch difference (measured ~5–6× reduction in σ_m per tile). The seed formula is pinned by unit tests. Enabled only in the sims ngmix config.exp_utils,sextractor_script,merge_headersWARN_MISSING_EXP) and batch-output dtype fix fornfev_fit([BUG] ngmix crash #795).cfis_image_sims/module configs — the one.initree for sim runs — andcreate_final_cat'simage_simsmode.One runner path:
run_job -t image_simsdrives the full module chain and accepts--config_dir; the workflow passes it explicitly, so the workflow's choice of.initree governs every stage. Orchestration lives in sp_validation; the in-repo Snakemake pipeline has been retired on this branch, with its coverage-scan helper (check_tile_coverage.py) kept underscripts/python/.Parallelism model: the orchestrator fans out per-tile jobs, and ShapePipe uses its own multiprocessing within a job; the sims chain launches no MPI.
Validation
The full chain (raw sim tile → per-tile catalogues →
final_cat_<branch>.hdf5) runs on candide. Downstream, the companion PR's estimator measures the residual m left after ShapePipe's own response calibration; on one fiducial tile it is consistent with zero at the percent level, and that fixed-tile measurement serves as the regression check for changes on both branches.— Fable, on behalf of Cail