Psf leakage#27
Merged
Merged
Conversation
This comment has been minimized.
This comment has been minimized.
Unit Test Report 1 files 1 suites 5s ⏱️ For more details on these failures, see this check. Results for commit 138637c. |
cailmdaley
added a commit
that referenced
this pull request
Jul 1, 2026
shear_psf_leakage #27 removed the square_size parameter from build_cat_to_compute_{rho,tau} and CovTauTh (HSM_T_* already stores the area T=2σ², so the flag was dead weight). sp_validation still threaded square_size into those leakage calls; once #27's container lands, passing it would raise TypeError. Drop the square_size key from both param builders (rho_tau.get_params_rho_tau, cosmo_val/compute_theory_cov.py) and stop passing it to the handlers. Behavior-identical against the current container (square_size defaults to False), required for the post-#27 one. Also correct the migration doc: galsim shapes are not producible by ShapePipe (no shape-measurement runner; make_cat's galsim mode reads an external catalogue nothing generates; production configs pin ngmix) — the decisive reason the dead consumer path was removed rather than migrated; and reframe the shear_psf_leakage coordination now that #27 is a sibling PR landing the same grammar. Suite: 137 passed, 1 skipped (unrelated cmss12.tfm font gap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnCCTxKE13cHzPr8BUeKZE
cailmdaley
added a commit
that referenced
this pull request
Jul 3, 2026
* docs: record the ngmix PSF/shape column migration plan Downstream of CosmoStat/shapepipe#761, which makes the ngmix output columns a single source of truth (true original-PSF fit stored separately, ELL split into scalar G1/G2, ESTIMATOR_COMPONENT_OBJECT grammar). Captures the old->new column map and the sp_validation consumer checklist. Blocked on #761 landing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: rewrite column-migration map to the as-shipped v2 grammar The plan committed 2026-06-20 predated the "drop GAL" decision on shapepipe#761: it showed every galaxy column gaining a _GAL token and said nothing about HSM. Rewrite it to the authoritative as-shipped grammar, grounded in the #761 producer source: - galaxy is the implicit object (no _GAL token); ellipticity splits into named scalars NGMIX_G1/G2_{shear}, errors likewise; - NGMIX_Tpsf -> NGMIX_T_PSF_RECONV (value-safe); ELL_PSFo/T_PSFo -> G1/G2/T_PSF_ORIG (value change, shapepipe#749); - HSM E*_{PSF,STAR} -> HSM_G*_{PSF,STAR} (native g, pure rename), SIGMA_*_HSM -> HSM_T_* (units: now T=2sigma^2), FLAG_*_HSM -> HSM_FLAG_*; - spread_model removed; sizes route through cs_util.size; square_size retired. e->g conversion is not needed anywhere (both estimators store g). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv * refactor(src): adopt ShapePipe-v2 column grammar in the package Consumer half of shapepipe#761. Every shape-column read in the package moves onto the one grammar: - metacal (calibration.py): galaxy ellipticity read as named scalars NGMIX_G1/G2_{shear} and errors NGMIX_G1/G2_ERR_{shear} (was packed NGMIX_ELL_*[:, c]); NGMIX_Tpsf -> NGMIX_T_PSF_RECONV. The obsolete col_2d flag (packed-vs-split ellipticity) is removed from the metacal ctor and get_variance_ivweights — v2 is always named scalars. - galaxy.py: NGMIX_ELL_PSFo[:, 0] -> NGMIX_G1_PSF_ORIG scalar; the spread_model branch/param dropped (shapepipe stopped writing it) — classification falls back to the size-based path. - catalog.py: match_spread_class deleted (SPREAD_CLASS gone); match_subsample takes two scalar keys (g1_key, g2_key) instead of a packed ell_key; check_invalid drops the now-meaningless comp arg. - glass_mock.py: HSM E1/E2_PSF_HSM -> HSM_G1/G2_PSF. - rho_tau.py: sizes are always T=2sigma^2 now, so square_size is False and the not_square_size list is gone. - tests: synthetic catalogues + configs rebuilt on the v2 grammar in lock-step (the migration's internal-consistency check). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv * refactor(config): rename HSM/ngmix columns to v2 grammar, drop square_size - cat_config.yaml: every HSM block onto the v2 grammar — E*_{PSF,STAR}_HSM -> HSM_G*_{PSF,STAR}, SIGMA_*_HSM / T_*_HSM -> HSM_T_*, FLAG_*_HSM -> HSM_FLAG_*. The DES/piff block is untouched (Piff isn't this grammar). The dead per-dataset square_size flags are dropped: sizes are always T=2sigma^2 now, so nothing squares (this also fixes a latent factor error in the SP_v1.3 block, which squared its already-T T_PSF_HSM). - mask_v1.X.{2..11}.yaml: NGMIX_ELL_PSFo_NOSHEAR_0/_1 -> NGMIX_G1/G2_PSF_ORIG_NOSHEAR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv * refactor(scripts): migrate scripts, papers, cosmo_inference to v2 grammar Non-library consumers onto the same grammar: - scripts/calibration/{extract_info,params,calibrate_comprehensive_cat}.py: Tpsf -> T_PSF_RECONV; ELL_PSFo -> G1/G2_PSF_ORIG; T_PSFo -> T_PSF_ORIG; bare galaxy ELL -> G1/G2 scalars (match_subsample/check_invalid callers updated for the new signatures); HSM E*_PSF -> HSM_G*_PSF; spread_model wiring removed; col_2d args dropped. - scripts/apply_alpha_snr_size_bin.py, scripts/examples/demo_*.py: same Tpsf/PSFo renames, col_2d dropped. - cosmo_val/compute_theory_cov.py, cosmo_inference/scripts/masking.py: Tpsf -> T_PSF_RECONV; square_size hardcoded False. - papers/catalog/*, papers/harmonic/*: HSM ellipticity renames and the sigma->T units fix (hand-rolled SIGMA_*_HSM**2 -> read HSM_T_* directly). Out of scope, left as-is: GALSIM_* columns, scratch/, and the cosmo_inference notebooks (own migration fiber). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv * refactor(src): remove dead galsim path, finish v2 grammar tightening Two tightening-pass changes to complete ShapePipe-v2 column adoption in the package: - Remove the galsim estimator path as dead code. `shape` is hardcoded to "ngmix", extract_info.py raises for any other value, and nothing outside scratch/ instantiates metacal(prefix="GALSIM") or calls classification_galaxy_galsim. Migrating it would carry an untestable path whose shared `col_1p = {prefix}_T_PSF_RECONV_1P` read never matched the galsim producer output (GALSIM_T_PSF_*, not ..._T_PSF_RECONV_*) — already broken. Removed: metacal._read_data_galsim, the prefix=="GALSIM" dispatch (now else: raise — unknown prefixes fail loudly), the two galsim ellipticity sign flips in _shear_response/_selection_response, galaxy.classification_galaxy_galsim, the sh=="galsim" branch in catalog.get_snr (now else: raise), and the unused shape_method arg on get_calibrated_quantities/get_calibrated_m_c. ngmix is the sole estimator; the `prefix` param stays (names the column family; a future NGMIXm moments family could reuse it). - Rename the galaxy failure-flag read NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL in classification_galaxy_ngmix. The v1 column counted moments-initial-guess failures (get_guess, gone in v2); the producer reused the slot for a failed-metacal-types count. sp_validation cuts on == 0 either way (keep fully-measured objects), so the read migrates cleanly; the underlying failure mode changed, so this is the first line to check if the post-cut galaxy count looks off against a regenerated v2 catalogue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW * refactor(config): NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL in mask configs The v1 failure-flag column no longer exists in the ShapePipe-v2 header; the producer renamed the slot to NGMIX_MCAL_TYPES_FAIL. Propagate the rename into all ten calibration mask configs so the flag column resolves against a v2 catalogue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW * refactor(scripts,papers): finish v2 grammar tightening in scripts - NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL in params.add_cols_pre_cal (and its int-format set), the two demo column lists, and hist_mag's read lists. - Drop the galsim mentions left in params.py (shape comment) and extract_info.py (the stale "cuts common to ngmix and galsim" comment, whose spread-model line also went with the earlier spread_model removal). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW * refactor(notebooks): migrate cfis_analysis HSM PSF columns to v2 grammar cfis_analysis.ipynb read E1/E2_PSF_HSM into treecorr for the C_sys PSF test; rename to HSM_G1/G2_PSF (native g, pure rename). A sweep of all tracked notebooks found no other old column tokens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW * docs: record galsim removal, MOM_FAIL rename, and *_PSF_ORIG reframe Update the authoritative migration map for the tightening pass: - *_PSF_ORIG reframed as a value change that is NOT a code blocker (the rename is correct as-is; a v2 catalogue only enables a look-at-numbers check). The real merge gate is cutover timing — merging makes develop require v2 columns. - NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL added to the ngmix map with a semantics-change note. - galsim path documented as removed dead code (was "left untouched, flagged"). - notebooks added to the consumer-sites list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW * refactor(rho_tau,cov): drop square_size, coordinate with leakage #27 shear_psf_leakage #27 removed the square_size parameter from build_cat_to_compute_{rho,tau} and CovTauTh (HSM_T_* already stores the area T=2σ², so the flag was dead weight). sp_validation still threaded square_size into those leakage calls; once #27's container lands, passing it would raise TypeError. Drop the square_size key from both param builders (rho_tau.get_params_rho_tau, cosmo_val/compute_theory_cov.py) and stop passing it to the handlers. Behavior-identical against the current container (square_size defaults to False), required for the post-#27 one. Also correct the migration doc: galsim shapes are not producible by ShapePipe (no shape-measurement runner; make_cat's galsim mode reads an external catalogue nothing generates; production configs pin ngmix) — the decisive reason the dead consumer path was removed rather than migrated; and reframe the shear_psf_leakage coordination now that #27 is a sibling PR landing the same grammar. Suite: 137 passed, 1 skipped (unrelated cmss12.tfm font gap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnCCTxKE13cHzPr8BUeKZE --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request implements the modification in #26 in the function used for the PSF leakage.
replace=Falsein the jackknife weighted average.I don't know how to pass both option
parallel=Trueand setn_jobsin the function may be @sfarrens has some idea how to do that?Parts of the code are commented to keep it transparent with the previous implementation. But they should be un-commented in the future once the rest of the code is up-to-date.