Image simulations workflow

auxdir/CFIS/im_sims_202606/tile_numbers_1.txt

@@ -0,0 +1,40 @@
+233.293
+237.292
+238.292
+239.291
+239.292
+239.293
+240.291
+240.292
+240.293
+241.293
+242.290
+242.291
+242.292
+242.293
+243.290
+243.291
+243.292
+243.293
+244.293
+245.290
+245.291
+245.292
+246.290
+246.291
+247.289
+247.291
+247.292
+247.293
+248.288
+248.289
+248.290
+248.291
+249.289
+249.290
+249.291
+249.292
+250.289
+250.290
+250.292
+251.287

auxdir/CFIS/im_sims_202606/tile_numbers_2.txt

@@ -0,0 +1,43 @@
+251.288
+251.289
+251.291
+252.287
+252.288
+252.289
+252.291
+253.287
+253.288
+253.291
+254.286
+254.287
+254.288
+255.286
+255.288
+255.290
+256.286
+256.287
+257.285
+257.286
+257.287
+257.288
+258.287
+258.289
+259.285
+259.286
+259.287
+259.288
+259.289
+260.284
+260.285
+260.286
+260.287
+260.288
+260.289
+261.284
+261.285
+261.286
+261.287
+261.288
+262.285
+262.286
+262.287

auxdir/CFIS/im_sims_202606/tile_numbers_3.txt

@@ -0,0 +1,46 @@
+262.288
+263.283
+263.284
+263.285
+263.286
+264.283
+264.284
+264.286
+264.287
+265.284
+265.285
+265.286
+265.287
+266.282
+266.283
+266.285
+266.286
+267.283
+267.284
+267.285
+267.286
+268.282
+268.283
+268.284
+268.285
+269.281
+269.282
+269.283
+269.284
+269.286
+270.281
+270.282
+270.283
+271.281
+271.282
+271.283
+271.284
+271.285
+272.281
+272.282
+272.283
+272.285
+273.280
+273.282
+273.283
+273.284

auxdir/CFIS/im_sims_202606/tile_numbers_4.txt

@@ -0,0 +1,37 @@
+274.279
+274.280
+274.282
+274.283
+275.279
+275.280
+275.282
+275.283
+276.279
+276.280
+276.281
+276.282
+276.283
+277.279
+277.281
+277.282
+278.279
+278.281
+278.282
+279.281
+279.282
+280.280
+280.281
+280.282
+281.280
+281.281
+282.280
+282.281
+283.280
+284.279
+285.279
+285.280
+286.279
+286.280
+287.279
+288.279
+289.279

example/cfis_image_sims/README.md

@@ -0,0 +1,79 @@
+# cfis_image_sims — module configs for ShapePipe on simulated tiles
+
+This directory is the single `.ini` tree for image-simulation runs. It is
+selected by `run_job_sp_canfar_v2.0.bash -t image_sims` (which sets
+`config_dir = example/cfis_image_sims`, `retrieve = symlink`, and forces
+`tile_det = sx`) and consumed stage-by-stage by `job_sp_canfar_v2.0.bash`.
+
+The pipeline runs as a bit-coded chain of jobs. `run_job_sp_canfar_v2.0.bash`
+loops over the bits set in `-j`, initialises the tile/exposure work directories,
+runs each completeness check, and delegates the actual `shapepipe_run` calls to
+`job_sp_canfar_v2.0.bash -j <bit>`, which is where the `.ini` file for each bit
+is selected. A few bits are special-cased for sims *before* that delegation —
+those cases are documented in the last column below and, at more length, under
+[Sim special-casing](#sim-special-casing).
+
+## Job-bit dispatch
+
+Every row is derived from the two bash scripts. "Module(s)" is the ShapePipe
+runner(s) the selected `.ini` names; "`.ini` selected" is what
+`job_sp_canfar_v2.0.bash` picks for that bit under sim settings
+(`retrieve=symlink`, `psf=psfex`, `tile_det=sx`, `star_cat_for_mask=onthefly`).
+
+| Bit | Stage | Module(s) | `.ini` selected (sim settings) | Sim special-casing |
+|----:|-------|-----------|--------------------------------|--------------------|
+| 1 | retrieve tile image + weight | `get_images_runner` | `config_tile_Git_symlink.ini` | symlink retrieval (see below); completeness check expects 2 files (image + weight) vs. 4 for data |
+| 2 | uncompress tile weight | *(none — faked)* | *(none)* | **Faked weight-uncompress.** run_job skips the `config_tile_Uz.ini` run entirely; sim weights are already uncompressed, so it fabricates a `run_sp_tile_Uz*/uncompress_fits_runner/output` dir and symlinks `input_tiles/CFIS_simu_weight-<ID-dashed>.fits` into it, satisfying downstream `last:uncompress_fits_runner` lookups |
+| 4 | find exposures | `find_exposures_runner` | `config_tile_Fe.ini` | — |
+| 8 | retrieve exposure images | `get_images_runner` | `config_exp_Gie_symlink.ini` | symlink retrieval; completeness check expects 3 files vs. 6 for data |
+| 16 | split exposures, merge WCS headers | `split_exp_runner` | `config_exp_Sp.ini` | — |
+| 32 | mask exposures | `mask_runner` | `config_exp_Ma_onthefly.ini` | — |
+| 64 | exposure PSF model | *(none — placeholder)* | *(none)* | **Placeholder.** For data this runs full exposure PSF modelling. For sims run_job writes a placeholder log and does nothing here; the sim PSF (`fake_psf_runner`) actually runs inside bit 512 |
+| 128 | merge exposure WCS headers → tile sqlite log | `merge_headers_runner` | `config_tile_Mh_exp.ini` | — |
+| 256 | object detection on tiles | `sextractor_runner` | `config_tile_Sx_nomask.ini` | `tile_det` is forced to `sx`, so the SExtractor-no-mask branch is always taken (the `uc` external-catalogue branch is never reached for sims) |
+| 512 | fake PSF + postage stamps | `fake_psf_runner`, then `vignetmaker_runner` ×2 | `config_exp_psfex.ini` (fake PSF), then `config_tile_PiViVi_canfar_sx.ini` (vignets) | **Two sub-runs.** run_job first calls the job script with `-j 64` → `config_exp_psfex.ini`, which despite its name runs `fake_psf_runner` (needs the sexcat from bit 256; run dir `run_sp_tile_fpsf`), then `-j 512` → `config_tile_PiViVi_canfar_sx.ini` for the two `vignetmaker_runner` runs. Data instead runs `psfex_interp_runner` + vignets here |
+| 1024 | multi-epoch shape measurement | `ngmix_runner` | `config_tile_Ng_batch_psfex_sx.ini` | — |
+| 2048 | create final catalogue | `make_cat_runner` | `config_tile_Mc_psfex.ini` | — |
+
+Notes on cross-cutting conventions used above:
+
+- **Dashed tile-id convention.** Data tile IDs use dot format (`233.293`); sim
+  input files use dash format (`233-293`). run_job writes `tile_numbers.txt` in
+  dash form for `image_sims` (`${ID//./-}`), and the faked bit-2 weight symlink
+  and sim input patterns all key off the dashed ID.
+- **Symlink retrieval.** `-t image_sims` sets `retrieve=symlink`, so bits 1 and 8
+  select the `*_symlink.ini` get-images configs (`RETRIEVE = symlink`), which
+  link sim images out of `$SP_DIR/input_tiles` / `$SP_DIR/input_exp` rather than
+  downloading from VOSpace.
+- **Forced `tile_det=sx`.** The `image_sims` type branch forces `tile_det=sx`, so
+  bit 256 always runs SExtractor detection and bit 512 selects the `_sx`-suffixed
+  vignet/ngmix/PiViVi configs.
+
+## Sim special-casing
+
+Three bits diverge from the data chain, all handled in
+`run_job_sp_canfar_v2.0.bash` before the per-bit delegation:
+
+- **Bit 2 — faked weight-uncompress.** Sim weights ship uncompressed, so instead
+  of running `uncompress_fits_runner`, run_job fabricates the expected Uz output
+  directory and symlinks `CFIS_simu_weight-<ID-dashed>.fits` in, so downstream
+  `last:uncompress_fits_runner` references resolve.
+- **Bit 64 — placeholder.** run_job writes a placeholder completeness log and
+  runs nothing; the sim PSF is produced by `fake_psf_runner` inside bit 512.
+- **Bit 512 — fake PSF + vignets.** run_job runs `fake_psf_runner`
+  (`config_exp_psfex.ini`) followed by the two `vignetmaker_runner` runs
+  (`config_tile_PiViVi_canfar_sx.ini`). `fake_psf_runner` consumes the sexcat
+  from bit 256, so bit 256 must precede it.
+
+## Parallelism model and the `PSF_DICT` requirement
+
+The orchestrator (the sp_validation Snakemake workflow) fans out one job per
+tile; within a single job ShapePipe uses its own multiprocessing, controlled by
+`-N/--N_SMP` (threaded through to each module's `SMP_BATCH_SIZE`). Every sim
+config sets `MODE = SMP` — the sims chain launches no MPI.
+
+`fake_psf_runner` reads the sims' pickled PSF dictionary from the `$PSF_DICT`
+environment variable (`config_exp_psfex.ini` sets `PSF_DICT_PATH = $PSF_DICT`,
+expanded from the environment). The orchestrating workflow exports it (from its
+`psf_dict` config key); on candide it points at the SKiLLS PSF dictionary. Bit
+512 will fail if `$PSF_DICT` is unset.

example/cfis_image_sims/config_exp_Gie_symlink.ini

@@ -0,0 +1,90 @@
+# ShapePipe configuration file for: get images
+
+
+## Default ShapePipe options
+[DEFAULT]
+
+# verbose mode (optional), default: True, print messages on terminal
+VERBOSE = False
+
+# Name of run (optional) default: shapepipe_run
+RUN_NAME = run_sp_exp_Gie
+
+# Add date and time to RUN_NAME, optional, default: False
+RUN_DATETIME = True
+
+
+## ShapePipe execution options
+[EXECUTION]
+
+# Module name, single string or comma-separated list of valid module runner names
+MODULE = get_images_runner
+
+# Parallel processing mode, SMP or MPI
+MODE = SMP
+
+
+## ShapePipe file handling options
+[FILE]
+
+# Log file master name, optional, default: shapepipe
+LOG_NAME = log_sp
+
+# Runner log file name, optional, default: shapepipe_runs
+RUN_LOG_NAME = log_run_sp
+
+# Input directory, containing input files, single string or list of names
+INPUT_DIR = $SP_RUN
+
+# Output directory
+OUTPUT_DIR = $SP_RUN/output
+
+
+## ShapePipe job handling options
+[JOB]
+
+# Batch size of parallel processing (optional), default is 1, i.e. run all jobs in serial
+SMP_BATCH_SIZE = 1
+
+# Timeout value (optional), default is None, i.e. no timeout limit applied
+TIMEOUT = 96:00:00
+
+
+## Module options
+
+# Get exposures
+[GET_IMAGES_RUNNER]
+
+INPUT_MODULE = last:find_exposures_runner
+
+FILE_PATTERN = exp_numbers
+
+FILE_EXT = .txt
+
+# NUMBERING_SCHEME (optional) string with numbering pattern for input files
+NUMBERING_SCHEME = -000-000
+
+
+# Paths
+
+# Output path (optional, default is [FILE]:OUTPUT_DIR
+# OUTPUT_PATH = input_images
+
+# Input path where original images are stored. Can be local path or vos url.
+# Single string or list of strings
+INPUT_PATH = $SP_DIR/input_exp/,$SP_DIR/input_exp/,$SP_DIR/input_exp/
+
+# Input file pattern including tile number as dummy template
+INPUT_FILE_PATTERN = simu_image-0000000,simu_weight-0000000,simu_flag-0000000
+
+# Input file extensions
+INPUT_FILE_EXT = .fits, .fits,.fits
+
+# Input numbering scheme, python regexp
+INPUT_NUMBERING = \d{7}
+
+# Output file pattern without number
+OUTPUT_FILE_PATTERN = image-,weight-,flag-
+
+# Method to retrieve images, one in 'vos', 'symlink'
+RETRIEVE = symlink