Repository navigation
DiffAE / OPSin interpretability release - #22
Merged
Merged
Conversation
… wraps so it never runs off-screen
…f the ML techniques (SetTransformer, DiffAE traversals, DDIM inversion, attention, virtual staining, transcriptional next step) with core-paper links
…(MIL/ISAB×2/PMA/cosine), Top-cells explaining-by-removing cell selection, DDIM named+broken-down, denoising-chain animation, per-slide Key-terms glossaries
…ms open by default, +25% text, wider layout, Screen slide poses the core question, Embedding/LatentLens slide, Classifier as mini neural net, define 'semantic'+'spatial conditioning', deck reordered (attention after classifier), fix score-legend leaking onto all tabs
… VS rebuilt from schematic (semantic Cell-DINO->FiLM + spatial phase-concat U-Net), Screen well+debarcode animation, Classifier mini neural-net, soften unverified attention-head claims, scroll-to-top on slide change, spell out OPSin title, Funk+Chad paper links
…mbedding & Embedding->Montage (+viewer tab), ViT/Classifier/Top-cells/Attention/Montage/VS animations redrawn with targeted highlighting (no fade-all), Screen well+debarcode w/ one-group-is-a-mix, Diffusion per-section text + x_T no longer collapses, accessible Classifier copy, wider marker search, Feldman link restored
…ed)+alternating phenotypes, Embedding vector jitter, Classifier 2x2 color sets into net, Top-cells rank sizing, tubules inside cell, Traversal z=CellDINO vector bars, Montage size progression along arms, VS outputs=compartment cells w/ alternating highlight; drop shuffle line
…lasses (fixes SVG rotate() flying tubules/montage/mito), slower+subtler animations, marker bar back to ~50% bump, Screen non-blue subtle, Top-cells rank reversed (orange right, one only), Montage branch-alternation, VS phase-cell compartments synced to stain highlight cycle
…se it + z-conditioning viz), DDIM broken down word-by-word, What's-next architecture depth (reuse-map vs CPA designs), mid-ground animation timings + VS compartment fill-mode fix, move Zoom-detail slider under Renderer toggle in Montage tab
…op cells/Traversal/Montage/PCs/Attn/How-it-works), CZ Biohub wordmark, Top-cells 'accuracy score' label, Classifier says CellDINO features (not images), sweep stops on last tile (--sw), Montage per-cell wave fade, VS full compartment names + static phase cell, What's-next interleukin traversal + CPA explained
…ge via middle-anchor; wording now 'one per cell; a set = one gene')
…ebrand CZ Biohub -> Biohub); Top-cells fixes (accuracy-score label no longer overlaps score line; ranked dots spaced by radius so big dots don't overlap)
…center cell matches arm cadence (mth-branch) and labeled 'Single Control Cell' (3 lines)
…two lines + larger, box extended to enclose it
…refs, DiffEx (Bourou 2025) in Traversal refs
…ene-KO mAP/distinctiveness order) + tooltip
…er L->R like transcription) to What's-next slide, feeding the growing interleukin expression bar
…tention heads/Virtual staining/mRNA phenotypes as 'Extra' add-ons (unnumbered, + marker); rename 'What's next' -> 'mRNA phenotypes'; counter numbers only core methods (n/7, Extras show 'Extra')
…-express, same 3s envelope as mth-grow) so the strand is fully visible at peak expression
…l ('How would this cell look if we applied a given perturbation?')
… spiral arm's outer cell isn't clipped
…s to Embedding tab, defined in ViT patch-token terms
Add end-to-end pipeline for the multi-bag SHAP re-scoring of the v5
SetTransformer checkpoints (v5/multi_rank/):
- build_multibag_sidecars.py — build per-experiment sidecar parquets
(v5m_{gko,ebionly,ebifb}) with 1-rank/N score + cutoff_20k / cutoff_10k
- launch_multibag_sidecars.py — fan out via submit_parallel_jobs
- attach_multibag_ranks.py + launcher — attach gko/ebionly/ebifb ranks to
per-exp PCA h5ads for the sliding-window sweep (dedup by (exp,well,seg)
to handle ~0.06% dual-gene crosstalk rows)
run_weighted_pca_monkey_patch: register v5m_* strategies (pure, cutoff_20k,
cutoff_10k) and add --slurm-memory / --phase-memory passthroughs so
Phase-1 vstack (65M × 1024 float32 ≈ 247 GiB peak) can fit on high-mem
nodes without hitting the default 600GB.
plot_sweep_curves: force mAP y-axis to [0, 1.0] in overlay layout, move
metric label to y-axis (was title).
METHODS_final §13.4: describe the set-accuracy distilled Phase mAP
recomputation (rank-derived weight, top-20k cutoff, mean=1 per
(sgRNA, experiment) normalization); note gene-KO / EBI classifier
matched to gene-level / complex-level readout in Fig. 4D.
…e pop wave, drop sweep bar); Top-cells input as clean 2x2 set w/ orange bigger cell x; Montage per-arm perturbation labels + 'same control cell' note; Attention-heads mitochondria as irregular worms (not rice) + attention patches recolored to accent (distinct from orange mitos)
…ning compartments; Classifier final layer highlights only the predicted class's edges+node (others stay static)
…it covers both nucleoli
…rbation, gene A/B on cells, nearby=similar) + gene B reposition; nucleoli high-contrast white
…n search bar + real annotation in info panel; succinct message; restore Info button blue/gray toggle
…port CORS fix + 50% slower default - phase Traversal gets the marker-normalized background clip (PHASE_CLIP=0.20) - fix clip/toggle desync across Traversal<->Top Cells tabs (updateTravClip on tab entry, resync both pills) - Top Cells pop-out follows the marker-norm toggle (crops<->crops_norm via syncTcPopouts) - GIF export works on prod: crossOrigin+cache-buster so CloudFront frames don't taint the canvas - GIF export default 50% slower (deployed to prod/staging as sha-b852b70, app.js?v=270)
…ing Alex Lin's set-accuracy score with our v5 model Physically-remove-cell LOO marginal (mean over bag sizes) reproduces the viewer 'accuracy confidence' (pma_attention) ranking; an important cross-check of the LOO/Metric-B removal-based scoring. (renamed from alex_rank.py)
…AMM50 morpho target _resolve_seg was silently defaulting structure_type to "tubular" for every channel (mTOR's vesicular/blob config was resolving as tubular/frangi with the wrong max_radius_um), run_seg's parallel joblib workers raced writing to the same zarr store and randomly corrupted alpha=0's labels, and VS-H2B nucleus selection picked the largest object instead of the one closest to the crop center. Also adds CHROMALIVE_SAMM50 alongside the existing CHROMALIVE_MICOS13/TIM23 targets for the Figure-4 F-score work.
… filter SAMM50/MICOS13 (MICOS complex, mitochondria ChromaLIVE561) join TIM23 as native-160 measurement groups, sharing its seg params/checkpoint/NTC anchor pool, geneKO (not complex-pooled) ranking. Own centroid-recovery f values (SAMM50=1.55, MICOS13=2.0), not borrowed from TIM23. min_obj_px=15 added to TIM23/SAMM50/MICOS13 (+ TAF1B's tim23-style fscore companion): real NTC segmentation picks up ~2x more sub-visual noise fragments than gen (sensor noise vs. diffusion smoothing), inflating the "count" feature without being visually apparent -- this floor drops those before counting, on both real and gen.
…eatures alpha grid now includes the 0.5-step points (0.5, 1.5, 2.5) alongside the integer ones, and TIM23 gains area/connectivity/branches/nodes on top of degree/count. Widened figure + smaller tick labels to fit the extra columns; dropped the mean-diamond/IQR-bar styling in favor of a plain median tick, matching the F-score violins built on top of this module.
…date Adds SAMM50/MICOS13 to bruno_real_panels.GROUPS (geneKO, ChromaLIVE561 mitochondria, no anchor offset) so they render through the same unmodified image_panels/render_images path as every other Figure-4 gene. SAMM50's primary candidate moves from local rank 0 to rank 1 after review.
phi = alpha/f, where f is each perturbation's own centroid-recovery peak-alpha, so phi=0/1/2/3 land on the same biological milestone (none / half-recovered / fully-recovered / beyond-KO) across every gene instead of comparing raw alpha values that saturate at different points per gene. bruno_fscore.py renders the violins (merging each group's base stats.npz with the one new alpha grid point its f needs); bruno_fscore_panels.py renders the matching single-cell traversal panel through the same unmodified image_panels/render_images code in figure4_morpho_traversal.py that the raw-alpha reference page uses, just fed phi-snapped alphas.
…ers) Per-(marker, class) centroid-recovery: embed each class's top-k real crops (CellDINO) into class centroids, embed the target's generated frames per alpha, and find the fraction of generated cells whose nearest real centroid (top-1) is the true class. Peak-alpha of that top-1 curve is the f value used to phi-rescale each perturbation's traversal (bruno_fscore.py). f_centroid_recovery.py covers phase targets; centroid_recovery_fluor.py covers fluor markers (TIM23, SAMM50, MICOS13, mTOR, EMC8).
…egies
Alex's shap_screen_fluor_full_20k_all.csv + shap_screen_ebi_fluor_full_20k_all.csv
now cover every (gene, marker) pair up to 20k cells (previously only ~11-14%
of pairs had rankings). Fair-coverage runs on the old partial sidecar were
therefore silently falling back to uniform weights on most (gene, marker)
pairs, diluting the top-K signal.
Sidecar builder (build_multibag_fluor_sidecar.py):
- Refactor into a VARIANTS registry ({partial, full20k}) so both source sets
can be built to distinct parquet output paths side-by-side.
- Added --variant CLI arg (default keeps old behavior).
Monkey-patch (run_weighted_pca_monkey_patch.py):
- New V5M_SIDECAR_FLUOR_FULL path constant.
- Four new strategies mirroring the existing cutoff_500 / cutoff_100 variants
but routed to the full-coverage sidecar: v5m_{gko,ebionly}_fluor_full_cutoff_{500,100}.
- Dispatch: added is_v5m_fluor_full branch in both the fluor_path selection
and the phase_sidecar selection (the missed second branch was the source
of the FAILED-at-0:00 crash on first launch — phase-1 workers were
looking up the weight column in the phase parquet, which doesn't have it).
Result on paper_v2, no_phase, 2nd-pass PCA consensus:
gene-level distinctiveness: 0.396 → 0.903 (v5m_gko_fluor_full_cutoff_500)
protein-complex EBI: 0.488 → 0.900 (v5m_ebionly_fluor_full_cutoff_500)
Fluor now essentially matches phase on both axes; the coverage gap explained
the previous ~0.45 / 0.61 ceiling.
…e them bruno_final.py -> raw_alpha_violins.py bruno_real_panels.py -> raw_alpha_panels.py bruno_fscore.py -> fscore_violins.py bruno_fscore_panels.py -> fscore_panels.py Updates the import aliases in fscore_violins.py/fscore_panels.py and the comment references in morpho_native.py accordingly. Output directory paths (bruno/, bruno_fscore/ under figure4_traversals_violin/) are left as-is -- they're already linked from the live Confluence page and hold generated deliverables, not code.
…ubdirs metric_b_loo/ centroid_recovery/ anchor_standardization/ valid200/ bag_sweep/ embedding_viz/ figure4/ cache_utils/ - relocate the whole generation-validation package out of figures/ (it's scoring/validation, not figure code) - fix the 6 intra-package imports for the new subdir paths; update publish_multibag_page doc path (40 scripts; 30 tracked renames + 10 previously-untracked test scripts now kept)
…tability, diffex -> diffae, inner diffae -> generator Structural/code rename only (public-release org). Updated all ~140 python import paths (absolute + relative inner-diffae -> generator). On-disk data paths, OPS_DIFFEX_ASSETS env var, and SLURM job names intentionally left as 'diffex' (they reference real external resources / deployed opsin infra). 234 file renames; interpretability tree compiles clean; ops_process unaffected.
… (PR #19) Vendored from alexhillsley/public-release interpretability/classifier -> models/interpretability/classifier (sibling of diffae). Standalone set-classifier train/eval/score + configs (1K/EBI x phase/fluor, ebi_complexes.csv). Self-contained (relative imports); fixed the one absolute self-ref to the models/ path. Distinct from the older diffae/classifier.
…hap/ It documents the SHAP pipeline (run_all_shap, ko_shap_features, 6-way compare) with atlas render as one step, so it belongs with the shap/ scripts it drives.
Required webapp asset referenced by index.html and baked into every prod/staging image (13,487 keys, CORAL-degapped centroid-recovery [f,peak]). Was untracked in the source repo though tracked in the deploy repo — track it here so webapp/ is complete for the sync pipeline.
…ote staging is Okta-gated (no curl check)
…2->Okta so browser-only verify
…ames + regen tooling Viewer (webapp): - traversal φ hint (was α) + 'negative φ shows the opposite phenotype'; 'play φ ≥ 0' - Plausible analytics snippet - hide NTC from the perturbation list (no traversal assets) - prominent Marker/Perturbation search bars: 🔍 icon, equal widths, +10% text, bright border+glow, bold labels - rename tabs: Top cells→Top-Predictive Cells, Traversal→Image Generation (+ About panel + tooltip) - match on/off toggle text size (14px) across all tabs Stale-traversal fix tooling: - pearson_gen0_bymarker.py: detect stale (pre-DDIM-inversion) traversals via α=0-vs-anchor Pearson - regen_stale_traversals.py: force-regen the r<0.5 subset (chunked, h100/h200 pin)
…oupled CFM direction test
Adds FLOW_OT_CELLSTATE_PROGRAM.md scoping flow matching, optimal transport, cell-state
transition mapping, and pathway morphological mapping as extensions of the DiffEx/
SetTransformer stack. flow.py gains an additive `coupling` arg to train_flow ('independent',
unchanged default, vs 'ot' via exact Hungarian assignment) to fix the crossing-path noise
that got the unsupervised CFM shelved 2026-07-03. ot_cfm_test.py is the multi-metric A/B/C
harness (half-axis monotonicity, overshoot ratio, faithfulness-to-real-population distance,
pixel-localization proxy, cross-seed reproducibility) validating it against mean_diff.
…alidation Adds flow_field.py/submit_flow_fields.py to build validated OT-coupled flow fields for every geneKO (~1000) and EBI complex (~98) target, with a flow_advantage_ko_delta_ratio screening signal per target. flow_field_compute.py is the SLURM-shardable compute unit (checkpoint load + Euler integration) factored out so it's importable on compute nodes. ot_cfm_test.py gains: a cross-source gather for pooled complex targets (the complex ranking parquet carries no NTC rows), a visual montage renderer, and run_v5_score — scoring mean_diff/flow-independent/flow-OT trajectories with the actual production classifier (v5 no-mask SetTransformer) instead of the ad-hoc per-test LR, since the LR is itself linear and fit fresh per comparison. Program doc updated with the pooled-complex and low-mAP validation results this uncovered, including the real-vs-generated bag-size sensitivity findings.
gav-sturm
added a commit
that referenced
this pull request
Aug 20, 2026
Squash of the diffex-interpretability branch: DiffAE-based OPSin interpretability viewer, scoring, traversal generation, classifier, and supporting tooling — plus the latest main integrated. Replaces the 391-commit merge with one commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge the diffex-interpretability (opsin) branch into main — the DiffAE-based OPSin interpretability viewer, scoring, traversal generation, and supporting tooling. Includes the latest main (per-exp embedding, batch_process_embeddings fail-loud) merged in cleanly.
Recent viewer work: φ traversal labels, Plausible analytics, hide-NTC, prominent search bars, tab renames (Top-Predictive Cells / Image Generation), stale-traversal detection + regen tooling.