Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 25 additions & 5 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ Acceptance (met)
Composition/space/volume coverage maps; energy and per-element force distributions
with outlier flags; extrapolation grade; redundancy report.

### 🚧 Chunk 3 — Validation (`validation/`) *(parity + quality report done)*
### 🚧 Chunk 3 — Validation (`validation/`) *(parity, NVE probe, spectra, learning curves done)*

- ✅ `validation/` — parity metrics (RMSE/MAE for E [total + per-atom], forces
[overall + per-element], stress/dipole/polarizability) against reference
Expand All @@ -240,15 +240,35 @@ with outlier flags; extrapolation grade; redundancy report.
training holdout `model/valid.xyz` (never trained on), overridable via
`test_file`. Slurm executor maps it to `traincraft-mlip.sif`. Failed frames
skip-and-record (same philosophy as labeling).
- 🔜 Learning curves (error vs dataset size), NVE/MD stability, EOS/phonons.
- 🔜 **IR and Raman spectra reconstruction** from MLIP-driven MD vs
DFT/experiment.
- ✅ NVE stability probe (`[validation.stability]`): Langevin equilibration →
Velocity-Verlet production with an explosion guard; `nve_aborted`,
`nve_drift_mev_per_atom_ps` etc. fold into the quality report as
thresholdable metrics. An exploding model is a *result*, not a crash.
- ✅ **Spectra reconstruction from MLIP-driven MD** (`[validation.spectra]`,
TCF route, clean-room from published theory): VDOS (mass-weighted,
per-element), IR (dipole-derivative ACF), Raman (iso/aniso invariants →
VV/VH/unpolarized/depolarization, laser + Bose prefactors, selectable QCF),
and SFG (Morita–Hynes cross-TCF, signed Im χ⁽²⁾). Dielectric model loaded
via `MaceCalc.model_type`; artifacts under `validation/spectra/`; optional
reference files → `<name>_spectral_error` metrics (the AL convergence hook).
SERS = documented recipe (chemical enhancement via adsorbed-system training
data + optional `enhancement_file` EM profile; TrainCraft computes no EM
fields).
- ✅ Learning curves: `traincraft learning-curve` retrains on nested random
subsets against one fixed holdout → `validation/learning_curve.json`
(diagnostic command, not a stage).
- 🔜 EOS/phonons; harmonic (Hessian-based) spectra path.

Acceptance
- ✅ A model is validated per property with recorded pass/fail vs thresholds
(covered by `test_validation`, EMT standing in for the MLIP).
- ✅ The physics probes are exercised end-to-end scheduler-free
(`test_spectra` against synthetic signals with known answers;
`test_dynamics`/`test_validation_science`/`test_learning_curve` with EMT and
an analytic dielectric stand-in).
- 🔜 A fine-tuned model trains on a seed set and produces a quality report
covering every property, including reconstructed IR/Raman spectra.
covering every property, including reconstructed IR/Raman spectra
(needs real MACE + machine access).

---

Expand Down
153 changes: 153 additions & 0 deletions docs/how-to/validation-science.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
# Validation Science: NVE Probe, Spectra, Learning Curves

Parity metrics (RMSE/MAE against reference labels) tell you the model
interpolates; they don't tell you it will **survive dynamics** or **reproduce
observables**. Three tools close that gap, and all of them write into the same
`validation/quality_report.json`, so their numbers threshold exactly like
parity metrics — including as convergence criteria for the future
active-learning loop.

## NVE stability probe

Add a `[validation.stability]` section and the `validate` stage runs short MD
with the model under test: Langevin equilibration to `temperature`, then
**NVE (Velocity-Verlet) production** — no thermostat to hide a broken
potential-energy surface.

```toml
[validation.stability]
steps = 2000 # NVE production steps
timestep = 1.0 # fs
temperature = 300.0
equilibration_steps = 500
n_structures = 1 # probe the first N reference frames; worst case reported

[validation]
thresholds = { nve_aborted = 0.0, nve_drift_mev_per_atom_ps = 1.0 }
```

Metrics folded into the report (thresholdable, `value <= limit`):

| Metric | Meaning |
|---|---|
| `nve_aborted` | `1.0` if any probe trajectory exploded (force/energy guard) — threshold at `0.0` |
| `nve_drift_mev_per_atom_ps` | \|slope\| of total energy vs time — the standard NVE health number |
| `nve_energy_std_mev_per_atom` | detrended energy fluctuation |
| `nve_temp_mean` | mean production temperature (K) |

An exploding trajectory is a **result** (`nve_aborted = 1`), not a crash: the
run completes and the report shows why the model failed.

## Spectra from MLIP-driven MD (VDOS, IR, Raman, SFG)

A `[validation.spectra]` section reconstructs vibrational spectra from one
production trajectory via time-correlation functions: velocities → VDOS,
dipole → IR, polarizability → Raman (VV/VH/depolarization), and the
dipole–polarizability cross-correlation → the resonant SFG response.

```toml
[validation.spectra]
which = ["vdos", "ir", "raman"] # any of vdos | ir | raman | sfg
temperature = 300.0
timestep = 0.5 # fs — resolve the fastest mode (H stretches!)
equilibration_steps = 2000
steps = 20000 # production; longer = sharper lines
laser_wavelength = 532.0 # nm, Raman prefactor
qcf = "harmonic" # quantum correction: harmonic|standard|schofield|none

# the dielectric model evaluating dipole/polarizability along the trajectory
[validation.spectra.calculator]
type = "mace"
model_path = "runs/dielectric/model/mace_model.model"
model_type = "DipoleMACE" # MACE calculator class for dielectric models

# optional: reference spectra -> <name>_spectral_error metrics
[validation.spectra.reference]
ir = "experiment_ir.dat" # two columns: freq_cm1 intensity
raman = "dft_raman.dat"
```

What lands in `runs/<name>/validation/spectra/`: one two-column `.dat` per
spectrum (`vdos.dat` + per-element `vdos_<El>.dat`, `ir.dat`, `raman_vv.dat`,
`raman_vh.dat`, `raman_unpolarized.dat`, `raman_depolarization.dat`,
`raman_iso.dat`, `raman_aniso.dat`, `sfg_re.dat`, `sfg_im.dat`,
`sfg_abs2.dat`), plus `spectra.json` (parameters, peak tables, metrics). Set
`save_series = true` to also keep the raw time series (`series.npz`).

Practical guidance:

- **Which model computes what.** The model under test drives the MD (energy/
forces). Dipole/polarizability come from `[validation.spectra.calculator]` —
typically a separately trained dielectric MACE (`AtomicDipolesMACE` /
`AtomicDielectricMACE` heads, loaded with `model_type`). If omitted, the
model under test is asked for those properties directly (works only for
models with dielectric heads). VDOS needs neither.
- **MD length sets resolution.** The frequency resolution is roughly
`1 / (correlation window)`: 20 000 steps × 0.5 fs → 10 ps → ~3 cm⁻¹ grid
before broadening. Aim for at least ~10× the period of your slowest mode.
- **Timestep sets the ceiling.** The Nyquist limit is `1/(2·dt)`: 0.5 fs
covers ~33 000 cm⁻¹ — fine; 4 fs caps at ~4 200 cm⁻¹ — too close to O–H
stretches for comfort.
- **`qcf`** applies a quantum correction on top of the derivative-ACF spectra
(which already correspond to the *harmonic* correction, hence
`harmonic` = default and a no-op). Change it only if you know which
convention your reference uses.
- **SFG** is surface-specific: pick the χ⁽²⁾ component with `sfg_component`
(default `"xxz"` = the ssp geometry with the surface normal along z) and
keep your slab's normal along z. The sign of `sfg_im.dat` carries the net
orientation of each mode.
- **References** are compared after Gaussian broadening (`broadening`, cm⁻¹)
and normalization on a common grid over `freq_range`; each contributes
`<name>_spectral_error = 1 − cosine similarity` (0 = same shape) — this is
the "spectral error" convergence criterion. Stick spectra (e.g. harmonic
DFT output) work as references: both sides go through the same broadening.

Slurm note: the spectra MD runs *inside* the `validate` job
(`traincraft-mlip.sif`, 1 GPU by default) — size `stages.validate.time`
accordingly.

## SERS (surface-enhanced Raman): what is and isn't computable

SERS enhancement has two parts, and it's important to be honest about which
one an MD/TCF tool can give you:

- **Chemical enhancement** (adsorption-induced changes of the polarizability)
is captured *implicitly*: train the dielectric model on DFT
dipoles/polarizabilities of the **molecule + metal cluster/slab** system,
then run `raman` spectra of the adsorbed geometry. The difference to the
gas-phase spectrum *is* the chemical enhancement.
- **Electromagnetic enhancement** (the plasmonic 10⁴–10⁸ factor) is classical
electrodynamics of the nanostructure (Mie/BEM/FDTD) — TrainCraft does not
compute it. If you have an enhancement profile from such a tool (or an
empirical one), point `enhancement_file` at a two-column
`freq_cm1 enhancement` file: it is interpolated onto the Raman grid and
multiplied onto the VV/VH/unpolarized spectra, written as
`raman_*_enhanced.dat`.

```toml
[validation.spectra]
which = ["raman"]
enhancement_file = "em_enhancement.dat" # from your EM solver, or empirical
```

## Learning curve: error vs training-set size

How data-efficient is your setup — and is more data still buying accuracy?

```bash
pixi run -e mace traincraft learning-curve my_run.toml --fractions 0.125,0.25,0.5,1.0
```

The command retrains the `[training]` model on **nested** random subsets of
the training frames (smaller ⊂ larger — one shuffle, seeded by `[run].seed`)
and validates every point against **one fixed holdout** (split off once with
the trainer's `valid_fraction`; never trained on at any fraction). Each point
validates its own trained model. Results:
`runs/<name>/validation/learning_curve.json` with
`points: [{fraction, n_train, metrics{...}}, ...]`, plus per-point model dirs
under `validation/learning_curve/`.

It retrains once per fraction — run it where training runs (a GPU allocation);
it is a diagnostic command, not a pipeline stage, so it never touches the
stage cache. A curve still steep at 100% of your data says "label more"; a
flat one says the model, not the data, is the limit.
25 changes: 24 additions & 1 deletion docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,34 @@ stage 'train': 42 frames
The `train` stage writes `runs/<name>/model/<name>.model` and a `manifest.json`.
The `validate` stage writes `runs/<name>/validation/quality_report.json` —
parity metrics + threshold checks for the trained model (see
[`[validation]`](config.md#validation)).
[`[validation]`](config.md#validation)), plus the NVE stability metrics and
`validation/spectra/` artifacts when configured (see
[Validation Science](../how-to/validation-science.md)).
See [Tutorial 10 · Training](../tutorials/10-training.md).

---

## `traincraft learning-curve`

Validation error vs training-set size: retrain the `[training]` model on
nested random subsets and validate every point against one fixed holdout.

```bash
pixi run -e mace traincraft learning-curve CONFIG [--fractions 0.125,0.25,0.5,1.0]
```

| Option | Description |
|---|---|
| `--fractions` | Comma-separated training-set fractions in `(0, 1]` |

Prints a table (`n_train` vs the headline metrics) and writes
`runs/<name>/validation/learning_curve.json`. It retrains once per fraction —
run it where training runs (e.g. inside a GPU allocation). A diagnostic
command, not a pipeline stage: no events, no cache keys. See
[Validation Science](../how-to/validation-science.md#learning-curve-error-vs-training-set-size).

---

## `traincraft submit`

Render (and, unless `--dry-run`, submit) the workflow as dependency-chained
Expand Down
52 changes: 50 additions & 2 deletions docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,7 @@ Same as `tblite`. Uses the `xtb-python` binding instead.
| `model_path` | `str \| null` | `null` | Path to a local `.model` checkpoint (overrides `model`) |
| `device` | `str` | `"cpu"` | `"cpu"` or `"cuda"` |
| `default_dtype` | `str` | `"float32"` | `"float32"` or `"float64"` |
| `model_type` | `str \| null` | `null` | MACE calculator class for dielectric models trained with dipole/polarizability heads (e.g. `"DipoleMACE"`, `"EnergyDipoleMACE"`). Requires `model_path`; verify the name against your installed MACE version |

---

Expand Down Expand Up @@ -618,11 +619,15 @@ what the future active-learning loop will use as its convergence criterion.
| `properties` | list \| null | `null` | Which properties to check (`energy`/`forces`/`stress`/`dipole`/`polarizability`). Default: everything the reference frames carry |
| `max_frames` | `int \| null` | `null` | Cap the number of frames compared (quick checks) |
| `thresholds` | `dict` | `{}` | Metric name → maximum allowed value; the report records each check and an overall `passed`. Unknown metric names fail fast |
| `stability` | table \| null | `null` | NVE stability probe — see [`[validation.stability]`](#validationstability) |
| `spectra` | table \| null | `null` | MD spectra reconstruction — see [`[validation.spectra]`](#validationspectra) |

Metric names for `thresholds`: `energy_rmse`, `energy_mae`,
`energy_rmse_per_atom`, `energy_mae_per_atom` (eV, eV/atom), `forces_rmse`,
`forces_mae` (eV/Å), and `<property>_rmse`/`<property>_mae` for
`stress`/`dipole`/`polarizability` when present.
`forces_mae` (eV/Å), `<property>_rmse`/`<property>_mae` for
`stress`/`dipole`/`polarizability` when present, the `nve_*` metrics when
`stability` is set, and `spectra_md_aborted` / `<spectrum>_spectral_error`
when `spectra` is set.

```toml
[validation]
Expand All @@ -633,3 +638,46 @@ thresholds = { forces_rmse = 0.05, energy_mae_per_atom = 0.005 }
A frame whose energy/forces prediction fails is skipped and recorded (same
philosophy as `[labeling]`'s `on_error="skip"`); optional properties the model
cannot produce are dropped per property without voiding the frame.

### `[validation.stability]`

NVE stability probe: Langevin equilibration then Velocity-Verlet (NVE)
production with the model under test; energy-conservation metrics
(`nve_aborted`, `nve_drift_mev_per_atom_ps`, `nve_energy_std_mev_per_atom`,
`nve_temp_mean`) join the quality report. An exploding trajectory sets
`nve_aborted = 1` instead of failing the stage. See
[Validation Science](../how-to/validation-science.md).

| Field | Type | Default | Description |
|---|---|---|---|
| `steps` | `int` | `2000` | NVE production steps |
| `timestep` | `float` | `1.0` | fs |
| `temperature` | `float` | `300.0` | Init + equilibration target (K) |
| `equilibration_steps` | `int` | `500` | Langevin steps before production |
| `n_structures` | `int` | `1` | Probe the first N reference frames; worst case reported |

### `[validation.spectra]`

Vibrational spectra reconstructed from one MLIP-MD trajectory via
time-correlation functions: velocities → VDOS, dipole → IR, polarizability →
Raman (VV/VH/depolarization), dipole×polarizability → SFG. Artifacts land in
`validation/spectra/` (two-column `.dat` per spectrum + `spectra.json`). See
[Validation Science](../how-to/validation-science.md) for the physics knobs.

| Field | Type | Default | Description |
|---|---|---|---|
| `which` | `list` | `["vdos", "ir", "raman"]` | Any of `vdos`/`ir`/`raman`/`sfg` |
| `temperature` | `float` | `300.0` | K |
| `timestep` | `float` | `0.5` | fs (sets the Nyquist ceiling) |
| `equilibration_steps` | `int` | `2000` | Langevin steps before production |
| `steps` | `int` | `20000` | NVE production steps (sets resolution) |
| `sample_every` | `int` | `1` | Record every Nth step |
| `calculator` | calculator config \| null | `null` | Dielectric model evaluating dipole/polarizability along the trajectory (e.g. `mace` with `model_type`). Default: the model under test |
| `qcf` | `str` | `"harmonic"` | Quantum correction: `harmonic`/`standard`/`schofield`/`none` |
| `laser_wavelength` | `float` | `532.0` | nm — Raman Stokes prefactor |
| `sfg_component` | `str` | `"xxz"` | χ⁽²⁾ indices (`"xxz"` = ssp, surface normal ∥ z) |
| `freq_range` | `[lo, hi]` | `[0.0, 4000.0]` | cm⁻¹ — written + compared range |
| `broadening` | `float` | `10.0` | cm⁻¹ Gaussian width for the similarity metric |
| `reference` | `dict` | `{}` | Spectrum name → two-column file (`freq_cm1 intensity`); each adds a `<name>_spectral_error` metric |
| `enhancement_file` | path \| null | `null` | Frequency-dependent EM enhancement profile multiplied onto Raman spectra (the SERS recipe) |
| `save_series` | `bool` | `false` | Also keep the raw time series as `spectra/series.npz` |
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ nav:
- Write a Custom Builder: how-to/custom-builder.md
- Write a Custom Sampler: how-to/custom-sampler.md
- Train & Use Your Own MACE Model: how-to/own-mace-model.md
- Validation Science (NVE, Spectra, Learning Curves): how-to/validation-science.md
- Manage Datasets: how-to/datasets.md
- Run on HPC (Slurm + Apptainer): how-to/hpc.md
- Serve Runs to Agents & UIs: how-to/agent-service.md
Expand Down
21 changes: 20 additions & 1 deletion skills/traincraft/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,11 @@ Map the user's words to config keys (run `traincraft plugins` for the live list)
optional `thresholds` → `passed`). Defaults to the model `[training]` just
produced and its holdout `valid.xyz`. **Read `quality_report.json` and report
the metrics + pass/fail to the user** — never claim a model is good without it.
Physics probes are opt-in sub-sections: `[validation.stability]` (NVE probe →
`nve_*` metrics; threshold `nve_aborted = 0.0`) and `[validation.spectra]`
(MD-reconstructed VDOS/IR/Raman/SFG → `validation/spectra/*.dat` +
`<name>_spectral_error` vs reference files). See
`docs/how-to/validation-science.md`.

## ⚠️ Gotchas (learned the hard way)

Expand All @@ -126,6 +131,14 @@ Map the user's words to config keys (run `traincraft plugins` for the live list)
variants (stochastic builders get derived seeds); `smiles` with
`n_conformers > 1` and `file` with `index = ":"` also yield several seeds.
Every seed is sampled and the candidates pooled before selection.
- **IR/Raman/SFG spectra need a dielectric model.** `[validation.spectra]` with
`ir`/`raman`/`sfg` requires dipole/polarizability predictions — point
`[validation.spectra.calculator]` at a trained dielectric MACE (`model_path`
+ `model_type = "DipoleMACE"` etc.). `vdos` works with any model. The MD runs
inside the `validate` job: size `stages.validate.time` for it.
- **`traincraft learning-curve` retrains once per fraction** — never launch it
casually; run it where `stage train` runs (a GPU allocation), with the `mace`
env.

## 📂 Outputs (under `runs/<run.name>/`)

Expand All @@ -143,7 +156,12 @@ Map the user's words to config keys (run `traincraft plugins` for the live list)
- `model/<name>.model` (+ `manifest.json`, `valid.xyz` holdout) — a
trained/fine-tuned MACE model.
- `validation/quality_report.json` — parity metrics, threshold checks, and the
overall `passed` verdict for the validated model.
overall `passed` verdict for the validated model (plus `nve_*` and spectral
metrics when those probes are configured).
- `validation/spectra/*.dat` + `spectra.json` — reconstructed spectra
(two-column, plottable as-is) and their peak tables.
- `validation/learning_curve.json` — error vs training-set size from
`traincraft learning-curve`.

## 🖼 Visualising (headless-VM friendly)

Expand All @@ -169,6 +187,7 @@ traincraft validate <file> # validate + show resolved stages
pixi run -e <env> traincraft run <file> # run the pipeline
pixi run -e <env> traincraft stage <name> <file> # one stage standalone
# (geometry|sample|select|label|dataset|train|validate)
pixi run -e mace traincraft learning-curve <file> [--fractions 0.25,0.5,1.0] # error vs data size
traincraft runs [outdir] [--json] # list all runs + status (default outdir: runs/)
traincraft status <run-dir> [--json] # per-stage status of one run (live label progress)
```
Expand Down
Loading
Loading