Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 84 additions & 130 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ Iris is a Python package for analysing and visualising Earth science data,
built around CF-compliant multi-dimensional arrays ("Cubes").

Subdirectory AGENTS.md files take precedence for their subtrees:
- [`changelog/AGENTS.md`](changelog/AGENTS.md) — documentation on changelog
- [`changelog/AGENTS.md`](changelog/AGENTS.md) — changelog fragments
- [`docs/AGENTS.md`](docs/AGENTS.md) — documentation-specific rules
- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — test-specific rules

- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — testing rules; read
this before writing or running any test

## Project Overview

Expand All @@ -24,87 +24,51 @@ Subdirectory AGENTS.md files take precedence for their subtrees:
| **Test runner** | pytest + pytest-xdist (`-n auto`) |
| **Env management** | nox + conda |


### Main source layout

```
lib/iris/
cube.py # Core Cube / CubeList data structures
coords.py # DimCoord, AuxCoord, CellMeasure, AncillaryVariable
loading.py # File-loading entry points
analysis/ # Collapse, regrid, statistics, calculus
fileformats/ # NetCDF, PP, GRIB, NIMROD format handlers
io/ # I/O registry and URI handling
common/ # Shared metadata, mixins, resolvers
mesh/ # Unstructured grid (UGRID) support
experimental/ # Unstable / in-progress features
tests/ # All tests (unit/, integration/, graphics/)
changelog/ # changelog fragments
docs/src/ # Sphinx documentation source
benchmarks/ # ASV performance benchmarks
requirements/ # Conda environment specs and lock files
lib/iris/ cube.py, coords.py, loading.py; analysis/ (collapse, regrid,
statistics), fileformats/ (NetCDF, PP, GRIB, NIMROD), io/,
common/ (metadata, mixins, resolvers), mesh/ (UGRID),
experimental/, tests/ (unit/, integration/, graphics/)
changelog/ changelog fragments
docs/src/ Sphinx documentation source
benchmarks/ ASV performance benchmarks
requirements/ conda environment specs and lock files
```


## Setup

### Conda environment (recommended)

Always use a conda environment, reuse the iris-dev conda environment if it
already exists but confirm with the user before installing or removing packages.

If a package cannot be installed via conda then you can use pip that is in the
conda environment.

```bash
# Create and activate a development environment
conda env create -f requirements/iris.yml
conda activate iris-dev
pip install --no-build-isolation -e .
```

Alternatively, use lock files for exact reproducibility:
Always work in a conda environment. Reuse `iris-dev` if it exists, but confirm
with the user before installing or removing packages; if a package is not on
conda, use the `pip` inside the conda environment.

```bash
# Preferred - exact reproducibility from a lock file.
conda create -n iris-dev --file requirements/locks/py314-linux-64.lock
# Alternative - solve fresh: conda env create -f requirements/iris.yml
conda activate iris-dev
pip install --no-build-isolation -e .
```


### Environment variables

```bash
# Disable CPU features that can cause SIGILL in some CI environments
export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"

# Point to iris-test-data for tests that need external data files
export OVERRIDE_TEST_DATA_REPOSITORY=/path/to/iris-test-data/test_data

# Override Cartopy cache directory if needed
export CARTOPY_CACHE_DIR=~/.local/share/cartopy
```


## Testing

- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — test-specific


## Code Style

```bash
# Lint
ruff check lib/iris

# Auto-fix safe lint issues
ruff check --fix lib/iris

# Format
ruff format lib/iris

# Check formatting without writing
ruff format --check lib/iris
ruff check lib/iris # lint
ruff check --fix lib/iris # auto-fix safe lint issues
ruff format lib/iris # format
ruff format --check lib/iris # check formatting without writing
```

- **Line length**: 88 characters (Ruff default).
Expand All @@ -121,7 +85,6 @@ ruff format --check lib/iris
- **Imports**: Ruff-managed ordering. No direct `import netCDF4` — always use
`iris.fileformats.netcdf._thread_safe_nc` for thread safety.


## Development Conventions

### Core data model
Expand All @@ -130,108 +93,99 @@ ruff format --check lib/iris
- Coordinates: `DimCoord` (regular), `AuxCoord` (auxiliary), `CellMeasure`,
`AncillaryVariable`.
- Data may be **lazy** (Dask array). Always preserve laziness; never call
`.data`
unnecessarily inside library code.
`.data` unnecessarily inside library code.
- Operations return **new** Cubes (functional style); do not mutate in place.
- All metadata must be **CF-convention** compliant.


### Deprecation

- Use `iris._deprecation.warn_deprecated()` or issue a custom warning class.
- Warning classes live in `iris.warnings` (e.g., `IrisUserWarning`,
`IrisCfWarning`).
`IrisCfWarning`). All `UserWarning` subclasses must ultimately inherit from
`IrisUserWarning`.
- Follow the NEP29 deprecation schedule (same as NumPy).
- All `UserWarning` subclasses must ultimately inherit from `IrisUserWarning`.


### Exception hierarchy

Base class: `iris.exceptions.IrisError`. Common subclasses:
`CoordinateNotFoundError`, `CoordinateCollapseError`, `IgnoreCubeException`.


### Versioning

Version is derived from git tags via `setuptools_scm`. Do not hard-code version
strings.
### Exceptions and versioning

- Base class: `iris.exceptions.IrisError`. Common subclasses:
`CoordinateNotFoundError`, `CoordinateCollapseError`, `IgnoreCubeException`.
- Version is derived from git tags via `setuptools_scm`. Do not hard-code
version strings.

## Changelog

Changelog fragments lives under `chngelog/` and is built with towncrier via
Changelog fragments live under `changelog/` and are built with towncrier via
sphinx. See [`changelog/AGENTS.md`](changelog/AGENTS.md) for full rules.


## Documentation

Documentation lives under `docs/` and is built with Sphinx. See
[`docs/AGENTS.md`](docs/AGENTS.md) for full rules.

Agreed design specs live in `docs/src/developers_guide/specs/` as
`YYYY-MM-DD-<topic>-design.md`. They are **published** MyST Markdown and are
**living documents**, revised as a design evolves. Each declares a citation
prefix (e.g. `merge spec §5.7`) so its sections can be cited from issues and
pull requests, and carries explicit `(prefix-N-N)=` anchors. Markdown is used
*only* for specs; all other documentation is RST.

Implementation plans live in the sibling `docs/src/developers_guide/plans/` as
`YYYY-MM-DD-<topic>.md`. A plan is point-in-time and frozen once its pull
request merges, so plans are tracked in the repository but excluded from the
Sphinx build via `exclude_patterns` in `docs/src/conf.py`.

## Lock-file Maintenance

Adding a dependency means editing `requirements/py*.yml` **and** the matching
`requirements/locks/*.lock`. Repository precedent (PRs #7095, #7100) is a
*minimal* lock edit — insert only the new package URLs and update `input_hash`
— rather than a full re-solve, which bundles dozens of unrelated version bumps
into an otherwise focused PR. Full re-solves are left to the weekly
`refresh-lockfiles` workflow, or run locally with:

```bash
# Regenerate lock files for all supported Python versions
python tools/update_lockfiles.py -o requirements/locks requirements/py*.yml
# Shortcut via Makefile
make lockfiles
make lockfiles # python tools/update_lockfiles.py -o requirements/locks requirements/py*.yml
```


## Pull Request Guidelines

- When creating a pull request a template is provided to ensure all checks are
considered.
- This project is configured to use pre-commit tht will ensure some checks are
performed automatically.
- Keep changes focused; avoid unrelated refactors in the same PR.
- Add or update tests for every change to production code.
- Ensure a whatsnew fragment is added, see
[`changelog/AGENTS.md`](changelog/AGENTS.md)

## Contribution Workflow

- Push development branches to `origin` (the `bjlittle/iris` fork). Raise pull
requests from there against the relevant **feature branch** on `upstream`
(`SciTools/iris`) — never against `upstream/main`. The `upstream` remote is
push-disabled, which enforces this.
- **Attribute agentic work clearly.** Say in the body of every pull request and
issue that it is agentic, and end commit messages with the `Co-Authored-By`
trailer. Never let agentic work read as hand-written.
- Label such pull requests and issues `Agentic` and `Type: Feature Branch`,
plus whichever `Feature: …` label fits the subject.
- Use the pull request template; keep changes focused and avoid unrelated
refactors; add or update tests for every production change; ship a changelog
fragment (see [`changelog/AGENTS.md`](changelog/AGENTS.md)).
- **If in doubt about anything outward-facing** — base branch, labels, whether
to post at all — **ask before pushing.**

## Critical Development Gotchas

1. **xfail_strict Behavior**: Tests marked `@pytest.mark.xfail` that now PASS
become FAILURES -> **remove xfail immediately when bug is fixed**

2. **Pre-commit Auto-fixes**: Hooks may auto-fix ISC001/COM812 conflicts ->
re-stage files: `git add . && git commit`

3. **Lockfile Rebuilds**: Updating `requirements/locks/*.lock` files triggers
slow conda environment rebuilds - only update when deps genuinely change

4. **Lazy Data Pitfalls**: Check `cube.has_lazy_data()` before operations; use
`cube.lazy_data()` when appropriate; `cube.data` materializes arrays

5. **Coordinate Scope**: Not all coordinates span all dimensions -> use
`cube.coords(dimensions=dim_indices)` to query specific axes

6. **Dask/NumPy Compatibility**: Some operations fail on lazy (Dask) arrays
-> materialize with `cube.compute()` or check dask support before use

7. **Optional Dependencies**: cartopy, matplotlib lazy-imported -> handle
ImportError gracefully for optional visualization features

8. **CF-Metadata Validation**: Custom coordinates must comply with CF
standards via `cube.metadata`

9. **Cube Immutability**: Operations return new cubes; don't modify in-place
(`cube.data[...] = x` won't affect the cube)

10. **Environment Reproducibility**: Set CPU features for consistent results:
`export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"`


## Getting Help

- **Discussions**:
[GitHub Discussions](https://github.com/SciTools/iris/discussions)
- **Contributing Guide**: [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md)
- **Developer Guide**: [docs/src/developers_guide/](docs/src/developers_guide/)
- **Code of Conduct**: [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)

1. **xfail_strict**: `@pytest.mark.xfail` tests that now PASS become FAILURES —
remove the xfail immediately when the bug is fixed.
2. **Pre-commit auto-fixes**: hooks may auto-fix ISC001/COM812 conflicts —
re-stage with `git add . && git commit`.
3. **Lock-file rebuilds**: lock changes trigger slow conda environment rebuilds
— only update when deps genuinely change.
4. **Lazy data**: check `cube.has_lazy_data()`; use `cube.lazy_data()` where
appropriate; `cube.data` materialises arrays.
5. **Coordinate scope**: not all coords span all dims — use
`cube.coords(dimensions=dim_indices)` to query specific axes.
6. **Dask/NumPy compatibility**: some operations fail on lazy arrays —
materialise with `cube.compute()` or check dask support first.
7. **Optional dependencies**: cartopy and matplotlib are lazy-imported — handle
`ImportError` gracefully.
8. **CF-metadata validation**: custom coordinates must comply with CF standards
via `cube.metadata`.
9. **Cube immutability**: operations return new cubes; `cube.data[...] = x`
will not affect the cube.
10. **Environment reproducibility**: set `NPY_DISABLE_CPU_FEATURES` as above for
consistent results.

## ⚠️ Meta-Instruction: Auto-Update Rule
- **Trigger**: You MUST automatically update this `AGENTS.md` file if a new
Expand Down
3 changes: 3 additions & 0 deletions changelog/7274.dependency.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
:user:`bjlittle` added `myst-parser <https://myst-parser.readthedocs.io/>`_ as a
documentation dependency, so that developer design specs may be authored in
MyST Markdown. All other documentation remains reStructuredText.
6 changes: 6 additions & 0 deletions changelog/7274.doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
:user:`bjlittle` added a design spec for a programme of work to make
:meth:`~iris.cube.CubeList.merge` and :meth:`~iris.cube.CubeList.concatenate`
more robust, efficient, configurable, extensible, lazy and lenient, along with a
new :ref:`developers_guide_specs` section of the Developers Guide to publish it
and any future specs. (:issue:`2761`, :issue:`5375`, :issue:`6790`,
:issue:`7063`, :issue:`7241`)
17 changes: 17 additions & 0 deletions docs/src/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ def _dotv(version):
extensions = [
"matplotlib.sphinxext.mathmpl",
"matplotlib.sphinxext.plot_directive",
"myst_parser",
"readingtime",
"readingtime_validator",
"sphinx.ext.autodoc",
Expand Down Expand Up @@ -222,6 +223,15 @@ def _dotv(version):
)
)

# -- myst-parser extension ----------------------------------------------------
# See https://myst-parser.readthedocs.io/en/latest/configuration.html
# Markdown is reserved for developer design specs, see
# developers_guide/specs/index. All other documentation remains
# reStructuredText.
myst_enable_extensions = [
"colon_fence",
]

# -- Napoleon extension -------------------------------------------------------
# See https://sphinxcontrib-napoleon.readthedocs.io/en/latest/sphinxcontrib.napoleon.html
napoleon_google_docstring = True
Expand Down Expand Up @@ -458,6 +468,13 @@ def _dotv(version):
# list of sources to exclude from the build.
exclude_patterns = [
"**/highlights.rst",
# Implementation plans are tracked in the repository but not published;
# see developers_guide/specs/index.
"developers_guide/plans/**",
# myst-parser makes ".md" a source suffix, which would otherwise pull the
# README of each of these non-document directories into the build.
"_build/**",
"_static/**",
]

# -- sphinx-gallery config ----------------------------------------------------
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ If you are new to using GitHub we recommend reading the
contributing_changes
github_app
release
specs/index


.. toctree::
Expand Down
10 changes: 10 additions & 0 deletions docs/src/developers_guide/plans/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Implementation plans

Implementation plans for the design specs in [`../specs/`](../specs).

Unlike a spec, a plan is **not** a living document. It records the sequence of
steps agreed at the time it was written, and is frozen once the work it
describes has merged. Plans are tracked in the repository for provenance, but
are excluded from the Sphinx build via `exclude_patterns` in `docs/src/conf.py`.

Naming: `YYYY-MM-DD-<topic>.md`.
Loading
Loading