Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .agents/skills/python-ort-tests/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
name: python-ort-tests
description: Write tests for the python-ort library (a pydantic port of the Kotlin OSS Review Toolkit / ORT). Use this whenever creating or updating tests under the tests/ folder, especially when porting behavior from the upstream Kotlin ORT test suite.
---

# Writing tests for python-ort

`python-ort` is a Python (pydantic v2) port of the Kotlin **OSS Review Toolkit (ORT)**
model. When adding tests for a model class, mirror the corresponding upstream Kotlin
test so behavior stays in parity.

## Hard rules

1. **Never use the `assert` statement.** Always use pytest functions instead:
- Use `pytest.fail("message")` to report unexpected values (this is the dominant
pattern in the existing suite — see `tests/test_vulnerability_reference.py`).
- Use `with pytest.raises(ValidationError):` (or `pytest.raises(...)`) for expected
exceptions. For pydantic models, validation failures surface as
`pydantic.ValidationError`; when you need to inspect the message, catch it and
check substrings of `str(exc)`.
- Use `@pytest.mark.parametrize` for table-style cases.
2. **Follow the existing patterns in `tests/`.** Group related cases in `Test...`
classes with `test_...` methods, give every test a one-line docstring, and import
models by their full module path, e.g.
`from ort.models.licenses.license_classifications import LicenseClassifications`.
3. **Place tests in the `tests/` folder** named `test_<thing>.py`. YAML-backed tests
load fixtures from `tests/data/` via `tests.utils.load_yaml_config.load_yaml_config`.

## Basing tests on the upstream Kotlin ORT suite

The reference Kotlin checkout lives **outside** this project at:

```
https://github.com/oss-review-toolkit/ort
```

- Main sources: `<ort>/model/src/main/kotlin/...`
- Tests (kotest `WordSpec`): `<ort>/model/src/test/kotlin/...`

Always read the upstream **main source** too, not just the test — the Python model
may be missing behavior (validators, derived properties, helper methods) that the
Kotlin tests exercise. If so, port that behavior onto the pydantic model first, then
write the tests. (Example: `LicenseClassifications` needed its consistency validator,
`licenses_by_category` / `categories_by_license` / `category_names` properties,
`merge()`, and `__getitem__` ported before the tests were meaningful.)

### Translating kotest -> pytest

| kotest (Kotlin) | python-ort (pytest) |
|----------------------------------------------|-------------------------------------------------------|
| `"feature" should { "does X" { ... } }` | `class TestFeature:` with `def test_does_x(self):` |
| `shouldThrow<IllegalArgumentException> {}` | `pytest.raises(ValidationError)` / catch + `str(exc)` |
| `x shouldBe y` | `if x != y: pytest.fail(...)` |
| `msg shouldContain "s"` / `shouldNotContain` | `if "s" not in msg:` / `if "s" in msg:` + `pytest.fail` |
| `coll should containExactlyInAnyOrder(...)` | helper comparing elements ignoring order |
| `x should beNull()` / `shouldNotBeNull` | `if x is not None:` / `if x is None:` + `pytest.fail` |
| `coll should beEmpty()` | `if coll != <empty>: pytest.fail(...)` |

Note Kotlin `IllegalArgumentException`/`require(...)` maps to a `ValueError` raised in
a pydantic `@model_validator`, which pydantic wraps in `ValidationError`.

## Running the tests

```
uv run pytest tests/test_<thing>.py -v
uv run ruff check tests/test_<thing>.py
```

There is one pre-existing, unrelated failure in
`tests/test_repo_config_curations.py::test_curations_yml_package_curations`; ignore it.

## Reference example

`tests/test_license_classifications.py` is a complete worked example of porting
`<ort>/model/src/test/kotlin/licenses/LicenseClassificationsTest.kt`, including a
`classify(...)` builder helper and a `contains_exactly_in_any_order(...)` helper.
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true
python-version: ${{ matrix.python-version }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/commit_checks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- uses: webiny/action-conventional-commits@faccb24fc2550dd15c0390d944379d2d8ed9690e # v1.3.1
- uses: webiny/action-conventional-commits@7f91b1595ca1951cdb671ddc9f07a49081ec5b69 # v1.4.2

typecheck:
needs: commitlint
Expand All @@ -38,7 +38,7 @@ jobs:
- name: Check out code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true
activate-environment: "true"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true
python-version: "3.10" # Minimal supported Python
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/scorecard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,6 @@ jobs:
# Upload the results to GitHub's code scanning dashboard (optional).
# Commenting out will disable upload of results to your repo's Code Scanning dashboard
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@v3
uses: github/codeql-action/upload-sarif@v4
with:
sarif_file: results.sarif
19 changes: 11 additions & 8 deletions .github/workflows/testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
runs-on: ubuntu-24.04
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']

steps:
- name: Harden Runner
Expand All @@ -18,16 +18,19 @@ jobs:

- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
- uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true

- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405
with:
python-version: ${{ matrix.python-version }}

- name: Install the project
run: uv sync --locked --all-extras --dev
activate-environment: true

- name: Sync dependencies
run: |
uv lock
uv sync \
--locked \
--all-extras \
--dev
Comment on lines +27 to +33

- name: Test with python ${{ matrix.python-version }}
run: uv run --frozen pytest
33 changes: 33 additions & 0 deletions examples/licenses_classification.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# SPDX-FileCopyrightText: 2025 Helio Chissini de Castro <heliocastro@gmail.com>
# SPDX-License-Identifier: MIT
#
import logging
import sys
from pathlib import Path

import click
from pydantic import ValidationError
from rich.pretty import pprint

from ort import ort_yaml_load
from ort.models import LicenseClassifications

logger = logging.getLogger()


@click.command()
@click.argument("datafile")
def main(datafile: str) -> None:
try:
with Path(datafile).open() as fd:
data = ort_yaml_load(fd)
parsed = LicenseClassifications(**data)
pprint(parsed)
except ValidationError as e:
logger.error("Validation error while parsing the ORT result:")
Comment thread
heliocastro marked this conversation as resolved.
Comment thread
heliocastro marked this conversation as resolved.
pprint(e.errors())
sys.exit(1)


if __name__ == "__main__":
main()
1 change: 1 addition & 0 deletions examples/ort_result.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ def main(
else:
pprint(parsed)
except ValidationError as e:
logger.error("Validation error while parsing the ORT result:")
pprint(e.errors())
sys.exit(1)

Expand Down
1 change: 1 addition & 0 deletions examples/repo_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ def main(datafile: str) -> None:
parsed = RepositoryConfiguration(**data)
pprint(parsed)
except ValidationError as e:
logger.error("Validation error while parsing the ORT result:")
Comment thread
heliocastro marked this conversation as resolved.
Comment thread
heliocastro marked this conversation as resolved.
pprint(e.errors())
sys.exit(1)

Expand Down
Loading
Loading