Skip to content

Make the Python unit-test job fail when a test fails - #4236

Merged
ekluzek merged 2 commits into
ESCOMP:b4b-devfrom
Santoshkumarpuppala:python-tests-fail-on-failure
Sep 30, 2026
Merged

ekluzek merged 2 commits into
ESCOMP:b4b-devfrom
Santoshkumarpuppala:python-tests-fail-on-failure

Conversation

@Santoshkumarpuppala

@Santoshkumarpuppala Santoshkumarpuppala commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes

python/ctsm/run_ctsm_py_tests.py dropped the return value of pytest.main(), so ./run_ctsm_py_tests exited 0 on failures and the python-unit-tests job stayed green. The runner now calls sys.exit(pytest.main(pytest_args)).

That alone would turn the job red at collection. The checkout had no cime, so 8 test modules failed to import CIME and pytest stopped before running a test, and the .nc inputs were LFS pointers. The job now checks out with lfs: true and runs bin/git-fleximod update cime, the same pattern as docs-common.yml:24-32 (which updates all submodules with -o). Background in #4230.

Until #4231 is merged, this job fails on test_inputdata_setup_files_gswp3_error (it looks for /glade). Merging #4231 first avoids that on b4b-dev.

Specific notes

Contributors other than yourself, if any:

  • None

CTSM issues resolved or otherwise addressed, if any:

Any user interface changes (namelist or namelist defaults changes)?
No namelist changes. ./run_ctsm_py_tests and make utest now exit non-zero when a test fails.

Testing planned or performed, if any:
Local only: a depth-1 checkout of this branch on b4b-dev 8e3f959, pip venv with ctsm_pylib's test-dependency pins (Python 3.13, pytest 8.3.5), network blocked. No conda, and the workflow itself has not run.

  • Stub test: failing exits 0 before this change and 1 after; passing exits 0. make utest with the failing stub: 0 before, 2 after.
  • The CI command with today's checkout: collected 558 items / 8 errors, exit 0 before, 2 after.
  • bin/git-fleximod update cime as written: cime at 92ae0fd (cime6.5.5), no other submodule checked out.
  • cime plus the LFS .nc inputs (sha256-checked against the pointers): 602 collected, 5 deselected (below), 596 passed, 1 failed, the gswp3 test. With test(subset_data): make unit test work outside NCAR cluster, like its siblings do #4231's line applied, 597 passed, exit 0. Without the LFS inputs, 35 tests fail on pointer files.
  • pylint 3.3.4 (10.00/10) and black 25.1.0 --check on run_ctsm_py_tests.py.
  • Not run: the 5 tests that download from NEON (test_check_neon_time, test_get_neon, test_get_neon_false_site, test_check_neon_listing, test_check_neon_listing_misspelled).

Requirements before merge:

  • I have followed the CTSM contribution guidelines.
  • The code in this PR branch builds with no errors. Not applicable: Python test runner and CI only; CTSM itself was not built.
  • The code in this PR branch runs with no errors. Not ticked: one test fails until test(subset_data): make unit test work outside NCAR cluster, like its siblings do #4231 is merged (noted above). Briefly describe tested configuration(s): macOS arm64, pip venv as above.
  • This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. Which?: (a)
  • I have reviewed relevant parts of the CLM documentation Tech Note or User's Guide to determine if anything needs to be changed or added. If it does, describe: Nothing under doc/ covers the Python test runner or this job.
  • This PR either (a) does not create a need to update the documentation or (b) includes required documentation updates (see guidelines for contributing documentation). Which?: (a)

run_ctsm_py_tests.py called pytest.main() without passing its return
code on, so ./run_ctsm_py_tests exited 0 even when tests failed or
collection was interrupted, and the python-unit-tests job could not go
red.

That job also checked out without the cime submodule and without the
Git LFS test inputs. Test modules that import CIME failed at
collection, pytest stopped before running any test, and the netCDF
inputs were LFS pointer files. Check out with lfs: true and run
bin/git-fleximod update cime, as docs-common.yml does for the docs
build.

See ESCOMP#4230.
@ekluzek ekluzek added enhancement new capability or improved behavior of existing capability testing additions or changes to tests next this should get some attention in the next week or two. Normally each Thursday SE meeting. b4b bit-for-bit devops Development Operations to improve development throughput, E.g., adding GitHub Workflows labels Sep 30, 2026

@ekluzek ekluzek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ekluzek ekluzek linked an issue Sep 30, 2026 that may be closed by this pull request
@ekluzek
ekluzek merged commit b550b26 into ESCOMP:b4b-dev Sep 30, 2026
5 of 6 checks passed
@samsrabin

Copy link
Copy Markdown
Member

@Santoshkumarpuppala Thanks for your contribution! Would you mind emailing ctsm-support@ucar.edu so we can discuss possible future contributions?

@slevis-lmwg slevis-lmwg mentioned this pull request Oct 1, 2026
12 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

b4b bit-for-bit devops Development Operations to improve development throughput, E.g., adding GitHub Workflows enhancement new capability or improved behavior of existing capability next this should get some attention in the next week or two. Normally each Thursday SE meeting. testing additions or changes to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI for python tests maybe broken/missing?

3 participants