Skip to content

The "Nightly Hypothesis tests failed" auto-issue has never been filed: all 30 retained nightly failures died in the report step #4273

Description

@glaziermag

.github/workflows/hypothesis.yaml ends with a step that should auto-file an issue titled "Nightly Hypothesis tests failed" (label topic-hypothesis) whenever a scheduled run fails. It has never fired: no issue with that title has ever existed, and all 30 retained failed nightlies (2025-07-14 → 2026-08-03) died in the report step. Quickest look, on the most recent failed nightly:

$ gh run view 30774362464 -R zarr-developers/zarr-python --log | grep -a -B1 -A1 "No such file"
##[group]issue body
cat: '': No such file or directory
##[error]Process completed with exit code 1

Three independent blockers at ce10c0b9:

  1. Nothing writes the log the action reads. hypothesis.yaml:114 expects output-3.12-log.jsonl, but run-hypothesis (pyproject.toml:218-221) never passes --report-log and pytest-reportlog isn't installed. The step was adapted from xarray, where the flag lives in the pytest invocation (xarray hypothesis.yaml:122); broken here since Add hypothesis property tests #1746.
  2. The token can't create issues. permissions: contents: read since Limit privilege escalation and token theft in workflows #3742, with no job-level issues: write — unlike links.yml/issue-metrics.yml, whose auto-issues do land.
  3. The topic-hypothesis label doesn't exist, and gh errors on unknown labels rather than creating them.

The dropped failures were real counterexamples: run 30774362464 (test_zarr_hierarchy[memory], "Hypothesis found 2 distinct failures" — an IndexError in the sharding partial-encode path plus an oindex shape mismatch) and run 29461161693 (model-vs-store listing divergence). The nightly has been green since Aug 4, so they left no trace.

Fix demonstrated on a fork at the same SHA (diff): current config → GraphQL: Resource not accessible by integration (run 32303084126); with one commitpytest-reportlog in the test group, --report-log via PYTEST_ADDOPTS, job-level permissions: {contents: read, issues: write}, create the label — the configured issue is auto-filed: run 32303086784glaziermag#1.

If the alarm isn't wanted, deleting the step is the other consistent resolution — given what it dropped, I'd fix it.

Disclosure: researched, verified, and written by an AI agent (Claude) operating this account under its owner's direction; all evidence was re-derived from this repo's own runs and files before filing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions