Skip to content

[Bug]: slurm_job_evidence returns HTTP 400 when /var/log or /etc/slurm is not an available evidence root #117

Closed

Description

@0z5a

AI Factory Operations Agent version

main @ 0509477 (chart mosaic-stack 0.0.1)

Kubernetes version

Not cluster-specific; reproduced against the collector script directly (Node.js 24, macOS arm64).

Installation method

Helm chart, Slurm module with the vanilla evidence collector.

Enabled modules and relevant configuration

modules.slurm.enabled=true, modules.slurm.backend=vanilla, and either the default slurmEvidenceCollector.roots or a site-specific list as shown in docs/installation.md:

--set-json "slurmEvidenceCollector.roots=[\"$SLURM_EVIDENCE_ROOT\"]"

Description

slurmJobEvidence() in helm/mosaic-stack/files/slurm-evidence/server.mjs first finds the job's output files, then always searches two hard-coded roots:

slurm_logs: await searchFiles(..., '/var/log', isSlurmPath),
slurm_config: await searchFiles(..., '/etc/slurm'),

With an explicit root, searchFiles() calls hostPath(root) without a catch. If that root is not in EVIDENCE_ROOTS, or does not exist on the node, the whole /slurm/job request fails with HTTP 400. The job files that were already found are discarded. This hits two setups:

  1. Site-specific roots (documented). With slurmEvidenceCollector.roots=["/shared/slurm"], /var/log is not allowlisted, so every slurm_job_evidence call returns 400 path is outside allowed roots: /var/log.
  2. Default roots on a node without /etc/slurm. The DaemonSet mounts the host / and runs on every node. On a node where /etc/slurm does not exist, realpath throws ENOENT, and the call returns 400.

docs/installation.md says: "Missing paths are expected on many clusters; the skill continues with whatever evidence is present." findJobFiles() and searchFiles() with root / already skip missing roots. Only these two call sites do not.

Reproduction steps

Fake host tree in a temp dir, one job output file slurm-4242.out under the first root, then GET /slurm/job?id=4242 with the collector started by createEvidenceServer() (EVIDENCE_TEST_MODE=1):

EVIDENCE_ROOTS /etc/slurm on host status body
7 chart defaults present 200 matched_files 1, slurm_logs 1, slurm_config 1
7 chart defaults absent 400 ENOENT: no such file or directory, realpath '<host>/etc/slurm'
/shared/slurm absent 400 path is outside allowed roots: /var/log

Relevant sanitized logs

{"error": "path is outside allowed roots: /var/log"}

Code of Conduct

  • I agree to follow the AI Factory Operations Agent Code of Conduct.
  • I searched open issues and found no duplicate.

I have a three-line fix with a regression test and can open a PR. It skips an unavailable fixed root and returns [] for that section, so out-of-root paths are still never read.

This issue was drafted with Claude Code (claude-opus-5-5); the results above are from the runs described.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions