Skip to content

Skip unavailable fixed roots in Slurm job evidence - #118

Merged
RohanAdwankar merged 1 commit into
NVIDIA:mainfrom
0z5a:fix/slurm-evidence-fixed-roots
Nov 17, 2026
Merged

RohanAdwankar merged 1 commit into
NVIDIA:mainfrom
0z5a:fix/slurm-evidence-fixed-roots

Conversation

@0z5a

@0z5a 0z5a commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

slurm_job_evidence fails with HTTP 400 whenever /var/log is not an allowlisted root, which is the case for the site-specific slurmEvidenceCollector.roots setup in docs/installation.md. It also fails when /etc/slurm does not exist on the collector's node. In both cases, the job output files the collector already found are discarded.

slurmJobEvidence() searches the fixed roots /var/log and /etc/slurm for the slurm_logs and slurm_config sections. searchFiles() with an explicit root throws when the root is outside EVIDENCE_ROOTS or missing, and nothing caught that error. This change returns [] for such a section, which matches how findJobFiles() and searchFiles('/') already treat missing roots ("Missing paths are expected on many clusters" in docs/installation.md).

The fix does not widen access. A path outside EVIDENCE_ROOTS still yields no data, and explicit /grep or /read requests outside the roots still return 400.

Before and after, GET /slurm/job?id=4242 with one matching job file (Node.js 24, macOS arm64):

EVIDENCE_ROOTS /etc/slurm before after
chart defaults present 200 200 (unchanged)
chart defaults absent 400 (ENOENT) 200, matched_files 1, slurm_config []
/shared/slurm absent 400 (outside allowed roots: /var/log) 200, matched_files 1, slurm_logs [], slurm_config []

New tests in helm/mosaic-stack/tests/slurm-evidence.test.mjs cover both setups. They fail on main (400 instead of 200) and pass with this change. The file runs 4 pass / 0 fail.

Closes #117

Checklist

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

Summary by CodeRabbit

  • Bug Fixes
    • Slurm job evidence requests now continue when searches for system logs or configuration files fail, while still returning available job evidence. Missing search results no longer prevent a successful response.

slurmJobEvidence() always searched /var/log and /etc/slurm. When either
root was outside EVIDENCE_ROOTS (site-specific roots, as documented) or
missing on the node, searchFiles() threw and /slurm/job returned 400,
dropping the job files already found. Return an empty section instead,
as findJobFiles() already does for missing roots. Paths outside the
allowed roots are still never read.

Closes NVIDIA#117

Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
@RohanAdwankar

Copy link
Copy Markdown
Member

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Nov 17, 2026

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

slurmJobEvidence now returns empty arrays when searches of /var/log or /etc/slurm fail. Regression tests cover site-specific evidence roots and a missing /etc/slurm root.

Changes

Slurm job evidence

Layer / File(s) Summary
Handle unavailable evidence roots
helm/mosaic-stack/files/slurm-evidence/server.mjs, helm/mosaic-stack/tests/slurm-evidence.test.mjs
Search failures for /var/log and /etc/slurm now produce empty arrays. Tests verify that /slurm/job returns status 200 with matching job evidence and empty Slurm logs and configuration when those roots are unavailable.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Minimal · up to deaef

The change makes the Slurm job evidence endpoint return matching job files even when /var/log or /etc/slurm is unavailable, instead of failing with HTTP 400. Regression tests cover this behavior and access to out-of-root paths is not widened. No actionable merge risk was identified.

Security architecture risk: Minimal · up to deaef

The change preserves authentication and configured filesystem access restrictions while allowing available job evidence to be returned. No introduced security, ownership, or failure-containment risk was identified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The changed outcome is confined to returning available evidence through the existing collector authority. Although previously failing requests can now return job data, the same bearer-authorized service already exposes reads and searches under its configured roots. No additional filesystem roots, credentials, or write authority are introduced.

Trust Boundaries and Controls

  • observed — Requests must pass bearer-token validation before job collection. The request-provided job ID is format-checked; discovered snippets are resolved through hostPath, and fixed-root searches also invoke hostPath before traversal. Neither rejection handler bypasses authentication or supplies a fallback filesystem location.

Resilience and Maintainability Implications

  • observed — Optional search rejection changes response contents without adding a persistent state transition. Opened file handles retain finally-based closure, and the existing per-server busy gate retains finally-based release after request completion or handled failure.
Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "Skip unavailable fixed roots in Slurm job evidence" matches the main change. The summary says slurmJobEvidence now catches failures from searches of /var/log and /etc/slurm and returns an e…
Linked Issues check Passed Issue #117 (direct) requires that /slurm/job not return HTTP 400 when /var/log or /etc/slurm is missing or outside EVIDENCE_ROOTS, that job files already found are kept, and that paths outside the all…
Out of Scope Changes check Passed The reviewed head changes are limited to the two .catch() guards in slurmJobEvidence() in server.mjs and the new tests in slurm-evidence.test.mjs. The guards directly implement #117. The tests exercis…
Full details: Title check

Explanation

The title "Skip unavailable fixed roots in Slurm job evidence" matches the main change. The summary says slurmJobEvidence now catches failures from searches of /var/log and /etc/slurm and returns an empty array for the matching slurm_logs or slurm_config field. The PR objectives describe the same behavior, and the added tests cover missing-root setups. The title is concise, specific, and free of noise.

Full details: Linked Issues check

Explanation

Issue #117 (direct) requires that /slurm/job not return HTTP 400 when /var/log or /etc/slurm is missing or outside EVIDENCE_ROOTS, that job files already found are kept, and that paths outside the allowed roots are not read. At head, slurmJobEvidence() in helm/mosaic-stack/files/slurm-evidence/server.mjs appends .catch(() => []) to both the slurm_logs search (root /var/log) and the slurm_config search (root /etc/slurm). A missing root or an out-of-roots error therefore yields an empty section, and the job files from findJobFiles() still return. hostPath() and uncheckedHostPath() are unchanged, so the allowlist check still applies. The regression tests in helm/mosaic-stack/tests/slurm-evidence.test.mjs cover a site-specific root (/shared/slurm, where /var/log is not allowed) and a roots list that includes /etc/slurm while it is missing on the host. Both assert HTTP 200, the matched job file, and empty slurm_logs and slurm_config. The chart-default case with /etc/slurm absent is covered by the second test's roots list. Limitation: the whole-PR base-to-head diff was unavailable, so the pre-change behavior is taken from the issue and PR text. The author's test run results were not reproduced.

Full details: Out of Scope Changes check

Explanation

The reviewed head changes are limited to the two .catch() guards in slurmJobEvidence() in server.mjs and the new tests in slurm-evidence.test.mjs. The guards directly implement #117. The tests exercise the same fixed-root behavior and are supporting validation. Explicit /grep and /read paths still call hostPath() without a catch, so out-of-root requests still return HTTP 400, as the PR states. No unrelated code changes were identified in the inspected files. The base-to-head diff tool was unavailable, so the claim of no other changed files relies on the supplied PR summary.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checked the job log trail,
Found its matching file without fail.
When fixed roots could not be found,
Empty lists came back, safe and sound.
The rabbit thumped: “The tests prevail!”

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: be247b93-83ff-47e8-81ac-0523537376f2
Commits

Reviewing files that changed from the base of the PR and between 0509477 and deaef4b.

Files selected for processing: 2

  • helm/mosaic-stack/files/slurm-evidence/server.mjs
  • helm/mosaic-stack/tests/slurm-evidence.test.mjs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@RohanAdwankar
RohanAdwankar merged commit 4d48262 into NVIDIA:main Nov 17, 2026
4 checks passed
@RohanAdwankar

Copy link
Copy Markdown
Member

Thanks for the PR! Feel free to share if you have any other feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: slurm_job_evidence returns HTTP 400 when /var/log or /etc/slurm is not an available evidence root

2 participants