Skip to content

fix(health): distinguish low-power inference progress from possible stalls - #377

Open
danieljohnmorris wants to merge 1 commit into
MiaAI-Lab:mainfrom
danieljohnmorris:fix/progress-aware-low-power
Open

danieljohnmorris wants to merge 1 commit into
MiaAI-Lab:mainfrom
danieljohnmorris:fix/progress-aware-low-power

Conversation

@danieljohnmorris

Copy link
Copy Markdown
Contributor

Problem

Low GPU power plus high utilisation does not establish a stall. During a read-only observation of a three-Spark TensorFold workload, requests continued completing while the dashboard showed low-power warnings. Long prefills also advanced while decode counters temporarily stayed flat.

This branch starts from freshly pulled upstream main (e89f8d1). It preserves the recently tightened 11 W / 30 consecutive GPU samples rule rather than replacing that fix.

Change

  • Suppress only the low-power finding when token counters have measurably advanced within the last 60 seconds. Keep the GPU streak so a sustained low-power condition becomes eligible again when progress stops.
  • Track endpoint identity and fresh LLM polls. Initial baselines, resets, invalid/unavailable readings, busy flags and positive/held rates are not progress evidence.
  • Expose TensorFold's live prefill position and completed-plus-running generation count, allowing long in-flight requests to demonstrate progress before completion totals move. Missing/partial measurements remain null; existing throughput calculations are unchanged.
  • Let configured workers inherit measured progress only from their explicitly paired, online head. Do not propagate progress to unrelated units.
  • Clear a retained low-power warning immediately when progress resumes; leave thermal/error findings and their retention unchanged.
  • Document the behavior and add regressions for stall/recovery, stale evidence, raw counter gaps/resets and head/worker propagation.

Unknown progress still permits the existing observational warning; this does not claim to diagnose a GPU fault or prove that every request is healthy.

Verification

  • 99 targeted tests pass across HealthEvaluator, SparkMonitor lifecycle and TensorFold probing.
  • Full server suite: 1,679 passed, 16 skipped, 0 failed.
  • i18n catalog check and all 11 checker tests pass.
  • Read-only live TensorFold smoke: fed actual probe snapshots into latest-main and patched evaluators with a deliberately simulated 96% / 9 W GPU condition. Latest main kept warning as counters advanced; the patch cleared head and worker warnings. Frozen observations restored both warnings after the 60-second evidence window. No inference requests, workload changes or restarts were performed.

Existing main-branch validation blocker

npm run typecheck, npm run build, and 10 frontend test suites cannot resolve ../../shared/mainFilesystem imported by src/components/OverviewPage/SparkCard.tsx. The import and missing module are present in baseline e89f8d1, not introduced here; #372 already includes a separate repair. The remaining 131 frontend suites / 897 tests passed. That unrelated repair is intentionally not bundled into this PR. Browser verification of the rebuilt frontend is blocked by the same missing module.

No production deployment is included.

@danieljohnmorris

Copy link
Copy Markdown
Contributor Author

Follow-up verification: built a separate test copy of this PR with only the existing main-filesystem build repair from #372 (81360aa) applied locally. That repair remains outside this PR.

  • npm run typecheck and npm run build pass.
  • Full server suite: 1,679 passed, 16 environment-dependent skips, 0 failures.
  • Frontend suite: all 141 suites / 962 tests pass.
  • i18n catalog check and all 11 checker tests pass.
  • Started the built dashboard alongside the existing installation with separate configuration/history and checked it in Chromium against three live DGX Sparks. All three were online, TensorFold prefill/generation positions advanced, and no low-power warnings appeared during the observation. No workload restarts or inference requests were performed.

The live readings in this browser check were above the preserved 11 W threshold, so the progressing-versus-stalled distinction remains separately verified by the earlier real-probe replay with a simulated 9 W condition and the head/worker regression tests—not inferred solely from the absence of a warning in the browser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant