Skip to content

feat(health): make the GPU temperature warning and critical limits configurable - #273

Closed
jammsen wants to merge 3 commits into
MiaAI-Lab:mainfrom
jammsen:feat/configurable-gpu-temp-thresholds
Closed

jammsen wants to merge 3 commits into
MiaAI-Lab:mainfrom
jammsen:feat/configurable-gpu-temp-thresholds

Conversation

@jammsen

@jammsen jammsen commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Closes #233

What

The GPU temperature Health finding fired at fixed 85/90 °C (TEMP_WARN / TEMP_CRITICAL in server/health/HealthEvaluator.js). Both are now settings, following the shape agreed in the issue.

  • Settings: healthTempWarnC and healthTempCriticalC, defaults 85 and 90, so behaviour is identical until someone edits them.
  • Validation: each must be a number from 40 to 110 and warn must be lower than critical. PUT /api/settings refuses anything else with a 400 and saves nothing. A hand-edited settings.json with unusable values falls back to 85/90 together.
  • Evaluator: reads the limits on every sample (via a thresholds option on HealthEvaluator, wired to getSettings() in SparkMonitor), so a change applies on the next sample without a restart. The finding detail shows the configured values.
  • Alerts and Activity: both reuse the finding, so they follow the configured limits with no change of their own.
  • UI: a "GPU temperature alerts" row in Settings with two °C inputs and a Reset to Defaults button. The dialog checks the same rules before saving. English and 简体中文 strings added.
  • Docs: README settings table row and a CHANGELOG entry.

Per-unit overrides are left for a later step, as agreed.

Notes

  • The limits are always in °C, whatever the display unit; the label says so.
  • The 40–110 °C range is my choice; the issue only said "sensible range". Happy to change it.
  • The 简体中文 text should be checked by a fluent speaker.
  • Existing Activity events keep their old text ("warns at 85°C"); only new events use the configured values.

Testing

npm run typecheck, npm run build and npm test pass (1331 server, 720 frontend). New tests cover the evaluator, settings validation and fallback, SparkMonitor Activity events with non-default limits (including a live change), and the dialog (save, refusal of bad input, reset). I also tried the dialog in a running instance.

jammsen and others added 3 commits October 24, 2026 16:21
…nfigurable

The thermal finding fired at fixed 85/90 °C, which does not suit units in
enclosed rooms or on hardware with its own thermal policy. Settings now has
healthTempWarnC and healthTempCriticalC (defaults 85 and 90, so behaviour is
unchanged). Each must be a number from 40 to 110 and warn must be below
critical; PUT /api/settings refuses anything else with 400, and an unusable
settings.json falls back to 85/90. The evaluator reads the limits on every
sample, the finding text shows them, and alerts follow because they reuse
the finding. Per-unit overrides are left for later.

Closes MiaAI-Lab#233

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Drive a SparkMonitor with non-default limits and check the Activity event
text, and that a limit changed in Settings applies to the next sample.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The button sits under the help text, puts 85 and 90 back in the two fields
(applied on Save) and is disabled while they already hold the defaults.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@MiaAI-Lab

Copy link
Copy Markdown
Owner

Reviewed and ported: now on main as 36d3567, 9407d41 and 3ad5ddc, with your authorship (cherry-picked with -x) and a CHANGELOG credit. All three commits were needed — the tip alone does not apply — and they went in cleanly on current main. I verified the parts that matter rather than taking the description on trust: the three new threshold tests fail against the unchanged evaluator, updateSettings is reached before the merge so a bad patch persists nothing (server/settings.js:329), and the existing PUT /api/settings handler already turns that throw into a 400 (server/index.js:760-777), so your claim holds without touching the route. _clampSettings falling both values back together is the right call for a hand-edited file, since it cannot leave warn above critical.

The 40–110 °C range is fine as proposed. The thresholds callback reading getSettings() on every evaluation is also the right shape — it keeps one evaluator per unit and applies a change on the next sample, as agreed in #233. Full suite on Linux after the port: 1331 server and 732 frontend tests pass, typecheck and build clean. Thank you for the patience with the issue thread, and for matching the agreed scope exactly, including leaving per-unit overrides out.

(Review notes prepared with Claude Code on behalf of the maintainer.)

@MiaAI-Lab MiaAI-Lab closed this Oct 26, 2026
pavelkubicek added a commit to pavelkubicek/sparkDash that referenced this pull request Oct 27, 2026
78 upstream commits: MQTT publishing (MiaAI-Lab#248), harness occupancy helpers (MiaAI-Lab#208),
GPU clock cap from Settings (MiaAI-Lab#218), settings i18n foundation (MiaAI-Lab#226),
configurable GPU temperature limits (MiaAI-Lab#273), Locals/Splash engines,
TensorFold cache-counter fixes (MiaAI-Lab#252/MiaAI-Lab#260/MiaAI-Lab#272) and the zsh-safe remote probes (MiaAI-Lab#255/MiaAI-Lab#257).

Conflict resolutions:
- server/collectors/LlmProbe.js + its TensorFold test: took main's implementation,
  which supersedes this fork's /health `live` + /metrics work with the cache-counter
  precedence, running/waiting aliases, histogram validation and prefill holds. This
  drops the fork's parallel /metrics fetch (micro-optimisation only).
- SystemCollector: main's zsh-safe find probes merged into the fork's build/parse
  refactor, keeping the fork-only RAPL package-power reads; _getRemoteNetwork now uses
  the injected executor (the merge had left the fork body calling sshExec, which made
  main's zsh-glob corpus test run real SSH).
- OverviewPage: FleetKpis keeps the fork's placement but takes all sparks, so hidden
  cluster workers count in the KPI tiles (main's fix).
- SparkPage: main's column layout (History in the hardware column, storage/network in
  the right column) over the fork's full-width History panel.
- SettingsDialog: fork VulnHunter/REA opt-in toggles kept, metrics-history toggle
  switched to main's i18n keys.
- index.js: fork vulnHunt/rea spark cleanup kept alongside MQTT removeUnit/applySettings.
- CHANGELOG/README/package.json: both sides' entries kept; test globs unioned so the
  fork-only models/autopower/vulnhunt suites still run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Make the GPU temperature warning and critical thresholds configurable in Settings

2 participants