Skip to content

feat: alert notifications over Health findings and Overview exceptions - #164

Closed
Lesilva wants to merge 2 commits into
mainfrom
feat/alerts
Closed

Lesilva wants to merge 2 commits into
mainfrom
feat/alerts

Conversation

@Lesilva

@Lesilva Lesilva commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Reworked on 2.x (single commit e2c9b59). Opt-in (alertsEnabled, default off): a notification layer, not a second rule set.

How it coexists with HealthEvaluator and the Activity log (also at the top of server/alerts/engine.js): detection stays where it is — the rules are HealthEvaluator findings (same ids, severities, thresholds) plus the Overview strip's exceptions, whose derivation moved to src/shared/fleetExceptions.js so FleetAlertStrip and the server run the same code (throttle = thermal/hardware slowdown only, never the power cap). This layer adds delivery: pending → firing → resolved with a for wait and hysteresis, grouping, reminders, firing set kept across restarts. Every send, delivered or failed, is an alert.notify.sent/.failed Activity event (no URL, no body).

Outbound (server/alerts/outbound.js, one place, mirrors validate.js): isAllowedTargetHost + DNS check on save, assertAllowedTarget on every send, connection pinned to the checked addresses; LAN/loopback allowed as for units. redirect: "manual" — any 3xx fails. Response bodies are never read. While off: no evaluation, and PUT /api/alerts/config / POST /api/alerts/test answer 409. Remote strings capped (120/60). Channel URLs live in the encrypted secrets store (alertChannels), masked in the API; old plaintext URLs migrate on load.

UI: Settings → Alert notifications · Configure… (2.0 SettingRow/ToggleRow, modal sheet); the strip shows server alerts with real startsAt while on.

Test plan

  • typecheck, build ✓; server 839 (+51; only the same 22 macOS llmlaunch/tooleval failures as main); frontend 508 (+14)
  • notifier over real sockets: stub DNS → 169.254.169.254 refused before any request; 302 never followed; a body containing a fake key never reaches API, status, logs or events
  • isolated instance against a real fleet: an offline unit fired after 60 s and was delivered to a local receiver (webhook + ntfy), two Activity events; a HealthEvaluator memory finding (stubbed input) fired once, survived a flap, resolved once; URLs never on disk in plaintext

🤖 Generated with Claude Code

@MiaAI-Lab

Copy link
Copy Markdown
Owner

Thanks, this is a big and careful piece of work (rule state machine, masked URLs, throttled logging, good tests). Before it can merge: (1) channel URLs should get the same outbound-target validation as other targets in server/validate.js (private LAN hosts such as a self-hosted ntfy should stay allowed); (2) the notifier should not follow redirects automatically; (3) /api/alerts/test should not return response bodies. Details are in the code review thread if you want them and we are happy to pair on it. It also overlaps HealthEvaluator and the Activity event log that landed in 2.0, so a short note on how they should coexist would help, and it needs a rebase onto 2.0 (11 files).

(Review notes prepared with Claude Code on behalf of the maintainer.)

@MiaAI-Lab

Copy link
Copy Markdown
Owner

Re-reviewed against main; the three requests from 2026-10-09 are still open. (1) Channel URL validation: server/alerts/store.js:39-54 checks only scheme and hostname; it should apply the outbound-target policy from server/validate.js, resolve at send time and reject disallowed addresses. (2) Redirects: notifier.js:242 follows them; use manual redirect handling and treat 3xx as failure. (3) Response bodies: notifier.js:243-250 puts the first 200 characters of a response into error, which runtime.js:142 returns and stores in status.lastError. Also: PUT /api/alerts/config and POST /api/alerts/test are registered regardless of alertsEnabled, so the opt-in does not gate outbound requests; summaries embed remote strings (model id, error text) without a length cap; channel URLs sit in config/alerts.json rather than the encrypted secrets store; and gpu_throttle fires on the software power cap, which main's strip deliberately ignores, with thresholds that differ from HealthEvaluator. It still needs the note on how it coexists with HealthEvaluator and the Activity log, and a rebase (16 files conflict; the backend is mechanical, the Overview/Settings UI is real work). What outbound targets the dashboard allows is a maintainer decision. Leaving this open for the author.

(Review notes prepared with Claude Code on behalf of the maintainer.)

@Lesilva Lesilva changed the title feat: server-side alert rules with ntfy, Discord, Slack and webhook notifications feat: alert notifications over Health findings and Overview exceptions Oct 12, 2026
@Lesilva

Lesilva commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator Author

Reworked on 2.x (single commit e2c9b59), addressing every point:

  1. Channel URLs use validate.js's policy: isAllowedTargetHost + DNS check on save, assertAllowedTarget on every send, connection pinned to the checked addresses (server/alerts/outbound.js — one place; LAN/loopback still allowed as for units, your call what else).
  2. redirect: "manual"; any 3xx = failure "redirect not followed (HTTP 3xx)".
  3. No response body is ever read: /api/alerts/test, status.lastError, logs and Activity carry a status code or fixed reason only (tested with a body containing a fake key).
  • alertsEnabled now gates outbound: off = no evaluation, and PUT /api/alerts/config + POST /api/alerts/test answer 409 (save does DNS checks, so refusing keeps off = no DNS, no HTTP).
  • Remote strings (unit names, SSH errors, Xid lines, model ids) capped at 120/60 chars.
  • Channel URLs moved to the encrypted secrets store (alertChannels section, same key as LLM API keys); alerts.json has no URLs; old plaintext URLs migrate on load; the API masks them. Note: that section is a persisted-state change — a downgrade would drop it on the next password save.
  • gpu_throttle and our own thresholds are gone: rules are HealthEvaluator findings + the strip's exceptions (now src/shared/fleetExceptions.js, shared by FleetAlertStrip and the server) — throttle = thermal/HW only.
  • Coexistence (also atop engine.js): detection stays in HealthEvaluator/the strip; Activity keeps its events; this layer adds the for/hysteresis delivery and logs alert.notify.sent/.failed per channel.
  • Settings/Overview re-done in 2.0 style (SettingRow/ToggleRow, modal sheet).

— Wei

@MiaAI-Lab

Copy link
Copy Markdown
Owner

Re-reviewed at e2c9b59 and the earlier requests are fixed in code: channel URLs go through the validate.js policy on save and again at send with the connection pinned to the checked addresses (server/alerts/outbound.js:47-84, notifier.js:294-295), redirects are not followed and a 3xx is a failure (notifier.js:298-310), response bodies are never read or stored (notifier.js:304), PUT /api/alerts/config and POST /api/alerts/test answer 409 while alertsEnabled is off (runtime.js:177, :256), remote strings are capped, URLs live in the encrypted store, and the throttle rule matches main's deriveAlerts. It applies to main cleanly and tests, typecheck and build pass. I am not merging it myself because it adds an outbound network path and stores new secrets, which needs an explicit maintainer decision. Before that: the Slack payload (notifier.js:166) sends unit names and remote error text unescaped, so channel-wide mentions and links can be injected; please escape &, < and > there. Also make maskUrl (store.js:75) show a fixed-length mask instead of the last four characters, and note in the docs that a failed send is not retried.

(Review notes prepared with Claude Code on behalf of the maintainer.)

@Lesilva

Lesilva commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 2966826 (on top of a rebase onto main):

  • Slack escaping: slackEscape() escapes &, < and > in the title and every line, so a unit name or remote text like <!channel>, <!here>, <@U123> or <https://x|click> reaches Slack as literal text. Tested with exactly those strings.
  • maskUrl: now shows scheme, host and a fixed /… (e.g. https://ntfy.sh/…). Nothing of the path is shown, its length included, and credentials are still never shown. Sending the masked value back still keeps the stored URL.
  • Docs: README has a No retries item. Each message is sent once; a failure is logged, shown on the channel and recorded in Activity, but the alert counts as notified, so the channel next hears about it on escalation, clearing or a reminder. The CHANGELOG says the same.

Server 863 (same 22 pre-existing macOS failures), frontend 539, typecheck clean.

— Wei

@Lesilva

Lesilva commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator Author

Force-pushed 39561c8: rebased onto d94450b, and restored a closing brace in dialogs.css that my earlier conflict resolution dropped (the build failed; tests did not catch it). No other changes.

— Wei

Lesilva and others added 2 commits October 15, 2026 15:21
Nothing told anyone about a problem unless a dashboard was open. The
server can now notify ntfy, Discord, Slack or a JSON webhook - as a
notification layer over what 2.0 already detects, not a second rule set:

- conditions are HealthEvaluator's findings (snapshot.health: same ids,
  severities, thresholds) and the Overview strip's exceptions, whose
  derivation moves to src/shared/fleetExceptions.js so FleetAlertStrip
  and the server run the same code (offline, thermal/hardware throttle -
  never the software power cap - disk >= 90%, LLM, tailnet)
- the engine adds delivery: pending -> firing -> resolved with a per-rule
  wait and hysteresis (clear for the same wait, at least 30 s), grouping
  per evaluation, optional reminders, firing set persisted across restarts
- every send, delivered or failed, is an alert.notify.sent / .failed
  Activity event (no URL, no response body); new Activity category
- outbound: channel URLs pass validate.js's target policy at save and are
  re-resolved with assertAllowedTarget on every send, the connection
  pinned to the checked addresses; LAN/loopback stay allowed
- redirects are never followed (3xx = failure); response bodies are never
  read - errors are a status code or a fixed reason
- alertsEnabled (default off) gates everything outbound: tick does not
  evaluate, PUT /api/alerts/config and POST /api/alerts/test answer 409
- remote-derived text (unit names, SSH errors, Xid lines, model ids) is
  capped to 120 chars (names 60)
- channel URLs live in the encrypted secrets store (new alertChannels
  section; saveSecrets keeps it); alerts.json holds no URLs, and a
  plaintext URL from the earlier build is migrated on load
- Settings -> Alert notifications -> Configure... in the 2.0 dialog style
  (SettingRow/ToggleRow moved to SettingRows.tsx); the exceptions strip
  lists the server's alerts with real durations while it is on

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tries

- Slack payload: `&`, `<` and `>` are escaped in the title and every line, so
  a unit name or remote error text cannot produce `<!channel>`, `<!here>`,
  user mentions or `<url|text>` links.
- maskUrl() shows scheme + host and a fixed `/…`, nothing of the path (no
  last four characters, no length); saving the masked value still keeps the
  stored URL, and credentials in the URL are never shown.
- README: a failed send is not retried; the alert counts as notified and the
  channel hears about it again on escalation, clearing or a reminder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MiaAI-Lab

Copy link
Copy Markdown
Owner

Re-verified at 1a77eb7 against main 2c005e3 (before today's pushes). It merges cleanly and its 51 alert tests pass. The earlier requests are fixed: Slack text is escaped (notifier.js:51-53,176), the connection is pinned to the addresses the outbound policy checked (outbound.js:61-84, notifier.js:305-306), redirects are not followed (notifier.js:309,319), response bodies are never read (notifier.js:313-318), the config and test routes return 409 while alerts are off (runtime.js:177,256), and channel URLs live in the encrypted store and are masked in API output. One item to fix: secretsStore.js:267-275 treats an unreadable secrets file as empty, so a later save would rewrite it without the other sections; it should refuse to write instead, and copy through ciphertext it cannot decrypt. Because this adds an outbound network path and new stored secrets, and it also rewrites the same part of secretsStore.js as #208, it is waiting for a maintainer decision rather than being merged in this pass. Leaving it open.

(Review notes prepared with Claude Code on behalf of the maintainer.)

@MiaAI-Lab

Copy link
Copy Markdown
Owner

Ported to main with your two commits and authorship (rebased by hand: the settings, config, README and CHANGELOG hunks kept both sides). The maintainer has accepted the outbound path and the new stored secrets. One defect fixed in a follow-up: server/secretsStore.js treated a secrets file that exists but cannot be parsed as empty, so the next save rewrote it without the SSH passwords and LLM keys; every save now fails with a clear error and leaves the file alone, and an alert URL that cannot be decrypted is kept while its channel is configured (saveAlertChannelUrls with keepIds). Both have regression tests that fail without the change. The README now says that the alert test route reveals LAN reachability to a token holder. Typecheck, build, 1002 server tests and 788 frontend tests pass. Now on main as 2c2a1af, 94b31fc (yours) and 61e9183 (the follow-up).

(Review notes prepared with Claude Code on behalf of the maintainer.)

@MiaAI-Lab MiaAI-Lab closed this Oct 15, 2026
@Lesilva
Lesilva deleted the feat/alerts branch October 16, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants