Repository navigation
Add optional authenticated Alertmanager webhook receiver - #78
JeffPengCoder wants to merge 1 commit into
Conversation
|
Validation evidence for this PR (2026-10-29) All UI records below come from a disposable isolated container with synthetic alerts and disposable test credentials. They contain no production alert data, registry credentials, or UI tokens. The default chart image was The test has two complementary parts: the committed optional test starts a real Alertmanager 0.28.1 and sends standard grouped notifications to the adapter with a fake UI API; the isolated private-image check runs the adapter against the real UI API without an Alertmanager in that Pod. Production network/ingress/routing and unattended investigation are outside those checks. Full isolated UI API results: every case and complete returned event metadata{
"success": true,
"image": "nvcr.io/0948643769302270/afoa-release/mosaic-ui:348d4635",
"nodeVersion": "v22.23.2",
"rows": [
{
"label": "grouped firing",
"responseStatus": 200,
"response": {
"success": true,
"firing": 2,
"resolved": 0
},
"alerts": [
{
"id": "alertmanager:0123456789abcdef",
"payload": "MosaicWebhookSmokeTest | Isolated synthetic compatibility test | severity=warning",
"source": "alertmanager",
"createdAt": "2026-10-29T08:31:25.830Z",
"stage": "active",
"labels": {
"alertname": "MosaicWebhookSmokeTest",
"severity": "warning"
},
"annotations": {
"summary": "Isolated synthetic compatibility test"
},
"status": "firing",
"startsAt": "2026-10-29T08:31:25.830Z",
"endsAt": "0001-01-01T00:00:00Z",
"fingerprint": "0123456789abcdef",
"generatorURL": "https://example.com/test"
},
{
"id": "alertmanager:fedcba9876543210",
"payload": "MosaicWebhookSmokeTest | Isolated synthetic compatibility test | severity=warning",
"source": "alertmanager",
"createdAt": "2026-10-29T08:31:25.830Z",
"stage": "active",
"labels": {
"alertname": "MosaicWebhookSmokeTest",
"severity": "warning"
},
"annotations": {
"summary": "Isolated synthetic compatibility test"
},
"status": "firing",
"startsAt": "2026-10-29T08:31:25.830Z",
"endsAt": "0001-01-01T00:00:00Z",
"fingerprint": "fedcba9876543210",
"generatorURL": "https://example.com/test"
}
]
},
{
"label": "repeat delivery",
"responseStatus": 200,
"response": {
"success": true,
"firing": 2,
"resolved": 0
},
"alerts": [
{
"id": "alertmanager:0123456789abcdef",
"payload": "MosaicWebhookSmokeTest | Isolated synthetic compatibility test | severity=warning",
"source": "alertmanager",
"createdAt": "2026-10-29T08:31:25.830Z",
"stage": "active",
"labels": {
"alertname": "MosaicWebhookSmokeTest",
"severity": "warning"
},
"annotations": {
"summary": "Isolated synthetic compatibility test"
},
"status": "firing",
"startsAt": "2026-10-29T08:31:25.830Z",
"endsAt": "0001-01-01T00:00:00Z",
"fingerprint": "0123456789abcdef",
"generatorURL": "https://example.com/test"
},
{
"id": "alertmanager:fedcba9876543210",
"payload": "MosaicWebhookSmokeTest | Isolated synthetic compatibility test | severity=warning",
"source": "alertmanager",
"createdAt": "2026-10-29T08:31:25.830Z",
"stage": "active",
"labels": {
"alertname": "MosaicWebhookSmokeTest",
"severity": "warning"
},
"annotations": {
"summary": "Isolated synthetic compatibility test"
},
"status": "firing",
"startsAt": "2026-10-29T08:31:25.830Z",
"endsAt": "0001-01-01T00:00:00Z",
"fingerprint": "fedcba9876543210",
"generatorURL": "https://example.com/test"
}
]
},
{
"label": "mixed status",
"responseStatus": 200,
"response": {
"success": true,
"firing": 1,
"resolved": 1
},
"alerts": [
{
"id": "alertmanager:0123456789abcdef",
"payload": "MosaicWebhookSmokeTest | Isolated synthetic compatibility test | severity=warning",
"source": "alertmanager",
"createdAt": "2026-10-29T08:31:25.830Z",
"stage": "active",
"labels": {
"alertname": "MosaicWebhookSmokeTest",
"severity": "warning"
},
"annotations": {
"summary": "Isolated synthetic compatibility test"
},
"status": "firing",
"startsAt": "2026-10-29T08:31:25.830Z",
"endsAt": "0001-01-01T00:00:00Z",
"fingerprint": "0123456789abcdef",
"generatorURL": "https://example.com/test"
}
]
},
{
"label": "resolved",
"responseStatus": 200,
"response": {
"success": true,
"firing": 0,
"resolved": 1
},
"alerts": []
},
{
"label": "repeat resolved",
"responseStatus": 200,
"response": {
"success": true,
"firing": 0,
"resolved": 1
},
"alerts": []
}
]
}
Exact isolated UI API test harnessThe receiver source was mounted as import assert from 'node:assert/strict';
import { webhookServer } from './receiver.mjs';
const uiUrl = 'http://127.0.0.1:3000';
const headers = { authorization: `Bearer ${process.env.MOSAIC_UI_MACHINE_TOKEN}`, 'content-type': 'application/json' };
const token = process.env.MOSAIC_ALERTMANAGER_WEBHOOK_TOKEN;
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
let ready = false;
for (let n = 0; n < 100; n++) {
try {
const response = await fetch(uiUrl + '/api/alerts', { headers });
if (response.ok && (await response.json()).success) { ready = true; break; }
} catch {}
await sleep(500);
}
assert.equal(ready, true, 'isolated UI ingestion API must become available');
const server = webhookServer({ token, uiToken: process.env.MOSAIC_UI_MACHINE_TOKEN, uiUrl });
await new Promise(resolve => server.listen(3004, '127.0.0.1', resolve));
const startsAt = new Date(Date.now() - 1000).toISOString();
const alert = (fingerprint, status = 'firing') => ({ fingerprint, status, labels: { alertname: 'MosaicWebhookSmokeTest', severity: 'warning' },
annotations: { summary: 'Isolated synthetic compatibility test' }, startsAt,
endsAt: status === 'resolved' ? new Date().toISOString() : '0001-01-01T00:00:00Z', generatorURL: 'https://example.com/test' });
const rows = [];
async function send(label, alerts, expectedCount) {
const response = await fetch('http://127.0.0.1:3004/api/alertmanager/webhook', {
method: 'POST', headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
body: JSON.stringify({ version: '4', status: 'firing', truncatedAlerts: 0, alerts }),
});
const body = await response.json();
assert.equal(response.status, 200, JSON.stringify(body));
const listResponse = await fetch(uiUrl + '/api/alerts', { headers });
const list = await listResponse.json();
assert.equal(list.success, true);
assert.equal(list.alerts.length, expectedCount);
rows.push({ label, responseStatus: response.status, response: body, alerts: list.alerts });
}
try {
await send('grouped firing', [alert('0123456789abcdef'), alert('fedcba9876543210')], 2);
await send('repeat delivery', [alert('0123456789abcdef'), alert('fedcba9876543210')], 2);
await send('mixed status', [alert('0123456789abcdef'), alert('fedcba9876543210', 'resolved')], 1);
await send('resolved', [alert('0123456789abcdef', 'resolved')], 0);
await send('repeat resolved', [alert('0123456789abcdef', 'resolved')], 0);
console.log(JSON.stringify({ success: true, image: process.env.TEST_UI_IMAGE, nodeVersion: process.version, rows }, null, 2));
} finally {
server.closeAllConnections();
await new Promise(resolve => server.close(resolve));
}Complete repository test output: 236 tests, 234 passed, 2 conditional skips, 0 failedRun with Node 24, the chart test dependencies prepared as in CI (the private research dependency is a render-only placeholder), and node --test utils/*.test.mjs helm/mosaic-stack/tests/*.test.mjs tests/docs/*.test.mjs |
|
@coderabbitai review |
Action not completed
|
|
Warning Review limit reachedThis review ran on the open-source allowance, not this organization's plan, because the pull request author doesn't have an assigned seat. Waiting won't change this — ask an organization admin to assign them a seat, or add seats in Billing if every seat is already assigned, then retry. Next included review available in 26 minutes. View limit details
|
Signed-off-by: Jeff Peng <jepeng@nvidia.com>
70696fb to
c074536
Compare
Description
Mosaic currently pulls active alerts from Alertmanager. Add an optional push receiver so operators can route standard Alertmanager v4 notifications into the existing Mosaic alert list: firing notifications upsert
alertmanager:<fingerprint>, repeated deliveries update the same entry, and resolved notifications remove it. Closes #77.The default-off
alertmanagerWebhookcomponent uses a dedicated inbound bearer-token Secret and the existing UI machine-token Secret for internal delivery. It provides a ClusterIP Service and optional TLS ingress restricted to the exact webhook path, validates the complete group before forwarding, bounds request size and delivery time, and returns retryable errors for downstream failure or backpressure. The installation builder includes the receiver and its resource footprint. The guide covers credentials, Alertmanager routes and Secret mounts, administrator responsibilities, and firing/resolution smoke tests.This change delivers alerts. Existing Auto investigate triggers still come from the open Mosaic browser page; it does not add an unattended investigation worker or Teams notification delivery. It also does not configure an external Alertmanager on the operator's behalf. The UI's process-local alert store requires one UI replica, has no durable incident history, and remains subject to the documented retry/reordering/restart limits. Use webhook delivery with Auto pull off.
Validation
70696fbc8064d13eef438a6912482c597b1f8e65: CI / Tests and Documentation / build.ALERTMANAGER_BINARYset: 236 tests, 234 passed, 2 conditional skips, 0 failures. The skips are the existing OpenClaw-runtime native-memory and external VictoriaMetrics integration checks.-W --keep-goingbuild passed; all four YAML examples parsed;amtool check-configand route testing passed with both Mosaic and an existing notification receiver preserved; regenerated agent-documentation bundle passed freshness check.mosaic-ui:348d4635image, Node v22.23.2:The compatibility check used synthetic alerts, isolated UI state, and separate disposable credentials. Its Pod and ConfigMap were removed. Existing Mosaic deployments and live Alertmanager routing were not changed. Full synthetic result records, the exact API-test harness, and complete local test output are attached to this PR. This validates the receiver protocol and the UI API separately; a customer's network, TLS ingress, routing, browser-triggered investigation, and Teams delivery still need deployment-specific validation.
Checklist