Skip to content

feat: loki alert rules - #124

Merged
SimonRichardson merged 1 commit into
juju:mainfrom
SimonRichardson:loki-alert-rules
Jul 14, 2026
Merged

feat: loki alert rules#124
SimonRichardson merged 1 commit into
juju:mainfrom
SimonRichardson:loki-alert-rules

Conversation

@SimonRichardson

Copy link
Copy Markdown
Member

The Juju controller charm forwards its workload logs to Loki via the loki-push-api relation, and the LokiPushApiConsumer library defaults to forwarding alert rules from ./src/loki_alert_rules/.

Adds src/loki_alert_rules/juju_controller_alerts.yaml with three LogQL alert rules, each scoped to the charm's Juju topology via the %%juju_topology%% placeholder (automatically resolved by the library):

  • JujuControllerHighErrorRate — fires when >5% of controller log lines contain ERROR over a 10-minute window. Catches sustained operational issues like failed API requests, model errors, or database connectivity problems.
  • JujuControllerFatalError — fires when the controller logs a FATAL error or panic. Indicates a critical failure warranting immediate investigation.
  • JujuControllerLogsStopped — fires when no logs are received from the controller for 15 minutes. Catches controller outages or logging misconfiguration that might otherwise go unnoticed.

Adds default loki alert rules to help when setting up loki for juju.
Comment on lines +6 to +9
sum(rate({%%juju_topology%%} |= "ERROR" [5m])) by (juju_model, juju_model_uuid, juju_application)
/
sum(rate({%%juju_topology%%}[5m])) by (juju_model, juju_model_uuid, juju_application)
> 0.05

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generally correct. Probably there are always enough logs that we don't need to consider low-volume scenario like there are 3 logs for 5 minutes and 1 of the is error level.

Comment on lines +30 to +32
The Juju controller has logged a FATAL error or panic in the last
5 minutes. This indicates a critical failure that may require
immediate investigation.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not in last 5 minutes to be exact. This will fire as soon as possible. You could also change the range from 5m to 1m, but that also doesn't matter so much.

@SimonRichardson
SimonRichardson merged commit 7b94ff6 into juju:main Jul 14, 2026
@SimonRichardson
SimonRichardson deleted the loki-alert-rules branch July 14, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants