feat(platform): Add AllQuiet alert integration alongside Discord alerts#11234
feat(platform): Add AllQuiet alert integration alongside Discord alerts#11234ntindle wants to merge 19 commits into
Conversation
- Added system_alert method to NotificationManager that sends both Discord and AllQuiet alerts - Implemented correlation IDs for all system alerts to prevent duplicate incidents: - Late executions: Based on threshold, count, and affected users - Block errors: Based on affected blocks and date - Balance alerts: Based on user ID - Retry failures: Based on function, context, and error type - Updated all alert locations to use NotificationManager.system_alert() method - Added AllQuiet webhook URL configuration in settings - Maintained backward compatibility with existing Discord alerts AllQuiet alerts are only sent when correlation_id is provided, ensuring controlled rollout. Severity levels (critical/warning/minor) and extra attributes provide better incident management and debugging context. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Applied isort for import sorting - Applied black for code formatting - Applied ruff for linting - Fixed type annotations for severity parameter in retry.py - All formatters and linters now pass successfully 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
✅ Deploy Preview for auto-gpt-docs-dev canceled.
|
✅ Deploy Preview for auto-gpt-docs canceled.
|
github-actions
Bot
added
the
conflicts
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThis pull request refactors the alerting system across multiple monitoring and notification modules to replace single-channel Discord alerts with a multi-channel system that supports both Discord and AllQuiet webhooks. The changes introduce structured alert payloads with correlation_id, severity, status, and extra_attributes metadata throughout the monitoring infrastructure. Changes
Sequence Diagram(s)sequenceDiagram
participant Caller as Monitoring/Executor
participant Manager as NotificationManager
participant AllQuiet as AllQuiet Webhook
participant Discord as Discord API
Caller->>Manager: system_alert(content, channel, correlation_id, severity, status, extra_attributes)
activate Manager
Manager->>Manager: Create AllQuietAlert from content & metadata
Manager->>AllQuiet: send_allquiet_alert(alert)
activate AllQuiet
AllQuiet-->>Manager: POST response
deactivate AllQuiet
note over Manager: Log if AllQuiet fails, continue flow
Manager->>Discord: discord_send_alert(content, channel)
activate Discord
Discord-->>Manager: response
deactivate Discord
Manager-->>Caller: Alert dispatched
deactivate Manager
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thanks for this PR adding AllQuiet alert integration alongside Discord alerts. The implementation looks solid with correlation IDs to prevent duplicate incidents, and the code changes are well-structured. However, I need to flag that the PR checklist is completely unchecked. Before we can merge this PR, please complete the checklist in the PR description:
Once you've addressed the checklist items, this PR should be ready for approval. |
|
Here's the code health analysis summary for commits Analysis Summary
|
|
Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly. |
github-actions
Bot
removed
the
conflicts
|
Thank you for implementing the AllQuiet alert integration alongside Discord alerts! The changes look well-structured and consistent across the codebase. Your implementation of correlation IDs for different alert types is thorough, and I appreciate the backward compatibility with existing Discord alerts. Before this can be merged:
The code implementation itself looks good, but the PR process requirements need to be addressed before this can be merged. Please update the PR description with the completed checklist and any additional configuration changes information. |
| # Send AllQuiet alert if correlation_id is provided | ||
| if correlation_id: | ||
| # Extract title from content (first line or first sentence) | ||
| lines = content.split("\n") | ||
| title = lines[0] if lines else content[:100] | ||
| # Remove Discord formatting from title | ||
| title = ( | ||
| title.replace("**", "") | ||
| .replace("🚨", "") | ||
| .replace("⚠️", "") | ||
| .replace("❌", "") | ||
| .replace("✅", "") | ||
| .replace("📊", "") | ||
| .strip() | ||
| ) |
There was a problem hiding this comment.
wdythink on alwasy sending even if no correlation id? (could cause duplicate notifications)
There was a problem hiding this comment.
Kept AllQuiet gated on correlation_id so Discord still always sends, but AllQuiet only opens incidents when we have a stable dedupe key. Verified in 6ca0ad7.
| async with httpx.AsyncClient() as client: | ||
| response = await client.post(hook_url, json=alert.model_dump()) | ||
| response.raise_for_status() |
There was a problem hiding this comment.
@claude replace this with Requests() like we do elsewhere in the codebase. search Requests() to make sure you know what i mean
There was a problem hiding this comment.
Claude finished @ntindle's task —— View job
✅ Done! I've replaced the httpx implementation with the Requests() helper.
Changes Made:
- Replaced
httpx.AsyncClient()withRequests()helper frombackend.util.request - The
Requests()helper provides built-in SSRF protection and automatic retry logic for transient failures - Simplified the code by removing the context manager since
Requests()handles session management internally
Todo List:
- Search for
Requests()usage in the codebase to understand the pattern - Read the current implementation in metrics.py
- Replace the HTTP request with
Requests()pattern - Commit and push changes
The change maintains the same functionality while using the consistent HTTP request pattern established in the codebase.
| Branch: ntindle/systemallquietalerts
There was a problem hiding this comment.
Handled in 60cdcba — send_allquiet_alert now uses Requests().post(...) like the rest of the backend.
| title=title[:100], # Limit title length | ||
| description=content, | ||
| correlation_id=correlation_id, | ||
| extra_attributes=extra_attributes or {}, |
There was a problem hiding this comment.
@claude all quiet should take in the channel too and add that as an attribute to AllQuietAlert since we already know what it is. then we can filter better in all quiet
There was a problem hiding this comment.
Claude finished @ntindle's task —— View job
✅ Done! I've added the channel information as an attribute to AllQuiet alerts.
Changes Made:
- Modified the
system_alertmethod in/autogpt_platform/backend/backend/notifications/notifications.py - The channel value (e.g., "platform" or "product") is now included in the
extra_attributessent to AllQuiet - This enables better filtering and categorization of alerts in the AllQuiet dashboard
Todo List:
- Read the current implementation in notifications.py
- Identify how channel information flows through the system
- Update
system_alertmethod to pass channel tosend_allquiet_alert - Update
send_allquiet_alertto include channel in AllQuiet alert attributes - Commit and push changes
The implementation preserves backward compatibility while adding the channel information for better alert management in AllQuiet.
| Branch: ntindle/systemallquietalerts
There was a problem hiding this comment.
Handled in 5485a96 — channel is now passed as a first-class field on AllQuietAlert, so it can be filtered directly in AllQuiet.
Uses the existing Requests() helper from backend.util.request for consistency with the rest of the codebase. The Requests() helper includes built-in SSRF protection and retry logic. Co-authored-by: Nicholas Tindle <ntindle@users.noreply.github.com>
|
Thanks for implementing this AllQuiet alert integration! This will definitely help improve our incident management capabilities. However, there are a few things that need to be addressed before we can merge this PR:
The code changes themselves look good! The implementation with correlation IDs for deduplication and the extra context attributes will be very helpful for incident management. |
This allows for better filtering and categorization of alerts in AllQuiet. The channel (platform/product) is now included in the extra_attributes sent to AllQuiet, enabling teams to filter alerts by their source channel. Co-authored-by: Nicholas Tindle <ntindle@users.noreply.github.com>
|
Thank you for your PR implementing AllQuiet alerts alongside Discord alerts. The implementation looks well-structured with correlation IDs and severity levels to improve incident management. However, there are a couple of items that need to be addressed before this can be merged:
Once you've addressed these items, the PR should be ready for another review. |
| ) | ||
|
|
||
| # Add channel to extra attributes for better filtering in AllQuiet | ||
| all_attributes = extra_attributes or {} |
There was a problem hiding this comment.
@claude let's make the discord channel (product or platform) a first class member of the base model AllQuietAlert rather than an extra attribute
There was a problem hiding this comment.
Claude finished @ntindle's task —— View job
✅ Done! I've made the Discord channel a first-class member of the AllQuietAlert model.
Changes Made:
- Added
channelas a direct field in theAllQuietAlertmodel in/autogpt_platform/backend/backend/util/metrics.py:62 - Updated
system_alertmethod in/autogpt_platform/backend/backend/notifications/notifications.py:465to pass channel directly instead of in extra_attributes - This enables better filtering and categorization of alerts in the AllQuiet dashboard
Todo List:
- Read the AllQuietAlert model definition in metrics.py
- Add channel as a first-class field to the AllQuietAlert model
- Update send_allquiet_alert function to use channel field directly
- Remove channel from extra_attributes in system_alert method
- Verify changes with git diff
- Commit and push changes
The channel value (e.g., "platform" or "product") is now a proper field in the AllQuiet alert structure, making it easier to filter and manage alerts based on their source channel.
| Branch: ntindle/systemallquietalerts
There was a problem hiding this comment.
Handled in 5485a96 — channel now lives on the base AllQuietAlert model instead of being tucked into extra_attributes.
- Added channel as a first-class field in AllQuietAlert model - Updated system_alert to pass channel directly instead of in extra_attributes - Enables better filtering and categorization in AllQuiet dashboard Co-authored-by: Nicholas Tindle <ntindle@users.noreply.github.com>
|
Thank you for adding the AllQuiet alert integration! This looks like a valuable addition to provide better incident management alongside the existing Discord alerts. However, before we can approve this PR:
The code implementation looks thorough, with correlation IDs for different alert types and proper severity levels. Once you complete the checklist items, this PR should be ready for approval. |
github-actions
Bot
added
the
conflicts
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
|
|
allquiet-alertsLaunchDarkly feature flag (killswitch)AllQuiet alerts are only sent when correlation_id is provided, ensuring
controlled rollout. Severity levels (critical/warning/minor) and extra
attributes provide better incident management and debugging context.
🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com
Changes 🏗️
NotificationManager.system_alert()sends to both Discord and AllQuiet with error isolationAllQuietAlertmodel with severity, status, correlation_id, channel, and extra_attributesbilling.pymigrated fromdiscord_system_alerttosystem_alert_extract_clean_title()helper for stripping markdown/emoji from alert titlesallquiet-alertsLaunchDarkly feature flag as killswitch (default: on)Checklist 📋
For code changes:
For configuration changes:
.env.defaultis updated or already compatible with my changesdocker-compose.ymlis updated or already compatible with my changesPlatform deployment changes needed
ALLQUIET_WEBHOOK_URLto the infra repoallquiet-alertsboolean feature flag in LaunchDarkly (default: true) — this is the killswitchNote
Medium Risk
Touches cross-cutting alerting paths (executor billing alerts, monitoring, retry infrastructure) and adds a new outbound webhook; failures could drop or spam incidents if correlation IDs/severity are mis-set, though Discord behavior is preserved.
Overview
System alerting is upgraded from Discord-only to a unified
NotificationManager.system_alertthat always posts to Discord and optionally posts an AllQuiet incident when acorrelation_idis provided, with normalizedseverity/status, derived titles, and extra attributes.Low-balance/insufficient-funds, late execution, block error-rate, and retry-threshold alerts are migrated to use
system_alertand now emit correlation IDs to dedupe incidents and attach debugging metadata. The PR also adds AllQuiet webhook support via a newallquiet_webhook_urlsecret and updates affected tests to assertsystem_alertusage and correlation IDs.Written by Cursor Bugbot for commit 07f5913. This will update automatically on new commits. Configure here.