Create alert system with customizable rules

[TonsOfFun]

Description

Build alerting system for monitoring agent performance and triggering notifications based on conditions.

Alert Rule Features

• Metric selection (cost, latency, error rate, token usage)
• Operator configuration (>, <, ==, !=, contains)
• Threshold settings
• Duration requirements (sustained for X seconds)
• Alert actions (email, webhook, UI notification)

Alert Types

• Cost threshold exceeded
• Error rate spike
• Latency degradation
• Token limit approaching
• Agent failure
• Generation timeout

Implementation

• Background job for alert checking (using SolidQueue)
• Alert state machine (pending, triggered, acknowledged, resolved)
• Alert history tracking
• Cooldown periods to prevent spam
• Alert grouping and deduplication

Notification Channels

• In-app notifications with Turbo
• Email notifications (Action Mailer)
• Webhook integrations
• Slack/Discord webhooks (optional)

Acceptance Criteria

• Alerts trigger based on conditions
• Notifications are delivered
• Alert history is maintained
• No duplicate alerts within cooldown