Alerts

Alerts watch your project's metrics — latency, cost, token usage, error counts, and evaluation scores — and notify you when they cross a threshold. Each alert has a rule (what to watch) and one notification channel (where to send it).

How alerts work

1. You define a rule: a metric, an aggregation, a comparison operator, a threshold, and a look-back window.
2. The evaluator runs on a schedule, querying your project's traces or sessions over the window.
3. If the aggregated value breaches the threshold, an alert event is created and the notification is dispatched.
4. After firing, the rule enters a cooldown — it will not fire again for that duration even if the condition still holds.
5. Events progress through three states: Firing → Acknowledged → Resolved.

Opening Alerts

Open Alerts under the Observability section of the left sidebar. The page has two tabs:

• Alert Events — every event the alert engine has produced, with current state and notification delivery status.
• Alert Rules — every rule you've configured. Toggle enabled, edit, or delete from here.

Click Create Alert at the top right to open the rule editor — it's a dropdown offering Metric Alert (described next) and Model Deprecation (see model deprecation alerts).

Create an alert

Metrics reference

System metrics work on both Trace and Session scopes:

Score metrics appear as score:<score-name> and are populated from your Online Evaluations. Only scores produced by online evaluation (source = EVAL, environment = online_evaluation) are available — manual annotation scores and experiment scores do not feed alert rules.

Notification channels

Webhook payload

When an alert fires, the notification worker POSTs this JSON body to your webhook URL:

{
  "alertId": "cuid-of-the-rule",
  "alertName": "High latency on chat agent",
  "projectId": "your-project-id",
  "state": "FIRING",
  "message": "AVG latency exceeded 500 ms (actual: 742.31 ms) over the last 5 min",
  "condition": {
    "metric": "latency",
    "aggregation": "AVG",
    "operator": "GT",
    "threshold": 500,
    "scope": "TRACE",
    "windowMinutes": 5
  },
  "evaluation": {
    "metricValue": 742.31,
    "breached": true
  },
  "firedAt": "2026-05-11T14:00:00.000Z"
}

Delivery semantics:

• Method: POST
• Content-Type: application/json
• Your Custom Headers are merged into the request headers (your Content-Type will be overridden if you set one).
• Timeout: 10 seconds.
• Retries: up to 3 attempts with exponential backoff (handled by the queue).
• 4xx responses (except 429) are treated as permanent failures and skipped to the dead-letter queue immediately — fix your endpoint and resave the rule rather than waiting for a retry.

Email

Email recipients receive a formatted message containing the alert name, the breached condition, the actual metric value, and the time it fired. Email delivery uses the same retry policy as webhooks.

Slack

When an alert fires, a formatted message is posted to the Slack channel selected on the rule. Connect Slack once for your organization in Settings → Integrations (an admin authorizes the BrowserStack Slack app); after that, the Slack channel option lists your workspace's channels when you create or edit an alert.

Zenduty

When a metric alert fires, the notification worker POSTs a Zenduty Generic Integration payload to your configured URL:

{
  "alert_type": "critical",
  "message": "High latency on chat agent",
  "summary": "AVG latency exceeded 500 ms (actual: 742.31 ms) over the last 5 min\nCondition: AVG latency > 500 ms.\nValue: 742.31.\nhttps://<your-instance>/project/<project-id>/alerts?ref=alert_zenduty",
  "entity_id": "cuid-of-the-rule"
}

Delivery semantics:

• For a metric breach, message is the alert rule's name and entity_id is fixed to the rule's ID — repeated fires of the same rule collapse into one Zenduty incident instead of opening a new one each time. summary carries the breach message, the condition, the measured value, and a link back to the Alerts page.
• For a model deprecation alert, message is Model deprecation — <model name> and entity_id combines the rule and the model, so all reminder tiers for one model group into a single incident. summary names the deprecation date, where the model is used, and the suggested replacement when the provider announced one.
• alert_type is whatever you picked in Send as on the rule — it does not vary per fire.
• Zenduty is send-only: AI Evals never reads back incident state, priority, or acknowledgements from Zenduty.
• Delivery uses the same notification queue and retry policy as webhooks, and is isolated per channel — for metric alerts, a Zenduty failure shows up in the event's notificationResults without affecting anything else.

Alert events

The Alert Events tab shows every event the engine has produced. Each event has a state:

Each event also records notificationResults — a per-channel delivery log: { channel, status, error? }. This is how you debug a notification that didn't arrive.

Model deprecation alerts

A model deprecation alert watches your project for continued use of a model its provider has scheduled for retirement, and reminds you before it stops working. Unlike a metric alert, it has no threshold or SQL condition — it watches for usage, not a metric value.

How it works

• Deprecation lifecycle dates — the announced deprecation date, the hard-shutdown date, and a suggested replacement model where the provider names one — are tracked for models from OpenAI, Anthropic, Google (Gemini), Groq, and Mistral. See Model Deprecation on the Models settings page for how this status also appears in the model picker.
• A daily scan checks every deprecated model against four usage surfaces in your project: traces (LLM generations from the last 90 days), evaluators, prompts, and experiments. All four are always monitored — this isn't configurable in this release.
• If usage is found, reminders fire on an escalating schedule: 30, 7, and 1 day(s) before the deprecation date, plus one final notice on the hard-shutdown date (when the provider's API stops serving the model).
• Reminders are bucketed so exactly one tier is active at a time: an alert created when a model is already, say, 5 days from deprecation fires its current tier (the 7-day reminder), never a stale 30-day one. The notification names the tier and the real days remaining, and each reminder for a given alert, model, and tier fires at most once — rescans and retries never send duplicates.
• Each fired reminder also creates an entry in the Alert Events tab, with the same per-channel delivery log as metric alerts.

Create a model deprecation alert

Model deprecation alerts appear in the same Alert Rules table as metric alerts, tagged Model deprecation in the Alert type column (metric alerts are tagged Metric), with a condition summary like Model deprecation · all surfaces · reminders 30, 7, 1d before.

If you don't configure an alert

Projects are still covered by a baseline: if the daily scan finds usage of a deprecated model and the project has no active model deprecation alert, AI Evals emails the project owner directly (not the would-be alert creator) on the same 30/7/1-day schedule. Creating your own model deprecation alert takes over from this baseline — you won't get both for the same model and reminder tier — so make sure the alert's channel is actually filled in.

Permissions

Alerts follow your project role. Metric alerts and model deprecation alerts share the same permissions — there is no separate permission for either.

Limits

Use cases

• Cost guardrails — AVG total_cost > 0.05 on a 1-hour window catches a runaway pipeline cheaper than discovering it in next month's bill.
• Latency regression detection — P95 latency > 2000 ms on a 15-minute window over production only.
• Hallucination spike — AVG score:faithfulness < 0.6 on a 15-minute window, paired with an online evaluation that scores faithfulness.
• Error surge — SUM error_count > 10 on a 5-minute window — fastest signal that something is misconfigured downstream.
• Model deprecation reminders — get notified 30/7/1 day(s) before a provider retires a model your traces, evaluators, prompts, or experiments still reference.
• On-call paging — route a production-only latency or error rule to Zenduty so a sustained breach opens an incident and pages whoever is on call.