Manage alerts, automation rules, and online-eval rules from the terminal — create thresholds on trace metrics, evaluate traces automatically, and read back rule results.

Alerts & Rules

Three command groups cover the rules that watch your traces and act on them:

Command	What it does
`aievals alert`	Threshold alerts on trace and session metrics
`aievals automation-rule`	Multi-action rules — online evals plus dataset capture
`aievals online-eval-rule`	Online-eval rules, plus the scores and metrics they produce

Add --format json to any command below. The default table view truncates nested objects such as notification targets and action configs.

None of these groups has a delete verb in the CLI. Stopping a rule is non-destructive: use alert pause, automation-rule update-status --status PAUSED, or online-eval-rule update --status=false.

Deletion itself is still available — through the dashboard, and through the SDK's delete methods. It is the CLI that deliberately omits it.

Alerts

Alerts evaluate a metric over a rolling window and notify you when it crosses a threshold.

List and read

aievals alert list --format json
aievals alert list --status active --scope TRACE --format json
aievals alert get alr_123 --format json

list accepts --page, --limit, --search (free-text over the alert name), --status (active, paused), --scope (TRACE, SESSION), and --metric.

Create

--name, --window-minutes, and --notifications are required.

aievals alert create --name "p95 latency" --window-minutes 60 \
  --environment prod \
  --notifications '[{"channel":"email","enabled":true,"config":{"to":"dev@example.com"}}]'

A fuller example with an explicit trigger:

aievals alert create --name "error rate" --window-minutes 15 --scope TRACE \
  --aggregation COUNT --metric errors --operator GT --threshold 10 \
  --environment prod --environment staging \
  --notifications '[{"channel":"slack","enabled":true,"config":{"webhookUrl":"https://hooks.example.com/x"}}]'

Trigger flags

Flag	Values
`--scope`	`TRACE`, `SESSION`
`--aggregation`	`AVG`, `MIN`, `MAX`, `P50`, `P95`, `P99`, `SUM`, `COUNT`
`--metric`	Metric name to evaluate
`--operator`	`GT`, `LT`, `GTE`, `LTE`
`--threshold`	Number the metric is compared against
`--unit`	Display-only label for the threshold
`--window-minutes`	`5`, `10`, `15`, `30`, `60`, `360`, `720`, `1440`, `4320`, `10080`
`--cooldown-minutes`	Minimum minutes between repeat notifications
`--environment`	Environment to scope to (repeatable)
`--notifications`	JSON array of `{channel, enabled, config}`
`--filter`	Additional filter criteria as a JSON object

Two constraints are checked locally, before the request is sent:

--environment is required for non-score metrics. An alert with no environments matches nothing and would never fire, so the CLI rejects it rather than saving a silent no-op.
Score metrics are capped at a 15-minute window. If --metric starts with score:, --window-minutes must be 15 or less.

Update, pause, resume

Update is a PATCH — only the flags you pass are sent, everything else is left alone. Every field is optional.

aievals alert update alr_123 --threshold 250 --cooldown-minutes 30

aievals alert pause alr_123     # stops evaluation, keeps the config
aievals alert resume alr_123

toggle sets the enabled state directly, which is what you want when scripting from a computed value:

aievals alert toggle alr_123 --status=false

Automation rules

An automation rule matches events and runs one or more actions against them. Actions are a sub-resource with their own endpoints, so they have their own action command group.

List and read

aievals automation-rule list --format json
aievals automation-rule list --action-type ONLINE_EVAL --format json
aievals automation-rule get aur_123 --format json

list accepts --page, --limit, --search, and --action-type (ONLINE_EVAL, DATASET).

Create

--name and at least one --action are required. --action is repeatable and takes inline JSON, because an action is a discriminated union on type: ONLINE_EVAL carries {config, metrics}, DATASET carries {config}.

aievals automation-rule create --name "eval prod traces" \
  --action '{"type":"ONLINE_EVAL","config":{"concurrency":2},"metrics":[{"evaluatorListId":"el_1"}]}'

aievals automation-rule create --name "capture to dataset" --sampling-rate 10 --status ACTIVE \
  --action '{"type":"DATASET","config":{"datasetId":"ds_1"}}' \
  --targeting-filter '{"fieldName":"environment","operator":"EQUALS","value":"prod"}'

Rule flags

Flag	Notes
`--name`	Rule name
`--description`	Rule description
`--sampling-rate`	Percentage of matching events to act on (1–100)
`--status`	`ACTIVE`, `PAUSED`
`--targeting-filter`	Targeting filter as inline JSON (repeatable)
`--source-type`	`TRACE`, `SESSION` — create only
`--action`	Action as inline JSON (repeatable) — create only

Update

update patches the rule's own fields. Actions are not editable here.

aievals automation-rule update aur_123 --name renamed --sampling-rate 50
aievals automation-rule update-status aur_123 --status PAUSED

update-status requires --status (ACTIVE or PAUSED).

Manage actions

aievals automation-rule action add aur_123 \
  --action '{"type":"DATASET","config":{"datasetId":"ds_1"}}'

aievals automation-rule action update aur_123 act_1 \
  --action '{"type":"DATASET","config":{"datasetId":"ds_2"}}'

action add appends; action update replaces one action by ID. Both require --action. There is no action delete in the CLI — the SDK's deleteAction and the dashboard both still offer it.

Online-eval rules

Online-eval rules score live traces as they arrive, and expose the resulting aggregates back through the CLI.

List and read

aievals online-eval-rule list --format json
aievals online-eval-rule list --search prod --format json
aievals online-eval-rule get oer_123 --format json

list accepts --page, --limit, and --search.

Create

Only --name is required.

aievals online-eval-rule create --name "score prod traces" --evaluator-list-id el_1

aievals online-eval-rule create --name "sample 10%" --sampling-rate 10 --delay 30 \
  --evaluator-list-id el_1 --evaluator-list-id el_2 \
  --filter '{"fieldName":"environment","operator":"EQUALS","value":"prod"}'

Rule flags

Flag	Notes
`--name`, `--description`	Rule identity
`--evaluator-list-id`	Evaluator list to run (repeatable)
`--filter`	Filter as inline JSON (repeatable)
`--sampling-rate`	Percentage of matching events to evaluate (1–100)
`--delay`	Seconds to wait before evaluating
`--trigger-trace-count`	Number of traces that triggers an evaluation
`--status`	Whether the rule is enabled
`--dataset-id`	Dataset to capture evaluated items into
`--dataset-mapping-type`	`EXPECTED_OUTPUT`, `LLM_OUTPUT`
`--dataset-extraction-mode`	`SINGLE_IO`, `FULL_CONVERSATION`
`--output-observation-name`	Observation name to read the output from
`--dataset-tag`	Tag applied to captured dataset items
`--source-type`	`TRACE`, `SESSION` — create only
`--online-eval-enabled`	Enable/disable the evaluation half — update only
`--dataset-enabled`	Enable/disable the dataset-capture half — update only

Update

aievals online-eval-rule update oer_123 --sampling-rate 50 --status=false

Read results

scores returns evaluator-score aggregates, metrics returns observation-metric aggregates. Both take the same time window:

aievals online-eval-rule scores oer_123 --granularity day --format json
aievals online-eval-rule metrics oer_123 --from-timestamp 2026-08-01T00:00:00Z --format json

Flag	Notes
`--from-timestamp`	Window start, RFC 3339 (e.g. `2026-08-01T00:00:00Z`)
`--to-timestamp`	Window end, RFC 3339
`--granularity`	`minute`, `hour`, `day`, `week`, `month`

list-traces lists the traces a rule has evaluated:

aievals online-eval-rule list-traces oer_123 --limit 20 --format json

It accepts --page, --limit, --search-query, and --search-type (content, repeatable).

online-eval-rule trace takes a trace ID, not a rule ID — it reads one trace with its online-eval context. Use list-traces <rule-id> to find trace IDs for a rule.

aievals online-eval-rule trace tr_123 --format json

Alerts & Rules

Alerts & Rules

Alerts

List and read

Create

Update, pause, resume

Automation rules

List and read

Create

Update

Manage actions

Online-eval rules

List and read

Create

Update

Read results

Automation Rules (UI)

Online Evaluations (UI)

Automation Rules (SDK)

Trace Management

On this page