Experiment Workflow
Run a full evaluation from the terminal — discover providers, build a dataset and evaluators, create a prompt and experiment, then read the scores.
Experiment Workflow
Provider discovery
Always run provider discovery before creating prompts or LLM evaluators:
aievals provider list --format jsonUse the exact name field (case-sensitive) for --model-params-provider. The
name and adapter fields can differ (e.g., name="OpenAI" vs.
adapter="openai").
Using the wrong provider value causes silent 0 scores on LLM evaluators.
Always copy values from provider list — never guess, hardcode, or lowercase
them.
The workflow
Discover providers
aievals provider list --format jsonDiscover available providers and models, and the exact values to use below.
Create a dataset
aievals dataset create
# then add items with:
aievals dataset-item createCreate evaluator(s)
aievals evaluator createCode evaluators must use function main(...), not function evaluate(...).
All params (input, output, expected) are strings — use JSON.parse() to
access fields.
Create a metric
aievals metrics createGroups evaluators (called "Metrics" in the UI, evaluator-list in the API).
Create a prompt
aievals prompt create \
--model-params '{"provider":"OpenAI","adapter":"openai","model":"gpt-4o-mini"}'Use exact values from Step 1.
Read the scores
aievals experiment-run compare <run-id> --format jsonReturns aggregate scores per evaluator. For individual scores:
# get score IDs from each trace's `scores` array
aievals trace list --format json
# then fetch a single score
aievals score get <score-id> --format jsonRunning experiments locally
Use experiment-run run when you have .experiment.ts or .experiment.py files that call the SDK's two-step API. The CLI discovers files, injects credentials, spawns the runner, and streams progress. Results are pushed to the platform automatically.
Use experiment-run run for local SDK files. experiment-run create is for triggering server-side runs on existing experiments — do not mix them.
Prerequisites
- Authenticate:
aievals auth login - Set your provider API key in the shell environment (
OPENAI_API_KEY,ANTHROPIC_API_KEY,GOOGLE_API_KEY, etc.). All environment variables are inherited by the spawned process. - Name experiment files
*.experiment.tsor*.experiment.py.
Basic usage
# Run a single file
aievals experiment-run run my-eval.experiment.ts
# Run all experiment files in a directory
aievals experiment-run run .
# List discovered files without executing
aievals experiment-run run . --listKey flags
| Flag | Description |
|---|---|
--first N | Process only the first N dataset items — useful during development |
--sample N | Randomly sample N items |
--sample-seed S | Seed for reproducible sampling (default: 0) |
--list | Discover and list experiment files without executing them |
--filter EXPR | Only run experiments whose name matches the pattern |
--no-send-logs | Dry run — execute locally without uploading to the platform |
--jsonl | Machine-readable output (one JSON object per experiment) for CI/CD |
--terminate-on-failure | Stop processing after the first failing item. On its own this does not fail CI — exit non-zero from your experiment file (e.g. on a REGRESSION verdict) to gate the pipeline |
--watch | Re-run on file save (incompatible with --matrix-param and --list) |
--runner CMD | Override auto-detected runner (tsx, node, bun, python3) |
--env-file PATH | Load environment from a specific .env file |
--metadata JSON | Attach metadata to the run, e.g. '{"branch":"main","pr":42}'. Git/CI context (commit, branch, author) is auto-collected separately as ciMetadata |
--comparison-timeout S | Max seconds to wait for baseline comparison data (default: 30) |
Parameter injection
Pass named parameters to the experiment without editing the file:
# Single parameter
aievals experiment-run run . --param model=gpt-4o
# Multiple parameters
aievals experiment-run run . --param model=gpt-4o --param temperature=0.7Parameters are injected as AIEVALS_PARAMS_JSON. The task function must read this env var explicitly; evaluators receive params automatically via a params argument.
Matrix sweep
Run a Cartesian product of parameter values in one command — the CLI creates a separate experiment run for each combination:
# 2 models × 3 temperatures = 6 runs
aievals experiment-run run . \
--matrix-param model=gpt-4o,gpt-4o-mini \
--matrix-param temperature=0.0,0.5,1.0Run aievals provider list --format json to discover valid model names before using --matrix-param model=....
Verifying results
aievals experiment list # find the experiment
aievals experiment-run list --experiment-id <id> # list runs
aievals experiment-run compare <run-id> # compare to baseline
aievals experiment-run get <run-id> --format json # full run detailsSee SDK: Local Experiment Runs for the SDK API used inside experiment files.
LLM evaluator tips
--score-range-promptmust be exactly:Provide a score ranging from 0 to 1orProvide a discrete score of 0 or 1.- Do not use
score listfor experiment scores — it returns empty. Use the trace-based flow in the final step. - Do not use
eval executeto debug experiment scores — useexperiment-run compareinstead.
Code evaluator tips
- Shell safety: avoid
!==(use!=), avoid!in strings (use== false). For complex code, write it to a file and pass it via--code "$(cat file.js)".
Long-running operations
Commands like experiment-run create --wait poll until completion (default
timeout: 10 minutes). Use --wait-timeout 300 to adjust.