Run a full evaluation from the terminal — discover providers, build a dataset and evaluators, create a prompt and experiment, then read the scores.

Experiment Workflow

Provider discovery

Always run provider discovery before creating prompts or LLM evaluators:

aievals provider list --format json

Use the exact name field (case-sensitive) for --model-params-provider. The name and adapter fields can differ (e.g., name="OpenAI" vs. adapter="openai").

Using the wrong provider value causes silent 0 scores on LLM evaluators. Always copy values from provider list — never guess, hardcode, or lowercase them.

The workflow

Discover providers

aievals provider list --format json

Discover available providers and models, and the exact values to use below.

Create a dataset

aievals dataset create
# then add items with:
aievals dataset-item create

Create evaluator(s)

aievals evaluator create

Code evaluators must use function main(...), not function evaluate(...). All params (input, output, expected) are strings — use JSON.parse() to access fields.

Create a metric

aievals metrics create

Groups evaluators (called "Metrics" in the UI, evaluator-list in the API).

Create a prompt

aievals prompt create \
  --model-params '{"provider":"OpenAI","adapter":"openai","model":"gpt-4o-mini"}'

Use exact values from Step 1.

Create the experiment

aievals experiment create

Links dataset + prompt + metrics.

Run it

aievals experiment-run create --experiment-id <id> --wait

Runs and waits for results.

Read the scores

aievals experiment-run compare <run-id> --format json

Returns aggregate scores per evaluator. For individual scores:

# get score IDs from each trace's `scores` array
aievals trace list --format json
# then fetch a single score
aievals score get <score-id> --format json

LLM evaluator tips

--score-range-prompt must be exactly: Provide a score ranging from 0 to 1 or Provide a discrete score of 0 or 1.
Do not use score list for experiment scores — it returns empty. Use the trace-based flow in the final step.
Do not use eval execute to debug experiment scores — use experiment-run compare instead.

Code evaluator tips

Shell safety: avoid !== (use !=), avoid ! in strings (use == false). For complex code, write it to a file and pass it via --code "$(cat file.js)".

Long-running operations

Commands like experiment-run create --wait poll until completion (default timeout: 10 minutes). Use --wait-timeout 300 to adjust.

Experiment Workflow

On this page