Create and manage evaluation datasets, items with pre-captured outputs, and experiments that score them.

Evaluation Datasets API

Evaluation datasets store collections of input + output pairs that have already been generated. Evaluators score the existing outputs directly — no prompt or API is re-run when an experiment is launched.

Note: Evaluation datasets share the same HTTP endpoints as regular Datasets, discriminated by the kind field. There is no separate /api/public/evaluation-datasets URL prefix.

Evaluation Datasets

Create Evaluation Dataset

POST /api/public/datasets

Field	Type	Required	Description
`name`	`string`	Yes	Unique dataset name
`kind`	`string`	Yes	Must be `"EVALUATION"` (defaults to `"REGULAR"` if omitted)
`description`	`string`	No	Human-readable description
`metadata`	`object`	No	Arbitrary metadata

kind is immutable after creation — it cannot be changed via update.

curl -X POST https://evals-api.browserstack.com/api/public/datasets \
  -u "pk-lf-...:sk-lf-..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-prod-traces",
    "kind": "EVALUATION",
    "description": "Captured support responses from production",
    "metadata": { "source": "prod-traces", "window": "2026-04" }
  }'

Response:

{
  "id": "ds-uuid-eval-1",
  "name": "support-prod-traces",
  "kind": "EVALUATION",
  "description": "Captured support responses from production",
  "metadata": { "source": "prod-traces", "window": "2026-04" },
  "projectId": "proj-xyz",
  "createdAt": "2026-04-03T10:00:00.000Z",
  "updatedAt": "2026-04-03T10:00:00.000Z"
}

List Evaluation Datasets

GET /api/public/datasets?kind=EVALUATION

Parameter	Type	Description
`kind`	`string`	Pass `EVALUATION` to return only evaluation datasets. Omit to default to `REGULAR`.
`page`	`integer`	Page number
`limit`	`integer`	Items per page

curl "https://evals-api.browserstack.com/api/public/datasets?kind=EVALUATION&page=1&limit=20" \
  -u "pk-lf-...:sk-lf-..."

Get Evaluation Dataset

GET /api/public/datasets/{datasetName}

The response includes kind: "EVALUATION" for evaluation datasets.

curl "https://evals-api.browserstack.com/api/public/datasets/support-prod-traces" \
  -u "pk-lf-...:sk-lf-..."

Evaluation Dataset Items

Items in an evaluation dataset must include an output field — the captured response that evaluators will score.

Create Evaluation Dataset Item

POST /api/public/dataset-items

Field	Type	Required	Description
`datasetName`	`string`	Yes	Target evaluation dataset name
`input`	`any`	Yes	Input data for this case
`output`	`any`	Yes	The captured response that evaluators score
`expectedOutput`	`any`	No	Reference answer for evaluators that need ground truth
`context`	`any`	No	Context information
`metadata`	`any`	No	Arbitrary metadata
`id`	`string`	No	Custom item ID
`sourceTraceId`	`string`	No	Trace this item was sourced from
`sourceObservationId`	`string`	No	Observation this item was sourced from
`status`	`string`	No	`ACTIVE` (default) or `ARCHIVED`

output is the field that makes an item suitable for an evaluation dataset. It is silently ignored on REGULAR dataset items.

curl -X POST https://evals-api.browserstack.com/api/public/dataset-items \
  -u "pk-lf-...:sk-lf-..." \
  -H "Content-Type: application/json" \
  -d '{
    "datasetName": "support-prod-traces",
    "input": { "question": "How do I cancel my subscription?" },
    "output": { "answer": "Go to Settings → Billing → Cancel." },
    "expectedOutput": { "answer": "Cancel from Settings → Billing." },
    "metadata": { "source_trace": "trace_123" }
  }'

Response:

{
  "id": "item-uuid-1",
  "datasetId": "ds-uuid-eval-1",
  "datasetName": "support-prod-traces",
  "status": "ACTIVE",
  "input": { "question": "How do I cancel my subscription?" },
  "output": { "answer": "Go to Settings → Billing → Cancel." },
  "expectedOutput": { "answer": "Cancel from Settings → Billing." },
  "context": null,
  "metadata": { "source_trace": "trace_123" },
  "sourceTraceId": null,
  "sourceObservationId": null,
  "createdAt": "2026-04-03T10:00:00.000Z",
  "updatedAt": "2026-04-03T10:00:00.000Z"
}

List Evaluation Dataset Items

GET /api/public/dataset-items?datasetName={datasetName}

Same endpoint as regular dataset items — filter by datasetName to scope to a single evaluation dataset. The output field is populated in the response.

curl "https://evals-api.browserstack.com/api/public/dataset-items?datasetName=support-prod-traces" \
  -u "pk-lf-...:sk-lf-..."

Get Evaluation Dataset Item

GET /api/public/dataset-items/{datasetItemId}

curl "https://evals-api.browserstack.com/api/public/dataset-items/item-uuid-1" \
  -u "pk-lf-...:sk-lf-..."

Delete Evaluation Dataset Item

DELETE /api/public/dataset-items/{datasetItemId}

curl -X DELETE "https://evals-api.browserstack.com/api/public/dataset-items/item-uuid-1" \
  -u "pk-lf-...:sk-lf-..."

Response:

{ "message": "Dataset item successfully deleted" }

Run an Experiment on an Evaluation Dataset

Evaluation datasets do not use Dataset Runs. Instead, you create an experiment that points at the evaluation dataset; the platform scores each item's pre-captured output directly with the chosen evaluator list. No prompt or API record is needed — the service layer enforces that combination automatically based on the dataset's kind.

POST /api/public/experiments

Field	Type	Required	Description
`name`	`string`	Yes	Experiment name
`datasetId`	`string`	Yes	ID of the evaluation dataset
`evaluatorListId`	`string`	Yes	Evaluator list to score the items with
`description`	`string`	No	Human-readable description
`concurrency`	`integer`	No	Items scored in parallel (1–10)
`datasetVersion`	`integer \| null`	No	Pin to a version. `null`/omit = latest

Do not send promptId, apiRecordId, or datasetRunTagId when the dataset is kind=EVALUATION — the request is rejected by the service layer because the output is already on the item.

curl -X POST https://evals-api.browserstack.com/api/public/experiments \
  -u "pk-lf-...:sk-lf-..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "april-replay-with-v2-rubric",
    "datasetId": "ds-uuid-eval-1",
    "evaluatorListId": "evl_answer_quality",
    "description": "Re-score April production traces with the v2 rubric",
    "concurrency": 5
  }'

Response:

{
  "id": "exp-uuid-1",
  "name": "april-replay-with-v2-rubric",
  "datasetId": "ds-uuid-eval-1",
  "evaluatorListId": "evl_answer_quality",
  "status": "PROCESSING",
  "createdAt": "2026-04-03T10:00:00.000Z"
}

For polling, listing runs, and reading scored results, see the Experiments API.

Evaluation Datasets API

On this page