Annotation Queues
Create and manage annotation queues for routing traces to human reviewers.
Annotation Queues
An annotation queue is an ordered list of traces (or observations) that need human review. Each queue has:
- A name and optional instructions for reviewers
- One or more score configs that define what reviewers should score
- Items in
PENDINGorCOMPLETEDstate - Organized into runs (batches of annotation work)
Creating a Queue
Open Annotation Queues
Navigate to Annotation Queues in the left sidebar.
Create a new queue
Click + New Queue and enter a name, optional description, and reviewer instructions.
Select score configs
Choose which score configurations reviewers will fill in for each item. Score configs define the type of label (numeric, categorical, boolean, or text) and any constraints.
Save the queue
Click Create. The queue appears in your queue list and is ready to receive items.
curl -X POST https://evals-api.browserstack.com/api/public/annotation-queues \
-u "pk-lf-...:sk-lf-..." \
-H "Content-Type: application/json" \
-d '{
"name": "Production Quality Review",
"description": "Manual review of customer-facing traces",
"scoreConfigIds": ["config-uuid-1", "config-uuid-2"]
}'Response:
{
"id": "queue-uuid-1",
"name": "Production Quality Review",
"description": "Manual review of customer-facing traces",
"scoreConfigIds": ["config-uuid-1", "config-uuid-2"],
"createdAt": "2026-04-07T10:00:00.000Z",
"updatedAt": "2026-04-07T10:00:00.000Z"
}Listing and Fetching Queues
# List all queues
curl "https://evals-api.browserstack.com/api/public/annotation-queues?page=1&limit=50" \
-u "pk-lf-...:sk-lf-..."
# Get a specific queue
curl "https://evals-api.browserstack.com/api/public/annotation-queues/queue-uuid-1" \
-u "pk-lf-...:sk-lf-..."Managing Queue Items
Queue items are the individual traces or observations assigned for review.
Add an Item to a Queue
curl -X POST "https://evals-api.browserstack.com/api/public/annotation-queues/queue-uuid-1/items" \
-u "pk-lf-...:sk-lf-..." \
-H "Content-Type: application/json" \
-d '{
"objectId": "trace-uuid-1",
"objectType": "TRACE"
}'| Field | Type | Required | Description |
|---|---|---|---|
objectId | string | Yes | ID of the trace or observation to review |
objectType | string | Yes | TRACE or OBSERVATION |
runId | string | No | Assign to a specific run (auto-created if omitted) |
status | string | No | Initial status — PENDING (default) |
List Items in a Queue
curl "https://evals-api.browserstack.com/api/public/annotation-queues/queue-uuid-1/items?status=PENDING&page=1&limit=50" \
-u "pk-lf-...:sk-lf-..."| Parameter | Type | Description |
|---|---|---|
page | integer | Page number (default: 1) |
limit | integer | Items per page (default: 50) |
status | string | Filter by PENDING or COMPLETED |
runIds | string | string[] | Filter to specific runs |
Delete an Item
curl -X DELETE "https://evals-api.browserstack.com/api/public/annotation-queues/queue-uuid-1/items/item-uuid-1" \
-u "pk-lf-...:sk-lf-..."Item Status Values
| Status | Description |
|---|---|
PENDING | Waiting for a reviewer |
COMPLETED | Reviewer has submitted a score |
Object Types
objectType | Description |
|---|---|
TRACE | Item references a top-level trace |
OBSERVATION | Item references a specific span or generation within a trace |
Annotation Runs
Runs are named batches of annotation work within a queue. They let you group items by time period, reviewer assignment, or data slice — making it easier to track progress and compare results across rounds.
When you add an item to a queue without specifying a runId, the platform automatically creates or reuses the default run for that queue.
Run Structure
Each run tracks:
- Total items assigned
- Pending vs. completed item counts
- Completion percentage
- Score averages and distributions for completed items
Viewing Runs in the Dashboard
Open a queue and switch to the Runs tab to see all runs with their completion stats. Click a run to filter the item list to only items in that run.
Use multiple runs to represent annotation rounds. For example: round-1-sample, round-2-validation, adversarial-cases.