EvaluationDifference between Datasets and Evaluation Datasets
Difference between Datasets and Evaluation Datasets
Side-by-side comparison of Datasets and Evaluation Datasets and when to use which.
Difference between Datasets and Evaluation Datasets
Both Datasets and Evaluation Datasets live in the same Dataset Library and feed the same experiment pipeline. The difference is whether an experiment needs to generate outputs or just score them.
Comparison
| Datasets | Evaluation Datasets | |
|---|---|---|
Items have an output column | No | Yes (required) |
| What an experiment does | Runs a prompt or API on each input to generate outputs | Forwards the captured output straight into the run — no model call |
| LLM cost per experiment | One call per item | Zero |
| Best for | Comparing prompts, models, or APIs against the same inputs | Re-scoring an existing output set with new evaluators or metrics |
| Typical source of items | Hand-authored, imported from CSV, or pulled from traces | Production traces, an offline batch run, or an external system you want to grade |
| CSV upload requires | input column | input and output columns |
| Has Dataset Runs | Yes — runs group execution results per dataset | No — experiments are the entry point instead |
| SDK client | testOps.datasets / client.datasets / sdk.datasets() | testOps.evaluationDataset / client.evaluation_dataset / sdk.evaluationDataset() |
When to use which
- Pick a Dataset when you want to compare how different prompts, models, or APIs perform on the same set of inputs. Each experiment will execute the prompt or API per item and produce fresh outputs.
- Pick an Evaluation Dataset when the outputs already exist and you want to score them — for example, trying new evaluators on yesterday's production responses, replaying a captured set against an updated rubric, or grading outputs from a system you don't want to re-invoke.
Quick rule of thumb: if you can answer "what produces the output for each item?" with a prompt or API in TestAI, use a Dataset. If the outputs already exist and you just want to grade them, use an Evaluation Dataset.