EvaluationEvaluators

Evaluators

Create evaluator lists and run inline evaluations across TypeScript, Python, and Java SDKs.

Evaluator lists group one or more evaluation metrics that can be attached to experiments or run inline against LLM outputs.

Build evaluator lists with metrics and parameter configurations.

Run evaluations directly against LLM outputs without an experiment.

A kind of evaluator that the SDK enforces around live model calls — redact, substitute, block, or re-ask.

Run the simulator

Run a conversation-simulation experiment against your agent using a scenarios dataset, then review per-conversation results.

Evaluator Types

Reference for all evaluator types in TestAI — LLM-as-a-Judge, code-based (heuristic, tool-call, and custom), and aggregate evaluators.