EvaluationConversation Simulator
Conversation Simulator
Test multi-turn conversational agents by simulating realistic users against your scenarios and scoring the whole conversation.
Conversation Simulator
The Conversation Simulator tests conversational agents — chatbots, copilots, and deployed assistants — the way real users exercise them: over a full multi-turn conversation, not a single request/response. You define scenarios (who the user is, what they want, and how they open), point the simulator at your agent, and it drives each scenario as a back-and-forth conversation, then scores the result with conversation-level evaluators.
When to use
- You are building a chatbot, copilot, or agent that talks with users over multiple turns.
- Single-turn dataset experiments don't capture failures that only appear across a conversation — losing context, drifting off policy, or never reaching the user's goal.
- You want to test the same agent against many user types (cooperative, confused, adversarial) without hand-writing every conversation.
Core concepts
| Concept | Description |
|---|---|
| Persona | A reusable user profile the simulator roleplays — its traits, tone, and an example opening utterance. Personas are referenced by scenario rows, not chosen at run time. |
| Scenario | One test case: a persona, a goal the user is trying to achieve, and an opening message that starts the conversation. Optionally an expected outcome. |
| Scenarios dataset | A conversation-shape dataset whose rows are scenarios. This is what an experiment runs against. |
| Agent | The system under test. The agent is an API endpoint that carries its own grounding (a description plus uploaded docs), so it doubles as the target you point the simulator at. |
| Simulator model | The LLM that plays the user in each simulated conversation, driving turns toward the scenario's goal. |
How it works end-to-end
- Define personas — the user profiles the simulator will roleplay. See Personas.
- Build a scenarios dataset — add scenario rows manually, or generate a set with Assist. Generated scenarios are based on the linked agent's description, so they stay on-topic for what your agent does. See Generate scenarios.
- Run the simulator via an experiment — pick the agent to simulate, the scenarios dataset, a simulator model, and the metrics to score. See Run the simulator.
- Review results — inspect per-conversation transcripts, per-persona breakdowns, and aggregate scores.
Related
- Experiments — how experiments provide and score outputs.
- Datasets — regular input/expected-output datasets.
- Evaluator Types — including the conversational LLM-as-a-Judge evaluator.