Auto-Stop & Pre-Run Checks

TestAI includes two safety mechanisms to help you catch problems early and avoid wasting compute on a doomed run:

1. Fail-fast threshold — automatically stops a run (status STOPPED) once a configurable percentage of work fails, rather than processing every dataset item.
2. Pre-run API healthcheck — for API/agent experiments, probes the configured endpoint when you start a run so a dead endpoint is caught before any items are dispatched.

---

Fail-Fast Threshold

What it does

When the percentage of failed dataset items or evaluations crosses the threshold, the run transitions to STOPPED and no further items are dispatched. Items already in progress finish first, so the final failed count can slightly exceed the threshold. Scores are hidden for auto-stopped runs because the evaluation was incomplete.

The STOPPED status is intentional — it is not FAILED. Individual items keep their own status; the run itself is marked STOPPED with the reason recorded in the run details.

Enable fail-fast for an experiment

Fail-fast is set on the experiment when you create it:

The threshold is checked live during the run, not snapshotted at run start.

What a stopped run looks like

When the threshold is crossed, the run page shows a banner:

Stopped: Auto-stopped: [unit] failures crossed the [N]% failure threshold. Scores are hidden because evaluation was incomplete.

The banner also links to filter the run's items by failure stage:

• View N failed dataset items — items that failed during generation/dispatch
• View N failed evaluations — items that failed during the evaluation phase

In the run list, the run's status shows STOPPED.

Disabling fail-fast

Leave the Enable fail-fast for this experiment toggle off when creating the experiment (this is the default). An experiment with fail-fast off processes every dataset item regardless of how many fail.

---

Pre-Run API Healthcheck

For experiments that call an API/agent endpoint, TestAI checks that the endpoint is reachable before a run starts — so a dead endpoint is caught up front instead of failing every item.

When it appears

The check runs when you start a run (the Run action on the experiments list, the run analytics header, or the experiment's runs page). It does not run while you are creating the experiment.

• Reachable — the run starts immediately; you see no dialog.
• Unreachable — a Check your agent endpoint dialog appears before the run is created.

Reachability, not correctness

The probe only confirms the endpoint is reachable — it is not a validation of your request or response:

• Any HTTP response counts as reachable, including 4xx and 5xx. The empty probe is expected to be rejected by your endpoint; the real run sends the proper request.
• Only a connection failure is treated as unreachable: connection refused, timeout, DNS failure, or a TLS error. The dialog surfaces the specific reason (for example, ECONNREFUSED).
• The probe is a single lightweight request used only to confirm the endpoint answers; it is not a test of your actual request or response.
• For private environments using BrowserStack Local, the probe routes through the same tunnel the run uses.
• The check is bounded by a 15-second timeout and is advisory — you can always proceed.

The unreachable dialog

When the endpoint can't be reached, the Check your agent endpoint dialog offers:

• Retry — probe again (useful after restarting the agent or fixing its config).
• Create run anyway — start the run despite the warning. The run is created regardless of the result.
• Don't show this again for this experiment — suppress the dialog for this experiment. The dismissal is tied to the linked API record, so the dialog re-appears if that API record is changed.

---

Related

• Create an Experiment — full create wizard walkthrough
• Experiment Runs — run status values, polling, and auto-stopped runs