BrowserStack Local & API Registration

When something you want to evaluate isn't publicly reachable — it runs on localhost, inside a VPN, or behind a corporate firewall — BrowserStack Local bridges your private network to the BrowserStack infrastructure so evals can call it. You run the Local binary on your machine and bind a private environment to it; requests to that environment are then proxied through your tunnel.

A private environment can front two kinds of endpoint:

• Agents — your application's own API (the system under test) for experiments and the Conversation Simulator.
• Private LLM models — a self-hosted or on-prem model behind an OpenAI-compatible endpoint.

1. Download BrowserStack Local

The BrowserStack Local binary establishes a secure connection between your private network and the BrowserStack infrastructure.

Download BrowserStack Local Binary

2. Run BrowserStack Local

Authenticate and run the binary to open the tunnel:

./BrowserStackLocal <your-access-key>

3. Point an agent at your private endpoint

With the tunnel open, register the endpoint as an agent and bind it to a private environment so its requests are routed through your tunnel:

Once the agent responds, use it as the API Configuration in an experiment or as the target of a conversation simulation.

Point an LLM connection at a private model

To evaluate a self-hosted or on-prem model, create the private environment on the Private Environments tab of the LLM Connections page, run the Local binary, then add an LLM connection with your model's API Base URL (for example http://localhost:11434/v1) and set its Connection to that environment. The model is then usable in the playground, evaluators, and experiment runs.

Register an API with an OpenAPI spec

As an alternative to registering through the dashboard, you can define your endpoint in an api.yaml file and submit it via the API — useful for scripted or CI-driven setup.

Step A: Define the API (api.yaml)

Modify and save the following configuration as api.yaml based on your API schema. This defines the endpoints for query execution and retrieval, in standard OpenAPI format.

openapi: 3.0.3
info:
  title: Local URL for testing
  description: >
    A RESTful API for running LLM queries.
  version: 1.0.0
servers:
  - url: http://localhost:8000
    description: Staging server
paths:
  /query:
    post:
      summary: Run Query
      operationId: query
      parameters:
        - name: Authorization
          in: header
          required: true
          description: Bearer token for authentication
          schema:
            type: string
            example: "Bearer <access-token>"
      requestBody:
        description: Query request
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - input
              properties:
                input:
                  type: string
                  example: "{{input}}"
      responses:
        '200':
          description: Query successful
          content:
            application/json:
              schema:
                type: object
                required:
                  - llmOutput
                  - retrievedContexts
                properties:
                  llmOutput:
                    type: string
                    # Defines which JSON node in the response will contain the LLM response
                    example: response.output
                  retrievedContexts:
                    type: string
                    # Optional: Defines which JSON node contains the retrieval context
                    example: response.retrieval_context
        '401':
          description: Unauthorized - Invalid or missing bearer token

Input Configuration

To feed data into your API during a run, map the input field to a column in your dataset.

Syntax: Use double curly braces (e.g., {{input}}) to reference the specific dataset column.

Output Configuration

There are two methods to capture the output from your LLM:

Option A: Direct API Response

Use this method if your API returns the LLM response directly in the JSON body.

• Configuration: Define the llmOutput property in the api.yaml.
• Mapping: Specify which JSON node contains the text (e.g., response.output). Refer to the example api.yaml provided above.

Option B: Async SDK Trace

Use this method if the SDK is integrated and the output needs to be picked up from a trace generated during the experiment or dataset run.

• Configuration: Include the AsyncTraceObservationName property in the api.yaml.
• Mapping: Specify the name of the observation trace. Example snippet below.

AsyncTraceObservationName:
   type: string
   description: An optional field for tracing async operations
   example: chat_completion

Step B: Submit the Registration

Once the file is ready, register it using the command line.

curl -u "<public-key>:<secret-key>" -X POST \
https://evals.browserstack.com/api/public/experiments/openapi \
-F "file=@./api.yaml"

Related

• Agents — register and manage the endpoints evals calls.
• LLM Connections — connect provider and self-hosted models.
• Create an Experiment — evaluate an agent against a dataset.