Set up BrowserStack Local to test internal servers and register your API with AI Evals for LLM testing.

BrowserStack Local & API Registration

This guide details how to set up BrowserStack Local to test internal servers and register your API with AI Evals for LLM testing.

Use this when you need to test servers hosted within your private network or register a local API for automated LLM evaluations.

1. Download BrowserStack Local

The BrowserStack Local binary establishes a secure connection between your private network and the BrowserStack infrastructure.

BrowserStack Local architecture

Download BrowserStack Local Binary

2. Run BrowserStack Local

Once downloaded, you need to authenticate and run the binary to bridge the connection.

Fetch Access Key: Go to Profile Details > Authentication & Security in your BrowserStack account settings.

Link: https://www.browserstack.com/accounts/profile/details
Run the Binary: Open your terminal and execute the following command:

./BrowserStackLocal <your-access-key>

Running BrowserStack Local is required only when your API endpoint is private (i.e., running locally or behind a VPN).

3. Register Your API with AI Evals

To run LLM queries against your local setup, you must define and register your API using an api.yaml file.

Step A: Define the API (`api.yaml`)

Modify and save the following configuration as api.yaml based on your API schema. This defines the endpoints for query execution and retrieval. This is in standard OpenAPI format.

openapi: 3.0.3
info:
  title: Local URL for testing
  description: >
    A RESTful API for running LLM queries.
  version: 1.0.0
servers:
  - url: http://localhost:8000
    description: Staging server
paths:
  /query:
    post:
      summary: Run Query
      operationId: query
      parameters:
        - name: Authorization
          in: header
          required: true
          description: Bearer token for authentication
          schema:
            type: string
            example: "Bearer <access-token>"
      requestBody:
        description: Query request
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - input
              properties:
                input:
                  type: string
                  example: "{{input}}"
      responses:
        '200':
          description: Query successful
          content:
            application/json:
              schema:
                type: object
                required:
                  - llmOutput
                  - retrievedContexts
                properties:
                  llmOutput:
                    type: string
                    # Defines which JSON node in the response will contain the LLM response
                    example: response.output
                  retrievedContexts:
                    type: string
                    # Optional: Defines which JSON node contains the retrieval context
                    example: response.retrieval_context
        '401':
          description: Unauthorized - Invalid or missing bearer token

Input Configuration

To feed data into your API during a run, you must map the input field to a column in your dataset.

Syntax: Use double curly braces (e.g., {{input}}) to reference the specific dataset column.

Output Configuration

There are two methods to capture the output from your LLM:

Option A: Direct API Response

Use this method if your API returns the LLM response directly in the JSON body.

Configuration: Define the llmOutput property in the api.yaml.
Mapping: Specify which JSON node contains the text (e.g., response.output). Refer to the example api.yaml provided above.

Option B: Async SDK Trace

Use this method if the SDK is integrated and the output needs to be picked up from a trace generated during the experiment or dataset run.

Configuration: Include the AsyncTraceObservationName property in the api.yaml.
Mapping: Specify the name of the observation trace. Example snippet below.

AsyncTraceObservationName:
   type: string
   description: An optional field for tracing async operations
   example: chat_completion

Step B: Submit the Registration

Once the file is ready, register it using the command line.

Get Credentials: Create and fetch the Public Key and Secret Key for your project from the BrowserStack dashboard.
Run Curl Command: Navigate to the folder containing api.yaml and run:

curl -u "<public-key>:<secret-key>" -X POST \
https://evals.browserstack.com/api/public/experiments/openapi \
-F "file=@./api.yaml"

Please connect with the AI Evals team for a one-time setup process before running experiments with any private API.

Updating Tokens: To update your Authentication access token later, simply edit the api.yaml file with the new value and resubmit the curl request above.

BrowserStack Local & API Registration

On this page