# Scenarios
Source: https://docs.arklex.ai/v0.3.x/build-scenario
Define test scenarios in ArkSim by specifying the simulated user's profile, goal, and prior knowledge for realistic agent conversations.
## What is a Scenario?
A scenario is a test case that describes who the simulated user is, what they want to accomplish, and what they already know going into the conversation with the agent.
![ArkSim Workflow]()
## The Three Core Pillars
### Goal
Describes the user's objective, guiding their actions and decisions throughout the conversation.
> Plan a purchase of a service robot, focusing on delivery timelines and suitability for commercial use.
### User Profile (Persona)
Defines who the simulated user is — their personality, demographics, and behavior — as a natural-language description written in second person.
> You are Alex, a 26-year-old operations analyst at a mid-sized logistics company in Chicago. You are analytical, detail-oriented, budget-conscious, and risk-aware.
### Knowledge
Provides context that shapes the simulated user's questions and expectations. Also used during **evaluation** to check whether the agent's responses were accurate and consistent with the source.
> * `home_insurance_overview.md` \
> The service robot is a 40 kg payload autonomous service robot with 8-hour battery life and \$70,000 price…
> * `purchase_options.pdf` \
> The payment plans include full purchase, 12-month leasing, and vendor financing…
Think of a scenario as answering three questions: **Who** is this user? **What** do they want? **What** do they know?
## Why ArkSim Scenarios?
ArkSim Scenarios let you test and evaluate your agent using realistic, profile-driven simulations of user interactions. Each scenario is built around clearly defined goals and behaviors, enabling structured performance evaluation without requiring a real-world dataset.
By using synthetic users, you can run consistent and reproducible tests across a wide range of interaction types, making it easy to scale evaluations and understand how your agent performs before deploying it to real users.
***
## Scenario File Structure
A scenario file is a single JSON document containing a list of scenario objects. Each scenario represents one simulated user and one conversation session.
```json theme={null}
{
"schema_version": "v1",
"scenarios": [
{ ... },
{ ... }
]
}
```
For the full scenario schema (all fields and types), see [Schema Reference](./schema-reference#scenarios-scenariosjson).
***
## Full Example
```json theme={null}
{
"schema_version": "v1",
"scenarios": [
{
"scenario_id": "123456",
"user_id": "123456",
"goal": "You want to ask detailed questions about XYZ Insurance car coverage options in your province, including limits, deductibles, and what's actually covered.",
"knowledge": [
{
"content": "Car Insurance Overview: XYZ Insurance offers mandatory and optional coverages including third-party liability, collision, and comprehensive...",
"metadata": {
"title": "Car Insurance Overview",
"source": "https://www.xyzinsurance.com/products-services/auto-car-insurance",
"knowledge_type": "car_insurance_overview",
"doc_id": "car_insurance_overview_001"
}
}
],
"user_profile": "You are Priya Chen who is open to experience, conscientious, introverted, agreeable, and emotionally stable. You are a 34-year-old single woman living in Toronto, ON, Canada. As a prospective customer who found XYZ Insurance through an online search, you are analytical and value-oriented with moderate spending habits.",
"origin": {
"target_agent_capability": "Answer questions about XYZ Insurance products and coverage",
"goal_raw": "Ask question about XYZ insurance",
"user_attributes": {
"sex": "Female",
"age": "34",
"marital_status": "single",
"household_size": "1",
"location": "Toronto, ON, Canada",
"Household_income": "$50,000 - $74,999",
"education": "Bachelor's degree",
"Openness": "open to experience",
"Conscientiousness": "conscientious",
"Extraversion": "introverted",
"Agreeableness": "agreeable",
"Neuroticism": "emotionally stable",
"customer_type": "prospective customer",
"discovery_type": "online search",
"decision_making_style": "analytical",
"purchasing_preference": "value oriented",
"spending_behavior": "moderate spending",
"loyalty_level": "neutral"
}
}
}
]
}
```
***
## Writing Scenarios
You can write scenario files manually. The required fields are `scenario_id`, `user_id`, `goal`, `agent_context`, and `user_profile`:
```json theme={null}
{
"schema_version": "v1",
"scenarios": [
{
"scenario_id": "my-scenario-001",
"user_id": "user-001",
"goal": "You want to find out whether your home insurance policy covers water damage from a burst pipe.",
"agent_context": "You are a helpful home insurance assistant for XYZ Insurance.",
"user_profile": "You are Jordan, an existing customer from Calgary, AB, Canada. You are detail-oriented and want a clear answer.",
}
]
}
```
**Tips**
* Write `goal` in second person ("You want to...") so it reads naturally as a simulator instruction.
* `knowledge` can be omitted if the simulated user has no specific background information.
* `origin` can be omitted for hand-authored scenarios.
***
## Multi-Knowledge Scenarios
A scenario can include multiple knowledge items by adding additional entries to the `knowledge` list. These items are directly injected into the simulated user's context — there is no retrieval step or external knowledge base involved.
```json theme={null}
"knowledge": [
{
"content": "Home Insurance Basics: XYZ Insurance provides coverage for your dwelling, contents, and liability...",
"metadata": { "title": "Home Insurance Basics" }
},
{
"content": "Claims Process: To file a claim, you can call the XYZ Insurance claims line or use the online claims portal...",
"metadata": { "title": "How to File a Claim" }
}
]
```
***
## How Fields Are Used Downstream
Understanding how each field is used helps you decide what to include.
| Field | Used By | Description |
| ---------------------- | --------------------- | -------------------------------------------------------------------- |
| `goal` | Simulator & Evaluator | Directly used in the simulator prompt; the most important field. |
| `knowledge[].content` | Simulator & Evaluator | Provides simulated user's background knowledge for the conversation. |
| `user_profile` | Simulator | Used as-is in the simulated user prompt to define the persona. |
| `knowledge[].metadata` | Not used by ArkSim | For labeling and traceability only. |
| `origin` | Not used by ArkSim | Tracks scenario provenance; reference only. |
## Next Steps
Once your scenario file is ready, you're ready to run your scenarios and review the results.
Execute your scenarios and simulate conversations for review and analysis.
# CI Integration
Source: https://docs.arklex.ai/v0.3.x/ci-integration
Run ArkSim as an automated quality gate in your CI pipeline so every code change is tested against your agent's quality bar before it ships.
## Why run ArkSim in CI?
Agents break in non-obvious ways. A prompt tweak, a model upgrade, or a dependency change can silently degrade helpfulness, introduce false information, or cause goal failures — without triggering any unit test.
Running ArkSim on every pull request turns quality regression into a CI signal:
| Without ArkSim in CI | With ArkSim in CI |
| ------------------------------------- | ------------------------------------- |
| Regressions found in production | Regressions caught before merge |
| Manual spot-checking before releases | Automated evaluation on every PR |
| Subjective "looks good to me" reviews | Objective, reproducible metric scores |
| No record of quality over time | Artifact history per commit |
***
## Choose your approach
| | pytest (custom agent) | HTTP server |
| ----------------- | --------------------------------------------- | --------------------------------------------------------- |
| **How it works** | ArkSim loads your agent class in-process | ArkSim calls your agent over HTTP |
| **Agent type** | Python class subclassing `BaseAgent` | Any HTTP server (any language or framework) |
| **CI complexity** | Simple — just run `pytest` | Requires starting, health-checking, and stopping a server |
| **Template** | `arksim-pytest.yml` + `test_agent_quality.py` | `arksim.yml` |
***
## Approach 1: pytest with a custom agent
Your agent is a Python class that subclasses `BaseAgent`. ArkSim loads it in-process — no HTTP server needed. The pytest test runs simulation and evaluation directly and asserts scores.
### How it works
```
GitHub Actions runner
│
├── 1. Checkout + install dependencies
├── 2. python -m pip install arksim pytest pytest-asyncio
├── 3. python -m pip install -r requirements.txt
├── 4. pytest tests/test_agent_quality.py
│ ├── Simulate N conversations (agent runs in-process)
│ ├── Evaluate each conversation with an LLM judge
│ ├── Generate HTML report
│ └── Assert quality thresholds → pass or fail
└── 5. Upload artifacts (always runs)
├── arksim-html-report (final_report.html)
└── arksim-full-results (simulation + evaluation JSONs)
```
### Setup
Your agent class must implement two methods:
```python theme={null}
from arksim.simulation_engine.agent.base import BaseAgent
class MyAgent(BaseAgent):
async def get_chat_id(self) -> str:
# Return a stable ID for the current conversation session
return str(id(self))
async def execute(self, user_query: str, **kwargs) -> str:
# Your agent logic — return the agent's reply as a string
...
```
If you already have an existing agent, wrap it in a thin adapter:
```python theme={null}
class MyAgentAdapter(BaseAgent):
def __init__(self, agent_config):
super().__init__(agent_config)
self.agent = MyExistingAgent()
async def get_chat_id(self) -> str:
return str(id(self))
async def execute(self, user_query: str, **kwargs) -> str:
return await self.agent.process(user_query)
```
Download the GitHub Actions example:
```bash theme={null}
arksim examples ci
```
Copy the pytest workflow and test template into your repo:
```bash theme={null}
mkdir -p .github/workflows tests
cp examples/ci/pytest/arksim-pytest.yml .github/workflows/arksim-pytest.yml
cp examples/ci/pytest/test_agent_quality.py tests/test_agent_quality.py
```
Create a `conftest.py` in your `tests/` directory so pytest can resolve your agent and metrics modules:
```python theme={null}
# tests/conftest.py
import sys
import os
# Add the project root to sys.path so imports like
# `from my_agent import MyAgent` resolve correctly.
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
```
Create `tests/arksim/scenarios.json` with test cases representative of your agent's real usage:
```
tests/
├── test_agent_quality.py
└── arksim/
├── scenarios.json ← test inputs (committed)
└── results/ ← outputs (add to .gitignore)
```
See [Build Scenarios](./build-scenario) for the schema, and [`examples/bank-insurance`](https://github.com/arklexai/arksim/tree/main/examples/bank-insurance) for a full example.
Open `tests/test_agent_quality.py` and update:
* Import your agent class (replace `from my_agent import MyAgent`)
* Import your custom metrics if needed (replace `from my_metrics import ...`), or remove the custom metrics block entirely to use built-in metrics only
* Set `NUMERIC_THRESHOLDS` and `QUALITATIVE_FAILURE_LABELS` to match your quality bar
* Adjust `num_convos_per_scenario`, `max_turns`, and `num_workers` as needed
In your repo → **Settings → Secrets and variables → Actions**:
| Secret | Purpose |
| ---------------- | -------------------------------------- |
| `OPENAI_API_KEY` | LLM ArkSim uses to evaluate your agent |
The workflow triggers automatically on every push to `main` and on every PR. A failing assertion fails the job.
***
## Approach 2: HTTP server
Your agent runs as an HTTP server exposing an OpenAI-compatible chat completions endpoint. ArkSim calls it over HTTP during CI — works with any language or framework.
### How it works
```
GitHub Actions runner
│
├── 1. Checkout + install dependencies
├── 2. pip install arksim
├── 3. pip install -r requirements.txt
├── 4. Start your agent as an HTTP server (background)
├── 5. Poll /health until HTTP 200
├── 6. arksim simulate-evaluate config.yaml
│ ├── Simulate N conversations with your agent
│ ├── Evaluate each conversation with an LLM judge
│ └── Check configured quality thresholds
├── 7. Upload artifacts (always runs)
│ ├── arksim-html-report (final_report.html)
│ └── arksim-full-results (simulation + evaluation JSONs)
└── 8. Exit code → job passes or fails
```
The CLI exits non-zero if any threshold is not met, which fails the job.
| Exit code | Meaning |
| --------- | ------------------------------------------- |
| `0` | All evaluations passed |
| `1` | Evaluation failed — a threshold was not met |
| `2` | Config / usage error |
| `3` | Internal / engine error |
### Setup
Download the GitHub Actions example:
```bash theme={null}
arksim examples ci
```
Copy the workflow template into your repo:
```bash theme={null}
mkdir -p .github/workflows
cp examples/ci/github-actions/arksim.yml .github/workflows/arksim.yml
```
Create `arksim/config.yaml` in your repo. Point the `endpoint` at the port where your agent server will listen:
```yaml theme={null}
agent_config:
agent_type: chat_completions
agent_name: my-agent
api_config:
endpoint: http://localhost:8888/v1/chat/completions
headers:
Content-Type: application/json
body:
model: my-model
messages: []
scenario_file_path: ./arksim/scenarios.json
num_conversations_per_scenario: 3
max_turns: 5
output_file_path: ./arksim/results/simulation.json
output_dir: ./arksim/results/evaluation
metrics_to_run:
- faithfulness
- helpfulness
- coherence
- relevance
- goal_completion
- agent_behavior_failure
generate_html_report: true
numeric_thresholds:
overall_score: 0.6
goal_completion: 0.8
model: gpt-4o
provider: openai
num_workers: 10
```
See [Evaluation](./evaluate-conversation) for all configuration options and the full list of available metrics.
If your agent's `requirements.txt` conflicts with ArkSim's dependencies, install them in separate virtual environments so they don't interfere:
```yaml theme={null}
- name: Install agent dependencies
run: |
python -m venv .venv-agent
.venv-agent/bin/pip install -r requirements.txt
- name: Install ArkSim
run: pip install arksim
- name: Start agent server
run: .venv-agent/bin/python my_agent_server.py --port 8888 &
```
Create `arksim/scenarios.json` with test cases representative of your agent's real usage. See [Build Scenarios](./build-scenario) for the schema, and [`examples/bank-insurance`](https://github.com/arklexai/arksim/tree/main/examples/bank-insurance) for a full example.
If you have custom metrics, add them to `arksim/custom_metrics/` and reference them in your `arksim/config.yaml`. See [Custom Metrics](./evaluate-conversation#custom-metrics) for the schema.
Open `.github/workflows/arksim.yml` and update the two `TODO` sections:
**Start agent server** — replace with your framework's startup command:
```yaml theme={null}
- name: Start agent server
run: uvicorn my_agent:app --host 0.0.0.0 --port 8888 &
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```
```yaml theme={null}
- name: Start agent server
run: python -m uvicorn my_chain_server:app --port 8888 &
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```
```yaml theme={null}
- name: Start agent server
run: python my_agent_server.py --port 8888 &
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```
```yaml theme={null}
- name: Start agent server
run: adk api_server --port 8888 &
env:
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
```
**Wait for agent** — update the health-check URL if your server uses a different route:
```yaml theme={null}
- name: Wait for agent to be healthy
run: |
for i in $(seq 1 30); do
if curl -sf http://localhost:8888/health > /dev/null 2>&1; then
exit 0
fi
sleep 2
done
exit 1
```
In your repo → **Settings → Secrets and variables → Actions**:
| Secret | Purpose |
| ---------------- | -------------------------------------------- |
| `OPENAI_API_KEY` | LLM ArkSim uses to evaluate your agent |
| `AGENT_API_KEY` | *(optional)* API key your agent server needs |
The workflow triggers automatically on every push to `main` and on every PR. The job passes or fails based on your configured thresholds.
***
## Quality gates
Configure pass/fail thresholds in your config or test file. If any gate is not met the job fails.
### Per-metric thresholds
Fail if any conversation's per-metric score falls below the minimum. Built-in turn-level metrics use a 1–5 scale; `goal_completion` and `overall_score` use 0–1:
```yaml theme={null}
numeric_thresholds:
overall_score: 0.6
goal_completion: 0.8
faithfulness: 3.5
helpfulness: 3.0
```
### Qualitative failure labels
Fail if any evaluated turn returns one of these labels:
```yaml theme={null}
qualitative_failure_labels:
agent_behavior_failure: ["false information", "disobey user request"]
prohibited_statements: ["violated"]
```
See [Threshold Gates](./evaluate-conversation#threshold-gates) for the full reference on all three gate types.
***
## Viewing the evaluation report
After every run — pass or fail — two artifacts are uploaded so you can inspect results regardless of outcome.
1. Open the workflow run in GitHub Actions.
2. Scroll to **Artifacts** at the bottom.
3. Download **arksim-html-report** → unzip → open `final_report.html` in your browser.
4. Download **arksim-full-results** for the raw simulation and evaluation JSONs (useful for programmatic analysis or diffing metric trends across runs).
The HTML report shows metric averages, per-conversation scores, and the top failure patterns, giving you enough context to know exactly what to fix.
# Contact
Source: https://docs.arklex.ai/v0.3.x/contact-email
Get in touch with the ArkSim team for support and inquiries.
## Get Help
Need assistance with ArkSim? For technical support, bug reports, or questions about using ArkSim, contact us:
**Email:** [support@arklex.ai](mailto:support@arklex.ai)
***
## Connect With Us
You can also find us on GitHub and LinkedIn.
Star the repository, open issues, and contribute.
Company news, product updates, and announcements.
# E-Commerce Customer Service Agent
Source: https://docs.arklex.ai/v0.3.x/e-commerce-customer-service-agent-evaluation
End-to-end example of using ArkSim to simulate and evaluate an e-commerce shopping assistant agent with product recommendation and order handling scenarios.
## Overview
This example walks through running ArkSim against a shopping assistant agent built for an e-commerce use case. The agent is designed to help customers navigate product discovery, orders, returns, and general shopping queries.
The example includes two ready-to-run agent setups you can test against out of the box, and a guide for plugging in your own agent once you're familiar with the setup.
***
## Example Agents
You can run the example in two ways: **Option 1** uses the OpenAI API directly (no server, minimal setup). **Option 2** runs a local agent built with the OpenAI Agents SDK (e-commerce knowledge base) via A2A, Chat Completions, or a custom agent connector.
A lightweight agent that calls the OpenAI API directly. Quick start with minimal setup.
Agent built with the OpenAI Agents SDK, backed by an e-commerce knowledge base; supports A2A, Chat Completions, or custom agent connector.
***
## Scenarios
The example ships with a set of pre-built scenarios in `scenarios.json` representing realistic e-commerce customer interactions. Each scenario defines a simulated user with a distinct persona, goal, and background knowledge drawn from product and policy documentation.
Sample goals include:
* Asking about product availability, specifications, and delivery timelines
* Checking order status and tracking a shipment
* Initiating or following up on a return or refund request
* Comparing products and asking for recommendations
Scenarios are defined in `scenarios.json` in the example directory and can be edited or extended to reflect your own use case.
***
Before following either path, ensure [ArkSim is installed](./installation) (`pip install arksim`).
## Option 1: OpenAI Agent
This agent calls the OpenAI API directly, with no server setup required.
```bash theme={null}
export OPENAI_API_KEY=""
```
Run from the `examples/e-commerce` directory:
```bash theme={null}
cd examples/e-commerce
arksim simulate-evaluate config.yaml
```
***
## Option 2: OpenAI Agents SDK
This agent is a shopping assistant built with the OpenAI Agents SDK, backed by an e-commerce knowledge base. It can be exposed via the A2A Protocol, a Chat Completions-compatible endpoint, or loaded directly as a Python class.
In the example directory, use the config file for your chosen interface:
* **A2A:** `config_a2a.yaml` (inline agent config; uses `${A2A_API_KEY}`)
* **Chat Completions:** `config_chat_completions.yaml` (inline agent config; uses `${AGENT_API_KEY}`)
* **Custom agent connector:** `config_custom.yaml` (loads agent directly as a Python class — no server needed)
Set the matching environment variable before running.
```bash theme={null}
uv venv --python 3.11
source .venv/bin/activate
uv pip install -r examples/e-commerce/agent_server/requirements.txt
```
```bash theme={null}
conda create -n ecommerce_venv python=3.11 -y
conda activate ecommerce_venv
pip install -r examples/e-commerce/agent_server/requirements.txt
```
Exposes an A2A-compatible agent on port `9999`. Run from the repository root:
```bash theme={null}
export OPENAI_API_KEY=""
export A2A_API_KEY=1234-4567-8910
python -m examples.e-commerce.agent_server.a2a.server
```
Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/a2a/server.py` under the e-commerce example.
Exposes a Chat Completions–compatible endpoint on port `8888` at `/chat/completions`. Run from the repository root:
```bash theme={null}
export OPENAI_API_KEY=""
export AGENT_API_KEY=123456
python -m examples.e-commerce.agent_server.chat_completions.server
```
Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/chat_completions/server.py` under the e-commerce example.
No server needed. The custom agent connector loads `custom_agent.py` directly as a Python class at runtime. Skip this step and proceed to run simulation.
In a **new terminal**, from `examples/e-commerce`, run the commands for the same interface you used in step 3. Use the same API key you set there.
```bash theme={null}
cd examples/e-commerce
export OPENAI_API_KEY=""
export A2A_API_KEY=1234-4567-8910
arksim simulate-evaluate config_a2a.yaml
```
```bash theme={null}
cd examples/e-commerce
export OPENAI_API_KEY=""
export AGENT_API_KEY=123456
arksim simulate-evaluate config_chat_completions.yaml
```
No separate server terminal is needed. From `examples/e-commerce`:
```bash theme={null}
cd examples/e-commerce
export OPENAI_API_KEY=""
arksim simulate-evaluate config_custom.yaml
```
Or run the full pipeline as a Python script:
```bash theme={null}
export OPENAI_API_KEY=""
python run_pipeline.py
```
***
## Running with Your Own Agent
To test your own backend agent against these scenarios:
* **Chat Completions:** Follow the comments in `agent_server/chat_completions/server.py` to swap in your own backend logic or point to your endpoint.
* **A2A:** Implement your own A2A executor in `agent_server/a2a/agent_executor.py`.
* **Custom:** Subclass `BaseAgent` and point `config_custom.yaml` at your module. See [Custom agent configuration](./simulate-conversation#connection-types) for details.
Make sure the `agent_config` field in the config YAML is updated. Then run simulation and evaluation as above.
See [Agent configuration](./simulate-conversation#agent-configuration) for supported protocols and how to configure agent connections.
***
## Configuration
The example uses a single config file for both simulation and evaluation.
```yaml theme={null}
# AGENT CONFIGURATION
agent_config:
agent_type: chat_completions
agent_name: e-commerce
api_config:
endpoint: https://api.openai.com/v1/chat/completions
headers:
Content-Type: application/json
Authorization: "Bearer ${OPENAI_API_KEY}"
body:
model: gpt-5.1
messages:
- role: system
content: >-
You are a shopping assistant. Please provide a complete answer
to the user's question based on your knowledge. Response within
50 words.
# SIMULATION SETTINGS
# Path to the scenarios file
scenario_file_path: ./scenarios.json
# Number of conversations per scenario to generate
num_conversations_per_scenario: 1
# Maximum turns per conversation
max_turns: 5
# Output file path for simulation results
output_file_path: ./results/simulation/simulation.json
# Jinja template for the simulated user's system prompt
simulated_user_prompt_template: null
# EVALUATION SETTINGS
# Output directory for evaluation results
output_dir: ./results/evaluation
# Paths to Python files defining custom
# QuantitativeMetric or QualitativeMetric subclasses
custom_metrics_file_paths:
- ./custom_metrics.py
# Built-in metrics to run; if empty, all built-in metrics run
metrics_to_run:
- faithfulness
- helpfulness
- coherence
- verbosity
- relevance
- goal_completion
- agent_behavior_failure
# Generate HTML report
generate_html_report: true
# Numeric thresholds: per-metric minimum scores on native scale
# numeric_thresholds:
# overall_score: 0.7
# goal_completion: 0.6
# SHARED SETTINGS
# LLM model
model: gpt-5.1
# LLM provider
provider: openai
# Workers for parallel processing
num_workers: 50
```
***
## Output
Results are written under the example directory:
| Location | Contents |
| ---------------------------------------- | ------------------------------------------------------------------------ |
| `./results/simulation/simulation.json` | Simulated conversations from the simulation step |
| `./results/evaluation/evaluation.json` | Evaluation results (per-turn and per-conversation scores, unique errors) |
| `./results/evaluation/final_report.html` | Interactive HTML report for browsing and sharing results |
# Evaluation
Source: https://docs.arklex.ai/v0.3.x/evaluate-conversation
Use ArkSim to score your agent's responses on helpfulness, coherence, relevance, faithfulness, and goal completion, and surface failure patterns with actionable fixes.
## What is Evaluation?
Evaluation analyzes the conversation transcripts produced by simulation and scores your agent across multiple dimensions.
The **output** is a set of files summarizing your agent's performance across turns, conversations, and error types.
![Evaluation]()
***
## What Gets Evaluated
Each agent response is scored on five metrics per turn:
| Metric | What it measures | Scale |
| ---------------- | ------------------------------------------------------ | ----- |
| **Helpfulness** | How effectively the response addresses the user's need | 1–5 |
| **Coherence** | Logical flow and clarity of the response | 1–5 |
| **Relevance** | How directly the response addresses the user's input | 1–5 |
| **Verbosity** | Conciseness; penalizes unnecessarily long responses. | 1–5 |
| **Faithfulness** | Whether the response contradicts the user's knowledge | 1–5 |
Scores range from **1 (poor)** to **5 (excellent)**, with 3–4 considered good and 4–5 excellent. You can add domain-specific [custom metrics](#custom-metrics) (e.g. product suitability, compliance) via Python files.
At the conversation level, two additional scores are computed on a 0–1 scale:
* **Goal Completion**: whether the user's goal was fully addressed by the end of the conversation.
* **Turn Success Ratio**: proportion of turns with no detected behavior failure.
These combine into an **Overall Agent Score** per conversation:
`overall_agent_score = turn_success_ratio × 0.75 + goal_completion_score × 0.25`
Each conversation is assigned a status based on this score. For the exact status values and thresholds, see [Evaluation output](./schema-reference#evaluation-output-evaluationjson) in the Schema Reference.
***
## Behavior Failure Detection
Beyond numeric scores, evaluation detects the type of failure in each underperforming turn:
| Failure type | Description |
| ---------------------------------- | ------------------------------------------------------------------------- |
| `lack of specific information` | Correct but incomplete answer that omits details present in the knowledge |
| `failure to ask for clarification` | Proceeds with an assumption instead of asking the user |
| `disobey user request` | Completely ignores what the user asked |
| `repetition` | Restates previous content without adding value |
| `false information` | Directly contradicts what is stated in the knowledge |
| `no failure` | No failure detected |
Across all conversations, duplicate failures are deduplicated into a **unique errors** list with occurrence counts. When errors are found and a scenario file is available, the evaluator automatically generates **focus files** for targeted reruns (see [Focus Files](#focus-files) below).
***
## Inputs
The input to evaluation is the simulation output file (e.g. `simulation.json`) from the [Simulation](./simulate-conversation) step. Knowledge referenced during evaluation comes from your [Scenarios](./build-scenario) file.
Evaluation is configured via a YAML file:
```yaml theme={null}
# Inputs
scenario_file_path: ./examples/path/to/scenario.json
simulation_file_path: ./examples/path/to/simulation.json
# Model
provider: openai
model: gpt-5.1
# Execution
num_workers: 50
# Metrics
metrics_to_run: [
faithfulness,
helpfulness,
coherence,
verbosity,
relevance,
goal_completion,
agent_behavior_failure
]
custom_metrics_file_paths: []
# Output
output_dir: ./examples/path/to/evaluation
generate_html_report: true
# Threshold gates (optional — omit or set to null to disable)
numeric_thresholds:
overall_score: 0.7
faithfulness: 3.5
helpfulness: 3.0
goal_completion: 0.8
qualitative_failure_labels:
agent_behavior_failure: ["false information", "disobey user request"]
prohibited_statements: ["violated"]
```
For all evaluation config keys, types, and defaults, see [Run configuration → Evaluation keys](./schema-reference#evaluation-keys) in the Schema Reference.
***
## Custom Metrics
In addition to the built-in metrics (helpfulness, coherence, relevance, verbosity, faithfulness, goal completion, behavior failure), you can define **custom metrics** in Python and load them via config.
List of paths to Python files. Each file is loaded and every public `QuantitativeMetric` or `QualitativeMetric` subclass is instantiated and run. Custom metrics **always run** — they are not filtered by `metrics_to_run`.
Names of built-in metrics to run. If empty, all built-in metrics run. Use this to restrict evaluation to a subset of built-ins while still running all custom metrics from `custom_metrics_file_paths`.
### Scope
Every custom metric has a **scope** that controls when it runs:
| Scope | Runs | Receives |
| ------------------ | -------------------------- | -------------------------------------------------------- |
| `"turn"` (default) | Once per agent response | The current turn + conversation history up to that point |
| `"conversation"` | Once per full conversation | The complete conversation history |
Use `"turn"` for response-quality checks (clarity, tone, compliance statements). Use `"conversation"` for end-to-end assessments that only make sense after the whole conversation (goal completion, needs assessment, product suitability).
### Metric Types
Produces a **numeric score** (e.g. 0–5).
| Step | Detail |
| --------- | ---------------------------------------------------- |
| Subclass | `QuantitativeMetric` |
| Implement | `score(score_input: ScoreInput) -> QuantResult` |
| Return | `QuantResult` with `name`, `value` (float), `reason` |
```python theme={null}
from arksim.evaluator import (
QuantitativeMetric,
QuantResult,
ScoreInput,
)
# Turn-level: scored on every agent response
class ClarityScore(QuantitativeMetric):
def __init__(self, llm=None):
super().__init__(name="clarity", score_range=(0, 5), llm=llm, scope="turn")
def score(self, score_input: ScoreInput) -> QuantResult:
# self.llm is the evaluator's configured LLM, injected automatically.
# score_input.chat_history, .current_turn, .user_goal, .knowledge, etc.
value = ... # your scoring logic using self.llm or any other logic
return QuantResult(name="clarity", value=value, reason="...")
# Conversation-level: scored once after the full conversation
class ProductSuitabilityScore(QuantitativeMetric):
def __init__(self, llm=None):
super().__init__(name="product_suitability", score_range=(0, 5), llm=llm, scope="conversation")
def score(self, score_input: ScoreInput) -> QuantResult:
# score_input.chat_history contains the full conversation.
value = ...
return QuantResult(name="product_suitability", value=value, reason="...")
```
Produces a **categorical label** (e.g. `"complete"` / `"partial"`).
| Step | Detail |
| --------- | ------------------------------------------------- |
| Subclass | `QualitativeMetric` |
| Implement | `evaluate(score_input: ScoreInput) -> QualResult` |
| Return | `QualResult` with `name`, `value` (str), `reason` |
```python theme={null}
from arksim.evaluator import (
QualitativeMetric,
QualResult,
ScoreInput,
)
# Turn-level: checked on every agent response
class ProhibitedStatements(QualitativeMetric):
def __init__(self, llm=None):
super().__init__(name="prohibited_statements", llm=llm, scope="turn")
def evaluate(self, score_input: ScoreInput) -> QualResult:
# self.llm is the evaluator's configured LLM, injected automatically.
# score_input.chat_history, .current_turn, .user_goal, .knowledge, etc.
label = ... # "ok" | "violated"
return QualResult(name="prohibited_statements", value=label, reason="...")
# Conversation-level: checked once after the full conversation
class NeedsAssessment(QualitativeMetric):
def __init__(self, llm=None):
super().__init__(name="needs_assessment", llm=llm, scope="conversation")
def evaluate(self, score_input: ScoreInput) -> QualResult:
# score_input.chat_history contains the full conversation.
label = ... # "complete" | "partial" | "missing"
return QualResult(name="needs_assessment", value=label, reason="...")
```
### Available fields
Both metric types receive a `ScoreInput`:
```python theme={null}
score_input.chat_history # Full conversation turns
score_input.user_goal # The goal the user was trying to achieve
score_input.knowledge # Retrieved knowledge / context passed to the agent
```
***
## Threshold Gates
Threshold gates let you fail the CLI with a non-zero exit code when evaluation results fall below acceptable levels. All three gate types are evaluated after every `arksim evaluate` or `arksim simulate-evaluate` run.
### `numeric_thresholds`
Per-metric minimum scores on each metric's native scale. The mean score across all turns per conversation is compared against the threshold. Any conversation that falls below the threshold fails the run with exit code 1.
| Metric | Scale | Notes |
| -------------------------------------------------------------------- | ---------- | -------------------------------------------------------- |
| `overall_score` | 0–1 | Per-conversation `overall_agent_score` compared directly |
| `faithfulness`, `helpfulness`, `coherence`, `verbosity`, `relevance` | 1–5 | Mean of per-turn scores |
| `goal_completion` | 0–1 | Per-conversation score compared directly |
| Custom quantitative metrics (`scope="turn"`) | Your scale | Mean of per-turn scores |
| Custom quantitative metrics (`scope="conversation"`) | Your scale | Per-conversation score compared directly |
```yaml theme={null}
numeric_thresholds:
overall_score: 0.7
faithfulness: 3.5
helpfulness: 3.0
goal_completion: 0.8
```
### `qualitative_failure_labels`
Hard-gate failure labels for qualitative metrics. Any evaluated turn whose label appears in the list fails the run with exit code 1. Turns where the metric did not run are skipped.
```yaml theme={null}
qualitative_failure_labels:
agent_behavior_failure: ["false information", "disobey user request"]
prohibited_statements: ["violated"]
```
If any gate fails, the CLI exits with code `1` and logs which conversations and turns did not meet the requirement. See [Exit codes](./schema-reference#exit-codes) for the full list of exit codes.
### Programmatic threshold gates
When running ArkSim programmatically (e.g. with a custom agent class), you can apply the same gates by importing `check_numeric_thresholds` and `check_qualitative_failure_labels` directly from `arksim.evaluator`:
```python theme={null}
import sys
from arksim.evaluator import (
Evaluator,
check_numeric_thresholds,
check_qualitative_failure_labels,
)
evaluator_output = evaluator.evaluate(simulation_output)
numeric_ok = check_numeric_thresholds(
evaluator_output,
numeric_thresholds={
"clarity": 3.5,
"goal_completion": 0.6,
},
)
qual_ok = check_qualitative_failure_labels(
evaluator_output,
qualitative_failure_labels={
"agent_behavior_failure": ["false information"],
"prohibited_statements": ["violated"],
},
)
if not numeric_ok or not qual_ok:
sys.exit(1)
```
You can also write your own custom gate by inspecting `evaluator_output.conversations` directly and applying any logic you need:
```python theme={null}
# Example: fail if any conversation has overall_agent_score below 0.7
for convo in evaluator_output.conversations:
if convo.overall_agent_score < 0.7:
print(f"Conversation {convo.conversation_id} failed: score {convo.overall_agent_score:.2f}")
sys.exit(1)
```
***
## Running Evaluation
```bash theme={null}
pip install arksim
```
```bash theme={null}
arksim evaluate config.yaml
```
```python theme={null}
from arksim.evaluator import run_evaluation, EvaluationInput
evaluation_results = run_evaluation(EvaluationInput(
simulation_file_path="./simulation.json",
scenario_file_path="./scenarios.json",
output_dir="./evaluation",
model="gpt-5.1",
provider="openai",
num_workers=50,
generate_html_report=True,
))
```
```python theme={null}
from arksim.scenario import Scenarios
from arksim.evaluator import run_evaluation, EvaluationInput
# simulation comes from run_simulation(...)
scenarios = Scenarios.load("./scenarios.json")
evaluation_results = run_evaluation(EvaluationInput(
output_dir="./evaluation",
model="gpt-5.1",
provider="openai",
num_workers="auto",
generate_html_report=True,
), simulation=simulation, scenarios=scenarios)
```
***
## Output Files
| File | Description |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `evaluation.json` | Full evaluation output: conversations with scores, failure labels, and unique errors. See [Evaluation output](./schema-reference#evaluation-output-evaluationjson) for the schema. |
| `final_report.html` | High-level summary with metric averages, overall assessment, and top unique errors, rendered for easier browsing and sharing. |
| `focus/error_N.json` | Filtered scenario file for error group N (sorted by severity). Only appears when unique errors are detected and a scenario file is provided. See [Focus Files](#focus-files). |
| `focus/all_failures.json` | Union of all failing scenarios across every error group. Use this to rerun everything that failed in one pass. |
***
## Reading the Output
Results go to `evaluation.json`: **conversations** (one per run), **unique\_errors** (deduplicated failures), and **error\_scenario\_mappings** (which scenarios triggered each error).
Each `conversations[]` entry has:
* **`overall_agent_score`** (0–1): Turn success + goal completion. Use to rank or filter.
* **`evaluation_status`**: `Done`, `Partial Failure`, `Failed`, or `Evaluation Failed`.
* **`goal_completion_score`** (0–1, or -1 if skipped): How fully the goal was addressed. **`goal_completion_reason`** explains it.
* **`turn_success_ratio`** (0–1): Share of turns with no behavior failure.
Use **`turn_scores[]`** for a turn: `turn_id`, **`scores`** (e.g. helpfulness, faithfulness), **`turn_behavior_failure`**, **`turn_behavior_failure_reason`**, **`unique_error_ids`**. Match IDs to **`unique_errors`** for category and description.
Each **`unique_errors[]`** entry has:
* **`unique_error_description`**: What went wrong.
* **`behavior_failure_category`**: e.g. `false information`, `failure to ask for clarification`, `repetition`.
* **`severity`**: e.g. `critical`, `high`, `medium`, `low`.
* **`occurrences`**: `{ "conversation_id", "turn_id" }` list. Use to find the conversation and turn in logs or CSV.
Use **`occurrences`** to go from an error to the exact turn; then check **`turn_scores`** and the transcript.
Each **`error_scenario_mappings[]`** entry maps one unique error to the scenarios that triggered it:
* **`unique_error_id`**: Links to the matching entry in `unique_errors`.
* **`scenario_ids`**: Which scenarios triggered this error. Use these IDs with `--scenario_file_path` to rerun only the affected scenarios.
* **`severity`** and **`error_description`**: Same as the linked `unique_errors` entry, included for convenience.
This field is empty when no errors are detected or no scenario file was provided. See [Focus Files](#focus-files) for the file-based workflow.
***
## Focus Files
When evaluation detects unique errors and a scenario file is provided, the evaluator generates **focus files** in a `focus/` subdirectory. These are filtered scenario files that let you rerun only the failing subset without any manual config.
### How it works
Each unique error group is mapped back to the scenarios that triggered it. The evaluator writes one JSON file per error group plus a combined file with all failing scenarios:
```
evaluation/
evaluation.json
final_report.html
focus/
error_1.json # scenarios for the highest-severity error
error_2.json # scenarios for the second error
all_failures.json # union of all failing scenarios
```
Error files are numbered by severity (critical first, then high, medium, low) and by occurrence count within the same severity level.
Each focus file uses the standard [Scenarios schema](./schema-reference#scenario-input-scenariosjson), so it works directly with `--scenario_file_path`:
```bash theme={null}
# Rerun only the scenarios that triggered error group 1
arksim simulate-evaluate config.yaml \
--scenario_file_path ./evaluation/focus/error_1.json
# Or rerun all failing scenarios at once
arksim simulate-evaluate config.yaml \
--scenario_file_path ./evaluation/focus/all_failures.json
```
Pass a separate `--output_dir` for each rerun so results do not overwrite the original evaluation. Each rerun produces its own evaluation, report, and focus files (or no focus files if everything passes).
### Iterative fix workflow
Focus files support an iterative fix-and-verify loop:
1. Run `arksim simulate-evaluate config.yaml` and review the errors
2. Fix the highest-severity bug in your agent
3. Rerun with the focus file: `--scenario_file_path ./evaluation/focus/error_1.json --output_dir ./results/rerun_1`
4. If the error is resolved, move to the next error group
5. When all targeted reruns pass, do a full battle test with the original scenario file
The focus set shrinks as you fix bugs. Scenarios that pass on rerun will not appear in the new run's focus files.
# Installation
Source: https://docs.arklex.ai/v0.3.x/installation
Install ArkSim and get ready to simulate and evaluate your AI agents.
## Requirements
* Python 3.10 or later (up to 3.13)
* An API key from a supported LLM provider ([OpenAI](https://developers.openai.com/api/docs), [Anthropic](https://platform.claude.com/docs/en/home), or [Google](https://ai.google.dev/gemini-api/docs/api-key))
## Install from PyPI
```bash theme={null}
pip install arksim
```
### Optional provider extras
ArkSim uses OpenAI by default. To use other providers, install the corresponding extra:
```bash theme={null}
# Anthropic
pip install "arksim[anthropic]"
# Google
pip install "arksim[google]"
# All providers
pip install "arksim[all]"
```
## Install from source
```bash theme={null}
git clone https://github.com/arklexai/arksim.git
cd arksim
pip install -e ".[dev]"
```
This installs in editable mode with test dependencies (`pytest`, `pytest-asyncio`).
## Verify installation
```bash theme={null}
arksim --help
```
You should see the available commands: `simulate`, `evaluate`, `simulate-evaluate`, `show-prompts`, `examples`, and `ui`.
You're all set! Continue to the [Quickstart guide](./quickstart) to run your first simulation.
# Insurance Customer Service Agent
Source: https://docs.arklex.ai/v0.3.x/insurance-customer-service-agent-evaluation
End-to-end example of using ArkSim to simulate and evaluate an insurance customer service agent with custom compliance metrics and adversarial scenarios.
## Overview
This example walks through running ArkSim against a customer service agent built for an insurance company (XYZ Insurance, part of XYZ Bank Group). The agent is designed to answer customer questions about insurance products and coverage, including topics like policy details, claims processes, deductibles, and coverage limits.
The example includes two ready-to-run agent implementations you can test against out of the box, and a guide for plugging in your own agent once you're familiar with the setup.
***
## Example Agents
A lightweight agent that calls the OpenAI API directly. Quick start with minimal setup.
Agent built with the OpenAI Agents SDK, backed by an insurance knowledge base; supports A2A, Chat Completions, or custom agent connector.
***
## Scenarios
The example ships with a set of pre-built scenarios in `scenarios.json` representing realistic insurance customer interactions. Each scenario defines a simulated user with a distinct persona, goal, and background knowledge drawn from insurance product documentation.
The scenario goals are:
* Learn how home insurance deductibles work, when to file a claim, and how they affect your premium and payout.
* Get a denied water damage claim (water heater 17 years old; policy excludes tanks 15+) overturned or learn how to fight it, including legal options; push back if the agent only repeats the policy.
* Renew Basic Form home insurance at the same price; push back if the agent upsells Broad or Comprehensive Form or extra features.
* Get a clear recommendation and dollar amount for personal condo insurance as a first-time buyer, without lengthy needs questions.
* Bundle home, two cars, and motorcycle with XYZ and get a specific savings number; resist needs questions or comparisons.
Scenarios are defined in `scenarios.json` in the example directory and can be edited or extended to reflect your own use case.
***
Before following either path, ensure [ArkSim is installed](./installation) (`pip install arksim`).
## Option 1: OpenAI Agent
This agent calls the OpenAI API directly, with no server setup required.
```bash theme={null}
export OPENAI_API_KEY=""
```
Run from the `examples/bank-insurance` directory:
```bash theme={null}
cd examples/bank-insurance
arksim simulate-evaluate config.yaml
```
***
## Option 2: In-house Agent (OpenAI Agents SDK)
This agent is a customer service agent built with the OpenAI Agents SDK, backed by an insurance knowledge base. It can be exposed via either the A2A Protocol or a Chat Completions-compatible endpoint.
In the example directory, use the config file for your chosen interface:
* **A2A:** `config_a2a.yaml` (inline agent config; uses `${A2A_API_KEY}`)
* **Chat Completions:** `config_chat_completions.yaml` (inline agent config; uses `${AGENT_API_KEY}`)
* **Custom agent connector:** `config_custom.yaml` (loads agent directly as a Python class — no server needed)
Set the matching environment variable before running.
```bash theme={null}
uv venv --python 3.11
source .venv/bin/activate
pip install -r examples/bank-insurance/agent_server/requirements.txt
```
```bash theme={null}
conda create -n bank_venv python=3.11 -y
conda activate bank_venv
pip install -r examples/bank-insurance/agent_server/requirements.txt
```
Or from inside the example directory: `pip install -r agent_server/requirements.txt`
Exposes an A2A-compatible agent on port `9999`. Run from the repository root:
```bash theme={null}
export OPENAI_API_KEY=""
export A2A_API_KEY=1234-4567-8910
python -m examples.bank-insurance.agent_server.a2a.server
```
Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/a2a/server.py` under the bank-insurance example.
Exposes a Chat Completions–compatible endpoint on port `8888` at `/chat/completions`. Run from the repository root:
```bash theme={null}
export OPENAI_API_KEY=""
export AGENT_API_KEY=123456
python -m examples.bank-insurance.agent_server.chat_completions.server
```
Run from the repository root (or the directory that contains the `examples` package). If your layout differs, use the module path that resolves to `agent_server/chat_completions/server.py` under the bank-insurance example.
No server needed. The custom agent connector loads `custom_agent.py` directly as a Python class at runtime. Skip this step and proceed to run simulation.
In a **new terminal**, from `examples/bank-insurance`, run the commands for the same interface you used in step 3. Use the same API key you set there.
```bash theme={null}
cd examples/bank-insurance
export A2A_API_KEY=1234-4567-8910
arksim simulate-evaluate config_a2a.yaml
```
```bash theme={null}
cd examples/bank-insurance
export AGENT_API_KEY=123456
arksim simulate-evaluate config_chat_completions.yaml
```
No separate server terminal is needed. From `examples/bank-insurance`:
```bash theme={null}
cd examples/bank-insurance
export OPENAI_API_KEY=""
arksim simulate-evaluate config_custom.yaml
```
Or run the full pipeline as a Python script:
```bash theme={null}
export OPENAI_API_KEY=""
python run_pipeline.py
```
***
## Running with Your Own Agent
To test your own backend agent against these scenarios:
* **Chat Completions:** Follow the comments in `agent_server/chat_completions/server.py` to swap in your own backend logic or point to your endpoint.
* **A2A:** Implement your own A2A executor in `agent_server/a2a/agent_executor.py`.
* **Custom:** Subclass `BaseAgent` and point `config_custom.yaml` at your module. See [Custom agent configuration](./simulate-conversation#connection-types) for details.
Make sure the `agent_config` field in the config YAML is updated. Then run simulation and evaluation as above.
See [Agent configuration](./simulate-conversation#agent-configuration) for supported protocols and how to configure agent connections.
***
## Configuration
The example uses a single config file for both simulation and evaluation.
```yaml theme={null}
# AGENT CONFIGURATION
agent_config:
agent_type: chat_completions
agent_name: bank-insurance
api_config:
endpoint: https://api.openai.com/v1/chat/completions
headers:
Content-Type: application/json
Authorization: "Bearer ${OPENAI_API_KEY}"
body:
model: gpt-5.1
messages:
- role: system
content: >-
You are a customer service chatbot for XYZ Bank insurance.
XYZ Insurance—a core business within XYZ Bank Group—is one of
Canada's leading providers of life, health, home, auto, and
travel insurance.
Rules:
1. Do not flip roles.
2. Avoid using bullet points or lists.
3. Never exceed 80 words.
# SIMULATION SETTINGS
# Path to the scenarios file
scenario_file_path: ./scenarios.json
# Number of conversations per scenario to generate
num_conversations_per_scenario: 1
# Maximum turns per conversation
max_turns: 5
# Output file path for simulation results
output_file_path: ./results/simulation/simulation.json
# Jinja template for the simulated user's system prompt
simulated_user_prompt_template: null
# EVALUATION SETTINGS
# Output directory for evaluation results
output_dir: ./results/evaluation
# Paths to Python files defining custom
# QuantitativeMetric or QualitativeMetric subclasses
custom_metrics_file_paths:
- ./custom_metrics.py
# Built-in metrics to run; if empty, all built-in metrics run
metrics_to_run:
- faithfulness
- helpfulness
- coherence
- relevance
- goal_completion
- agent_behavior_failure
# Generate HTML report
generate_html_report: true
# Numeric thresholds: per-metric minimum scores on native scale
# numeric_thresholds:
# overall_score: 0.7
# goal_completion: 0.6
# SHARED SETTINGS
# LLM model
model: gpt-5.1
# LLM provider
provider: openai
# Workers for parallel processing
num_workers: 50
```
***
## Output
Results are written under the example directory:
| Location | Contents |
| ---------------------------------------- | ------------------------------------------------------------------------ |
| `./results/simulation/simulation.json` | Simulated conversations from the simulation step |
| `./results/evaluation/evaluation.json` | Evaluation results (per-turn and per-conversation scores, unique errors) |
| `./results/evaluation/final_report.html` | Interactive HTML report for browsing and sharing results |
# Overview
Source: https://docs.arklex.ai/v0.3.x/overview
ArkSim is an open-source agent testing framework that simulates realistic multi-turn conversations with your AI agent and evaluates performance across built-in and custom metrics.
## What is ArkSim?
ArkSim is an open-source agent testing tool designed to help developers validate, simulate, and harden their agent systems. It focuses on reproducible interaction simulations to identify reliability and idempotency gaps before production deployment.
![ArkSim Workflow]()
![ArkSim Workflow]()
## Who is it for?
Developers looking to ensure their agents are robust, reliable, and ready for real-world interactions, without relying on manual testing.
## What makes it different?
Fully transparent and customizable, letting you extend or adapt the tool as needed.
Test your agent in realistic scenarios to uncover gaps in behavior or reliability before production.
Use predefined scenarios to quickly validate your agent and jump-start testing effectively.
Integrate with existing systems and tailor the testing environment to your specific needs.
## Core Capabilities
### Scenarios
A scenario is a test case that defines a simulated user’s attributes, goals, and prior knowledge. It allows you to test different versions of your agent against the same set of scenarios to compare behavior and catch regressions.
### Simulation
Simulation turns scenarios into live, multi-turn conversations with your agent. You get full transcripts (and optional metadata) that you can critically inspect or pass to evaluation.
### Evaluation
Evaluation scores your agent on those conversations. You get quantitative and qualitative metrics (e.g. helpfulness, faithfulness) and agent behavior failures that you can prioritize and fix.
## Frequently Asked Questions
ArkSim generates realistic multi-turn conversations between LLM-powered synthetic users and your agent, then evaluates every turn. Each synthetic user has a distinct profile, goal, and knowledge level. This reveals failures that only emerge across multiple conversation turns, like losing context, calling the wrong tool, or contradicting earlier responses.
Install with `pip install arksim`, run `arksim init` to scaffold a starter config and agent file, then run `arksim simulate-evaluate config.yaml`. See the [quickstart guide](./quickstart) for a full walkthrough.
ArkSim works with any AI agent, whether it is built with LangChain, CrewAI, OpenAI Agents SDK, or your own custom code. Connect through a Chat Completions HTTP endpoint, the A2A protocol, or load a Python agent class directly with no server needed. See the full list of [supported integrations](./integrations).
Yes. ArkSim runs as a CLI command that exits non-zero when quality thresholds are not met. Add it to any CI pipeline as a quality gate on every pull request. See the [CI integration guide](./ci-integration).
Built-in metrics include helpfulness, coherence, relevance, faithfulness, verbosity, goal completion, and agent behavior failure detection. You can also define custom quantitative and qualitative metrics with full access to the conversation context. See the [evaluation guide](./evaluate-conversation).
## Ready to start?
Test your AI agent with ArkSim. Follow the quick start or explore the source on GitHub.
Set up your first simulation in minutes.
Explore full implementation on GitHub.
# Personal AI Assistant (OpenClaw)
Source: https://docs.arklex.ai/v0.3.x/personal-ai-assistant-openclaw-evaluation
End-to-end example of using ArkSim to simulate and evaluate an OpenClaw personal AI assistant handling smart home, calendar, and messaging tasks.
## Overview
This example walks through running ArkSim against an **OpenClaw** personal AI assistant. OpenClaw is an open-source assistant that runs on your own hardware and can manage tasks, smart home devices, messaging, calendar, and files.
You use this setup to simulate and evaluate conversations against your own OpenClaw deployment. The agent is exposed via a Chat Completions-compatible endpoint; the example `config.yaml` is already configured for the OpenClaw gateway.
***
## Prerequisites
Before getting started:
* **OpenClaw** installed and configured: [openclaw.ai](https://openclaw.ai/)
* **OpenClaw gateway** running with the HTTP Chat Completions endpoint enabled
* **OpenAI API key**: used by ArkSim to power the simulated user
* **OpenClaw gateway token**: for authenticating to your OpenClaw instance (see `~/.openclaw/openclaw.json` under `gateway.auth.token`)
***
## Scenarios
The example ships with pre-built scenarios in `scenarios.json` representing realistic interactions with a personal AI assistant. Each scenario defines a simulated user with a distinct persona, goal, and background knowledge (files, smart home setup, contacts, tasks, etc.).
The scenario goals are:
* Manage reminders and to-do lists for files and storage (cleanup Downloads, cloud folders, storage alerts, important documents).
* Control and automate smart home devices and scenes (lights, thermostat, blinds, locks, scenes like Good Morning, Movie Night, Leaving Home, Goodnight).
* Send and manage messages across WhatsApp, iMessage, Slack, email, and Telegram using contacts and recent conversations.
* Manage calendar and schedule (e.g. call dentist to confirm appointment, schedule car maintenance, recurring reminders, upcoming tasks).
* Organize files and documents across local folders and cloud storage (cleanup Downloads, duplicate photos, Trash, important items and shared folders).
Scenarios are defined in `scenarios.json` in the example directory and can be edited or extended to reflect your assistant's capabilities.
***
## Setup
In `~/.openclaw/openclaw.json`, enable the HTTP Chat Completions endpoint:
```json theme={null}
{
"gateway": {
"http": {
"endpoints": {
"chatCompletions": {
"enabled": true
}
}
}
}
}
```
```bash theme={null}
openclaw gateway --port 18789 --verbose
```
```bash theme={null}
export OPENAI_API_KEY=""
export OPENCLAW_TOKEN=""
```
Your gateway token is in `~/.openclaw/openclaw.json` under `gateway.auth.token`.
Before running simulation, confirm the gateway is responding:
```bash theme={null}
curl -sS "http://127.0.0.1:18789/v1/chat/completions" \
-H "Authorization: Bearer $OPENCLAW_TOKEN" \
-H "Content-Type: application/json" \
-H "x-openclaw-agent-id: main" \
-d '{"model":"openclaw","messages":[{"role":"user","content":"Hello"}]}'
```
You should receive a valid JSON response from the gateway.
Run from the `examples/openclaw` directory:
```bash theme={null}
cd examples/openclaw
arksim simulate-evaluate config.yaml
```
The `config.yaml` has the `agent_config` pointed at `http://localhost:18789/v1/chat/completions` with `Authorization: Bearer ${OPENCLAW_TOKEN}` and `x-openclaw-agent-id: main`.
***
## Configuration
The example uses a single config file for both simulation and evaluation. Agent configuration is specified inline under the `agent_config` key:
```yaml theme={null}
# AGENT CONFIGURATION
agent_config:
agent_type: chat_completions
agent_name: openclaw
api_config:
endpoint: http://localhost:18789/v1/chat/completions
headers:
Content-Type: application/json
Authorization: "Bearer ${OPENCLAW_TOKEN}"
x-openclaw-agent-id: main
body:
model: openclaw
messages:
- role: system
content: "You are a helpful personal assistant."
enable_metadata: false
# SIMULATION SETTINGS
# Path to the scenarios file
scenario_file_path: ./scenarios.json
# Number of conversations per scenario to generate
num_conversations_per_scenario: 1
# Maximum turns per conversation
max_turns: 5
# Output file path for simulation results
output_file_path: ./results/simulation/simulation.json
# Jinja template for the simulated user's system prompt
simulated_user_prompt_template: null
# EVALUATION SETTINGS
# Output directory for evaluation results
output_dir: ./results/evaluation
# Paths to Python files defining custom
# QuantitativeMetric or QualitativeMetric subclasses
custom_metrics_file_paths: []
# Built-in metrics to run; if empty, all built-in metrics run
metrics_to_run:
- faithfulness
- helpfulness
- coherence
- verbosity
- relevance
- goal_completion
- agent_behavior_failure
# Generate HTML report
generate_html_report: true
# Numeric thresholds: per-metric minimum scores on native scale
# numeric_thresholds:
# overall_score: 0.7
# goal_completion: 0.6
# SHARED SETTINGS
# LLM model
model: gpt-5.1
# LLM provider
provider: openai
# Workers for parallel processing
num_workers: 50
```
***
## Output
Results are written under the example directory:
| Location | Contents |
| ---------------------------------------- | ------------------------------------------------------------------------ |
| `./results/simulation/simulation.json` | Simulated conversations from the simulation step |
| `./results/evaluation/evaluation.json` | Evaluation results (per-turn and per-conversation scores, unique errors) |
| `./results/evaluation/final_report.html` | Interactive HTML report for browsing and sharing results |
***
## Example Files
| File | Description |
| ---------------- | ---------------------------------------------------------------------- |
| `config.yaml` | Simulate and evaluate configuration with inline OpenClaw agent config. |
| `scenarios.json` | Pre-built scenarios for the personal assistant. |
***
## Adapting to Your Own Assistant
To use this example as a starting point for your own personal assistant:
1. **Scenarios**: Edit or add scenarios in `scenarios.json` so goals and knowledge reflect your assistant's capabilities.
2. **Agent config**: If your endpoint, port, or auth headers differ from the OpenClaw defaults, update the `agent_config` section within `config.yaml` file accordingly.
See [Agent configuration](./simulate-conversation#agent-configuration) for supported protocols and how to configure agent connections.
# Quickstart
Source: https://docs.arklex.ai/v0.3.x/quickstart
Get started with ArkSim in minutes. Test your own agent or explore a pre-built example with a single command.
## Prerequisites
* Python 3.10-3.13
* An API key from a supported provider ([OpenAI](https://developers.openai.com/api/docs), [Anthropic](https://platform.claude.com/docs/en/home), or [Google Gemini](https://ai.google.dev/gemini-api/docs/api-key))
## Option A: Test your own agent
If you already have an agent running, use `arksim init` to scaffold a starter config and scenarios file, then point it at your endpoint.
```bash theme={null}
pip install arksim
export OPENAI_API_KEY="your-api-key"
```
For other providers: `pip install "arksim[anthropic]"` or `pip install "arksim[google]"`.
```bash theme={null}
arksim init
```
This creates three files in the current directory:
* **config.yaml** pointing at `./my_agent.py` with sensible defaults
* **scenarios.json** with four domain-agnostic starter scenarios (happy path, out of scope, ambiguous intent, multi-step)
* **my\_agent.py** with a `BaseAgent` subclass ready to fill in (no server needed)
Open `my_agent.py` and replace the `execute()` body with your agent logic. All files include inline comments explaining each field.
For HTTP or A2A agents, use `arksim init --agent-type chat_completions` or `arksim init --agent-type a2a` instead. Use `--force` to re-scaffold if files already exist.
```bash theme={null}
arksim simulate-evaluate config.yaml
```
Open `results/final_report.html` in your browser for an interactive report with scores, failure categories, and full conversation transcripts.
## Option B: Explore a pre-built example
If you want to see ArkSim in action before connecting your own agent, try one of the included examples.
```bash theme={null}
pip install arksim
export OPENAI_API_KEY="your-api-key"
```
```bash theme={null}
arksim examples
```
This creates an `examples/` folder with ready-to-run projects (bank-insurance, e-commerce, customer-service, openclaw).
```bash theme={null}
cd examples/bank-insurance
arksim simulate-evaluate config.yaml
```
```python theme={null}
import asyncio
from arksim.simulation_engine import run_simulation, SimulationInput
from arksim.evaluator import run_evaluation, EvaluationInput
from arksim.config import AgentConfig
from arksim.scenario import Scenarios
EXAMPLE_DIR = "./examples/bank-insurance"
agent_config = AgentConfig(
agent_type="chat_completions",
agent_name="bank-insurance",
api_config={
"endpoint": "https://api.openai.com/v1/chat/completions",
"headers": {
"Content-Type": "application/json",
"Authorization": "Bearer ${OPENAI_API_KEY}",
},
"body": {
"model": "gpt-5.1",
"messages": [
{"role": "system", "content": "You are a customer service chatbot for XYZ Bank insurance."}
],
},
},
)
scenarios = Scenarios.load(f"{EXAMPLE_DIR}/scenarios.json")
simulation = asyncio.run(run_simulation(SimulationInput(
agent_config=agent_config,
num_conversations_per_scenario=1,
max_turns=5,
num_workers="50",
output_file_path=f"{EXAMPLE_DIR}/results/simulation.json",
)))
evaluation = run_evaluation(EvaluationInput(
scenario_file_path=f"{EXAMPLE_DIR}/scenarios.json",
output_dir=f"{EXAMPLE_DIR}/results",
custom_metrics_file_paths=[f"{EXAMPLE_DIR}/custom_metrics.py"],
metrics_to_run=[
"faithfulness", "helpfulness", "coherence",
"relevance", "goal_completion", "agent_behavior_failure",
],
model="gpt-5.1",
provider="openai",
num_workers=50,
generate_html_report=True,
), simulation=simulation, scenarios=scenarios)
print(f"Done. Simulated {len(simulation.conversations)} conversations.")
```
```bash theme={null}
arksim ui
```
Opens at [**http://localhost:8080**](http://localhost:8080). Load any example config and run from the browser.
Open `results/evaluation/final_report.html` in your browser for scores, failure analysis, and full conversation transcripts.
### Using other LLM providers
ArkSim uses OpenAI by default for both the simulated user and the evaluator. To use Anthropic or Google instead, set the provider in your `config.yaml`:
```bash theme={null}
pip install "arksim[anthropic]"
export ANTHROPIC_API_KEY="your-api-key"
```
```yaml theme={null}
model: claude-opus-4-6
provider: anthropic
```
```bash theme={null}
pip install "arksim[google]"
export GEMINI_API_KEY="your-api-key"
```
```yaml theme={null}
model: gemini-2.5-flash
provider: google
```
***
## Next Steps
Now that you've run your first simulation and evaluation, here's where to go next.
* **Explore the core concepts:** Dive deeper into [Scenarios](./build-scenario), [Simulation](./simulate-conversation), and [Evaluation](./evaluate-conversation) to understand how each piece works and how to configure them for your agent.
* **Explore the examples:** Run ArkSim against [E-commerce](./e-commerce-customer-service-agent-evaluation), [Insurance](./insurance-customer-service-agent-evaluation), [Customer Service (tool calling)](./customer-service-tool-calling-agent-evaluation), and [Personal AI assistant (OpenClaw)](./personal-ai-assistant-openclaw-evaluation) to see different use cases and configs.
# Schema Reference
Source: https://docs.arklex.ai/v0.3.x/schema-reference
JSON and YAML schemas used by ArkSim for scenarios, simulation output, evaluation output, and run configuration.
This page documents the structure of the main artifacts used and produced by ArkSim: scenario files, simulation output, evaluation output, and the run configuration file. Use it when building integrations, writing custom tooling, or debugging pipeline outputs.
For agent connection details (Chat Completions, A2A, and config), see [Agent configuration](./simulate-conversation#agent-configuration).
***
## Scenarios (`scenarios.json`)
The scenario file is a single JSON document that lists scenario objects. Each scenario describes one simulated user and one conversation session. It is the input to [Simulation](./simulate-conversation) and is referenced during [Evaluation](./evaluate-conversation).
### Schema
Schema version identifier. Use `"v1"`.
List of scenario objects.
Unique identifier for this scenario.
Unique identifier for the simulated user.
The user's conversational goal, passed into the simulated user prompt. Write in second person ("You want to...").
Context about the agent (e.g. company or product overview) supplied to the simulated user.
Natural-language persona description for the simulated user. Used directly in the simulation prompt. Write in second person ("You are...").
Knowledge items injected into the simulated user's context. Can be an empty list. Each item has:
* `content` (string, required): Text used as the user's background knowledge. (Additional keys such as `metadata` may be present in the file for traceability but are not part of the loaded model.)
Provenance metadata (e.g. `target_agent_capability`, `goal_raw`, `user_attributes`). Not used by the simulator or evaluator.
***
## Simulation output (`simulation.json`)
Written by `arksim simulate` or the simulation step of `arksim simulate-evaluate`. Path is set by `output_file_path` in your config.
### Schema
Output schema version (e.g. `"v1"`).
ArkSim package version that produced this file.
UUID that uniquely identifies this simulation run. Referenced by the evaluation output as `simulation_id` to link the two artifacts.
ISO-8601 UTC timestamp when the file was generated.
One record per simulated conversation.
Unique ID for this conversation.
ID of the scenario used for this conversation.
Ordered list of messages in the conversation.
Each message has: `turn_id` (int), `message_id` (string), `role` (`"simulated_user"` or `"assistant"`), `content` (string).
Rendered simulated user prompt and variables used to drive the conversation.
* `simulated_user_prompt_template`: The Jinja template used for the simulated user system prompt.
* `variables`: Dict used to render the template. Keys are `scenario.agent_context`, `scenario.goal`, `scenario.knowledge`, and `scenario.user_profile`.
***
## Evaluation output (`evaluation.json`)
Written by `arksim evaluate` or the evaluation step of `arksim simulate-evaluate`. Path is `{output_dir}/evaluation.json`.
### Schema
Output schema version (e.g. `"v1.1"`).
ISO-8601 UTC timestamp when the file was generated.
ArkSim package version that produced this file.
UUID that uniquely identifies this evaluation run. Generated fresh for each `arksim evaluate` or `arksim simulate-evaluate` invocation.
UUID of the simulation run that produced the conversations being evaluated. Copied from `simulation.json` to link the two artifacts.
One record per conversation evaluated.
ID of the conversation.
How fully the user goal was achieved (0–1).
Explanation for the goal completion score.
Proportion of turns with no detected behavior failure (0–1).
Combined score for the conversation (0–1). Computed from turn\_success\_ratio and goal\_completion\_score.
One of: `completed`, `partial_failure`, `failed`.
Per-turn scores and failure labels.
Each turn has: `turn_id`, `scores` (list of metric results with `name`, `value`, optional `reason`, optional `metadata`), `turn_score`, `turn_behavior_failure`, `turn_behavior_failure_reason`, `qual_scores`, `unique_error_ids`.
Deduplicated behavior failures across all conversations.
Unique ID (e.g. UUID) for this error pattern.
Category of the failure (e.g. lack of specific information, repetition, false information).
Human-readable description of the error.
Default `"medium"`. Severity of the error.
List of (conversation\_id, turn\_id) where this error occurred.
Maps each unique error to the scenarios that triggered it. Empty when no errors are detected or no scenario file was provided. Added in schema version `v1.1`.
Position in the list, ordered by severity (1 = most severe) then by occurrence count.
Links to the matching entry in `unique_errors`.
Human-readable description of the error.
`critical`, `high`, `medium`, or `low`.
IDs of scenarios that triggered this error, sorted alphabetically.
***
## Focus files (`focus/error_N.json`, `focus/all_failures.json`)
Written to `{output_dir}/focus/` when unique errors are detected and a scenario file is provided. Each file uses the standard [Scenario input](#scenarios-scenariosjson) schema.
Copied from the original scenario file (e.g. `"v1"`).
Filtered subset of the original scenarios. For `error_N.json`, only scenarios that triggered error group N. For `all_failures.json`, the union of all failing scenarios (deduplicated).
Files are numbered by severity (1 = highest) then by occurrence count. See [Focus Files](./evaluate-conversation#focus-files) for usage.
***
## Run configuration (`config.yaml`)
The same YAML file can be used for `arksim simulate`, `arksim evaluate`, and `arksim simulate-evaluate`. For `simulate-evaluate`, simulation and evaluation settings are merged from a single config.
### Agent configuration
| Key | Type | Description |
| -------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------ |
| `agent_config` | object | Inline agent configuration. See [Agent configuration](./simulate-conversation#agent-configuration) for `chat_completions` and `a2a`. |
### Simulation keys
Used by `simulate` and the simulation phase of `simulate-evaluate`.
| Key | Type | Description |
| -------------------------------- | ------------- | -------------------------------------------------------------- |
| `scenario_file_path` | string | Path to `scenarios.json`. |
| `num_conversations_per_scenario` | int | Number of conversations to run per scenario. |
| `max_turns` | int | Maximum turns per conversation. |
| `output_file_path` | string | Path where `simulation.json` is written. |
| `simulated_user_prompt_template` | string | Optional. Jinja template for the simulated user system prompt. |
| `model` | string | LLM model (e.g. `gpt-5.1`). |
| `provider` | string | LLM provider (e.g. `openai`). |
| `num_workers` | string \| int | Parallel workers (default `50`). Set `"auto"` to auto-scale. |
### Evaluation keys
Used by `evaluate` and the evaluation phase of `simulate-evaluate`.
| Key | Type | Description |
| ---------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scenario_file_path` | string | Path to `scenarios.json` (for goal and knowledge during evaluation). |
| `simulation_file_path` | string | Path to simulation output (`simulation.json`). Omitted when running `simulate-evaluate` (output passed in memory). |
| `output_dir` | string | Directory for evaluation results; `evaluation.json` and optional reports are written here. |
| `custom_metrics_file_paths` | list\[string] | Paths to Python files defining custom metrics. |
| `metrics_to_run` | list\[string] | Names of built-in metrics to run. If empty, all built-in metrics run. |
| `generate_html_report` | boolean | Whether to generate `final_report.html`. |
| `numeric_thresholds` | dict\[string, float] \| null | Per-metric thresholds on each metric's native scale. Keys are metric names; values are minimum required scores. Built-in turn-level metrics use a 1–5 scale (mean across turns per conversation). `goal_completion` and `overall_score` use 0–1 and are compared directly. |
| `qualitative_failure_labels` | dict\[string, list\[string]] \| null | Hard-gate failure labels for qualitative metrics. Keys are metric names; values are lists of labels that trigger failure (e.g. `agent_behavior_failure: ["false information", "disobey user request"]`). Any evaluated turn whose label appears in the list fails the run with exit code 1. |
| `model` | string | LLM model for scoring. |
| `provider` | string | LLM provider. |
| `num_workers` | string \| int | Parallel workers (default `50`). Set `"auto"` to auto-scale. |
`model`, `provider`, and `num_workers` are shared across simulation and evaluation when both run from the same config.
***
## Exit codes
The `arksim` CLI exits with one of the following codes, suitable for use in CI/CD pipelines:
| Code | Constant | When it is returned |
| ---- | --------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `0` | `EXIT_OK` | Run completed successfully and all threshold checks passed. |
| `1` | `EXIT_EVAL_FAILED` | Evaluation completed but one or more threshold checks failed (`numeric_thresholds` or `qualitative_failure_labels`). |
| `2` | `EXIT_CONFIG_ERROR` | Invalid or missing configuration (bad YAML, unknown keys, missing required fields). |
| `3` | `EXIT_INTERNAL_ERROR` | Unexpected internal error. |
# Simulation
Source: https://docs.arklex.ai/v0.3.x/simulate-conversation
Run ArkSim scenarios as live multi-turn conversations against your agent and collect transcripts for evaluation.
## What is Simulation?
Simulation is the process where ArkSim runs your pre-built scenarios as live conversations against your agent. Each scenario acts as a simulated user with a defined persona, goal, and prior knowledge who drives a multi-turn interaction with the agent until the user's goal is achieved or the turn limit is reached.
The **output** is a set of conversation transcripts you can inspect directly or pass into Evaluation.
![ArkSim Simulation Workflow]()
***
## Inputs
Before running a simulation, you need three things in place:
| Input | Description |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Scenarios** | A `scenarios.json` file defining user attributes, goals, knowledge, and scenario metadata. [Learn how to write scenarios →](./build-scenario) |
| **Agent config** | Inline `agent_config` in your config YAML. Defines how ArkSim connects to your agent. See [Agent configuration](#agent-configuration) below. |
| **Config file** | `config.yaml` controlling simulation parameters such as number of conversations, max turns, workers, and model settings. |
***
## Configuration
```yaml theme={null}
# Agent configuration
agent_config:
agent_type: chat_completions
agent_name: my-agent
api_config:
endpoint: https://api.openai.com/v1/chat/completions
headers:
Content-Type: application/json
Authorization: "Bearer ${OPENAI_API_KEY}"
body:
model: gpt-5.1
messages:
- role: system
content: "You are a helpful assistant."
# Scenario input
scenario_file_path: ./scenarios.json
# Simulation parameters
num_conversations_per_scenario: 5 # Number of conversations per scenario
max_turns: 5 # Maximum turns per conversation
num_workers: 50 # Parallel workers
# Output
output_file_path: ./simulation.json
# Model configuration (used for the simulated user, not your agent)
model: gpt-5.1
provider: openai
# Optional: custom Jinja2 template for the simulated user prompt
# simulated_user_prompt_template: null
```
The default `num_workers` is `50`. Set `num_workers: auto` to automatically parallelize across all conversations, or specify a fixed number to control load on your agent endpoint.
### Advanced: Custom simulated user prompt
By default, ArkSim uses a built-in system prompt to drive the simulated user. You can override it by setting `simulated_user_prompt_template` in your config to a **Jinja2** template string. The template is rendered per conversation with these variables:
**From the scenario file**
* **`scenario.agent_context`**: A description of the agent being simulated against (e.g., its role, domain, or business purpose).
* **`scenario.goal`**: The specific task or objective the simulated user is trying to accomplish during the conversation (e.g., "file an insurance claim").
* **`scenario.knowledge`**: Reference content (e.g., product details, policy documents) that the simulated user can draw on when answering or asking questions.
* **`scenario.user_profile`**: A second-person natural language persona description for the simulated user, used as-is in the prompt.
Use them in your template with `{{ scenario.goal }}`, `{{ scenario.user_profile }}`, and so on. If you omit `simulated_user_prompt_template`, the default prompt is used.
The built-in template used when `simulated_user_prompt_template` is not set:
```jinja2 theme={null}
You are a user interacting with an agent through multiple turns.
The agent is supplied by the following conversation context:
{{ scenario.agent_context }}
Your profile is:
{{ scenario.user_profile }}
You have the following goal when interacting with this agent:
{{ scenario.goal }}
{% if scenario.knowledge and scenario.knowledge|length == 1 %}
Here is the content that you might be interested in and might have questions about:
{{ scenario.knowledge[0].content }}
{% elif scenario.knowledge and scenario.knowledge|length > 1 %}
You will receive reference knowledge in a user message immediately before each of your replies; use it when relevant to achieve your goal.
{% endif %}
Rules:
- Do not give away all the instruction at once. Only provide the information necessary for the current step.
- Do not hallucinate information that is not provided in the instruction.
- If the instruction goal is satisfied, generate '###STOP###' as a standalone message without anything else.
- Do not repeat the exact instruction in the conversation.
- Avoid using bullet points or lists.
- Keep responses brief and under 50 words.
- You are the user and the agent is the assistant. Do not flip the roles.
{% if scenario.knowledge and scenario.knowledge|length > 1 %}
- Ask only one question per turn.
{% endif %}
```
***
## Agent configuration
Provide agent connection by defining `agent_config` inline in your config YAML.
### Configuration fields
| Field | Type | Required | Description |
| --------------- | ------ | ----------- | ----------------------------------------------------------------------------------------- |
| `agent_name` | string | Yes | Unique identifier for your agent (e.g. lowercase, no spaces). |
| `agent_type` | string | Yes | One of `chat_completions`, `a2a`, or `custom`. See [Connection types](#connection-types). |
| `api_config` | object | Conditional | Required for `chat_completions` and `a2a`. See [Connection types](#connection-types). |
| `custom_config` | object | Conditional | Required for `custom`. See [Connection types](#connection-types). |
### Connection types
ArkSim supports three ways to connect your agent:
| Type | When to use |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [**Chat Completions**](https://developers.openai.com/api/reference/resources/chat/) | Any agent that accepts OpenAI-compatible requests (OpenAI or a custom wrapper) |
| [**A2A**](https://a2a-protocol.org/latest/) | Agents exposed via the Agent-to-Agent protocol |
| **Custom** | Any Python agent loaded directly as a class — no HTTP server required |
**Type:** `chat_completions`
Connects to any OpenAI-compatible chat completions endpoint.
**Required fields:** `endpoint` (API URL), `headers` (e.g. `Content-Type`; `Authorization` optional), `body` (must include a `messages` array).
**Placeholders:** `${ENV_VAR}` is supported in header values for secrets.
**Example (YAML):**
```yaml theme={null}
agent_config:
agent_type: chat_completions
agent_name: my-agent
api_config:
endpoint: https://api.openai.com/v1/chat/completions
headers:
Content-Type: application/json
Authorization: "Bearer ${OPENAI_API_KEY}"
body:
model: gpt-5.1
messages:
- role: system
content: "You are a helpful assistant."
```
**Type:** `a2a`
For agents that speak the Agent-to-Agent (A2A) protocol.
**Required fields:** `endpoint` (A2A agent server URL). **Optional:** `headers` (e.g. for auth).
**Example (YAML):**
```yaml theme={null}
agent_config:
agent_type: a2a
agent_name: a2a-agent
api_config:
endpoint: http://localhost:9999
headers:
api-key: "${A2A_AUTH_TOKEN}"
```
**Type:** `custom`
Loads your agent directly as a Python class — no HTTP server needed. Your agent must subclass `BaseAgent` and implement `get_chat_id()` and `execute()`.
**Required fields:** `module_path` (path to a `.py` file containing your `BaseAgent` subclass). **Optional:** `class_name` (if the file contains multiple `BaseAgent` subclasses).
**Example agent (`my_agent.py`):**
```python theme={null}
from arksim.config import AgentConfig
from arksim.simulation_engine.agent.base import BaseAgent
class MyAgent(BaseAgent):
def __init__(self, agent_config: AgentConfig) -> None:
super().__init__(agent_config)
async def get_chat_id(self) -> str:
return "unique-conversation-id"
async def execute(self, user_query: str, **kwargs: object) -> str:
return "agent response"
```
**Example (YAML):**
```yaml theme={null}
agent_config:
agent_type: custom
agent_name: my-agent
custom_config:
module_path: ./my_agent.py
```
**Example (Python — no YAML needed):**
```python theme={null}
from arksim.config import AgentConfig, CustomConfig
from my_agent import MyAgent
agent_config = AgentConfig(
agent_type="custom",
agent_name=MyAgent.__name__,
custom_config=CustomConfig(agent_class=MyAgent),
)
```
### Environment variable support
Both types support `${ENV_VAR}` substitution in header values (and in the endpoint URL for chat completions where applicable). At runtime the value is replaced; if unset, it becomes an empty string. You can mix static text and variables (e.g. `"Bearer ${API_KEY}"`).
**Security:** Keep credentials in environment variables and out of committed config files.
***
## Running a Simulation
Create a `config.yaml` file with your agent config and simulation parameters:
```yaml theme={null}
agent_config:
agent_type: chat_completions
agent_name: my-agent
api_config:
endpoint: https://api.openai.com/v1/chat/completions
headers:
Content-Type: application/json
Authorization: "Bearer ${OPENAI_API_KEY}"
body:
model: gpt-5.1
messages:
- role: system
content: "You are a helpful assistant."
scenario_file_path: ./scenarios.json
num_conversations_per_scenario: 5
max_turns: 5
num_workers: 50
output_file_path: ./simulation.json
model: gpt-5.1
provider: openai
```
Run a simulation with the default configuration:
```bash theme={null}
arksim simulate config.yaml
```
You can override config values with CLI flags:
```bash theme={null}
arksim simulate config.yaml --max-turns 10 --num-workers 4
```
```python theme={null}
import asyncio
from arksim.simulation_engine import run_simulation, SimulationInput
from arksim.config import AgentConfig
agent_config = AgentConfig(
agent_type="chat_completions",
agent_name="my-agent",
api_config={
"endpoint": "https://api.openai.com/v1/chat/completions",
"headers": {
"Content-Type": "application/json",
"Authorization": "Bearer ${OPENAI_API_KEY}",
},
"body": {
"model": "gpt-5.1",
"messages": [
{"role": "system", "content": "You are a helpful assistant."}
],
},
},
)
simulation = asyncio.run(run_simulation(SimulationInput(
agent_config=agent_config,
scenario_file_path="./scenarios.json",
num_conversations_per_scenario=1,
max_turns=10,
num_workers=50,
output_file_path="./simulation.json",
)))
```
Simulation writes to the path set by `output_file_path` (default `./simulation.json`). The file contains the full transcript of every conversation, ready to inspect or pass into [Evaluation](./evaluate-conversation).
***
## Output
Simulation writes one file: the path set by `output_file_path` (default `./simulation.json`). It contains the full transcript of every conversation: message history, scenario ID, simulated user prompt (template and variables), and all agent and simulated user messages.
For the full structure and field order, see the [Schema Reference](./schema-reference#simulation-output-simulation-json).
### Example output
```json theme={null}
{
"schema_version": "v1",
"simulator_version": "v1",
"simulation_id": "a3f2c1d4-8e7b-4f9a-b6c2-1d0e5f3a8b7c",
"generated_at": "2025-04-10T14:32:00Z",
"conversations": [
{
"conversation_id": "00de685e-f76b-4a5f-a6e5-217cad777316",
"scenario_id": "8f4c2a91-3b7e-4d5f-a9c2-6e1b4d9f2037",
"conversation_history": [
{
"turn_id": 0,
"message_id": "3c9a7f12-6b4e-4d8a-b2f1-9e5c0a7d41e6",
"role": "simulated_user",
"content": "So this whole identity theft thing is stressing me out a bit...."
},
{
"turn_id": 0,
"message_id": "b7e2c9a4-1f6d-4c83-9a5b-2d8e7f0c4a91",
"role": "assistant",
"content": "I understand your concern about identity theft, but rest assured..."
}
],
"simulated_user_prompt": {
"simulated_user_prompt_template": "You are a customer interacting with an agent through multiple turns...",
"variables": {
"scenario.agent_context": "XYZ Bank Insurance is a Canadian provider of...",
"scenario.user_profile": "You are Priya Sharma, a 32-year-old from Toronto, ON...",
"scenario.goal": "You want to find out what documents are needed...",
"scenario.knowledge": ["Typical timelines: claim setup is usually same day..."]
}
}
}
]
}
```
***
## Next Steps
Once your conversations are simulated, you're ready to evaluate how well your agent performed.
Score your agent's responses against the simulated user's goals and knowledge.