> ## Documentation Index > Fetch the complete documentation index at: https://docs.arklex.ai/llms.txt > Use this file to discover all available pages before exploring further. # Simulation > Run scenarios against your agent and collect chats for evaluation. ## What is Simulation? Simulation is the process where ArkSim runs your pre-built scenarios as live conversations against your agent. Each scenario acts as a simulated user with a defined persona, goal, and prior knowledge who drives a multi-turn interaction with the agent until the user's goal is achieved or the turn limit is reached. The **output** is a set of conversation transcripts you can inspect directly or pass into Evaluation. ArkSim Simulation Workflow *** ## Inputs Before running a simulation, you need three things in place: | Input | Description | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **Scenarios** | A `scenarios.json` file defining user attributes, goals, knowledge, and scenario metadata. [Learn how to write scenarios →](./build-scenario) | | **Agent config** | Inline `agent_config` in your config YAML. Defines how ArkSim connects to your agent. See [Agent configuration](#agent-configuration) below. | | **Config file** | `config.yaml` controlling simulation parameters such as number of conversations, max turns, workers, and model settings. | *** ## Configuration ```yaml theme={null} # Agent configuration agent_config: agent_type: chat_completions agent_name: my-agent api_config: endpoint: https://api.openai.com/v1/chat/completions headers: Content-Type: application/json Authorization: "Bearer ${OPENAI_API_KEY}" body: model: gpt-5.1 messages: - role: system content: "You are a helpful assistant." # Scenario input scenario_file_path: ./scenarios.json # Simulation parameters num_conversations_per_scenario: 5 # Number of conversations per scenario max_turns: 5 # Maximum turns per conversation num_workers: 50 # Parallel workers # Output output_file_path: ./simulation.json # Model configuration (used for the simulated user, not your agent) model: gpt-5.1 provider: openai # Optional: custom Jinja2 template for the simulated user prompt # simulated_user_prompt_template: null ``` The default `num_workers` is `50`. Set `num_workers: auto` to automatically parallelize across all conversations, or specify a fixed number to control load on your agent endpoint. ### Advanced: Custom simulated user prompt By default, ArkSim uses a built-in system prompt to drive the simulated user. You can override it by setting `simulated_user_prompt_template` in your config to a **Jinja2** template string. The template is rendered per conversation with these variables: **From the scenario file** * **`scenario.agent_context`**: A description of the agent being simulated against (e.g., its role, domain, or business purpose). * **`scenario.goal`**: The specific task or objective the simulated user is trying to accomplish during the conversation (e.g., "file an insurance claim"). * **`scenario.knowledge`**: Reference content (e.g., product details, policy documents) that the simulated user can draw on when answering or asking questions. * **`scenario.user_profile`**: A second-person natural language persona description for the simulated user, used as-is in the prompt. Use them in your template with `{{ scenario.goal }}`, `{{ scenario.user_profile }}`, and so on. If you omit `simulated_user_prompt_template`, the default prompt is used. The built-in template used when `simulated_user_prompt_template` is not set: ```jinja2 theme={null} You are a user interacting with an agent through multiple turns. The agent is supplied by the following conversation context: {{ scenario.agent_context }} Your profile is: {{ scenario.user_profile }} You have the following goal when interacting with this agent: {{ scenario.goal }} {% if scenario.knowledge and scenario.knowledge|length == 1 %} Here is the content that you might be interested in and might have questions about: {{ scenario.knowledge[0].content }} {% elif scenario.knowledge and scenario.knowledge|length > 1 %} You will receive reference knowledge in a user message immediately before each of your replies; use it when relevant to achieve your goal. {% endif %} Rules: - Do not give away all the instruction at once. Only provide the information necessary for the current step. - Do not hallucinate information that is not provided in the instruction. - If the instruction goal is satisfied, generate '###STOP###' as a standalone message without anything else. - Do not repeat the exact instruction in the conversation. - Avoid using bullet points or lists. - Keep responses brief and under 50 words. - You are the user and the agent is the assistant. Do not flip the roles. {% if scenario.knowledge and scenario.knowledge|length > 1 %} - Ask only one question per turn. {% endif %} ``` *** ## Agent configuration Provide agent connection by defining `agent_config` inline in your config YAML. ### Configuration fields | Field | Type | Required | Description | | --------------- | ------ | ----------- | ----------------------------------------------------------------------------------------- | | `agent_name` | string | Yes | Unique identifier for your agent (e.g. lowercase, no spaces). | | `agent_type` | string | Yes | One of `chat_completions`, `a2a`, or `custom`. See [Connection types](#connection-types). | | `api_config` | object | Conditional | Required for `chat_completions` and `a2a`. See [Connection types](#connection-types). | | `custom_config` | object | Conditional | Required for `custom`. See [Connection types](#connection-types). | ### Connection types ArkSim supports three ways to connect your agent: | Type | When to use | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | [**Chat Completions**](https://developers.openai.com/api/reference/resources/chat/) | Any agent that accepts OpenAI-compatible requests (OpenAI or a custom wrapper) | | [**A2A**](https://a2a-protocol.org/latest/) | Agents exposed via the Agent-to-Agent protocol | | **Custom** | Any Python agent loaded directly as a class — no HTTP server required | **Type:** `chat_completions` Connects to any OpenAI-compatible chat completions endpoint. **Required fields:** `endpoint` (API URL), `headers` (e.g. `Content-Type`; `Authorization` optional), `body` (must include a `messages` array). **Placeholders:** `${ENV_VAR}` is supported in header values for secrets. **Example (YAML):** ```yaml theme={null} agent_config: agent_type: chat_completions agent_name: my-agent api_config: endpoint: https://api.openai.com/v1/chat/completions headers: Content-Type: application/json Authorization: "Bearer ${OPENAI_API_KEY}" body: model: gpt-5.1 messages: - role: system content: "You are a helpful assistant." ``` **Type:** `a2a` For agents that speak the Agent-to-Agent (A2A) protocol. **Required fields:** `endpoint` (A2A agent server URL). **Optional:** `headers` (e.g. for auth). **Example (YAML):** ```yaml theme={null} agent_config: agent_type: a2a agent_name: a2a-agent api_config: endpoint: http://localhost:9999 headers: api-key: "${A2A_AUTH_TOKEN}" ``` **Type:** `custom` Loads your agent directly as a Python class — no HTTP server needed. Your agent must subclass `BaseAgent` and implement `get_chat_id()` and `execute()`. **Required fields:** `module_path` (path to a `.py` file containing your `BaseAgent` subclass). **Optional:** `class_name` (if the file contains multiple `BaseAgent` subclasses). **Example agent (`my_agent.py`):** ```python theme={null} from arksim.config import AgentConfig from arksim.simulation_engine.agent.base import BaseAgent class MyAgent(BaseAgent): def __init__(self, agent_config: AgentConfig) -> None: super().__init__(agent_config) async def get_chat_id(self) -> str: return "unique-conversation-id" async def execute(self, user_query: str, **kwargs: object) -> str: return "agent response" ``` **Example (YAML):** ```yaml theme={null} agent_config: agent_type: custom agent_name: my-agent custom_config: module_path: ./my_agent.py ``` **Example (Python — no YAML needed):** ```python theme={null} from arksim.config import AgentConfig, CustomConfig from my_agent import MyAgent agent_config = AgentConfig( agent_type="custom", agent_name=MyAgent.__name__, custom_config=CustomConfig(agent_class=MyAgent), ) ``` ### Environment variable support Both types support `${ENV_VAR}` substitution in header values (and in the endpoint URL for chat completions where applicable). At runtime the value is replaced; if unset, it becomes an empty string. You can mix static text and variables (e.g. `"Bearer ${API_KEY}"`). **Security:** Keep credentials in environment variables and out of committed config files. *** ## Running a Simulation Create a `config.yaml` file with your agent config and simulation parameters: ```yaml theme={null} agent_config: agent_type: chat_completions agent_name: my-agent api_config: endpoint: https://api.openai.com/v1/chat/completions headers: Content-Type: application/json Authorization: "Bearer ${OPENAI_API_KEY}" body: model: gpt-5.1 messages: - role: system content: "You are a helpful assistant." scenario_file_path: ./scenarios.json num_conversations_per_scenario: 5 max_turns: 5 num_workers: 50 output_file_path: ./simulation.json model: gpt-5.1 provider: openai ``` Run a simulation with the default configuration: ```bash theme={null} arksim simulate config.yaml ``` You can override config values with CLI flags: ```bash theme={null} arksim simulate config.yaml --max-turns 10 --num-workers 4 ``` ```python theme={null} import asyncio from arksim.simulation_engine import run_simulation, SimulationInput from arksim.config import AgentConfig agent_config = AgentConfig( agent_type="chat_completions", agent_name="my-agent", api_config={ "endpoint": "https://api.openai.com/v1/chat/completions", "headers": { "Content-Type": "application/json", "Authorization": "Bearer ${OPENAI_API_KEY}", }, "body": { "model": "gpt-5.1", "messages": [ {"role": "system", "content": "You are a helpful assistant."} ], }, }, ) simulation = asyncio.run(run_simulation(SimulationInput( agent_config=agent_config, scenario_file_path="./scenarios.json", num_conversations_per_scenario=1, max_turns=10, num_workers=50, output_file_path="./simulation.json", ))) ``` Simulation writes to the path set by `output_file_path` (default `./simulation.json`). The file contains the full transcript of every conversation, ready to inspect or pass into [Evaluation](./evaluate-conversation). *** ## Output Simulation writes one file: the path set by `output_file_path` (default `./simulation.json`). It contains the full transcript of every conversation: message history, scenario ID, simulated user prompt (template and variables), and all agent and simulated user messages. For the full structure and field order, see the [Schema Reference](./schema-reference#simulation-output-simulation-json). ### Example output ```json theme={null} { "schema_version": "v1", "simulator_version": "v1", "simulation_id": "a3f2c1d4-8e7b-4f9a-b6c2-1d0e5f3a8b7c", "generated_at": "2025-04-10T14:32:00Z", "conversations": [ { "conversation_id": "00de685e-f76b-4a5f-a6e5-217cad777316", "scenario_id": "8f4c2a91-3b7e-4d5f-a9c2-6e1b4d9f2037", "conversation_history": [ { "turn_id": 0, "message_id": "3c9a7f12-6b4e-4d8a-b2f1-9e5c0a7d41e6", "role": "simulated_user", "content": "So this whole identity theft thing is stressing me out a bit...." }, { "turn_id": 0, "message_id": "b7e2c9a4-1f6d-4c83-9a5b-2d8e7f0c4a91", "role": "assistant", "content": "I understand your concern about identity theft, but rest assured..." } ], "simulated_user_prompt": { "simulated_user_prompt_template": "You are a customer interacting with an agent through multiple turns...", "variables": { "scenario.agent_context": "XYZ Bank Insurance is a Canadian provider of...", "scenario.user_profile": "You are Priya Sharma, a 32-year-old from Toronto, ON...", "scenario.goal": "You want to find out what documents are needed...", "scenario.knowledge": ["Typical timelines: claim setup is usually same day..."] } } } ] } ``` *** ## Next Steps Once your conversations are simulated, you're ready to evaluate how well your agent performed. Score your agent's responses against the simulated user's goals and knowledge.