Simulation

What is Simulation?

Simulation is the process where ArkSim runs your pre-built scenarios as live conversations against your agent. Each scenario acts as a simulated user with a defined persona, goal, and prior knowledge who drives a multi-turn interaction with the agent until the user’s goal is achieved or the turn limit is reached. The output is a set of conversation transcripts you can inspect directly or pass into Evaluation.

Inputs

Before running a simulation, you need three things in place:

Input	Description
Scenarios	A `scenarios.json` file defining user attributes, goals, knowledge, and scenario metadata. Learn how to write scenarios →
Agent config	Inline `agent_config` in your config YAML. Defines how ArkSim connects to your agent. See Agent configuration below.
Config file	`config.yaml` controlling simulation parameters such as number of conversations, max turns, workers, and model settings.

Configuration

# Agent configuration
agent_config:
  agent_type: chat_completions
  agent_name: my-agent
  api_config:
    endpoint: https://api.openai.com/v1/chat/completions
    headers:
      Content-Type: application/json
      Authorization: "Bearer ${OPENAI_API_KEY}"
    body:
      model: gpt-5.1
      messages:
        - role: system
          content: "You are a helpful assistant."

# Scenario input
scenario_file_path: ./scenarios.json

# Simulation parameters
num_conversations_per_scenario: 5    # Number of conversations per scenario
max_turns: 5                         # Maximum turns per conversation
num_workers: 50                      # Parallel workers

# Output
output_file_path: ./simulation.json

# Model configuration (used for the simulated user, not your agent)
model: gpt-5.1
provider: openai

# Optional: custom Jinja2 template for the simulated user prompt
# simulated_user_prompt_template: null

The default num_workers is 50. Set num_workers: auto to automatically parallelize across all conversations, or specify a fixed number to control load on your agent endpoint.

Advanced: Custom simulated user prompt

By default, ArkSim uses a built-in system prompt to drive the simulated user. You can override it by setting simulated_user_prompt_template in your config to a Jinja2 template string. The template is rendered per conversation with these variables: From the scenario file

scenario.agent_context: A description of the agent being simulated against (e.g., its role, domain, or business purpose).
scenario.goal: The specific task or objective the simulated user is trying to accomplish during the conversation (e.g., “file an insurance claim”).
scenario.knowledge: Reference content (e.g., product details, policy documents) that the simulated user can draw on when answering or asking questions.
scenario.user_profile: A second-person natural language persona description for the simulated user, used as-is in the prompt.

Use them in your template with {{ scenario.goal }}, {{ scenario.user_profile }}, and so on. If you omit simulated_user_prompt_template, the default prompt is used.

Show Show default prompt template

The built-in template used when simulated_user_prompt_template is not set:

You are a user interacting with an agent through multiple turns.
The agent is supplied by the following conversation context:
{{ scenario.agent_context }}

Your profile is:
{{ scenario.user_profile }}

You have the following goal when interacting with this agent:
{{ scenario.goal }}

{% if scenario.knowledge and scenario.knowledge|length == 1 %}
Here is the content that you might be interested in and might have questions about:
{{ scenario.knowledge[0].content }}

{% elif scenario.knowledge and scenario.knowledge|length > 1 %}
You will receive reference knowledge in a user message immediately before each of your replies; use it when relevant to achieve your goal.
{% endif %}

Rules:
- Do not give away all the instruction at once. Only provide the information necessary for the current step.
- Do not hallucinate information that is not provided in the instruction.
- If the instruction goal is satisfied, generate '###STOP###' as a standalone message without anything else.
- Do not repeat the exact instruction in the conversation.
- Avoid using bullet points or lists.
- Keep responses brief and under 50 words.
- You are the user and the agent is the assistant. Do not flip the roles.

{% if scenario.knowledge and scenario.knowledge|length > 1 %}
- Ask only one question per turn.
{% endif %}

Agent configuration

Provide agent connection by defining agent_config inline in your config YAML.

Configuration fields

Field	Type	Required	Description
`agent_name`	string	Yes	Unique identifier for your agent (e.g. lowercase, no spaces).
`agent_type`	string	Yes	One of `chat_completions`, `a2a`, or `custom`. See Connection types.
`api_config`	object	Conditional	Required for `chat_completions` and `a2a`. See Connection types.
`custom_config`	object	Conditional	Required for `custom`. See Connection types.

Connection types

ArkSim supports three ways to connect your agent:

Type	When to use
Chat Completions	Any agent that accepts OpenAI-compatible requests (OpenAI or a custom wrapper)
A2A	Agents exposed via the Agent-to-Agent protocol
Custom	Any Python agent loaded directly as a class — no HTTP server required

Chat Completions
A2A
Custom

Type: chat_completionsConnects to any OpenAI-compatible chat completions endpoint.Required fields: endpoint (API URL), headers (e.g. Content-Type; Authorization optional), body (must include a messages array).Placeholders: ${ENV_VAR} is supported in header values for secrets.Example (YAML):

agent_config:
  agent_type: chat_completions
  agent_name: my-agent
  api_config:
    endpoint: https://api.openai.com/v1/chat/completions
    headers:
      Content-Type: application/json
      Authorization: "Bearer ${OPENAI_API_KEY}"
    body:
      model: gpt-5.1
      messages:
        - role: system
          content: "You are a helpful assistant."

Type: a2aFor agents that speak the Agent-to-Agent (A2A) protocol.Required fields: endpoint (A2A agent server URL). Optional: headers (e.g. for auth).Example (YAML):

agent_config:
  agent_type: a2a
  agent_name: a2a-agent
  api_config:
    endpoint: http://localhost:9999
    headers:
      api-key: "${A2A_AUTH_TOKEN}"

Type: customLoads your agent directly as a Python class — no HTTP server needed. Your agent must subclass BaseAgent and implement get_chat_id() and execute().Required fields: module_path (path to a .py file containing your BaseAgent subclass). Optional: class_name (if the file contains multiple BaseAgent subclasses).Example agent (my_agent.py):

from arksim.config import AgentConfig
from arksim.simulation_engine.agent.base import BaseAgent

class MyAgent(BaseAgent):
    def __init__(self, agent_config: AgentConfig) -> None:
        super().__init__(agent_config)

    async def get_chat_id(self) -> str:
        return "unique-conversation-id"

    async def execute(self, user_query: str, **kwargs: object) -> str:
        return "agent response"

Example (YAML):

agent_config:
  agent_type: custom
  agent_name: my-agent
  custom_config:
    module_path: ./my_agent.py

Example (Python — no YAML needed):

from arksim.config import AgentConfig, CustomConfig
from my_agent import MyAgent

agent_config = AgentConfig(
    agent_type="custom",
    agent_name=MyAgent.__name__,
    custom_config=CustomConfig(agent_class=MyAgent),
)

Environment variable support

Both types support ${ENV_VAR} substitution in header values (and in the endpoint URL for chat completions where applicable). At runtime the value is replaced; if unset, it becomes an empty string. You can mix static text and variables (e.g. "Bearer ${API_KEY}").

Security: Keep credentials in environment variables and out of committed config files.

Running a Simulation

Set up your config file

Create a config.yaml file with your agent config and simulation parameters:

agent_config:
  agent_type: chat_completions
  agent_name: my-agent
  api_config:
    endpoint: https://api.openai.com/v1/chat/completions
    headers:
      Content-Type: application/json
      Authorization: "Bearer ${OPENAI_API_KEY}"
    body:
      model: gpt-5.1
      messages:
        - role: system
          content: "You are a helpful assistant."

scenario_file_path: ./scenarios.json
num_conversations_per_scenario: 5
max_turns: 5
num_workers: 50
output_file_path: ./simulation.json
model: gpt-5.1
provider: openai

Run the simulation

CLI
Python

Run a simulation with the default configuration:

arksim simulate config.yaml

You can override config values with CLI flags:

arksim simulate config.yaml --max-turns 10 --num-workers 4

import asyncio
from arksim.simulation_engine import run_simulation, SimulationInput
from arksim.config import AgentConfig

agent_config = AgentConfig(
    agent_type="chat_completions",
    agent_name="my-agent",
    api_config={
        "endpoint": "https://api.openai.com/v1/chat/completions",
        "headers": {
            "Content-Type": "application/json",
            "Authorization": "Bearer ${OPENAI_API_KEY}",
        },
        "body": {
            "model": "gpt-5.1",
            "messages": [
                {"role": "system", "content": "You are a helpful assistant."}
            ],
        },
    },
)

simulation = asyncio.run(run_simulation(SimulationInput(
    agent_config=agent_config,
    scenario_file_path="./scenarios.json",
    num_conversations_per_scenario=1,
    max_turns=10,
    num_workers=50,
    output_file_path="./simulation.json",
)))

Inspect your output

Simulation writes to the path set by output_file_path (default ./simulation.json). The file contains the full transcript of every conversation, ready to inspect or pass into Evaluation.

Output

Simulation writes one file: the path set by output_file_path (default ./simulation.json). It contains the full transcript of every conversation: message history, scenario ID, simulated user prompt (template and variables), and all agent and simulated user messages. For the full structure and field order, see the Schema Reference.

Example output

{
  "schema_version": "v1",
  "simulator_version": "v1",
  "simulation_id": "a3f2c1d4-8e7b-4f9a-b6c2-1d0e5f3a8b7c",
  "generated_at": "2025-04-10T14:32:00Z",
  "conversations": [
    {
      "conversation_id": "00de685e-f76b-4a5f-a6e5-217cad777316",
      "scenario_id": "8f4c2a91-3b7e-4d5f-a9c2-6e1b4d9f2037",
      "conversation_history": [
        {
          "turn_id": 0,
          "message_id": "3c9a7f12-6b4e-4d8a-b2f1-9e5c0a7d41e6",
          "role": "simulated_user",
          "content": "So this whole identity theft thing is stressing me out a bit...."
        },
        {
          "turn_id": 0,
          "message_id": "b7e2c9a4-1f6d-4c83-9a5b-2d8e7f0c4a91",
          "role": "assistant",
          "content": "I understand your concern about identity theft, but rest assured..."
        }
      ],
      "simulated_user_prompt": {
        "simulated_user_prompt_template": "You are a customer interacting with an agent through multiple turns...",
        "variables": {
          "scenario.agent_context": "XYZ Bank Insurance is a Canadian provider of...",
          "scenario.user_profile": "You are Priya Sharma, a 32-year-old from Toronto, ON...",
          "scenario.goal": "You want to find out what documents are needed...",
          "scenario.knowledge": ["Typical timelines: claim setup is usually same day..."]
        }
      }
    }
  ]
}

Next Steps

Once your conversations are simulated, you’re ready to evaluate how well your agent performed.

Evaluation →

Score your agent’s responses against the simulated user’s goals and knowledge.

Getting started

Core Capabilities

CI/CD

Examples

Help

What is Simulation?

Inputs

Configuration

Advanced: Custom simulated user prompt

Agent configuration

Configuration fields

Connection types

Environment variable support

Running a Simulation

Output

Example output

Next Steps

Evaluation →

​What is Simulation?

​Inputs

​Configuration

​Advanced: Custom simulated user prompt

​Agent configuration

​Configuration fields

​Connection types

​Environment variable support

​Running a Simulation

​Output

​Example output

​Next Steps

Evaluation →

What is Simulation?

Inputs

Configuration

Advanced: Custom simulated user prompt

Agent configuration

Configuration fields

Connection types

Environment variable support

Running a Simulation

Output

Example output

Next Steps