Skip to main content

​
What is a Scenario?

A scenario is a test case that describes who the simulated user is, what they want to accomplish, and what they already know going into the conversation with the agent.
ArkSim Workflow

​
The Three Core Pillars

​
Goal

Describes the user’s objective, guiding their actions and decisions throughout the conversation.
Plan a purchase of a service robot, focusing on delivery timelines and suitability for commercial use.

​
User Profile (Persona)

Defines who the simulated user is — their personality, demographics, and behavior — as a natural-language description written in second person.
You are Alex, a 26-year-old operations analyst at a mid-sized logistics company in Chicago. You are analytical, detail-oriented, budget-conscious, and risk-aware.

​
Knowledge

Provides context that shapes the simulated user’s questions and expectations. Also used during evaluation to check whether the agent’s responses were accurate and consistent with the source.
  • home_insurance_overview.md
    The service robot is a 40 kg payload autonomous service robot with 8-hour battery life and $70,000 price…
  • purchase_options.pdf
    The payment plans include full purchase, 12-month leasing, and vendor financing…
Think of a scenario as answering three questions: Who is this user? What do they want? What do they know?

​
Why ArkSim Scenarios?

ArkSim Scenarios let you test and evaluate your agent using realistic, profile-driven simulations of user interactions. Each scenario is built around clearly defined goals and behaviors, enabling structured performance evaluation without requiring a real-world dataset. By using synthetic users, you can run consistent and reproducible tests across a wide range of interaction types, making it easy to scale evaluations and understand how your agent performs before deploying it to real users.

​
Scenario File Structure

A scenario file is a single JSON document containing a list of scenario objects. Each scenario represents one simulated user and one conversation session. 
{
  "schema_version": "v1",
  "scenarios": [
    { ... },
    { ... }
  ]
}
For the full scenario schema (all fields and types), see Schema Reference.

​
Full Example

{
  "schema_version": "v1",
  "scenarios": [
    {
      "scenario_id": "123456",
      "user_id": "123456",
      "goal": "You want to ask detailed questions about XYZ Insurance car coverage options in your province, including limits, deductibles, and what's actually covered.",
      "knowledge": [
        {
          "content": "Car Insurance Overview: XYZ Insurance offers mandatory and optional coverages including third-party liability, collision, and comprehensive...",
          "metadata": {
            "title": "Car Insurance Overview",
            "source": "https://www.xyzinsurance.com/products-services/auto-car-insurance",
            "knowledge_type": "car_insurance_overview",
            "doc_id": "car_insurance_overview_001"
          }
        }
      ],
      "user_profile": "You are Priya Chen who is open to experience, conscientious, introverted, agreeable, and emotionally stable. You are a 34-year-old single woman living in Toronto, ON, Canada. As a prospective customer who found XYZ Insurance through an online search, you are analytical and value-oriented with moderate spending habits.",
      "origin": {
        "target_agent_capability": "Answer questions about XYZ Insurance products and coverage",
        "goal_raw": "Ask question about XYZ insurance",
        "user_attributes": {
          "sex": "Female",
          "age": "34",
          "marital_status": "single",
          "household_size": "1",
          "location": "Toronto, ON, Canada",
          "Household_income": "$50,000 - $74,999",
          "education": "Bachelor's degree",
          "Openness": "open to experience",
          "Conscientiousness": "conscientious",
          "Extraversion": "introverted",
          "Agreeableness": "agreeable",
          "Neuroticism": "emotionally stable",
          "customer_type": "prospective customer",
          "discovery_type": "online search",
          "decision_making_style": "analytical",
          "purchasing_preference": "value oriented",
          "spending_behavior": "moderate spending",
          "loyalty_level": "neutral"
        }
      }
    }
  ]
}

​
Writing Scenarios

You can write scenario files manually. The required fields are scenario_id, user_id, goal, agent_context, and user_profile: 
{
  "schema_version": "v1",
  "scenarios": [
    {
      "scenario_id": "my-scenario-001",
      "user_id": "user-001",
      "goal": "You want to find out whether your home insurance policy covers water damage from a burst pipe.",
      "agent_context": "You are a helpful home insurance assistant for XYZ Insurance.",
      "user_profile": "You are Jordan, an existing customer from Calgary, AB, Canada. You are detail-oriented and want a clear answer.",
    }
  ]
}
Tips
  • Write goal in second person (“You want to…”) so it reads naturally as a simulator instruction.
  • knowledge can be omitted if the simulated user has no specific background information.
  • origin can be omitted for hand-authored scenarios.

​
Multi-Knowledge Scenarios

A scenario can include multiple knowledge items by adding additional entries to the knowledge list. These items are directly injected into the simulated user’s context — there is no retrieval step or external knowledge base involved. 
"knowledge": [
  {
    "content": "Home Insurance Basics: XYZ Insurance provides coverage for your dwelling, contents, and liability...",
    "metadata": { "title": "Home Insurance Basics" }
  },
  {
    "content": "Claims Process: To file a claim, you can call the XYZ Insurance claims line or use the online claims portal...",
    "metadata": { "title": "How to File a Claim" }
  }
]

​
How Fields Are Used Downstream

Understanding how each field is used helps you decide what to include.
FieldUsed ByDescription
goalSimulator & EvaluatorDirectly used in the simulator prompt; the most important field.
knowledge[].contentSimulator & EvaluatorProvides simulated user’s background knowledge for the conversation.
user_profileSimulatorUsed as-is in the simulated user prompt to define the persona.
knowledge[].metadataNot used by ArkSimFor labeling and traceability only.
originNot used by ArkSimTracks scenario provenance; reference only.

​
Next Steps

Once your scenario file is ready, you’re ready to run your scenarios and review the results.

Simulations →

Execute your scenarios and simulate conversations for review and analysis.