> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arklex.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema Reference

> JSON and YAML schemas used by ArkSim for scenarios, simulation output, evaluation output, and run configuration.

This page documents the structure of the main artifacts used and produced by ArkSim: scenario files, simulation output, evaluation output, and the run configuration file. Use it when building integrations, writing custom tooling, or debugging pipeline outputs.

<Tip>
  For agent connection details (Chat Completions, A2A, and config), see [Agent configuration](./simulate-conversation#agent-configuration).
</Tip>

***

## Scenarios (`scenarios.json`)

The scenario file is a single JSON document that lists scenario objects. Each scenario describes one simulated user and one conversation session. It is the input to [Simulation](./simulate-conversation) and is referenced during [Evaluation](./evaluate-conversation).

### Schema

<ResponseField name="schema_version" type="string" required>
  Schema version identifier. Use `"v1"`.
</ResponseField>

<ResponseField name="scenarios" type="list[Scenario]" required>
  List of scenario objects.

  <Expandable title="Scenario fields">
    <ResponseField name="scenario_id" type="string" required>
      Unique identifier for this scenario.
    </ResponseField>

    <ResponseField name="user_id" type="string" required>
      Unique identifier for the simulated user.
    </ResponseField>

    <ResponseField name="goal" type="string" required>
      The user's conversational goal, passed into the simulated user prompt. Write in second person ("You want to...").
    </ResponseField>

    <ResponseField name="agent_context" type="string" required>
      Context about the agent (e.g. company or product overview) supplied to the simulated user.
    </ResponseField>

    <ResponseField name="user_profile" type="string" required>
      Natural-language persona description for the simulated user. Used directly in the simulation prompt. Write in second person ("You are...").
    </ResponseField>

    <ResponseField name="knowledge" type="list[KnowledgeItem]">
      Knowledge items injected into the simulated user's context. Can be an empty list. Each item has:

      * `content` (string, required): Text used as the user's background knowledge. (Additional keys such as `metadata` may be present in the file for traceability but are not part of the loaded model.)
    </ResponseField>

    <ResponseField name="origin" type="dict[string, Any]">
      Provenance metadata (e.g. `target_agent_capability`, `goal_raw`, `user_attributes`). Not used by the simulator or evaluator.
    </ResponseField>
  </Expandable>
</ResponseField>

***

## Simulation output (`simulation.json`)

Written by `arksim simulate` or the simulation step of `arksim simulate-evaluate`. Path is set by `output_file_path` in your config.

### Schema

<ResponseField name="schema_version" type="string" required>
  Output schema version (e.g. `"v1"`).
</ResponseField>

<ResponseField name="simulator_version" type="string" required>
  ArkSim package version that produced this file.
</ResponseField>

<ResponseField name="simulation_id" type="string" required>
  UUID that uniquely identifies this simulation run. Referenced by the evaluation output as `simulation_id` to link the two artifacts.
</ResponseField>

<ResponseField name="generated_at" type="string" required>
  ISO-8601 UTC timestamp when the file was generated.
</ResponseField>

<ResponseField name="conversations" type="list[Conversation]" required>
  One record per simulated conversation.

  <Expandable title="Conversation fields">
    <ResponseField name="conversation_id" type="string" required>
      Unique ID for this conversation.
    </ResponseField>

    <ResponseField name="scenario_id" type="string" required>
      ID of the scenario used for this conversation.
    </ResponseField>

    <ResponseField name="conversation_history" type="list[Message]" required>
      Ordered list of messages in the conversation.

      Each message has: `turn_id` (int), `message_id` (string), `role` (`"simulated_user"` or `"assistant"`), `content` (string).
    </ResponseField>

    <ResponseField name="simulated_user_prompt" type="SimulatedUserPrompt" required>
      Rendered simulated user prompt and variables used to drive the conversation.

      * `simulated_user_prompt_template`: The Jinja template used for the simulated user system prompt.
      * `variables`: Dict used to render the template. Keys are `scenario.agent_context`, `scenario.goal`, `scenario.knowledge`, and `scenario.user_profile`.
    </ResponseField>
  </Expandable>
</ResponseField>

***

## Evaluation output (`evaluation.json`)

Written by `arksim evaluate` or the evaluation step of `arksim simulate-evaluate`. Path is `{output_dir}/evaluation.json`.

### Schema

<ResponseField name="schema_version" type="string" required>
  Output schema version (e.g. `"v1"`).
</ResponseField>

<ResponseField name="generated_at" type="string" required>
  ISO-8601 UTC timestamp when the file was generated.
</ResponseField>

<ResponseField name="evaluator_version" type="string" required>
  ArkSim package version that produced this file.
</ResponseField>

<ResponseField name="evaluation_id" type="string" required>
  UUID that uniquely identifies this evaluation run. Generated fresh for each `arksim evaluate` or `arksim simulate-evaluate` invocation.
</ResponseField>

<ResponseField name="simulation_id" type="string" required>
  UUID of the simulation run that produced the conversations being evaluated. Copied from `simulation.json` to link the two artifacts.
</ResponseField>

<ResponseField name="conversations" type="list[ConversationEvaluation]" required>
  One record per conversation evaluated.

  <Expandable title="ConversationEvaluation fields">
    <ResponseField name="conversation_id" type="string" required>
      ID of the conversation.
    </ResponseField>

    <ResponseField name="goal_completion_score" type="float" required>
      How fully the user goal was achieved (0–1).
    </ResponseField>

    <ResponseField name="goal_completion_reason" type="string" required>
      Explanation for the goal completion score.
    </ResponseField>

    <ResponseField name="turn_success_ratio" type="float" required>
      Proportion of turns with no detected behavior failure (0–1).
    </ResponseField>

    <ResponseField name="overall_agent_score" type="float" required>
      Combined score for the conversation (0–1). Computed from turn\_success\_ratio and goal\_completion\_score.
    </ResponseField>

    <ResponseField name="evaluation_status" type="string" required>
      One of: `completed`, `partial_failure`, `failed`.
    </ResponseField>

    <ResponseField name="turn_scores" type="list[TurnEvaluation]" required>
      Per-turn scores and failure labels.

      Each turn has: `turn_id`, `scores` (list of metric results with `name`, `value`, optional `reason`, optional `metadata`), `turn_score`, `turn_behavior_failure`, `turn_behavior_failure_reason`, `qual_scores`, `unique_error_ids`.
    </ResponseField>
  </Expandable>
</ResponseField>

<ResponseField name="unique_errors" type="list[UniqueError]" required>
  Deduplicated behavior failures across all conversations.

  <Expandable title="UniqueError fields">
    <ResponseField name="unique_error_id" type="string" required>
      Unique ID (e.g. UUID) for this error pattern.
    </ResponseField>

    <ResponseField name="behavior_failure_category" type="string" required>
      Category of the failure (e.g. lack of specific information, repetition, false information).
    </ResponseField>

    <ResponseField name="unique_error_description" type="string" required>
      Human-readable description of the error.
    </ResponseField>

    <ResponseField name="severity" type="string">
      Default `"medium"`. Severity of the error.
    </ResponseField>

    <ResponseField name="occurrences" type="list[Occurrence]" required>
      List of (conversation\_id, turn\_id) where this error occurred.
    </ResponseField>
  </Expandable>
</ResponseField>

***

## Run configuration (`config.yaml`)

The same YAML file can be used for `arksim simulate`, `arksim evaluate`, and `arksim simulate-evaluate`. For `simulate-evaluate`, simulation and evaluation settings are merged from a single config.

### Agent configuration

| Key            | Type   | Description                                                                                                                          |
| -------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------ |
| `agent_config` | object | Inline agent configuration. See [Agent configuration](./simulate-conversation#agent-configuration) for `chat_completions` and `a2a`. |

### Simulation keys

Used by `simulate` and the simulation phase of `simulate-evaluate`.

| Key                              | Type          | Description                                                    |
| -------------------------------- | ------------- | -------------------------------------------------------------- |
| `scenario_file_path`             | string        | Path to `scenarios.json`.                                      |
| `num_conversations_per_scenario` | int           | Number of conversations to run per scenario.                   |
| `max_turns`                      | int           | Maximum turns per conversation.                                |
| `output_file_path`               | string        | Path where `simulation.json` is written.                       |
| `simulated_user_prompt_template` | string        | Optional. Jinja template for the simulated user system prompt. |
| `model`                          | string        | LLM model (e.g. `gpt-5.1`).                                    |
| `provider`                       | string        | LLM provider (e.g. `openai`).                                  |
| `num_workers`                    | string \| int | Parallel workers (default `50`). Set `"auto"` to auto-scale.   |

### Evaluation keys

Used by `evaluate` and the evaluation phase of `simulate-evaluate`.

| Key                         | Type          | Description                                                                                                        |
| --------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------ |
| `scenario_file_path`        | string        | Path to `scenarios.json` (for goal and knowledge during evaluation).                                               |
| `simulation_file_path`      | string        | Path to simulation output (`simulation.json`). Omitted when running `simulate-evaluate` (output passed in memory). |
| `output_dir`                | string        | Directory for evaluation results; `evaluation.json` and optional reports are written here.                         |
| `custom_metrics_file_paths` | list\[string] | Paths to Python files defining custom metrics.                                                                     |
| `metrics_to_run`            | list\[string] | Names of built-in metrics to run. If empty, all built-in metrics run.                                              |
| `generate_html_report`      | boolean       | Whether to generate `final_report.html`.                                                                           |
| `score_threshold`           | float \| null | If set, exit with non-zero code when any conversation score is below this value.                                   |
| `model`                     | string        | LLM model for scoring.                                                                                             |
| `provider`                  | string        | LLM provider.                                                                                                      |
| `num_workers`               | string \| int | Parallel workers (default `50`). Set `"auto"` to auto-scale.                                                       |

<Note>
  `model`, `provider`, and `num_workers` are shared across simulation and evaluation when both run from the same config.
</Note>