> ## Documentation Index > Fetch the complete documentation index at: https://docs.arklex.ai/llms.txt > Use this file to discover all available pages before exploring further. # Evaluation > Score your agent's responses across every conversation turn and surface failure patterns with actionable fixes. ## What is Evaluation? Evaluation analyzes the conversation transcripts produced by simulation and scores your agent across multiple dimensions. The **output** is a set of files summarizing your agent's performance across turns, conversations, and error types. Evaluation *** ## What Gets Evaluated Each agent response is scored on five metrics per turn: | Metric | What it measures | Scale | | ---------------- | ------------------------------------------------------ | ----- | | **Helpfulness** | How effectively the response addresses the user's need | 1–5 | | **Coherence** | Logical flow and clarity of the response | 1–5 | | **Relevance** | How directly the response addresses the user's input | 1–5 | | **Verbosity** | Conciseness; penalizes unnecessarily long responses. | 1–5 | | **Faithfulness** | Whether the response contradicts the user's knowledge | 1–5 | Scores range from **1 (poor)** to **5 (excellent)**, with 3–4 considered good and 4–5 excellent. You can add domain-specific [custom metrics](#custom-metrics) (e.g. product suitability, compliance) via Python files. At the conversation level, two additional scores are computed on a 0–1 scale: * **Goal Completion**: whether the user's goal was fully addressed by the end of the conversation. * **Turn Success Ratio**: proportion of turns with no detected behavior failure. These combine into an **Overall Agent Score** per conversation: `overall_agent_score = turn_success_ratio × 0.75 + goal_completion_score × 0.25` Each conversation is assigned a status based on this score. For the exact status values and thresholds, see [Evaluation output](./schema-reference#evaluation-output-evaluationjson) in the Schema Reference. *** ## Behavior Failure Detection Beyond numeric scores, evaluation detects the type of failure in each underperforming turn: | Failure type | Description | | ---------------------------------- | -------------------------------------------------------- | | `lack of specific information` | Generic or partial answer that omits important details | | `failure to ask for clarification` | Proceeds with an assumption instead of asking the user | | `disobey user request` | Completely ignores what the user asked | | `repetition` | Restates previous content without adding value | | `false information` | Contradicts/fabricates information relative to knowledge | | `no failure` | No failure detected | Across all conversations, duplicate failures are deduplicated into a **unique errors** list with occurrence counts. *** ## Inputs The input to evaluation is the simulation output file (e.g. `simulation.json`) from the [Simulation](./simulate-conversation) step. Knowledge referenced during evaluation comes from your [Scenarios](./build-scenario) file. Evaluation is configured via a YAML file: ```yaml theme={null} # Inputs scenario_file_path: ./examples/path/to/scenario.json simulation_file_path: ./examples/path/to/simulation.json # Model provider: openai model: gpt-5.1 # Execution num_workers: 50 score_threshold: null # Metrics metrics_to_run: [ faithfulness, helpfulness, coherence, verbosity, relevance, goal_completion, agent_behavior_failure ] custom_metrics_file_paths: [] # Output output_dir: ./examples/path/to/evaluation generate_html_report: true ``` For all evaluation config keys, types, and defaults, see [Run configuration → Evaluation keys](./schema-reference#evaluation-keys) in the Schema Reference. *** ## Custom Metrics In addition to the built-in metrics (helpfulness, coherence, relevance, verbosity, faithfulness, goal completion, behavior failure), you can define **custom metrics** in Python and load them via config. List of paths to Python files. Each file is loaded and every public `QuantitativeMetric` or `QualitativeMetric` subclass is instantiated and run. Custom metrics **always run** — they are not filtered by `metrics_to_run`. Names of built-in metrics to run. If empty, all built-in metrics run. Use this to restrict evaluation to a subset of built-ins while still running all custom metrics from `custom_metrics_file_paths`. ### Metric Types Produces a **numeric score** (e.g. 0–5). | Step | Detail | | --------- | ---------------------------------------------------- | | Subclass | `QuantitativeMetric` | | Implement | `score(score_input: ScoreInput) -> QuantResult` | | Return | `QuantResult` with `name`, `value` (float), `reason` | ```python theme={null} from arksim.evaluator import ( QuantitativeMetric, QuantResult, ScoreInput, ) class ClarityScore(QuantitativeMetric): def score(self, score_input: ScoreInput) -> QuantResult: # Use your own LLM or any logic; independent of evaluator model config. # score_input.chat_history, .user_goal, .knowledge, etc. value = ... # your scoring logic or LLM call return QuantResult(name="clarity", value=value, reason="...") ``` Produces a **categorical label** (e.g. `"complete"` / `"partial"`). | Step | Detail | | --------- | ------------------------------------------------- | | Subclass | `QualitativeMetric` | | Implement | `evaluate(score_input: ScoreInput) -> QualResult` | | Return | `QualResult` with `name`, `value` (str), `reason` | ```python theme={null} from arksim.evaluator import ( QualitativeMetric, QualResult, ScoreInput, ) class NeedsAssessment(QualitativeMetric): def evaluate(self, score_input: ScoreInput) -> QualResult: # Use score_input.chat_history, .user_goal, .knowledge, etc. label = ... # "complete" | "partial" | "missing" return QualResult(name="needs_assessment", value=label, reason="...") ``` ### Available fields Both metric types receive a `ScoreInput`: ```python theme={null} score_input.chat_history # Full conversation turns score_input.user_goal # The goal the user was trying to achieve score_input.knowledge # Retrieved knowledge / context passed to the agent ``` *** ## Running Evaluation ```bash theme={null} pip install arksim ``` ```bash theme={null} arksim evaluate config.yaml ``` ```python theme={null} from arksim.evaluator import run_evaluation, EvaluationInput evaluation_results = run_evaluation(EvaluationInput( simulation_file_path="./simulation.json", scenario_file_path="./scenarios.json", output_dir="./evaluation", model="gpt-5.1", provider="openai", num_workers=50, generate_html_report=True, )) ``` ```python theme={null} from arksim.scenario import Scenarios from arksim.evaluator import run_evaluation, EvaluationInput # simulation comes from run_simulation(...) scenarios = Scenarios.load("./scenarios.json") evaluation_results = run_evaluation(EvaluationInput( output_dir="./evaluation", model="gpt-5.1", provider="openai", num_workers="auto", generate_html_report=True, ), simulation=simulation, scenarios=scenarios) ``` *** ## Output Files | File | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `evaluation.json` | Full evaluation output: conversations with scores, failure labels, and unique errors. See [Evaluation output](./schema-reference#evaluation-output-evaluationjson) for the schema. | | `final_report.html` | High-level summary with metric averages, overall assessment, and top unique errors, rendered for easier browsing and sharing. | *** ## Reading the Output Results go to `evaluation.json`: **conversations** (one per run) and **unique\_errors** (deduplicated failures). Each `conversations[]` entry has: * **`overall_agent_score`** (0–1): Turn success + goal completion. Use to rank or filter. * **`evaluation_status`**: `Done`, `Partial Failure`, `Failed`, or `Evaluation Failed`. * **`goal_completion_score`** (0–1, or -1 if skipped): How fully the goal was addressed. **`goal_completion_reason`** explains it. * **`turn_success_ratio`** (0–1): Share of turns with no behavior failure. Use **`turn_scores[]`** for a turn: `turn_id`, **`scores`** (e.g. helpfulness, faithfulness), **`turn_behavior_failure`**, **`turn_behavior_failure_reason`**, **`unique_error_ids`**. Match IDs to **`unique_errors`** for category and description. Each **`unique_errors[]`** entry has: * **`unique_error_description`**: What went wrong. * **`behavior_failure_category`**: e.g. `false information`, `failure to ask for clarification`, `repetition`. * **`severity`**: e.g. `critical`, `high`, `medium`, `low`. * **`occurrences`**: `{ "conversation_id", "turn_id" }` list. Use to find the conversation and turn in logs or CSV. Use **`occurrences`** to go from an error to the exact turn; then check **`turn_scores`** and the transcript.