Agent Eval Task Spec

Obsidian Metadata

type	template
sources	Source - Demystifying Evals for AI Agents

Agent Eval Task Spec (YAML)

A starting point for defining one agent task (see Agentic Workflow Evaluation). Use only the graders you need.

task:
  id: "<unique-id>"
  desc: "<unambiguous task description: everything graders check must be stated here>"
  source: "bug-tracker #123 | support ticket | error-analysis failure mode X"
  reference_solution: "path/to/known-good-solution"   # proves solvable + graders configured
  environment: "clean container/snapshot per trial"
  trials: 5                                             # report pass@k and/or pass^k
  graders:
    - type: deterministic_tests          # fail-to-pass + pass-to-pass
      required: [test_x.py, test_y.py]
    - type: state_check                  # grade the OUTCOME, not the path
      expect: { orders: { status: cancelled } }
    - type: tool_calls                   # only for real requirements (authz, preconditions)
      required:
        - { tool: verify_identity }
        - { tool: cancel_order, params: { order_id: "<expected>" } }
    - type: llm_rubric                   # one criterion per validated judge
      rubric: prompts/<failure_mode>.md
      allow_unknown: true
    - type: transcript
      max_turns: 10
  scoring: all_must_pass                 # or weighted / partial credit
tracked_metrics:
  - type: transcript
    metrics: [n_turns, n_toolcalls, n_total_tokens]
  - type: latency
    metrics: [time_to_first_token, output_tokens_per_sec, time_to_last_token]
  - type: cost
    metrics: [usd_per_task]

Checklist:

• Would two domain experts independently agree on Pass/Fail?
• Is there a matching negative task (behavior should not happen)?
• Is the grader hack-resistant?
• Is the capability vs regression suite tagged? (Capability vs Regression Evals)