Thought UmweltThought Umwelt

thought umwelt

Coding Agent Evaluation

3 min read

Obsidian Metadata

typeguide
sourcesSource - AI Evals FAQ, Source - Demystifying Evals for AI Agents, Source - A Field Guide to Rapidly Improving AI Products

Coding Agent Evaluation

Broad vs narrow agents

  • Handles a wide variety of tasks → start with public benchmarks, as for a foundation model: SWE-bench (Verified), Terminal-Bench, Aider Polyglot, HumanEval.
  • Narrow workflow → product-specific evals are a better fit (see Product Evals vs Model Benchmarks).

Private benchmark

Build one from difficult real tasks in your org. OpenAI evaluated Codex on real internal SWE tasks at launch. Each task needs a working environment and code-based tests that establish success.

Choosing tasks = error analysis

Look at how people use the agent and where it fails. Review runs with engineers, group recurring problems, turn useful examples into tests. If existing tests already flag failures, use them to pick runs to investigate. Apply Clio-style clustering (Anthropic clusters conversations by topic) to coding sessions to see what kinds of work the benchmark should cover.

Beyond “tests pass”

Anthropic: start with clear tasks and a stable environment verified by unit tests (fail-to-pass and pass-to-pass), then add checks tests don’t capture: code quality, how the agent interacts with users, tool-use behavior. Claude Code’s team added evals for concision and file edits, later over-engineering. Starter metrics: net new lines of code and cyclomatic complexity. Also static analysis (ruff, mypy, bandit), state checks (e.g., security logs), transcript metrics (turns, tool calls, tokens), latency.

task:
  id: "fix-auth-bypass_1"
  desc: "Fix authentication bypass when password field is empty and ..."
  graders:
    - type: deterministic_tests
      required: [test_empty_pw_rejected.py, test_null_pw_rejected.py]
    - type: llm_rubric
      rubric: prompts/code_quality.md
    - type: static_analysis
      commands: [ruff, mypy, bandit]
    - type: state_check
      expect:
        security_logs: {event_type: "auth_blocked"}
    - type: tool_calls
      required:
        - {tool: read_file, params: {path: "src/auth/*"}}
        - {tool: edit_file}
        - {tool: run_tests}

In practice: unit tests for correctness plus an LLM rubric for quality, adding others only as needed.

GitHub Copilot examples (John Berryman & Shawn Simister)

  • Completions: remove function implementations from real repos, have the model regenerate them, run the repos’ existing tests. (This required cloning, environment setup, and test runs at massive scale; see GitHub Copilot.)
  • Chat: LLM judges with specific criteria plus separate checks for calling the right tool.
  • A/B tests tracking whether users accepted suggestions and kept the code afterward. Product metrics complement offline evals.

Pitfalls (Anthropic)

  • Ambiguous specs, e.g. tests assume a filepath the task never specified.
  • Rigid grading (“96.12” vs “96.124991…”). Opus 4.5 went from 42% to 95% on CORE-Bench after grading and scaffold fixes.
  • Shared state between trials (e.g., the agent reading git history from earlier trials).
  • Saturation: SWE-bench Verified went from ~30% to >80% within a year.

Agentic Workflow Evaluation · pass@k and pass^k

Graph View