Obsidian Metadata
| type | case-study |
| sources | Source - A Field Guide to Rapidly Improving AI Products, Source - AI Evals FAQ |
Case Study: GitHub Copilot
- Early Copilot invested heavily in offline evaluation infrastructure: testing completions against a large corpus of GitHub repos, using existing unit tests in high-quality codebases to verify completions automatically.
- A huge engineering effort: clone repos at scale, set up environments, run test suites, analyze results across many languages and frameworks.
- The payoff: thousands of experiments, and the ability to say “this change improved quality by X%” instead of arguing from gut feel.
- Completions eval: remove function implementations, regenerate them, run the tests. Chat: LLM judges with specific criteria plus right-tool checks. Online: A/B tests on acceptance and code retention.
- Hamel led CodeSearchNet, a precursor to Copilot.
Related
Coding Agent Evaluation · Capability Funnel and Experiment Roadmaps
More from the talk
See Video - The Evals That Made GitHub Copilot: the harness-lib repo selection (tests pass, coverage, docstrings, post-training-cutoff repos to avoid contamination, which skewed smaller than production), key A/B metrics (acceptance rate, characters retained, latency) plus dozens of diagnostic guardrail metrics, granular checklist judges for chat, tool-selection confusion matrices, and the finding that users preferred shorter completions.