Obsidian Metadata
| type | source |
| subtype | video |
| url | https://www.youtube.com/watch?v=LwLxlEwrtRA |
| speakers | John Berryman, Sean Simister, Hamel Husain |
| duration | 58:26 |
Video: The Evals That Made GitHub Copilot
Two early Copilot engineers describe the evaluation stack behind completions and chat. This extends GitHub Copilot and Coding Agent Evaluation.
A spectrum of eval types
Algorithmic (equality, JSON schema, length) → verifiable (compile, run unit tests, SQL returns the right rows, with freedom in how) → LLM-as-judge (subjective). A/B testing sits under all of it.
1. Verifiable offline evals: “harness lib”
- Mine public Python/JavaScript repos, run their tests, and keep repos where all tests pass.
- Use coverage to link functions to tests. Candidates must have test coverage, a docstring, and a bounded size.
- Scoop out the implementation, regenerate it with the model, and rerun the tests. Aggregated across many repos, this gives a directional quality number.
- Pitfalls: training contamination (use repos created after the model’s training cutoff, which OpenAI’s list made possible early on). Those newer repos skewed smaller than production code, so the eval set didn’t fully match real traffic.
- They debated headless-editor end-to-end integration tests (catch everything) vs flexible, Lego-like harnesses (can evaluate products that don’t exist yet). Both are needed.
- Later variants harvested one-line and partial-block completions, closer to real editing than whole functions.
- Experiments started in notebooks, graduated to scripts, and eventually to automated pipelines. Expensive setup (repo selection, function offsets) was cached in a SQL DB.
2. A/B tests (“ship room”)
- A few key metrics: completion acceptance rate, characters retained (people accept everything and then edit), and latency.
- Dozens to ~100 guardrail metrics (e.g., retention after N minutes, lines generated) used diagnostically, not as hard gates.
- Weird trade-offs: optimizing acceptance rate can produce tiny, useless completions. Surprising guardrail movement could delay a launch a week while someone did data-science detective work.
3. LLM-as-judge for chat
- Chat isn’t just code, and some answers have no runnable code. They started with side-by-side human comparison against a baseline, which was slow and hard even for the evaluator.
- GPT-4 as the comparator got distracted by trivia (brace placement).
- The fix: micromanage the judge with explicit, granular checklists (“Is there a test for null input? For an empty list?”). Checking items off is much easier than solving the task, which is why a model can judge its own family’s output reliably at this granularity.
- Alignment wasn’t rigorously measured in the rush to ship chat. The first version was a bet, refined afterward from telemetry.
4. Tool-use evals
Given context and tools, check the expected call. Build a confusion matrix of intended vs actual tool. Tools that get confused a lot overlap too much and need to partition the space more cleanly. This also works for agent handoffs.
Lessons
- Two evaluator personas: shippers who want one reliable number compared to the last one, and experimenters who want Lego blocks. Both kinds of pipeline are legitimate.
- Evals on Python/JS generalized surprisingly well to other languages’ acceptance rates.
- A counterintuitive finding: users preferred shorter completions they could tab through, not big one-shot blocks.
- When to start evals: after you have a prototype and are past pure vibe-checking. Don’t let evals block building something new.
- Many guardrail metrics were born from past breakages.