Obsidian Metadata
| type | source |
| url | https://hamel.dev/blog/posts/revenge/ |
| authors | Hamel Husain |
| published | 2026-03-26 |
Source: The Revenge of the Data Scientist (PyAI Conf talk)
URL: https://hamel.dev/blog/posts/revenge/
Training models was never most of the job. The job is experiment design, debugging stochastic systems, and metric design, and calling an LLM API doesn’t remove that work. The “harness” around agents (tests, specs, observability and metrics, as in OpenAI’s harness engineering and Karpathy’s auto-research) is data science.
Five pitfalls → the data-science fix
| Pitfall | Default behavior | Data scientist’s approach |
|---|---|---|
| Generic metrics | Off-the-shelf helpfulness/coherence dashboards | Explore traces, form hypotheses, build app-specific metrics (“Calendar Scheduling Failure”) |
| Unverified judges | Ask an LLM to rate on a scale | Treat the judge as a classifier: human labels, train/dev/test, precision/recall not accuracy |
| Bad experimental design | ”Give me 50 test queries”; whole rubric in one call; Likert | Ground synthetic data in real logs with dimensions and edge cases; binary, scoped, actionable metrics tied to outcomes |
| Bad data & labels | Delegate or outsource labeling | Domain experts label; be skeptical of labels; criteria drift means labeling reveals what matters |
| Automating too much | Let the LLM look at data for you | LLMs for plumbing, not for knowing what you want |
Speed-run of other pitfalls: misused similarity scores, vague judge questions (“is it helpful?”), annotators reading raw JSON, uncalibrated scores without CIs, data drift, overfitting, bad sampling, dashboards that don’t make sense.
The mapping
Reading traces = EDA · judge validation = model evaluation · representative test sets = experimental design · expert labeling = data collection · production monitoring = production ML. “None of this is new!”
Always look at the data.
Video of the talk: https://www.youtube.com/watch?v=lA4MfpgF91Y (PyAI Conf 2026, 18 min). The recording matches this annotated post.