Obsidian Metadata
| type | source |
| url | https://hamel.dev/blog/posts/evals/ |
| authors | Hamel Husain |
| published | 2024-03-29 |
Source: Your AI Product Needs Evals (Part 1)
URL: https://hamel.dev/blog/posts/evals/ · Hamel Husain, Mar 2024
Thesis: unsuccessful AI products almost always share one root cause, a failure to create robust evaluation systems. Success depends on iteration speed across evaluate → debug → change, and most people only do “change”.
Case study: Rechat Lucy (real-estate assistant that plateaued in whack-a-mole).
Key content:
- Three Levels of Evaluation: unit tests (every change) → human & model eval (cadence) → A/B tests (major changes).
- Scoped assertions per feature × scenario, the no-UUID regex, synthetic test-case generation, CI with Metabase tracking (Code-Based Evals).
- Logging traces (LangSmith), a custom viewer (Shiny), good/bad labels, “you can never stop looking at data”.
- Aligning an LLM critic with a human via spreadsheets; tracking agreement, and later precision/recall because raw agreement misleads (Honeycomb Query Assistant).
- Eval infrastructure unlocks fine-tuning data curation and debugging.
Takeaways: remove all friction from looking at data · keep it simple · look at lots of data · no generic frameworks · write lots of tests and update them · use LLMs to bootstrap the eval system · reuse the infrastructure.