Obsidian Metadata
| type | source |
| url | https://hamel.dev/blog/posts/field-guide/ |
| authors | Hamel Husain |
| published | 2025-03-24 |
Source: A Field Guide to Rapidly Improving AI Products (Part 3)
URL: https://hamel.dev/blog/posts/field-guide/ · Hamel Husain (patterns from 30+ implementations)
“Can you show me how you’re measuring if any of this actually works?” … room goes quiet. Successful teams barely talk about tools. They obsess over measurement and iteration.
- The most common mistake is skipping error analysis. Generic-metric dashboards create a false sense of progress and fragment attention. Nurture Boss: bottom-up analysis found 3 issues were >60% of problems; date handling improved 33% → 95% (Nurture Boss, Error Analysis).
- The most important investment is a simple data viewer. 10× faster iteration, built in hours; all context in one place, one-click feedback, open-ended notes, filters, hotkeys (Annotation Tools and Review Interfaces).
- Empower domain experts to write prompts. Integrated prompt environments; drop the jargon (Team Collaboration on Evals, Prompt Management).
- Synthetic data works, even with zero users. Features/scenarios/personas grounded in real constraints (Synthetic Data).
- Maintaining trust in evals is critical. Criteria drift, binary + critiques (15–20% higher agreement), alignment checks, scaling without losing trust (Maintaining Trust in Evals).
- Roadmaps should count experiments, not features. Capability funnel, timeboxes, fifteen-fives, GitHub Copilot’s offline eval infrastructure (Capability Funnel and Experiment Roadmaps).
Core principles: look at your data · build simple tools that remove friction · empower domain experts · use synthetic data strategically · maintain trust in evals · structure roadmaps around experiments.