Obsidian Metadata
| type | source |
| subtype | video |
| url | https://www.youtube.com/watch?v=XueTa4qrMpg |
| speakers | Aman Khan (Arize), Hamel Husain |
| duration | 33:57 |
Video: How Engineers and PMs Should Collaborate on Evals
A whiteboard-style discussion. The undercurrent: everyone tries to pass labeling off to someone else.
Points
- Three flavors of AI PM: core product (form factor, packaging), platform (infra, cost, reliability, model choice), and AI-powered PM (using AI in the job). Roles are converging, so think in skills, not titles, like a baseball card where pitchers still bat.
- A maturity arc: vibe-coded “looks good to me” (fine for internal tools, not where brand or regulatory risk exists) → you wrote an eval, but can you trust it? This is where the who-labels question comes up.
- Labels and the eval prompt are intertwined. The label defines “good”, and the judge prompt’s definitions and few-shot examples come from being in the data. So the PM should be close to both labeling and eval-prompt writing, collaborating with the AI engineer. Hamel: get labeling as close to the domain expert as possible, which in a well-run org is often the PM.
- Three questions to settle with an explicit responsible/accountable split:
- How many labels are enough, and do you require agreement per row across experts? (e.g., the engineer who’s accountable for eval quality says “we need N”, and the PM is responsible for getting them)
- When the eval says good but a human says bad, who reviews and who breaks the tie? (benevolent dictator)
- When the eval says good but the business metric drops, the PM is ultimately on the hook.
- You want a “single pane of glass” correlating eval metrics, human labels, and business metrics (e.g., conversions tied back to sessions, which often arrive days or weeks later). If there’s no analyst, the PM may have to roll up their sleeves. Hamel: it’s still “look at your data”, just more kinds of data.
- Traits of strong AI product teams: technical PMs who know what data they need, and well-scoped, simple AI products rather than throwing more agents at the problem.
- Engineers can build product sense by acting as the product person until it no longer scales, and by using the shared customer data (sales calls, session replays).