Obsidian Metadata
| type | source |
| subtype | video |
| urls | https://m.youtube.com/watch?v=4dUFIRj-BWo, https://www.youtube.com/watch?v=EEw2PpL-_NM |
| speakers | Han Lee (Moody's), Hamel Husain |
| event | Data Council 2025 |
Videos: “The Model Is the Product” vs “The Model Is NOT the Product”
The FAQ links this debate under “Will today’s evaluation methods still be relevant in 5–10 years?”
Han Lee: the model is the product (~39 min)
- Analogy: foundation models are the Intel CPU of the PC era. Application companies are “wrappers” riding the curve.
- Step-function capabilities create product categories: chat (GPT-3.5), tool calling (made glue code obsolete, enabled Perplexity-style products), agentic coding (Claude 3.5 Sonnet enabled Cursor’s agent mode and Lovable), and text-driven image editing.
- Smarter models absorb surrounding complexity, like CPUs absorbing the FPU and north/south bridges, or transformers absorbing the NLP pipeline. Scaffolding moats erode with each model release.
- Two strategies: model-first (frontier labs, now moving up the stack) or product-first (UX and integration on SOTA models).
- Where product-first companies can build a moat: distribution → data → your own eval harness and benchmarks. Public benchmarks lagged the real capabilities (a benchmark for long-horizon agentic work came ~9 months after the capability appeared). Owning evals lets you spot model gaps and new capabilities first, and hill-climb with prompts, fine-tuning, or RL.
Hamel: the model is NOT the product (~15 min)
- The argument really says labs will capture all value by moving up the stack. That’s the old “why won’t Amazon build it?” question.
- AI engineering is two things: getting the model to do what you want (which gets easier as models improve) and figuring out what you and your users want (taste, judgment, design, UX), which you can’t escape.
- Copilot and Cursor had the same models for long stretches and became very different products.
- A case study of Google’s Workspace AI (at the time): despite SOTA models, owned hardware, and huge distribution, Gemini in Docs, Calendar, Gmail, Sheets, and Slides often failed at basic tasks (“I can’t help with that”, wrong recipient), while third parties like Lindy succeeded. Microsoft 365 Copilot told him to read his own files.
- “You are the moat, not the model.” Your taste and product sense.
Why it matters for evals
Both talks land on the same point: your own evals and error analysis are durable value. Han frames them as the product-first moat, and Hamel as the way you articulate what users want. See Product Evals vs Model Benchmarks.