Obsidian Metadata
| type | source |
Video Library
Every video linked from the FAQ and the pages it links to. Titles are the actual YouTube titles. ✅ = captions pulled and summarized in Sources/Videos/. Raw transcripts are kept out of the vault (see “Regenerating transcripts” below).
Error analysis & automation
| Video | Length | Notes |
|---|---|---|
| How to Automate AI Evals (Correctly), Shreya Shankar | 27m | ✅ Video - How to Automate AI Evals Correctly |
| Error Analysis: The Highest ROI Technique In AI Engineering (Nurture Boss) | 12m | ✅ Video - Error Analysis Highest ROI Technique (Nurture Boss) |
| Intro To Error Analysis: Creating Custom Data Annotation Apps | 42m | ✅ Video - Intro to Error Analysis Custom Annotation Apps |
| Why AI evals are the hottest new skill for product builders (Lenny’s Podcast; the FAQ labels this link “Intro to Error Analysis With Just Spreadsheets”), https://www.youtube.com/watch?v=BsWxPI9UM4c | 1h47m | ⏳ captions rate-limited, see below |
| Carrying Out Error Analysis (Andrew Ng, C3W2L01), https://www.youtube.com/watch?v=JoAxZsdw_3w | 11m | ⏳ captions rate-limited |
Case studies & practitioners
| Video | Length | Notes |
|---|---|---|
| How To Build AI Evals (Nova Escola, Lucas Machado Rocha) | 34m | ✅ Video - How To Build AI Evals (Nova Escola) |
| From Noob to Automated Evals In A Week (as a PM), Teresa Torres | 1h10m | ✅ Video - From Noob to Automated Evals in a Week (Teresa Torres) |
| The Evals That Made GitHub Copilot | 58m | ✅ Video - The Evals That Made GitHub Copilot |
| Data and DevOps Tools for Evaluating and Productionizing LLMs (Vanishing Gradients, Rechat) | 1h05m | ✅ Video - Data and DevOps Tools for LLMs (Rechat, Vanishing Gradients) |
| How to Use LangSmith to Achieve a 30% Accuracy Improvement (Dosu) | 16m | ✅ Video - Continual In-Context Learning (Dosu, LangSmith) |
Judges, roadmaps, teams
| Video | Length | Notes |
|---|---|---|
| Reasoning Models & LLM-as-a-Judge, Alex Volkov | 1h03m | ✅ Video - Reasoning Models and LLM-as-a-Judge |
| Stop Managing AI Projects Like Traditional Software, Bryan Bischof | 56m | ✅ Video - Stop Managing AI Projects Like Traditional Software |
| How Engineers and PMs should collaborate on Evals, Aman Khan | 34m | ✅ Video - How Engineers and PMs Should Collaborate on Evals |
| The Revenge of the Data Scientist (PyAI Conf 2026) | 18m | ✅ matches Source - The Revenge of the Data Scientist |
Tools
| Video | Length | Notes |
|---|---|---|
| Build Your Own Eval Tools With Notebooks! (marimo, Vincent Warmerdam) | 1h14m | ✅ Video - Build Your Own Eval Tools With Notebooks (marimo) |
| Building Eval Tools with FastHTML (AnkiHub), Isaac Flath | 42m | ✅ Video - Building Eval Tools with FastHTML (AnkiHub) |
| Building custom eval tools with coding agents, Isaac Flath | 38m | ✅ Video - Building Custom Eval Tools With Coding Agents |
| LLM Eval Tools Compared: LangSmith / Braintrust / Arize Phoenix | 46m / 42m / 33m | ✅ Video - LLM Eval Tools Compared (LangSmith, Braintrust, Phoenix) |
Hard-to-evaluate products & verification
| Video | Length | Notes |
|---|---|---|
| How To Build AI That Earns User Trust (the eval-smell walkthrough), https://www.youtube.com/watch?v=0ASUz7gFRcw | 17m | ⏳ captions rate-limited; the written post is covered in Designing for Verification |
| How to Eval an AI Product That Seems Impossible to Eval, https://youtu.be/HJKiIBy43DM | 1m | ⏳ captions rate-limited (short clip) |
| Evals for AI Writing (clip) | 2m | ✅ Video - Short Clips (AI Writing Evals, Cursor Demos) |
| Cursor now shows you demos, not diffs | 2m | ✅ Video - Short Clips (AI Writing Evals, Cursor Demos) |
| Context Rot: When Long Context Fails, Kelly Hong | 31m | ✅ covered by Source - Context Rot (annotated slides) |
Strategy debate
| Video | Length | Notes |
|---|---|---|
| OpenAI Killed My Startup: How Product-First Companies Fight Back (“The model is the product”), Han Lee | 39m | ✅ Video - The Model Is (Not) the Product |
| Stop Worrying About Foundation Models: Your Taste, Judgment, and UX Are the Real Moat, Hamel | 15m | ✅ Video - The Model Is (Not) the Product |
Regenerating transcripts
The raw caption text isn’t stored in the vault (it’s the speakers’ content). To pull a transcript locally for personal reference:
yt-dlp --skip-download --write-auto-subs --sub-langs en --sub-format vtt -o "%(id)s.%(ext)s" "https://www.youtube.com/watch?v=<ID>"If YouTube returns HTTP 429, wait a while (or pass --cookies-from-browser chrome) and retry.