Obsidian Metadata
| type | source |
| subtype | video |
| url | https://youtu.be/CAxtf2nsnKE |
| speakers | Isaac Flath, Hamel Husain |
| duration | 38:26 |
Video: Building Custom Eval Tools With Coding Agents
Isaac rebuilds the same RAG annotation app (query → ~400 flashcards to mark relevant / not relevant with notes) using AI coding agents instead of by hand, and shares his workflow. Much of the session covers his AI-coding newsletter and tool reviews. The eval-relevant techniques are below.
Techniques
- Shared context files (“ruler”): keep framework and UI guidelines (e.g., FastHTML patterns, MonsterUI components) as markdown in one folder and compile them to each agent’s native rules format (Cursor, Claude Code, Gemini CLI, etc.). Teams share one context while everyone uses their preferred agent.
- Spec-driven development: have the agent write a plan in a markdown file, then edit the plan yourself (3–4 minutes) before any code. That gives better quality and more consistent results, and consistency is how you build intuition for what works.
- A custom MCP server for testing the app: start the app, navigate routes, click elements, and read browser console logs (e.g., HTMX errors that never reach server logs) via Playwright. The agent can then debug its own UI without you copying errors back and forth. It took ~10 minutes to build and is reusable across projects.
- Rule of thumb: if you’re copy-pasting something into the model more than once, fix your context file or build a small tool.
- For less mainstream libraries, curate docs-as-markdown context and personalize it over time. When an API changes, have the model update the context file.