Obsidian Metadata
| type | source |
| subtype | video |
| url | https://www.youtube.com/watch?v=fA4pe9bE0LY |
| speakers | Isaac Flath, Hamel Husain |
| duration | 41:32 |
Video: Building Eval Tools with FastHTML (the AnkiHub annotation app)
Isaac walks through a real commercial annotation app built for AnkiHub (medical flashcards) with Lab Codes, plus a minimal FastHTML version (companion repo, including a FastAPI + React variant).
The use case: why off-the-shelf didn’t fit
- Retrieval: medical students search for flashcards to study. Queries vary wildly, from one word to an uploaded 50-page PDF chapter, while the “documents” are one- or two-sentence cards. That’s the inverse of typical RAG.
- In a technical medical domain, exact terms matter and “similar” isn’t good enough.
App features that emerged from real use
- Dashboard of real production queries with per-annotator progress.
- Two annotators per query, then a finalize step only where they disagree: each sees the other’s rating and records a final rating plus a note on why. The team was still developing the rubric, so disagreements were meant to spark discussion, not just produce a κ score.
- Annotators are 4th-year medical students (domain experts) working with the PM.
- ~400 candidate cards per query, so keyboard navigation matters a lot at that volume.
- Rate each result good / bad (relevant / not relevant) plus a note, and also rate the query itself (e.g., “heart” is too broad). They only discovered they needed query-level coding after annotation started, which argues for shipping something simple quickly and iterating.
- Push to Phoenix from an admin page, so datasets and annotations live in the observability store.
- Build time: roughly 5 focused hours (spread over ~3 days).
FastHTML notes
- Python-only web framework (HTML elements as Python functions, HTMX for partial updates, routes from function names) with the fastlite DB and MonsterUI components. The minimal app is about 70–80 lines.
- HTMX patterns: auto-save notes
on change, and swap just the good/bad button pair to show the stored state. - Hamel’s question: is using a less mainstream framework “swimming upstream” with AI coding tools? Isaac’s answer: provide good context/rules files for the framework and stay in a middle ground (use AI heavily but understand the code). Use whatever you know; the principles are the same in any stack.
→ Annotation Tools and Review Interfaces · Benevolent Dictator (multi-annotator with finalize step) · RAG Evaluation