Obsidian Metadata
| type | reading-path |
| source | https://hamel.dev/notes/llm/evals/start/hard-to-evaluate/index.html |
Reading Path: My Product Feels Too Hard to Evaluate
“Our outputs are subjective, long, or involve many steps. Even a knowledgeable person has trouble deciding whether they’re right.”
- Source - Its Hard to Eval Is a Product Smell → Designing for Verification
- Videos: How to Eval an AI Product That Seems Impossible to Eval, Evals for AI Writing (Video Library)
Making human review easier: Designing for Verification · Reviewing Large Traces · Annotation Tools and Review Interfaces · Team Collaboration on Evals · Evals with Sensitive Data
Workflows and varied requests: Coding Agent Evaluation · Multi-Step Workflow Evaluation · Agentic Workflow Evaluation · Multi-turn Conversation Evaluation · Human Handoff Evaluation · Diverse Query Evaluation
Retrieval and evidence: RAG Evaluation · Document Processing and Chunk Size · Abstention and Uncertainty · Judge Context and Context Rot