Obsidian Metadata
| type | source |
Other Linked References
Smaller links from the FAQ and related pages, kept for completeness.
| Reference | Why it’s linked |
|---|---|
| LLM Tracing Tools’ Naming Conventions (Alex Strick van Linschoten) | Vendor terminology for traces/spans (see Traces) |
| Paul Graham, Writes and Write-Nots | ”Good writing is good thinking”, an argument for writing prompts yourself (Prompt Optimization) |
| Ben Balter, GitHub for non-technical roles | Getting non-engineers to manage prompts in Git (Prompt Management) |
| Anthropic Clio | Clustering conversations by topic, applied to coding sessions (Coding Agent Evaluation) |
| OpenAI: Introducing Codex | Private benchmark from real internal SWE tasks |
| Cyclomatic complexity | Starter metric for over-engineering |
| SWE-bench · Terminal-Bench · Aider leaderboards · HumanEval | Coding benchmarks |
| Jev by TypeSafe | General-purpose zero-shot classifier usable as an evaluator (Judge Validation) |
| Julius · Hex | AI-in-the-loop notebooks for annotation analysis |
| LangSmith · Arize · Braintrust | Vendors (Eval Tooling and Vendors) |
| AnkiHub | Hired 4th-year med students as domain experts |
| arXiv search: Abstention Ability | Abstention and Uncertainty |
| Failure is a Funnel deck (Bryan Bischof) | Transition Failure Matrix |
| Pash tweet on Claude Code retrieval | Claude Code uses agentic search (Is RAG Dead) |
| Eugene Yan: AlignEval · LLM-evaluators survey | Judge alignment tools and research |
| Databricks: Grading Notes | Critiques under another name |
| Dosu / LangSmith continual in-context learning | Dynamic few-shot examples for judges |
| applied-llms.org | What We’ve Learned From A Year of Building with LLMs |
| Jason Liu & Eugene Yan: 10 ways to be data illiterate | Data literacy for error analysis |
| AI Evals course (Maven) | Hamel & Shreya’s course; the source of most FAQ material |
| O’Reilly: Evals for AI Engineers | Book co-authored by Hamel |