Obsidian Metadata
| type | source |
| subtype | video |
| url | https://www.youtube.com/watch?v=tHZtq_pJSGo |
| speakers | Harrison Chase (LangChain) |
| duration | 15:50 |
Video: How Dosu Got a 30% Accuracy Improvement with No Prompt Engineering
Linked from the LLM-judge guide as the “continual in-context learning” / dynamic few-shot technique.
The pattern
- Log each prediction with a run ID so feedback can attach to it later.
- Capture user feedback: a positive score, or a correction (the value it should have been). In Dosu’s case, users relabeling a GitHub issue (“bug” → “documentation”) act as the correction.
- Automation rules send positively scored runs, and corrected runs (using the correction as the output), into a dataset.
- At inference, pull examples from that dataset into the prompt as few-shot examples. The model learns project-specific conventions (e.g., “fix bug in docs” is documentation for LangChain, but might be bug for another project).
- At scale, retrieve the ~5–10 most similar examples by embedding similarity instead of including everything.
Relevance to evals
Hamel adapts this to LLM judges: choose the judge’s few-shot examples dynamically based on the item being judged. That’s an advanced option once you have a pool of labeled critiques. Keep dev and test examples out of the pool (Judge Validation).