Obsidian Metadata

type	concept
sources	Source - AI Evals FAQ, Source - Context Rot

Judge Context and Context Rot

How much context should a judge get?

• Give each judge only the parts of the trace it needs for its failure mode. Don’t hand every judge the full trace by default.
• Extra context causes context rot and makes the judge worse.
• Finding the right context takes experimentation. Compare judge decisions with human labels, then inspect disagreements: was evidence missing, or was the judge distracted by irrelevant information?
• Ablation study: remove one piece of context at a time and check the change against human labels. If performance holds or improves, leave it out.
• Very long agent traces that fill the context window: consider giving the judge a search tool over the trace, but only if you absolutely need it, since it adds complexity, cost, and latency.

Context rot (Kelly Hong, Chroma)

LLM performance degrades as input length grows, even on simple tasks. 18 frontier models were tested.

• Needle-in-a-haystack is misleading. It mostly tests lexical matching. With semantic matching (little word overlap), performance falls sharply as context grows.
• Distractors (similar but wrong info, common in templated legal and financial docs) hurt more as length increases. When failing, GPT models tended to hallucinate from a distractor, while Claude models tended to abstain.
• Surprisingly, models did slightly better on shuffled haystacks than coherent ones. They don’t process context the way humans do.
• LongMemEval: focused (~100-token) history beat a full 120k-token history, clearly.
• Even replicating repeated words degrades with length (odd failures: refusals citing copyright, random output).
• No U-shaped position effect was found in these experiments. “Put important info at the start or end” isn’t reliable.
• No single model is best across tasks.

Takeaways: (1) performance isn’t uniform across input lengths; (2) how information is presented matters, not just whether it’s present; (3) context engineering is critical. Example: an orchestrator + subagents where each subagent has a clean, focused context and returns only what’s relevant.

Advice for finding context rot in your app: qualitatively compare outputs with short, focused context vs long, bloated context. What did the model miss? What could be removed?

Related

LLM-as-Judge · Document Processing and Chunk Size · RAG Evaluation