Obsidian Metadata
| type | guide |
| sources | Source - AI Evals FAQ |
Document Processing and Chunk Size
Unlike RAG (chunks optimized for retrieval), document processing assumes the model sees every chunk. The goal is to split text so the model reasons well without being overwhelmed. Even if a document fits in the context window, splitting may be better. Long inputs degrade performance, especially mid-context.
1. Fixed-output tasks → larger chunks
The output doesn’t grow with the input: extract a number, answer a specific question, classify a section.
- “What’s the penalty clause in this contract?” · “What was the CEO’s salary in 2023?”
- Use the largest chunk likely to contain the answer. That means fewer calls and less fragmentation.
- But avoid irrelevant text: models are distractible and under-attend the middle. If cost or latency matters, pre-filter with keyword search or a lightweight retriever.
2. Expansive-output tasks → smaller chunks
The output grows with the input: summarization, exhaustive extraction.
- “Summarize each section” · “List all customer complaints”
- Smaller chunks preserve reasoning quality and completeness. Process chunks independently, then aggregate (map-reduce). Respect content boundaries (paragraphs, sections, chapters). This also helps with output token limits.
Why chunk size matters
A bigger chunk means a heavier cognitive load. Models have limited capacity to retain and correlate details across long text and may favor the beginning or end, producing coarse summaries or missed facts. You’re trading global context for local focus (see Judge Context and Context Rot).
Treat chunk size as a hyperparameter and validate with experiments. The optimum varies by domain and model.