What is Chunk Size?
Chunk size is the target length of individual segments when dividing documents into chunks for embedding and retrieval in AI systems. It is typically measured in tokens or characters and represents a fundamental parameter that significantly affects the performance and behavior of retrieval-augmented generation (RAG) systems and agent memory architectures.
Selecting an appropriate chunk size involves balancing multiple competing concerns. Smaller chunks (e.g., 128-256 tokens) provide more granular retrieval and can surface very specific information, but may lack sufficient context to be independently meaningful. Larger chunks (e.g., 512-1024 tokens) preserve more context and relationships within the text, but may include irrelevant information that dilutes relevance scores and may exceed the input limits of some embedding models.
The optimal chunk size depends on several factors including the nature of the content (technical documentation vs. narrative text), the embedding model's capabilities, the expected query patterns, and how the retrieved chunks will be used. Many systems experiment with different chunk sizes to find what works best for their specific use case, and some advanced implementations use adaptive chunking strategies that vary chunk size based on document structure or content type.