Retrieval-Augmented Generation (RAG) systems are powerful tools for combining large-scale document retrieval with generative AI. However, traditional approaches to RAG can lead to significant storage inefficiencies. In this post, we explore a novel strategy called "chunk-on-demand," which promises to reduce storage costs while maintaining performance, inspired by a recent discussion on X by @femke_plantinga.
The Problem with Pre-Chunking
In traditional RAG pipelines, documents are processed through a method known as "pre-chunking." This involves:
- Ingesting Documents: Taking in large volumes of text data.
- Splitting into Chunks: Breaking documents into smaller, manageable pieces.
- Vectorizing: Converting each chunk into a vector embedding.
- Storing: Saving all chunk embeddings in a vector database.
This process happens upfront, regardless of whether the chunks are ever retrieved or used. As a result, storage requirements can balloon, especially for large datasets where many chunks may never be accessed. This inefficiency drives up costs and complexity, particularly for organizations managing vast document repositories.
Introducing Chunk-on-Demand
Chunk-on-demand offers a smarter alternative. Instead of pre-processing every document, this approach defers chunking until a document is deemed relevant to a user query. Here's how it works:
- Store Whole Documents: Initially, full documents are stored in their entirety and vectorized for high-level relevance search.
- Query-Driven Search: When a user submits a query, the system searches across whole documents to identify relevant ones.
- Selective Chunking: If a relevant document exceeds the model's context window, it is then chunked and vectorized.
- Compressed Storage: These chunks are stored in a separate, compressed collection with references to the original document.
- Organic Growth: Over time, the system only retains chunks for frequently accessed content, while unused documents remain whole.
This method significantly reduces storage needs by only processing and storing chunks for documents that are actually used. It also allows access to both full documents and their chunks, offering flexibility for various use cases.
Visualizing the Difference
To illustrate, consider the following comparison:
- Pre-Chunking: Data → Chunking → Vectorize → Store all chunks in a collection.
- Chunk-on-Demand: Data → Vectorize whole documents → Store documents → Query → Search → Identify relevant documents → Chunk and vectorize only those → Store chunks in a separate collection.
This diagram, shared in the original X post, highlights the streamlined process of chunk-on-demand, where chunking acts more like a re-ranking step triggered by actual demand.
Benefits and Trade-offs
The chunk-on-demand approach offers several advantages:
- Reduced Storage Costs: By only chunking relevant documents, storage requirements are minimized.
- Flexible Access: Both full documents and chunks are available, catering to different query needs.
- Scalability: Storage grows organically based on usage, making it ideal for large-scale systems.
However, there’s a trade-off: the first access to a document requiring chunking introduces additional latency due to on-the-fly processing. Subsequent searches benefit from cached chunks, mitigating this issue.
Community Insights
The X discussion also sparked interesting ideas. One commenter, @runonthespot, suggested storing "chunk ranges + metadata (e.g., vectors)" and yielding chunks only for matches. This variation could further optimize storage by pre-computing metadata for potential chunk boundaries without fully generating chunks until needed, potentially reducing latency and storage even more.
Try It Out
The chunk-on-demand approach is an exciting step toward more efficient RAG systems. Tools like Elysia, mentioned in the original post, are exploring this method. For those interested in experimenting, check out the full discussion on X: link to post.
Conclusion
As RAG systems become integral to AI-driven applications, optimizing storage is critical for cost and performance. Chunk-on-demand offers a promising solution, balancing efficiency with flexibility. By rethinking how we process and store data, we can build more scalable and cost-effective RAG pipelines. Stay tuned for more innovations in this space!
Originally inspired by a post from @femke_plantinga on X.