Chunking strategy

Structure-aware chunking uses a dynamic algorithm to determine the optimal chunk based on semantic boundaries, token limits, text formatting, and structure.

The following elements are taken into consideration when determining the chunk boundaries:

By using semantic, formatting, and structural markers to set the boundaries of each chunk, instead of a fixed word count, chunks are more coherent and contextually focused. This is especially true when using the Markdown data stream, which preserves the item’s structure and formatting, to create chunks. When using the body text data stream, structure-aware chunking can still perceive elements like headings, lists, and paragraphs by using newline characters and indentation patterns, but not as effectively as when using the Markdown data stream.

Because each chunk is created with a focus on maintaining semantic coherence and focus, the size of each chunk varies, while respecting token limits.

Complex items containing tables, hierarchical information, and structured data benefit significantly from this approach. Text that belong together stay together, improving the contextual relevance of each chunk. Semantic boundaries, which are natural breakpoints in the text where the subject shifts or completes, also help dictate where chunks begin and end. Table items, lists, and sections are preserved within a single chunk whenever possible, instead of being split across multiple chunks.

Unlike with fixed-size chunking, there’s no content overlap between chunks. Content overlap can sometimes lead to contradictory information during content retrieval because the same text can appear in more than one chunk and in different contexts. Therefore, the risk of contradictory information from retrieved chunks is reduced when using structure-aware chunking.

Structure-aware chunking is specifically optimized for large language models (LLMs) and RAG systems. Chunks created using semantic and structural markers, and with no content overlap, improve the semantic alignment between the query intent and the chunks retrieved by the CPR model. In the context of your RAG system, this results in more relevant, comprehensive, and coherent chunks that you can use in your LLM-powered application to generate higher-quality responses.

Structure-aware chunking requires more processing than fixed-size chunking, and typically results in more chunks per item. This impacts the number of chunks that count toward the model embedding limits.

As the name implies, fixed-size chunking creates chunks by splitting text into segments of a fixed number of words.

The text is split using a rolling window of 250 whitespace-delimited words. For a given item, the first chunk contains the first 250 words, the second chunk contains the next 250 words, and so on. The last chunk contains the remaining words, which may be fewer than 250 words.

Semantic boundaries (natural breakpoints in the text where the subject shifts or completes), formatting, and text structure, such as headings, paragraphs, and lists, aren’t taken into consideration when creating the chunks. Because of this, chunks are created with up to 10% overlap between chunks. This is done to preserve context continuity so that important context isn’t lost when the text is separated into chunks. Content overlap, however, can lead to contradictory information during content retrieval because the same text can appear in different contexts.

Fixed-size chunking requires less processing than structure-aware chunking, and typically results in less chunks per item, which may be a consideration given the model embedding limits.

The chunking strategy you choose for your model impacts how the text is segmented into chunks.

Choosing between structure-aware and fixed-size chunking comes down to the following considerations:

In summary, you should choose structure-aware chunking unless your dataset is too large and exceeds the chunk embedding limits, or if your model requires daily or frequent refreshes that can’t accommodate the longer processing times.

When items are indexed, the indexing pipeline processes each item into different data streams that are used for specific purposes. The data streams that pertain to the chunking process are the body text and body Markdown data streams:

For a given item, the model uses either the body text or body Markdown data stream to create the chunks. If a Markdown data stream exists for an item, the model automatically uses that data stream to create the chunks. There’s no configuration required to use the Markdown data stream. If a Markdown data stream isn’t available for an item, the model uses the body text data stream instead to create the chunks.