RAG Text Chunker

A rule and its exception can appear in neighboring sentences, yet a search result may contain only one of them. In retrieval-augmented generation (RAG), that separation matters because retrieved passages supply information for a language model's answer. Splitting a document determines which passages can be found and what context travels with them.

A chunk is a passage treated as an individual retrieval unit. In a typical vector-search workflow, an embedding model turns each chunk into a numerical representation that can be compared with a question. Relevant chunks are retrieved and supplied to the answering model. A handbook covering account access, backups, and billing can therefore contribute the relevant passage without requiring every section in the answer context.

Chunk size affects both the focus of a passage and how much explanation it carries. The useful balance depends on the document and the questions it needs to answer.

Overlap repeats part of one passage in the next. It can help preserve evidence that would otherwise straddle a boundary, but its value depends on what actually repeats. Embedding overlapping passages separately processes their shared text more than once. Including both passages unchanged in an answer prompt also spends context on that shared text twice.

Fixed windows advance through text by a length rule, even when a boundary cuts a word. Recursive separator splitting first tries larger breaks, such as paragraph boundaries, and uses smaller breaks when a passage is too long. This can preserve readable pieces, but literal separators do not understand an argument, a table, or the relationship between a rule and its exception.

Length units also matter. Models impose limits in tokens, the units produced by their tokenizer. A character count cannot establish a token count or guarantee that a passage fits a model. A document name and source offsets locate the exact occurrence of a passage so it can be checked against the original text.

A useful chunking choice needs testing with representative documents and questions. Inspect whether the passages needed for an answer are retrieved, whether their meaning survives the boundaries, and how much repeated context accompanies them.

How to Use This Tool:

Start with one document. The sample is editable, and the results update as the text or splitting settings change.

1. Paste your document into Source text, or choose Browse TXT/MD to read a TXT or Markdown file. Dropping a file onto the text area also works. Imports accept up to 100,000 bytes, and the loaded text must fit within 20,000 Unicode code points.
2. Choose Recursive to prefer the configured text separators, or Fixed size for regularly advancing character windows. Markdown is treated as text in both modes.
3. Set Chunk size and a smaller Requested overlap. The initial values are 300 and 50 code points. Use the number fields for exact values.
   If Chunk readiness shows Check inputs, follow the message beneath it. Overlap must be smaller than chunk size. A result exceeding 2,000 chunks needs a larger chunk size or less overlap.
4. Choose Chunk edges. Keep Trim edge whitespace for trimmed passage edges, or select Preserve edge whitespace when indentation, blank lines, or exact whitespace coverage matters.
5. Open Advanced when you need a different ordered separator list or a Source name. The name labels the records without changing their boundaries; importing a file does not fill it automatically.
6. Inspect Overlap preview and Chunk records before passing the text onward. Highlighted prefixes show actual repetition. Download JSONL provides one record per line with chunk text, splitting settings, and source positions for subsequent embedding work.

Interpreting Results:

Chunk readiness confirms that the splitting settings produced usable records. It does not assess whether those passages will answer a question. This tool creates neither embeddings nor retrieval-quality scores, so check the finished chunks with the intended model's tokenizer and evaluate them in the retrieval system you plan to use.

• Actual overlap describes shared source positions with the immediately preceding chunk. A zero value can be correct in Recursive mode even when overlap was requested. Check the passage boundary for any context the next chunk needs.
• Repeated totals the adjacent overlaps. It counts repeated occurrences, not distinct repeated words or phrases anywhere in the document.
• Edge characters omitted counts source characters absent from every retained chunk after trimming. If whitespace must be preserved, compare the result with Preserve edge whitespace.
• Smallest, Average, and Largest describe retained chunk lengths. A short final chunk or a shorter recursive chunk is expected; inspect whether its text still makes sense.

Chunk records and CSV use code-point offsets. JSONL also includes UTF-16 offsets for systems that index text in those units.

Size trade-offs compares record counts at nearby chunk sizes while keeping requested overlap, separators, and the edge-whitespace policy unchanged.

The chart compares record counts, not answer quality, token usage, or cost. An alternative can produce no retained text or exceed the 2,000-chunk limit even though its count is plotted. Inspect its boundaries and check the resulting validation message when entering that size.

Technical Details:

Character splitting produces ordered ranges within the original text. A size limit bounds each range, while the splitting rule determines where ranges start and end. Retaining the original positions makes repeated passages distinguishable and allows every chunk to be checked against the source.

Transformation Core

The default recursive separator order is ["\n\n", "\n", " ", ""]: two line-feed characters, one line feed, a literal space, then individual code points. The empty final string supplies the character fallback.

1. Select the first separator in the remaining list that occurs within the current passage.
2. Split at that separator while keeping it at the beginning of the following piece. Separator characters still count toward the size limit.
3. Collect pieces shorter than the size limit. When a piece is equal to or longer than the limit, finish the preceding collection and split that piece again using the remaining separators.
4. Merge collected pieces in order while the next piece fits. When adding it would exceed the limit, emit the current chunk. Remove whole pieces from its beginning until the retained length is at most the requested overlap and the next piece will fit.
5. Emit the remaining pieces. Apply the selected edge-whitespace policy to merged chunks.

Whole-piece retention explains overlap shortfalls. A trailing piece longer than the overlap allowance is removed in full, and separately split passages do not carry a retained tail between their merge operations. Increasing requested overlap therefore does not guarantee that every neighboring pair shares text.

Custom separators replace the default list and are matched literally in the supplied order. They are not regular expressions. Keeping separators attached to their following pieces preserves contiguous source slices, although edge trimming can later remove separator whitespace.

Formula Core

For nonempty text, fixed windows advance by the chunk size minus the requested overlap. Let N be the source length, C the chunk size, O the requested overlap, and d the advance, all measured in Unicode code points. Window numbers i begin at 1; positions begin at 0.

Each raw window starts at a and ends just before b. Splitting stops as soon as a window reaches the source end. The ceiling brackets round up to a whole number of windows. This count applies before trimming; whitespace-only windows can disappear afterward. With preserved edges, successive fixed windows share exactly the requested overlap, and only the final window can be shorter than the size limit.

For either mode, let s and e be a retained chunk's start and exclusive end after trimming, and let n be the number of retained chunks. Length and adjacent overlap follow directly from these positions.

Here l is retained length, o is actual overlap with the previous chunk, and the barred l is average length. Counts and offsets remain exact integers. The average uses all retained lengths and is rounded only for display, to one decimal place. The repeated total adds the individual overlap values. Omitted characters are counted by taking the union of retained source ranges and subtracting its length from the source length; a character kept in any chunk is covered.

Comparison sizes are half, the same, one and a half, and twice the selected size. Fractional sizes are rounded down, then limited to at least overlap plus one and at most 10,000 code points. Duplicate sizes are combined, so fewer than four comparisons can remain. Each count uses the same splitting rules; alternative counts can exceed the retained-output limit or be zero after trimming.

Code points and source offsets

A Unicode code point is a numbered element of Unicode text. It need not equal a whole visible character. A supplementary character such as 😀 uses one code point but two UTF-16 code units. A letter followed by a combining accent uses two code points, and some emoji use longer sequences. Character fallback and fixed windows can split those sequences even though they preserve each individual code point.

Offsets are zero-based and end-exclusive. A range from 2 to 5 includes positions 2, 3, and 4. Code-point offsets and UTF-16 offsets locate the same passage using different counting units. Both refer to positions in the supplied text, rather than file bytes.

For A😀BC😀D, recursive size 3 and overlap 1 produce the following exact slices with the default separators. The different offset columns locate the same text.

Changing line endings or normalizing Unicode after splitting can invalidate the recorded positions. Keep the exact source text associated with the records. When comparing runs, use the same text, length unit, separator order, and whitespace policy.

Whitespace and literal boundaries

Trimming affects only the edges of emitted chunks. Interior whitespace is preserved. The trimming set includes spaces, tabs, line breaks, nonbreaking spaces, and several other whitespace characters; it does not remove every invisible character. U+FEFF, for example, remains when present in the source text.

Recursive mode trims merged chunks. At the minimum size of 1, terminal code points are emitted unchanged, including whitespace. Fixed mode trims every window. An otherwise valid input that leaves no text after trimming needs preserved edges or non-whitespace content.

Line endings are not normalized during splitting. The default paragraph separator is specifically two line feeds. A blank line written as \r\n\r\n is a different sequence; custom separators can include "\r\n\r\n" and "\r\n" to match it literally. Neither mode parses Markdown headings, sentences, or tables into semantic units.

Accepted settings

Size and overlap use whole code-point counts. The following bounds keep the output finite and the separator rules unambiguous.

Privacy Notes:

Pasted text and imported files are split locally in your browser. Document contents are not uploaded for splitting or included in shared settings links. Reloading restores the settings and loads the sample text again, so keep your original document separately.

Shared settings can include custom separators and the optional Source name. Use a document label suitable for anyone who receives the link. Copied and downloaded records contain the retained document text.

The size chart may need a network connection to load its display resources. Local document processing does not imply that the entire page works without a connection.

Worked Examples:

A short note makes the boundary differences visible. Both cases use Reset password. Keep backup codes., which contains 34 code points, with size 18, requested overlap 4, and trimmed edges.

The summary shows 3 chunks, 0 repeated characters, and 2 omitted edge characters. Those omissions are the spaces at source positions 15 and 27. The retained lengths total 32, giving a displayed average of 10.7. Although no word is cut, the final chunk loses the nearby words that explain it; this is a reason to reconsider the size before using these passages for retrieval.

The chunk count remains 3, while repeated characters rise to 8 and omitted characters fall to 0. The average is 14.0. The second chunk repeats . Ke, and the third repeats code. These exact windows cut words, so equal chunk counts clearly do not imply equal boundaries or equally useful passages.

References:

• Chunk large documents for RAG and vector search in Azure AI Search, Microsoft Learn, July 2, 2026.
• Text-splitting reference, version 0.3.11, LangChain.
• UTF-8, UTF-16, UTF-32 & BOM, Unicode Consortium.
• How to configure a text splitter in LangChain, Simplified Guide; a hands-on guide using a newer splitter version.