Text splitter installation Python
Install langchain-text-splitters using pip with: pip install -U langchain-text-splitters, or with uv using: uv add langchain-text-splitters
LangChain & LangGraph · all subjects
text processing
31 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.
RecursiveCharacterTextSplitter recommended default
For most use cases, start with the RecursiveCharacterTextSplitter. It provides a solid balance between keeping context intact and managing chunk size. This default strategy works well out of the box, and should only be adjusted if needing to fine-tune performance for a specific application.
RecursiveCharacterTextSplitter hierarchical strategy
RecursiveCharacterTextSplitter attempts to keep larger units (e.g., paragraphs) intact. If a unit exceeds the chunk size, it moves to the next level (e.g., sentences). This process continues down to the word level if necessary.
RecursiveCharacterTextSplitter Python example
Example usage: from langchain_text_splitters import RecursiveCharacterTextSplitter text_splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=0) texts = text_splitter.split_text(document)
CharacterTextSplitter token-based splitting Python example
Example using token-based splitting: from langchain_text_splitters import CharacterTextSplitter text_splitter = CharacterTextSplitter.from_tiktoken_encoder( encoding_name="cl100k_base", chunk_size=100, chunk_overlap=0 ) texts = text_splitter.split_text(document)
Text splitter installation JavaScript
Install JavaScript text splitters using npm with: npm install @langchain/textsplitters @langchain/core (requires Node.js 22+), or with pnpm: pnpm add @langchain/textsplitters @langchain/core, or yarn: yarn add @langchain/textsplitters @langchain/core, or bun: bun add @langchain/textsplitters @langchain/core
Text splitter purposes
Text splitters break large documents into smaller chunks that will be retrievable individually and fit within model context window limits.
Text structure-based splitting strategies
Text splitters can leverage inherent hierarchical structures in text such as paragraphs, sentences, and words to create splits that maintain natural language flow, maintain semantic coherence within splits, and adapt to varying levels of text granularity.
Length-based text splitting benefits
Length-based splitting is straightforward to implement, ensures consistent chunk sizes, and is easily adaptable to different model requirements. It can be token-based (splits by number of tokens, useful with language models) or character-based (splits by number of characters, more consistent across different text types).
Document structure-based splitting benefits
Structure-based splitting preserves the logical organization of the document, maintains context within each chunk, and can be more effective for downstream tasks like retrieval or summarization. It works with documents like HTML, Markdown, JSON, and code that have inherent structure.
TokenTextSplitter JavaScript example
Example using token-based splitting: import { TokenTextSplitter } from "@langchain/textsplitters"; const splitter = new TokenTextSplitter({ encodingName: "cl100k_base", chunkSize: 100, chunkOverlap: 0 }) const texts = splitter.splitText(document)
RecursiveCharacterTextSplitter Python example
from langchain_text_splitters import RecursiveCharacterTextSplitter text_splitter = RecursiveCharacterTextSplitter( chunk_size=100, chunk_overlap=20, length_function=len, is_separator_regex=False, ) texts = text_splitter.create_documents([state_of_the_union]) print(texts[0]) # page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and' result = text_splitter.split_text(state_of_the_union)[:2] # ['Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and', # 'of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.']
RecursiveCharacterTextSplitter for languages without word boundaries
For languages like Chinese, Japanese, and Thai that lack word boundaries, the default separator list can cause words to be split across chunks. To keep words together, override the separators list to include additional punctuation: ASCII full-stop ".", Unicode fullwidth full stop "．", ideographic full stop "。", zero-width space "\u200b", ASCII comma ",", Unicode fullwidth comma "，", and Unicode ideographic comma "、".
RecursiveCharacterTextSplitter with CJK separators JavaScript
const splitter = new RecursiveCharacterTextSplitter({ separators: [ "\n\n", "\n", " ", ".", ",", "\u200b", // Zero-width space "\uff0c", // Fullwidth comma "\u3001", // Ideographic comma "\uff0e", // Fullwidth full stop "\u3002", // Ideographic stop "", ], });
RecursiveCharacterTextSplitter default separator list
The RecursiveCharacterTextSplitter uses a default separator list of ["\n\n", "\n", " ", ""] to split text. It attempts to split on separators in order until chunks are small enough, prioritizing keeping paragraphs, sentences, and words together as they are semantically related.
RecursiveCharacterTextSplitter JavaScript parameters
RecursiveCharacterTextSplitter in JavaScript accepts the following parameters: chunkSize (maximum chunk size determined by lengthFunction) and chunkOverlap (target overlap between chunks to mitigate information loss when context is divided).
RecursiveCharacterTextSplitter methods
RecursiveCharacterTextSplitter provides two methods: split_text (Python) or splitText (JavaScript) to obtain string content directly, and create_documents (Python) or createDocuments (JavaScript) to create LangChain Document objects for use in downstream tasks.
RecursiveCharacterTextSplitter JavaScript example
import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters"; const splitter = new RecursiveCharacterTextSplitter({ chunkSize: 100, chunkOverlap: 0 }) const texts = splitter.createDocuments([{ pageContent: "..." }]) // Returns: [{ pageContent: "...", metadata: {} }]
Token limits in language models require token-aware text splitting
Language models have token limits that must not be exceeded. When splitting text into chunks, the same tokenizer used by the language model should be used to count tokens for accurate splitting.
tiktoken BPE tokenizer for token counting
tiktoken is a fast BPE tokenizer created by OpenAI. It can be used to estimate tokens in text and is typically more accurate for OpenAI models than other tokenizers.
CharacterTextSplitter.from_tiktoken_encoder() method parameters
The CharacterTextSplitter.from_tiktoken_encoder() method accepts either encoding_name (e.g. cl100k_base) or model_name (e.g. gpt-4) as arguments. Additional arguments like chunk_size, chunk_overlap, and separators are used to instantiate CharacterTextSplitter. Splits from this method can be larger than the chunk size measured by the tiktoken tokenizer.
RecursiveCharacterTextSplitter.from_tiktoken_encoder() enforces hard chunk size constraint
RecursiveCharacterTextSplitter.from_tiktoken_encoder() implements a hard constraint on chunk size. Each split will be recursively split if it has a larger size than specified. It accepts model_name (e.g. gpt-4), chunk_size, and chunk_overlap as parameters.
TokenTextSplitter works with tiktoken for token-based splitting
TokenTextSplitter works with tiktoken directly and ensures each split is smaller than the specified chunk size. It accepts chunk_size and chunk_overlap as parameters.
Unicode handling pitfall with TokenTextSplitter
Using TokenTextSplitter directly with languages that have characters encoding to two or more tokens (such as Chinese and Japanese) can split the tokens for a character between two chunks, causing malformed Unicode characters. Use RecursiveCharacterTextSplitter.from_tiktoken_encoder() or CharacterTextSplitter.from_tiktoken_encoder() instead to ensure chunks contain valid Unicode strings.
js-tiktoken is JavaScript version of OpenAI BPE tokenizer
js-tiktoken is a JavaScript version of the BPE tokenizer created by OpenAI, used with TokenTextSplitter for token-based splitting in JavaScript environments.
TokenTextSplitter JavaScript API with encodingName parameter
In JavaScript, TokenTextSplitter accepts encodingName (e.g. cl100k_base), chunkSize, and chunkOverlap as parameters. Splits from this method can be larger than the chunk size measured by the tiktoken tokenizer.
spaCy text splitter behavior
SpacyTextSplitter splits text by spaCy tokenizer and measures chunk size by number of characters. It accepts chunk_size as a parameter.
SentenceTransformersTokenTextSplitter configuration parameters
SentenceTransformersTokenTextSplitter accepts the following optional parameters: chunk_overlap (integer count of token overlap), model_name (defaults to "sentence-transformers/all-mpnet-base-v2"), and tokens_per_chunk (desired token count per chunk).
NLTK text splitter behavior
NLTKTextSplitter splits text by NLTK tokenizer and measures chunk size by number of characters. It accepts chunk_size as a parameter.
KoNLPy Kkma analyzer for Korean text splitting
KonlpyTextSplitter uses KoNLPy's Kkma (Korean Knowledge Morpheme Analyzer) for Korean language processing. Kkma provides detailed morphological analysis, breaks down sentences into words and words into morphemes, and identifies parts of speech. It can segment text into individual sentences. Kkma is best suited for applications where analytical depth is prioritized over rapid text processing due to its lower speed.
Hugging Face GPT2TokenizerFast for token counting
CharacterTextSplitter can be instantiated with CharacterTextSplitter.from_huggingface_tokenizer() using a Hugging Face tokenizer like GPT2TokenizerFast. The chunk size is measured by number of tokens calculated by the Hugging Face tokenizer. It accepts tokenizer, chunk_size, and chunk_overlap as parameters.