What chunking strategy should I start with?

sentence,token_limit with limit=300 is the default for prose, and it stores no page number. Use separators=page with metadata=page when the row must record a page; that value is a 0-based page index. heading does nothing on a PDF. Do not request metadata=title: it is empty for a PDF and hides the parent title. Do not pass separators='sentence' with a separate limit=; combine them as sentence,token_limit.

Does chunking throw away the PDF?

It must not. Each passage row is a view over the document. A changed file rebuilds the units that came from it. Embedding the whole PDF as one vector is the other failure: you cannot cite a page.

Do I need LangChain text splitters?

No. document_splitter is the iterator. LangChain splitters produce lists you have to reload. Pixeltable vs LangChain covers that joint.

How to

How do I chunk PDFs for RAG in Python?

Chunking a PDF for RAG splits the file into retrieval units small enough to embed and cite: sentences, pages, or token windows. A sentence window does not store a page number. A page split stores a 0-based page index on the parent file. Chunking strategies trade recall against context size; the splitter is the strategy.

Updated 2026-10-14 · Part of How do I ... in Python?

Before you start

About 15 min.

Python 3.11+
pip install -U 'pixeltable[serve]'
pxt init in the project root
spacy, tiktoken, and en_core_web_sm for sentence splits

1. Store the PDF as a document column

Keep the file on the table. Text chunking that throws away the parent cannot cite a page. A Document column is the source the splitter iterates.

import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Docs(TableModel, name='docs'):
    title: pxt.String
    document: pxt.Document

2. Pick a chunking strategy, then split as a view

sentence,token_limit splits on sentences and caps each window at a token count. That strategy stores no page number. separators=page with metadata=page stores a 0-based page index. heading does nothing on a PDF. Do not request title metadata: on a PDF it is empty and hides the parent title. limit is the token cap; overlap is shared tokens between windows. A later schema update cannot change this iterator in place.

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='sentence,token_limit',
        limit=300,
        overlap=50,
    ),
):
    pass

class Pages(
    TableModel,
    name='pages',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='page',
        metadata='page',
    ),
):
    pass

3. Index the passage text for retrieval

Each chunk row has a text field. An embedding index on that field ranks nearest passages, still tied to the parent file. Keep the same iterator, including overlap. Insert a new PDF and only its windows are split and embedded.

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='sentence,token_limit',
        limit=300,
        overlap=50,
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

@pxt.query
def search_pdfs(question: str, limit: int = 5):
    sim = Chunks.text.similarity(string=question)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(text=Chunks.text, title=Docs.title, score=sim)
    )

Chunking strategies for RAG

Chunking strategies for RAG
Strategy	Unit	Tradeoff
sentence,token_limit	Sentences packed up to a token cap	Prose. Stores no page number
page + metadata=page	One row per page	page is a 0-based page index
token_limit only	Windows of N tokens	Ignores sentence boundaries and pages
title or heading metadata	A column the splitter fills	Empty on a PDF. title hides the parent title

Where Pixeltable fits

Pixeltable chunks PDFs with document_splitter on a Document column, so each passage row still points at the file. separators='sentence,token_limit' is the documented default strategy. The embedding index lives on that view. A RecursiveCharacterTextSplitter in a notebook is a list in memory, not a view that updates.

import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Docs(TableModel, name='docs'):
    document: pxt.Document

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=300
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

Questions

What chunking strategy should I start with?: sentence,token_limit with limit=300 is the default for prose, and it stores no page number. Use separators=page with metadata=page when the row must record a page; that value is a 0-based page index. heading does nothing on a PDF. Do not request metadata=title: it is empty for a PDF and hides the parent title. Do not pass separators='sentence' with a separate limit=; combine them as sentence,token_limit.
Does chunking throw away the PDF?: It must not. Each passage row is a view over the document. A changed file rebuilds the units that came from it. Embedding the whole PDF as one vector is the other failure: you cannot cite a page.
Do I need LangChain text splitters?: No. document_splitter is the iterator. LangChain splitters produce lists you have to reload. Pixeltable vs LangChain covers that joint.