---
title: "How do I chunk PDFs for RAG in Python?"
description: "Chunk PDFs for RAG in Python: sentence windows under a token cap store no page, while a page separator with page metadata stores a 0-based page index."
url: "https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag"
updated: "2026-10-14"
doc: "https://docs.pixeltable.com/howto/cookbooks/text/doc-chunk-for-rag"
---

# How do I chunk PDFs for RAG in Python?

Chunking a PDF for RAG splits the file into retrieval units small enough to embed and cite: sentences, pages, or token windows. A sentence window does not store a page number. A page split stores a 0-based page index on the parent file. Chunking strategies trade recall against context size; the splitter is the strategy.

Updated: 2026-10-14
Part of [How do I ... in Python?](https://www.pixeltable.com/learn/how-to).

## Before you start

About 15 min.

- Python 3.11+
- pip install -U 'pixeltable[serve]'
- pxt init in the project root
- spacy, tiktoken, and en_core_web_sm for sentence splits

## On this page

- [Before you start](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag#before-you-start)
- [Store the PDF as a document column](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag#step-1)
- [Pick a chunking strategy, then split as a view](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag#step-2)
- [Index the passage text for retrieval](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag#step-3)
- [Comparison](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag#comparison)
- [Where Pixeltable fits](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag#where-pixeltable-fits)
- [Questions](https://www.pixeltable.com/learn/how-to/chunk-pdfs-for-rag#questions)

## 1. Store the PDF as a document column {#step-1}

Keep the file on the table. Text chunking that throws away the parent cannot cite a page. A Document column is the source the splitter iterates.

```python
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Docs(TableModel, name='docs'):
    title: pxt.String
    document: pxt.Document
```

## 2. Pick a chunking strategy, then split as a view {#step-2}

sentence,token_limit splits on sentences and caps each window at a token count. That strategy stores no page number. separators=page with metadata=page stores a 0-based page index. heading does nothing on a PDF. Do not request title metadata: on a PDF it is empty and hides the parent title. limit is the token cap; overlap is shared tokens between windows. A later schema update cannot change this iterator in place.

```python
class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='sentence,token_limit',
        limit=300,
        overlap=50,
    ),
):
    pass

class Pages(
    TableModel,
    name='pages',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='page',
        metadata='page',
    ),
):
    pass
```

## 3. Index the passage text for retrieval {#step-3}

Each chunk row has a text field. An embedding index on that field ranks nearest passages, still tied to the parent file. Keep the same iterator, including overlap. Insert a new PDF and only its windows are split and embedded.

```python
class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document,
        separators='sentence,token_limit',
        limit=300,
        overlap=50,
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]

@pxt.query
def search_pdfs(question: str, limit: int = 5):
    sim = Chunks.text.similarity(string=question)
    return (
        Chunks.order_by(sim, asc=False)
        .limit(limit)
        .select(text=Chunks.text, title=Docs.title, score=sim)
    )
```

## Chunking strategies for RAG {#comparison}

| Strategy | Unit | Tradeoff |
| --- | --- | --- |
| sentence,token_limit | Sentences packed up to a token cap | Prose. Stores no page number |
| page + metadata=page | One row per page | page is a 0-based page index |
| token_limit only | Windows of N tokens | Ignores sentence boundaries and pages |
| title or heading metadata | A column the splitter fills | Empty on a PDF. title hides the parent title |

## Where Pixeltable fits {#where-pixeltable-fits}

Pixeltable chunks PDFs with document_splitter on a Document column, so each passage row still points at the file. separators='sentence,token_limit' is the documented default strategy. The embedding index lives on that view. A RecursiveCharacterTextSplitter in a notebook is a list in memory, not a view that updates.

```python
import pixeltable as pxt
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import sentence_transformer

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')

class Docs(TableModel, name='docs'):
    document: pxt.Document

class Chunks(
    TableModel,
    name='chunks',
    base=Docs,
    iterator=document_splitter(
        document=Docs.document, separators='sentence,token_limit', limit=300
    ),
):
    __indexes__ = [
        pxt.EmbeddingIndex(text, string_embed=text_embed),
    ]
```

## Concepts

- [What is chunking?](https://www.pixeltable.com/learn/what-is-chunking)
- [What is PDF RAG?](https://www.pixeltable.com/learn/what-is-pdf-rag)
- [What is a document splitter?](https://www.pixeltable.com/learn/what-is-a-document-splitter)

## Use case

- [Production RAG: From Documents to Answers in One System](https://www.pixeltable.com/use-cases/production-rag-implementation)

## Questions {#questions}

### What chunking strategy should I start with? {#faq-1}

sentence,token_limit with limit=300 is the default for prose, and it stores no page number. Use separators=page with metadata=page when the row must record a page; that value is a 0-based page index. heading does nothing on a PDF. Do not request metadata=title: it is empty for a PDF and hides the parent title. Do not pass separators='sentence' with a separate limit=; combine them as sentence,token_limit.

### Does chunking throw away the PDF? {#faq-2}

It must not. Each passage row is a view over the document. A changed file rebuilds the units that came from it. Embedding the whole PDF as one vector is the other failure: you cannot cite a page.

### Do I need LangChain text splitters? {#faq-3}

No. document_splitter is the iterator. LangChain splitters produce lists you have to reload. Pixeltable vs LangChain covers that joint.

## Related

- [What is chunking?](https://www.pixeltable.com/learn/what-is-chunking)
- [What is PDF RAG?](https://www.pixeltable.com/learn/what-is-pdf-rag)
- [What is a document splitter?](https://www.pixeltable.com/learn/what-is-a-document-splitter)
- [Production RAG implementation](https://www.pixeltable.com/use-cases/production-rag-implementation)
- [How do I build a multimodal RAG pipeline in Python?](https://www.pixeltable.com/learn/how-to/build-a-multimodal-rag-pipeline)
- [Pixeltable vs LangChain](https://www.pixeltable.com/compare/pixeltable-vs-langchain)
- [Split documents into chunks for RAG](https://docs.pixeltable.com/howto/cookbooks/text/doc-chunk-for-rag)