Extract text from PowerPoint, Word, and Excel files
Transform office documents into searchable, analyzable text data.
Updated Open in Colab
Split documents into chunks for RAG
Break PDFs and documents into searchable chunks for retrieval-augmented generation (RAG) pipelines.
Text · Notebook · Updated
Colab runs it in your browser. If it calls a model provider, it asks for that provider's API key in a cell.
How it works
You create a view with a document_splitter iterator that automatically breaks documents into chunks. Then you add an embedding index for semantic search.
What it covers
- Split PDFs into sentences with token limits
- Control chunk size with token limits
- Add embeddings for semantic search
What it uses
- Install
- terminalpip install pixeltable sentence-transformers spacy tiktoken
- Pixeltable calls
document.document_splitter()pxt.create_table()pxt.create_view()insert()add_embedding_index()similarity()
A look at the code
doc-chunk-for-rag.ipynb
# Add embedding index for semantic searchchunks.add_embedding_index(column='text',string_embed=sentence_transformer.using(model_id='all-MiniLM-L6-v2'),)
One cell of the notebook, copied as it is. Open it in Colab for the rest.
More in Text
Browse all 103Summarize text with LLMs
Generate concise summaries of long text, articles, or documents using large language models.
Updated Open in ColabTranslate text between languages
Automatically translate content into multiple languages using LLMs.
Updated Open in ColabExtract named entities from text
Identify and extract people, organizations, locations, dates, and other entities from text using LLMs.
Updated Open in Colab