Split documents into chunks for RAG
Break PDFs and documents into searchable chunks for retrieval-augmented generation (RAG) pipelines.
Updated Open in Colab
Insert a URL as a document, then compare chunking strategies on it
Text · Recipe · Updated
document.document_splitter()pxt.create_table()pxt.create_view()insert()add_embedding_index()similarity()from pixeltable.functions.huggingface import sentence_transformerembed = sentence_transformer.using(model_id='intfloat/e5-large-v2')by_paragraph.add_embedding_index('text', string_embed=embed)sim = by_paragraph.text.similarity(string='how do ants communicate')by_paragraph.order_by(sim, asc=False).limit(5).select(by_paragraph.text, sim).collect()
One block of the recipe, copied as it is. The rest is on the docs page.
Break PDFs and documents into searchable chunks for retrieval-augmented generation (RAG) pipelines.
Transform office documents into searchable, analyzable text data.
Generate concise summaries of long text, articles, or documents using large language models.
Automatically translate content into multiple languages using LLMs.