Ingest a web page

Insert a URL as a document, then compare chunking strategies on it

Text · Recipe · Updated

What it covers

  • Insert the URL
  • Chunk it, two ways
  • Make it searchable

What it uses

Pixeltable calls
document.document_splitter()pxt.create_table()pxt.create_view()insert()add_embedding_index()similarity()

A look at the code

doc-ingest-website.mdx
from pixeltable.functions.huggingface import sentence_transformer
embed = sentence_transformer.using(model_id='intfloat/e5-large-v2')
by_paragraph.add_embedding_index('text', string_embed=embed)
sim = by_paragraph.text.similarity(string='how do ants communicate')
by_paragraph.order_by(sim, asc=False).limit(5).select(by_paragraph.text, sim).collect()

One block of the recipe, copied as it is. The rest is on the docs page.

Browse all 103