---
title: "LectureSync: Slide-Grounded Lecture Q&A Without an Aligner Service"
date: "2026-09-17"
dateModified: "2026-09-17"
author: "Pierre Brunelle"
tags:
  - Education
  - Video
  - Document RAG
  - CLIP
  - TableModel
  - Multimodal AI
  - Pixeltable Cloud
description: "Pair an MP4 with a slide PDF. Split pages, sample frames, retrieve, then cite. Pixeltable is two views on one lecture — not PyMuPDF plus FFmpeg plus Pinecone."
url: "https://www.pixeltable.com/blog/lecturesync-slide-lecture-qa"
---

# LectureSync: Slide-Grounded Lecture Q&A Without an Aligner Service

**Summary:** LectureSync is the student loop: a lecture video and a slide deck become searchable pages and sampled frames, then an answer cites the slide and the transcript. You do not stitch a PDF splitter, a frame farm, and two vector databases. You declare `Lectures`, a `Slides` view, and a `Frames` view. Retrieve first. Generate second. `pip install pixeltable`.

 
## The Product

 

 - Ingest `lecture_video` + `slides_pdf`

 - One row per slide page (text + page image)

 - One row per sampled second of video

 - Ask “What does backpropagation compute?” and land on a page plus nearby speech

 

 
Passage RAG on a prospectus is [DocuVision](/blog/docuvision-pdf-chart-qa). Visual moments in a clip library are [ClipFinder](/blog/clipfinder-semantic-video-moment-search). This post is both media types on one lecture.

 
## The Stitch You Delete

 
PyMuPDF for pages. FFmpeg for JPEGs. Whisper in a GPU box. Chroma for slide text. CLIP in Pinecone for frames. A notebook that hopes the timestamps line up. Delete the PDF; the chunks in Chroma are your problem. In a catalog, delete the lecture; both views follow.

 
## The Receipt

 
```python
import pixeltable as pxt
from pixeltable.catalog.model import Column, EmbeddingIndex
from pixeltable.functions import whisper
from pixeltable.functions.document import document_splitter
from pixeltable.functions.huggingface import clip, sentence_transformer
from pixeltable.functions.video import extract_audio, frame_iterator
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()
text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')
clip_embed = clip.using(model_id='openai/clip-vit-base-patch32')

class Lectures(TableModel, name='lectures'):
 lecture_id = Column(pxt.String, primary_key=True)
 course_name: pxt.String
 title: pxt.String
 instructor: pxt.String
 lecture_video: pxt.Video
 slides_pdf: pxt.Document

 audio = extract_audio(lecture_video)
 transcript = whisper.transcribe(audio, model='base')

class Slides(
 TableModel,
 name='slides',
 base=Lectures,
 iterator=document_splitter(
 Lectures.slides_pdf,
 separators='page',
 elements=['text', 'image'],
 metadata='page',
 ),
):
 __indexes__ = [
 EmbeddingIndex(column=text, embedding=text_embed, name='ls_slide_text_idx'),
 EmbeddingIndex(column=image, embedding=clip_embed, name='ls_slide_image_idx'),
 ]

class Frames(
 TableModel,
 name='frames',
 base=Lectures,
 iterator=frame_iterator(video=Lectures.lecture_video, fps=1),
):
 __indexes__ = [
 EmbeddingIndex(column=frame, embedding=clip_embed, name='ls_frame_image_idx'),
 ]

@pxt.query
def retrieve_slides(question: str, top_k: int = 3):
 sim = Slides.text.similarity(string=question)
 return (
 Slides.order_by(sim, asc=False)
 .limit(top_k)
 .select(Lectures.lecture_id, Lectures.title, Slides.text, Slides.image, Lectures.transcript)
 )

api = FastAPIRouter(name='lecturesync')
api.add_insert_route(
 Lectures,
 path='/lectures',
 inputs=[
 Lectures.lecture_video,
 Lectures.slides_pdf,
 Lectures.lecture_id,
 Lectures.course_name,
 Lectures.title,
 Lectures.instructor,
 ],
)
api.add_query_route(path='/retrieve', query=retrieve_slides)

# After retrieve_slides(question):
# from pixeltable.functions import gemini
# gemini.generate_content([question, slide_text, transcript], model='gemini-2.5-flash')
```

 
Retrieval is the index. Generation is one Gemini call over those strings — not a shipped slide-aligner. `Frames` is the visual audit trail at 1 fps, same iterator as ClipFinder. Same `app.py` on Cloud: `pxt db update` → `pxt schema update` → `pxt service update`.

 
## What This Is Not

 
Not an LMS. Not Zoom. Not “we forced every slide to a timestamp with a private model.” You retrieve the page, you retrieve nearby speech, you generate. Auth and grades stay where they are.

 
## People Also Ask

 
**Why `separators='page'`?** Decks are pages, not 300-token passages. DocuVision uses `token_limit` for prospectuses. Change the iterator kwargs; do not change the table.

 
**Do I need two CLIP indexes?** Slide images and video frames are different rows. Each view gets its own index. Do not mix `embedding=` and `image_embed=` on one snippet.

 
**How do I go to Cloud?** Same file. `PIXELTABLE_API_KEY`, `pxt://org:db`, then db → schema → service.

 
**Is this an AI automation workflow?** Yes. Insert the deck and the lecture; splitter, frames, and retrieve-then-generate run as the table. The category is [AI automation workflow](/blog/ai-automation-workflow).

 
## Ground the Answer in the Deck

 
Declare the lecture. Split the PDF. Sample the video. Retrieve, then generate. Keep the citation. Delete the aligner service.

 

 - **[Pixeltable on GitHub](https://github.com/pixeltable/pixeltable)**: `pip install pixeltable`

 - **[ReelForge](/blog/reelforge-podcast-chaptering-viral-clips)** — podcast chapters

 - **[MedDossier](/blog/meddossier-clinical-intake-triage)** — clinical intake row

 - **[AdRadar](/blog/adradar-creative-fatigue-visual-search)** — creative fatigue

 - **[DocuVision](/blog/docuvision-pdf-chart-qa)** — PDF Q&A

 - **[ClipFinder](/blog/clipfinder-semantic-video-moment-search)** — NL video moments

 - **[Pixeltable documentation](https://docs.pixeltable.com)**