How to
How do I transcribe audio uploads and make them searchable?
Accept the upload, split it into timestamped slices, run speech-to-text (Whisper or a hosted API) on each slice, embed the text, and query by meaning; each hit points back to the file and the slice where it was said. When a new file arrives, only that file is transcribed and indexed.
Updated · Part of How do I ... in Python?
Before you start
About 30 min.
- Python 3.11+
- pip install -U 'pixeltable[serve]'
- pxt init in the project root
- ffmpeg
- openai-whisper
- sentence-transformers
1. Store uploads as a typed audio column
Put each file in a table as audio, plus any metadata you will filter on. A path string is not enough: splitting and transcription need a typed column.
import pixeltable as pxtfrom pixeltable.functions.audio import audio_splitterfrom pixeltable.functions.huggingface import sentence_transformerfrom pixeltable.functions.whisper import transcribeTableModel = pxt.model_base()text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')class Recordings(TableModel, name='recordings'):title: pxt.Stringaudio: pxt.Audio
2. Split long files into timestamped segments
audio_splitter emits one row per slice with segment_start, segment_end, and audio_segment. Pass exactly one of duration or max_size. duration=30.0 is a starting slice for local Whisper.
class Segments(TableModel,name='segments',base=Recordings,iterator=audio_splitter(audio=Recordings.audio, duration=30.0),):pass
3. Transcribe each segment
Whisper on audio_segment writes the text for that slice. A hosted API is an alternative model choice on the same column (openai.transcriptions). The catalog still holds the file and the timestamps.
class Segments(TableModel,name='segments',base=Recordings,iterator=audio_splitter(audio=Recordings.audio, duration=30.0),):transcript = transcribe(audio_segment, model='base').text.astype(pxt.String)
4. Embed the text and rank by meaning
Index the transcript with a sentence embedding model. Return the parent title, segment_start, text, and score. The start time is the second to seek to. Sentence splitting with string_splitter needs spaCy; segment rows are enough for a first index.
class Segments(TableModel,name='segments',base=Recordings,iterator=audio_splitter(audio=Recordings.audio, duration=30.0),):transcript = transcribe(audio_segment, model='base').text.astype(pxt.String)__indexes__ = [pxt.EmbeddingIndex(transcript, string_embed=text_embed),]@pxt.querydef search_audio(query: str, limit: int = 5):sim = Segments.transcript.similarity(string=query)return (Segments.order_by(sim, asc=False).limit(limit).select(title=Recordings.title,segment_start=Segments.segment_start,segment_end=Segments.segment_end,text=Segments.transcript,score=sim,))
5. Serve upload and search over HTTP
Expose an upload route for new files and a search route for the query. background=True on insert returns a job id and a job_url to poll, which fits a long transcription. Apply the schema, then start the service. A new file transcribes only that file.
from pixeltable.serving import FastAPIRouterapi = FastAPIRouter(name='audio_lib')api.add_insert_route(Recordings,path='/recordings',inputs=[Recordings.title],uploadfile_inputs=[Recordings.audio],background=True,)api.add_query_route(path='/search', query=search_audio)# pxt init# pxt schema update app.py audio_lib -f# pxt service update app.py audio_lib -f
The stack coding agents usually pick vs one schema
| Job | Glue stack | One schema |
|---|---|---|
| Upload and files | FastAPI plus MinIO or S3 | An Audio column; media under PIXELTABLE_HOME |
| Workers | Celery plus Redis between upload and Whisper | Computed columns and a splitter view on insert |
| Transcription | Whisper or a hosted STT API in the worker | whisper.transcribe on the segment, or openai.transcriptions |
| Index | Postgres plus pgvector | EmbeddingIndex on the transcript column |
| Timestamps | You store offsets next to the vector by hand | segment_start and segment_end on the view row |
| New file | Enqueue a task; retry the DAG | Insert a row; only that file is transcribed and indexed |
| Failed call | Celery retry policy you write | insert(..., on_error='ignore') stores None plus errortype; default on_error='abort' |
Where Pixeltable fits
Pixeltable is where the transcription call runs and the results live, not a speech-to-text model. Deepgram, AssemblyAI, Gladia, and other hosted STT APIs are fine model choices on a computed column; this page does not rank them or quote their prices. The glue stack is FastAPI upload, MinIO or S3, Celery plus Redis, Whisper, and Postgres plus pgvector. Pick that stack when those services are already the contract. Pick this schema when the file, the timestamped segments, the index, and HTTP should live in one app.py. Pixeltable Cloud hosts the same app file. Speakers are a WhisperX cookbook, not this first index.
import pixeltable as pxtfrom pixeltable.functions.audio import audio_splitterfrom pixeltable.functions.huggingface import sentence_transformerfrom pixeltable.functions.whisper import transcribefrom pixeltable.serving import FastAPIRouterTableModel = pxt.model_base()text_embed = sentence_transformer.using(model_id='all-MiniLM-L6-v2')class Recordings(TableModel, name='recordings'):title: pxt.Stringaudio: pxt.Audioclass Segments(TableModel,name='segments',base=Recordings,iterator=audio_splitter(audio=Recordings.audio, duration=30.0),):transcript = transcribe(audio_segment, model='base').text.astype(pxt.String)__indexes__ = [pxt.EmbeddingIndex(transcript, string_embed=text_embed),]@pxt.querydef search_audio(query: str, limit: int = 5):sim = Segments.transcript.similarity(string=query)return (Segments.order_by(sim, asc=False).limit(limit).select(title=Recordings.title,segment_start=Segments.segment_start,segment_end=Segments.segment_end,text=Segments.transcript,score=sim,))api = FastAPIRouter(name='audio_lib')api.add_insert_route(Recordings,path='/recordings',inputs=[Recordings.title],uploadfile_inputs=[Recordings.audio],background=True,)api.add_query_route(path='/search', query=search_audio)# pxt init# pxt schema update app.py audio_lib -f# pxt service update app.py audio_lib -f
Questions
Do I need Celery and Redis?
Local Whisper or a transcription API?
How do I get timestamps for a search hit?
What about speakers?
What happens if a transcription call fails?
Next step
Copy the commands into an agent, or run them locally with pip install pixeltable.