---
title: "Your Backend for Multimodal AI Applications"
description: "Every multimodal AI app needs the same five things: store media, run models, index embeddings, serve endpoints, version everything."
url: "https://www.pixeltable.com/notebooks/multimodal-backend"
updated: "2026-10-30"
---

# Your Backend for Multimodal AI Applications

Every multimodal AI app needs the same five things: store media, run models, index embeddings, serve endpoints, version everything.

- **Type:** Notebook
- **Group:** Core Operations

## What it covers

- Store: Multimodal Tables
- Orchestrate: AI as Computed Columns
- Insert: One Call Triggers the Full Pipeline
- Retrieve: Semantic Search
- Experiment on Media Data
- Serve: Queries Become API Endpoints
- Version: Automatic History
- Agents: Tool Calling and Memory as Computed Columns

## What it uses

- **Install:** `pip install pixeltable google-genai 'fastapi[standard]' torch torchvision transformers`
- **Environment variables it names:** `GEMINI_API_KEY`, `GOOGLE_API_KEY`
- **Pixeltable calls:** `uuid.uuid7()`, `gemini.generate_content()`, `huggingface.detr_for_object_detection()`, `vision.bboxes_draw()`, `anthropic.messages()`, `anthropic.invoke_tools()`, `pxt.create_table()`, `@pxt.query`, `pxt.tools()`, `add_computed_column()`

## A look at the code

```python
videos.add_computed_column(
    response=gemini.generate_content(
        [videos.video, 'Describe this video in detail.'],
        model='gemini-3-flash-preview',
    )
)

videos.add_computed_column(
    description=videos.response.candidates[0]
    .content.parts[0]
    .text.astype(pxt.String)
)

videos.add_embedding_index(
    'description',
    embedding=gemini.embed_content.using(
        model='gemini-embedding-2-preview'
    ),
)
```

## Open it

- **Open in Colab:** https://colab.research.google.com/github/pixeltable/pixeltable/blob/release/docs/release/howto/cookbooks/core/multimodal_backend.ipynb
- **Docs page:** https://docs.pixeltable.com/howto/cookbooks/core/multimodal_backend
- **Source on GitHub:** https://github.com/pixeltable/pixeltable/blob/release/docs/release/howto/cookbooks/core/multimodal_backend.ipynb