---
title: "vLLM: High-Throughput Local LLM Inference in Pixeltable"
date: "2026-06-13"
author: "Pixeltable Team"
tags:
  - vLLM
  - Local LLM
  - Self-Hosted
  - GPU Inference
  - HuggingFace
  - AI Integration
  - Pixeltable
  - Production AI
description: "Run HuggingFace models at production throughput with vLLM inside Pixeltable computed columns. Learn when to choose vLLM over Ollama or cloud APIs, and how batching and GPU utilization stay in the declarative pipeline."
url: "https://www.pixeltable.com/blog/vllm-high-throughput-local-inference-pixeltable"
---

# vLLM: High-Throughput Local LLM Inference in Pixeltable

Pixeltable 0.6.5 adds native vLLM integration. Run HuggingFace models at high throughput inside computed columns—with the same declarative pipeline, versioning, and per-row error handling as cloud provider calls.

 
## Why vLLM in a Pixeltable pipeline?

 
Pixeltable already supports local inference through [Ollama](/blog/ollama-local-llm-pixeltable) (easy model pulls, great for development) and [llama.cpp](/blog/llama-cpp-local-inference-pixeltable) (quantized models on modest hardware). **vLLM** targets a different constraint: **throughput on GPUs**.

 
vLLM's serving engine batches requests, manages KV cache efficiently, and scales across GPUs with tensor parallelism. When you need to classify thousands of documents, summarize a large backlog, or run batch evaluation—not chat with a single prompt interactively—vLLM is often the right local choice.

 
Combined with Pixeltable's [declarative infrastructure](/blog/unified-multimodal-ai-infrastructure-pixeltable), you get:

 

 - **Incremental computation** — only new or changed rows hit the model

 - **Per-row error containment** — one bad input does not fail the batch

 - **Version history** — compare outputs across model or prompt changes

 - **No glue code** — inference sits in a computed column next to embeddings and retrieval

 

 
## When to use vLLM vs Ollama vs cloud APIs

 
| Option | Best for | Tradeoff |
| --- | --- | --- |
| Ollama | Local dev, quick experiments, modest hardware | Lower throughput at scale |
| llama.cpp | Quantized models, CPU or Apple Silicon | Manual model management |
| vLLM | GPU clusters, batch inference, HF model catalog | Requires GPU + pip install vllm |
| Cloud APIs | Zero ops, latest frontier models | Cost, latency, data residency |

 
For hybrid strategies—cloud for quality, local for volume—see our [multi-provider comparison guide](/blog/multi-provider-ai-strategy-comparison-guide).

 
## Getting started

 
```bash
pip install -U pixeltable vllm
```

 
vLLM loads models from HuggingFace and caches them for reuse across computed-column evaluations on the same model.

 
## Chat completions

 
`chat_completions()` accepts OpenAI-style message lists—ideal when your table stores conversation history or prompt templates:

 
```python
import pixeltable as pxt
from pixeltable.functions.vllm import chat_completions

pxt.create_dir('vllm_demo')
t = pxt.create_table('vllm_demo.chat', {'messages': pxt.Json})

t.add_computed_column(
 result=chat_completions(
 t.messages,
 model='Qwen/Qwen2.5-0.5B-Instruct',
 sampling_params={'max_tokens': 256, 'temperature': 0.7},
 )
)

t.insert([{
 'messages': [{'role': 'user', 'content': 'Summarize declarative AI pipelines in one sentence.'}]
}])
t.select(t.messages, t.result).show()
```

 
Pass `engine_args` for vLLM constructor options—`dtype`, `max_model_len`, `gpu_memory_utilization`, `tensor_parallel_size`—and `sampling_params` for temperature, top-p, and max tokens. See the [vLLM SDK reference](https://docs.pixeltable.com/sdk/latest/vllm) for the full parameter list.

 
## Text generation

 
For single-string prompts, use `generate()`:

 
```python
from pixeltable.functions.vllm import generate

docs = pxt.create_table('vllm_demo.docs', {'prompt': pxt.String})

docs.add_computed_column(
 completion=generate(
 docs.prompt,
 model='Qwen/Qwen2.5-0.5B-Instruct',
 sampling_params={'max_tokens': 128},
 )
)
```

 
## Batch pipeline example

 
A typical pattern: ingest documents, chunk with an iterator, summarize each chunk with vLLM, embed for retrieval—all in one catalog:

 
```python
import pixeltable as pxt
from pixeltable.functions.vllm import chat_completions
from pixeltable.functions.huggingface import sentence_transformer

chunks = pxt.get_table('rag.chunks')

messages = [{'role': 'user', 'content': chunks.text}]
chunks.add_computed_column(
 summary=chat_completions(
 messages,
 model='Qwen/Qwen2.5-1.5B-Instruct',
 sampling_params={'max_tokens': 120, 'temperature': 0.3},
 )
)
chunks.add_embedding_index(
 'summary',
 embedding=sentence_transformer.using(model_id='BAAI/bge-small-en-v1.5'),
)
```

 
Insert new source documents and Pixeltable runs vLLM only on new chunks—see [incremental computation](/blog/dependency-graph-magic-computed-columns) for why that matters at scale.

 
## Errors, retries, and resource limits

 
GPU OOM and model load failures surface as per-row computation errors—the same pattern as cloud provider `429`s documented in our [rate limiting guide](/blog/production-rate-limiting). Use `pxt errors` or the [local dashboard](/blog/pixeltable-local-dashboard-browse-debug-pipelines) errors-only filter to inspect failures, then adjust `engine_args` (smaller model, lower `max_model_len`, higher `gpu_memory_utilization` headroom) and recompute affected rows.

 
## Learn more

 

 - [vLLM SDK documentation](https://docs.pixeltable.com/sdk/latest/vllm)

 - [vLLM project documentation](https://docs.vllm.ai/)

 - [May/June 2026 release highlights](/blog/pixeltable-may-june-2026-release-highlights)

 - [Ollama: run local LLMs](/blog/ollama-local-llm-pixeltable)

 - [llama.cpp: quantized local inference](/blog/llama-cpp-local-inference-pixeltable)