PixeltablePixeltable Logo

Working with vLLM

This tutorial demonstrates how to use Pixeltable's built-in vLLM integration to run local LLMs with high-throughput inference.

Providers · Notebook · Updated

Open in ColabRead in the docsView on GitHub

Colab runs it in your browser. If it calls a model provider, it asks for that provider's API key in a cell.

What it covers

  • Create a table for chat completions
  • Test chat completion
  • Comparing models
  • Using sampling parameters
  • Text generation

What it uses

Install
terminal
pip install pixeltable vllm
Pixeltable calls
vllm.chat_completions()vllm.generate()pxt.create_table()add_computed_column()insert()

A look at the code

working-with-vllm.ipynb
# Add a computed column that uses vLLM for chat completion
# against the input.


messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': t.input},
]


t.add_computed_column(
    result=vllm.chat_completions(
        messages, model='Qwen/Qwen2.5-0.5B-Instruct'
    )
)


# Extract the output content from the native vLLM response.


t.add_computed_column(output=t.result.outputs[0].text)

One cell of the notebook, copied as it is. Open it in Colab for the rest.

More in Providers

Browse all 103