PixeltablePixeltable Logo

Working with llama.cpp in Pixeltable

This tutorial demonstrates how to use Pixeltable's built-in llama.cpp integration to run local LLMs efficiently.

Providers · Notebook · Updated

Open in ColabRead in the docsView on GitHub

Colab runs it in your browser. If it calls a model provider, it asks for that provider's API key in a cell.

What it covers

  • Create a table for chat completions
  • Test chat completion
  • Comparing models

What it uses

Install
terminal
pip install pixeltable llama-cpp-python huggingface-hub
Pixeltable calls
llama_cpp.create_chat_completion()pxt.create_table()add_computed_column()insert()

A look at the code

working-with-llama-cpp.ipynb
# Add a computed column that uses llama.cpp for chat completion
# against the input.


messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': t.input},
]


t.add_computed_column(
    result=llama_cpp.create_chat_completion(
        messages,
        repo_id='Qwen/Qwen2.5-0.5B-Instruct-GGUF',
        repo_filename='*q5_k_m.gguf',
    )
)


# Extract the output content from the JSON structure returned
# by llama_cpp.


t.add_computed_column(output=t.result.choices[0].message.content)

One cell of the notebook, copied as it is. Open it in Colab for the rest.

More in Providers

Browse all 103