LLM as judge

Score model output with a second model, as a chain of computed columns

Agents · Recipe · Updated

What it covers

  • The table
  • The answer under test
  • The judge prompt
  • The verdict and the score

What it uses

Pixeltable calls
openai.chat_completions()pxt.create_table()@pxt.udfadd_computed_column()insert()

A look at the code

pattern-llm-judge.mdx
runs.add_computed_column(
response=openai.chat_completions(
messages=[{'role': 'user', 'content': runs.prompt}], model='gpt-4o-mini'
)
)
runs.add_computed_column(answer=runs.response.choices[0].message.content)

One block of the recipe, copied as it is. The rest is on the docs page.

Browse all 103