---
title: "ClaimBot: FNOL Triage on One Heterogeneous Row"
date: "2026-09-09"
author: "Pierre Brunelle"
tags:
  - Insurance
  - Multimodal AI
  - TableModel
  - Document
  - Audio
  - Pixeltable Cloud
  - Triage
description: "A claim is a photo, a voice memo, and a PDF on the same row. Whisper and Gemini fill columns. A UDF compares memo vs report. Not an image array. Not three microservices."
url: "https://www.pixeltable.com/blog/claimbot-multimodal-fnol-triage"
---

# ClaimBot: FNOL Triage on One Heterogeneous Row

**Summary:** ClaimBot is first notice of loss as a table row: scene photo, adjuster memo, written report. Whisper transcribes the memo. Gemini reads the photo and the PDF. A UDF (or a third Gemini call) compares spoken story vs written report and writes `triage_status`. You do not store “a list of images” and hope a pipeline joins them later. Three types. One insert. `pip install pixeltable`.

 
## The Product

 

 - Create a claim with `claim_id`

 - Attach `pxt.Image`, `pxt.Audio`, `pxt.Document`

 - Fill memo text, photo notes, report notes on insert

 - Flag `clear` / `mismatch` / `needs_adjuster`

 

 
The types lecture is [what is a multimodal data table](/blog/what-is-a-multimodal-data-table). The schema-as-backend recap is [Never Fuck With Your Backend Anymore](/blog/never-fuck-with-your-backend-anymore). The afternoon clone that dies in a container is the [Photos / AWS post](/blog/i-vibe-coded-google-photos-clone-then-tried-aws). This post is the heterogeneous claim, not an album.

 
## The Stitch You Delete

 
S3 for the JPEG. A second bucket for the WAV. A third for the PDF. Three workers. A case-management field you type by hand. Pixeltable is one row with three media types. Delete the claim; derived cells are not orphans in two other systems.

 
## The Receipt

 
```python
import pixeltable as pxt
from pixeltable.functions import gemini, whisper
from pixeltable.serving import FastAPIRouter

TableModel = pxt.model_base()

PHOTO_PROMPT = 'Describe visible damage and relevant objects in the scene photo.'
REPORT_PROMPT = 'Summarize the written FNOL report in one short paragraph.'
COMPARE_PROMPT = (
 'Compare the voice-memo transcript, the photo notes, and the report summary. '
 'Reply with one token: clear, mismatch, or needs_adjuster, then a short reason.'
)

@pxt.udf
def parse_triage(raw: str) -> str:
 blob = raw.lower()
 if 'mismatch' in blob:
 return 'mismatch'
 if 'needs_adjuster' in blob or 'adjuster' in blob:
 return 'needs_adjuster'
 if 'clear' in blob:
 return 'clear'
 return 'needs_adjuster'

class Claims(TableModel, name='claims'):
 claim_id: pxt.String
 photo: pxt.Image
 memo: pxt.Audio
 report: pxt.Document

 memo_text = whisper.transcribe(memo, model='base')
 photo_notes = gemini.generate_content(
 [photo, PHOTO_PROMPT],
 model='gemini-2.5-flash',
 )
 report_notes = gemini.generate_content(
 [report, REPORT_PROMPT],
 model='gemini-2.5-flash',
 )
 compare_raw = gemini.generate_content(
 [memo_text, photo_notes, report_notes, COMPARE_PROMPT],
 model='gemini-2.5-flash',
 )
 triage_status = parse_triage(compare_raw)

@pxt.query
def triage_queue():
 return (
 Claims.where(Claims.triage_status != 'clear')
 .select(
 Claims.claim_id,
 Claims.triage_status,
 Claims.photo_notes,
 Claims.report_notes,
 )
 )

api = FastAPIRouter(name='claimbot')
api.add_insert_route(
 Claims,
 path='/claims',
 inputs=[Claims.photo, Claims.memo, Claims.report, Claims.claim_id],
)
api.add_query_route(path='/triage', query=triage_queue)
```

 
Not an image array. Not a splitter view unless you want page-level RAG on the PDF — that receipt is [DocuVision](/blog/docuvision-pdf-chart-qa). Here Gemini reads the document as one column, same as the photo. Same file on Cloud: `pxt db update` → `pxt schema update` → `pxt service update`.

 
## What This Is Not

 
Not a claims system of record for payouts. Not a policy admin. Card TPV and coverage stay in the warehouse you already trust. Not medical or legal advice. The multimodal row is the intake graph.

 
## People Also Ask

 
**Why not `pxt.Image[]`?** FNOL here is one scene photo, one memo, one report. Arrays hide the join. If you need many photos, that is a child table, not a list cell you hope the LLM unpacks.

 
**Do I chunk the PDF?** Only if you need passage citations. Start with Gemini on `report`. Add a splitter view when the file is a prospectus, not a two-page FNOL.

 
**How do I go to Cloud?** Same `app.py`. `PIXELTABLE_API_KEY`, `pxt://org:db`, then db → schema → service. `pxt service run` is local only.

 
**Is this an AI automation workflow?** Yes. Insert photo, memo, and PDF; triage columns run on the row. The category is [AI automation workflow](/blog/ai-automation-workflow).

 
## Three Types, One Insert

 
Declare the claim. Apply the file. Insert photo, memo, PDF. Read `triage_status`. Keep the intake. Delete the three-bucket stitch.

 
The clinical analog — dictation, lab PDF, scan — is [MedDossier](/blog/meddossier-clinical-intake-triage).

 

 - **[Pixeltable on GitHub](https://github.com/pixeltable/pixeltable)**: `pip install pixeltable`

 - **[ClipFinder](/blog/clipfinder-semantic-video-moment-search)** — NL video moments

 - **[CallSense](/blog/callsense-sales-call-intelligence)** — sales-call auditor

 - **[DocuVision](/blog/docuvision-pdf-chart-qa)** — PDF Q&A

 - **[SnapCatalog](/blog/snapcatalog-product-visual-search)** — shop-the-look

 - **[SafeStream](/blog/safestream-ugc-video-moderation)** — UGC gate

 - **[What is a multimodal data table](/blog/what-is-a-multimodal-data-table)**

 - **[Pixeltable documentation](https://docs.pixeltable.com)**