Summary: An agent is not the model, the runtime, or the worker process currently executing a task. It is the durable history of everything that happened: every user input, model output, tool call, and tool result, plus a versioned session definition (system prompt, tools, model config). When that history is stored correctly, any fresh executor can reconstruct exactly where the agent was and continue. That single property unlocks reliability after crashes, horizontal scaling without sticky sessions, branching experiments, shared multi-user workflows, and migration between model providers. Compaction and summarization are lossy projections over the log—not replacements for it. Pixeltable treats session logs as queryable, versioned tables: the system of record your agent infrastructure should have been built on from the start.
What Is an Agent, Really?#
Think about a character you have spent dozens of hours building in a long-form RPG. What exactly is that character?
It is not the game engine. It is not the console. It is not the controller in your hands. Your character is the save file: the accumulated record of choices, inventory, quest progress, and world state that lets you drop back in exactly where you left off. Swap hardware, reinstall the game, and the character persists because the data persists.
AI agents are reaching the same inflection point. Most teams still treat an agent as the model weights, the orchestration loop, or the process currently holding open a WebSocket. Those things matter. They are interpreters and executors. But they are not the agent.
The agent is its data: specifically, the append-only history of events that accumulate as work gets done. We will call that the session log. Together with a session definition (the system prompt, tool schemas, model choice, and skills the agent runs under), it forms the agent's full state. Nothing essential lives in the runtime. A new worker reads the log, advances one step, appends the result, and moves on.
This is the same inversion databases settled on years ago: the durable write-ahead log is primary; indexes, caches, and materialized views are projections. Agents deserve the same architecture.
Defining the Agent Means Defining the Log#
At minimum, a session log records every meaningful state transition in order:
- User inputs — messages, files, approvals, rejections
- Model outputs — text, structured JSON, reasoning traces
- Tool calls — which tool, with what arguments
- Tool results — return values, errors, latency metadata
- System events — permission prompts, rate-limit pauses, human-in-the-loop checkpoints
The session definition sits alongside the log as a versioned constant: the system prompt, tool descriptions, model identifier, sampling parameters, and any skill definitions. It does not change turn-to-turn, but it does change over the life of a product—and those changes should be versioned and referenceable, the same way schema migrations are.
Together, log plus definition are enough to resume the agent from any point. The model reads a view of the log (often truncated or summarized for context limits), produces the next action, and that result is written back as durable data. The UI reads the same log and renders a timeline. The audit system reads it and reconstructs what happened. Tracing, evals, and debugging all become read paths over the same durable source.
In most agent frameworks, that read/write cycle is written imperatively: a while block reads state, calls the model, runs tools, and appends results. The architectural insight is separable from the syntax—each step should be fault-tolerant, and every meaningful transition should be durably recorded before moving on. If the worker dies mid-turn, another worker reconstructs from the log and continues. The worker died; the agent did not.
Recent academic work on event-sourced reactive graphs for auditable, forkable agentic systems arrives at a similar conclusion from a different angle: when coordination happens through persistent, replayable state, agents become inspectable infrastructure instead of opaque processes.
The Model and Runtime Are Not the Agent#
A fair objection: if you fork the same log and continue on Claude versus GPT, you may get different continuations. Sampling is non-deterministic. Tool environments differ. Model weights change between releases.
That is correct—and it does not weaken the claim. Those factors belong in the session definition, not in ephemeral process memory. Log plus definition captures everything needed to reconstruct what the agent knew and did at each step. Different models may interpret the same history differently, just as two players loading the same save file might make different choices—but the save file is still the character.
Execution policy (model, temperature, tools, sandbox environment) is versioned metadata referenced by the log. The log is the path-dependent record of how a particular configuration played out. That distinction matters when you debug regressions, compare providers, or audit a decision six months later.
The Problem: Logs as an Afterthought#
Most agent harnesses today treat the session log as exhaust—the messy byproduct of a running process—not as the system itself.
Common patterns:
- Transcripts written as JSONL files on local disk, sometimes fire-and-forget
- State trapped in a local SQLite database that corrupts under concurrent access
- Different shards of history scattered across tracing, chat UI, and tool-runner logs
- Checkpoint blobs tied to a specific framework version on a specific machine
The result is fragile. A process dies while waiting for a human approval—and on restart, the approval prompt is gone, the agent has moved on or stalled, and nobody can reconstruct what the user was asked. That is unacceptable in production, and it is exactly what happens when state lives in the executor instead of the log.
We have written about the broader production gap in The Invisible 80%: the work that separates a weekend demo from a system that survives real users. State management is Layer 2 in that stack, and it is where most teams underestimate the engineering cost. The Frankenstein Stack makes it worse: when memory lives in Redis, transcripts live in files, and tool results live in Postgres, reconstructing a coherent session history becomes a custom integration project.
Compaction Is a Projection, Not the Agent#
Session logs grow without bound. Context windows do not. Every production agent eventually compacts older history into summaries so the model can see a useful slice of the past.
That raises an obvious question: if compaction throws information away, does it destroy the claim that the log is the agent?
No—because compaction is lossy. A summary does not reproduce the agent's state in smaller form; it discards detail. That actually reinforces the distinction. The full log is the record. A compaction is one projection of it, the way a materialized view is not the database and a chat summary is not the conversation.
Keep the raw log and you can always generate new projections: a shorter summary for a cheaper model, a detailed trace for an auditor, a structured extract for eval. Throw the raw log away and keep only the compaction, and you have lost part of the agent—the path-dependent history that produced the current state.
The cleanest pattern treats compaction as a best-effort fork you resume from, while the canonical event history remains append-only and intact. In Pixeltable, that maps naturally to computed columns and views as projections over a source table—same pattern as the incremental computation engine uses everywhere else.
The Log Is Not the Whole World#
Tools change state outside the log. An agent edits a file, opens a GitHub issue, sends an email. That state lives elsewhere. Forking the log will not un-send the email; rolling back to an earlier event will not revert a file the agent already committed.
The log does not make the world deterministic or reversible. It keeps a faithful, resumable record of what the agent did and saw—which is exactly the artifact you cannot afford to lose. When the agent resumes and the world has changed underneath it, it re-engages with current reality the same way a game character reloads into a world that kept running.
This is the same principle behind decision traces as first-class data: the value is not controlling external systems retroactively, but capturing the reasoning context at decision time so you can audit, replay, and improve.
Properties That Fall Out#
Once you treat the session log as the agent, a set of system properties stops being bolt-on features and starts being structural.
Reliability#
Executors are allowed to be fallible. Workers crash, containers restart, sandboxes vanish, providers fail, users disconnect. If the agent is the running process, all of that is terrifying. If the agent is the log, it is execution detail—as long as each transition was written to durable storage before the worker moved on, a replacement process reads the same history and picks up from the last recorded event.
Scalability#
Most harnesses run one process per agent, tying identity to a machine. When state lives in the log, workers become stateless: they read history, run one step of orchestration, write the result back. Any process with access to the log can serve any session. No sticky sessions, no in-memory state migration.
Forking#
Instead of one linear path, branch the log. One branch runs on Claude, another on GPT, another on a local open-weight model—each exploring a different strategy in a different sandbox. Experimentation becomes a data operation, not a rewrite of orchestration code. Pixeltable's versioning and time travel make this concrete: every insert is a version; you can query historical states, compare branches, and roll back.
Multiplayer#
Sharing an agent should not mean pasting a transcript into Slack. That is a screenshot of a database, not replication. If the log is the agent, sharing means granting access to a durable history someone else can inspect, resume, or extend. Pixeltable data sharing and queryable tables turn multiplayer agents into a permissions problem instead of a copy-paste workflow.
Migration#
If agent identity is trapped inside provider-specific memory formats, switching models is painful. If the log is provider-agnostic, migration becomes an adapter problem: different models may want different projections of the same history, but the history itself is continuous. That is the architecture behind a multi-provider AI strategy—swap the model in your orchestrator, keep the session log intact.
Auditability#
Every UI, trace, eval, and compliance report becomes a read path over the same source. No reconciling three partial logs after an incident. When the log lives in Pixeltable, automatic lineage tracking links stored outputs and derived projections back to the rows that produced them.
Owning the Log Means Owning the Agent#
If the log is the agent, then whoever owns the log owns the agent.
Model lock-in is overrated as a long-term moat. Models can be swapped; APIs can be wrapped. The deepest lock-in is log lock-in. If a provider holds the only durable record of your agent's work—under their retention policies, queryable by their systems, exportable only as lossy transcripts—you do not merely host your agent there. They own the continuity of its identity.
That is not an argument against managed infrastructure; hosted loops, sandboxes, and background execution are useful and often inevitable. It is an argument for knowing where your log lives, who can inspect it, and whether you can replay, fork, migrate, and export it on your terms. For agents that touch personal data, company workflows, and operational decisions, the session log is among the most sensitive artifacts you will ever store.
This connects directly to the question of who owns the multimodal data plane. The next generation of AI infrastructure will not be defined by who runs the best model API. It will be defined by who holds the durable record of what agents did with that data.
Where Pixeltable Fits: The Log as a Table#
Pixeltable does not run your agent loop. The loop—call model, execute tools, decide whether to continue—lives in your orchestrator: Pixelagent, a framework like LangGraph, or application code you write yourself. What Pixeltable provides is the system of record for session state: queryable, versioned tables that any orchestrator can read from and write to. That is the foundation described in our Agent Harness and stateful agents posts—memory as data, not as a hidden file inside a running process.
The contract is simple. Your orchestrator reads session history before each inference step, and writes each new event (user message, assistant reply, tool call, tool result) back as a row. Pixeltable handles persistence, versioning, multimodal storage, and projections like compaction summaries. It does not replace the loop; it makes the loop's state durable and inspectable.
Example 1: Session Definition and the Event Log#
Two tables: one for session config, one for the append-only message history. Your orchestrator inserts a row after every meaningful event—never overwriting prior rows. This mirrors the chat-history table in our agent memory cookbook and the patterns in Building Memory-Powered AI.
Tool payloads and multimodal attachments can use typed columns (pxt.Image, pxt.Document, pxt.Json) alongside text. That is a practical advantage over transcript files that flatten everything to strings, and it connects to how we handle variable LLM outputs in Structure the Unstructured.
Example 2: Reading History Before Inference#
Before your orchestrator calls the model, it queries the log. A @pxt.query function returns ordered history for a session—the same mechanism used for memory retrieval elsewhere in Pixeltable:
The orchestrator owns the control flow; Pixeltable owns the durable record. After inference, write user and assistant turns back to memory—the same write-back step documented in the agent memory cookbook.
Example 3: Evals and Audits as Projections Over the Log#
Once events are stored, attach computed columns to a view of assistant messages for offline evals—so you do not re-score user or tool rows. This matches how the Agent Harness treats continuous evals: derived columns over stored output, not part of the runtime loop:
Change the eval prompt or model and Pixeltable's incremental engine recomputes only affected rows. The raw log in memory stays untouched.
Example 4: Compaction as a Lossy Projection#
When history exceeds the context window, summarize without deleting the raw log. A compactions table pulls history via the query function above; a UDF formats it for the model (same list-of-dicts pattern as context assembly in our memory recipes):
Your orchestrator prepends the compacted summary plus the last N raw rows from memory when building the next prompt. The full log stays intact for audit, fork, and re-summarization. Compaction is a projection; the table is the agent.
Where to Start#
For a working orchestrator that reads and writes Pixeltable tables, see Pixelagent and our launch post. For custom integrations, start with the practical guide to building agents and the AI agents with MCP use case. For deeper memory architecture (episodic, semantic, retrieval), see Building Memory-Powered AI.
Teams iterating on compaction strategies, eval hooks, or log schema benefit from treating those as data changes in Pixeltable—not infrastructure rewrites in the orchestrator. That is the experimentation loop we describe in iterate on your data, not your infrastructure.
Frequently Asked Questions#
Isn't the agent also the model and runtime?#
The model and runtime are necessary to advance the agent, but they are replaceable interpreters. Session definition plus event log capture everything needed to resume. Swap the model, restart the worker, migrate regions—the agent continues because its identity is in the data.
Doesn't compaction destroy the agent?#
Only if you delete the raw log and keep only the summary. Compaction is a lossy projection for model context, not a replacement for the canonical event history. Keep both.
What about side effects outside the log?#
The log records what the agent did and saw, not the entire world state. External effects (sent emails, merged PRs, charged payments) remain external. The log gives you an auditable, resumable record of the agent's trajectory—what you need for debugging, compliance, and human handoff.
How is this different from framework checkpointing?#
Framework checkpoints are often opaque blobs tied to a specific library version on a specific machine. Pixeltable stores the session log as queryable rows—versioned automatically, shareable across teams, multimodal-native, and independent of any orchestration SDK. Your orchestrator runs the loop; Pixeltable holds the history it reads and writes. Your framework becomes swappable; your session log does not.
Does Pixeltable run the agent loop?#
No. Pixeltable is the data layer for the log—storage, retrieval, versioning, compaction projections, and eval columns over stored events. The agent loop (model calls, tool execution, multi-step reasoning) runs in your orchestrator or in a layer like Pixelagent that uses Pixeltable tables as its system of record.
Conclusion: Make the Log the System#
The shift is conceptual but consequential. Stop treating the session log as exhaust the system emits. Start treating it as the system itself—the durable artifact your orchestrator reads and writes on every step. Compaction, UI rendering, tracing, and evals become projections over that log. Reliability, scaling, forking, sharing, and migration fall out of the same assumption.
Pixeltable was built to be that system of record: tables for session history, automatic versioning, multimodal columns, incremental projections for summaries and evals, and shareable data behind a Pythonic interface. The agent loop lives elsewhere today; the log should not. The model will keep getting smarter. The teams that ship production agents will be the ones who got the data layer right.
Ready to build? Start with the 10-minute Pixeltable tutorial, explore the Pixelagent repository, or browse the documentation for session and agent patterns.
