Orchestration
What is incremental computation?
Incremental computation recomputes only the rows and downstream columns a change actually affects, instead of rerunning the whole pipeline.
Updated
How it works
- Dependencies between columns are known.
- An edit marks the descendants of that row.
- Everything else stays cached.
What it is not
It is not a nightly full refresh, and it is not a cache you remember to invalidate.
incremental computation: this, and the thing it is confused with
| This | Not this | |
|---|---|---|
| Work | Changed rows | The whole table |
| Invalidation | Recorded dependencies | Hope |
| Cost | Model calls for the delta | Model calls for the corpus |
Where Pixeltable fits
Pixeltable tracks those dependencies on computed columns. A new file recomputes that row’s descendants.
Questions
- How does incremental computation work?
- Dependencies between columns are known. An edit marks the descendants of that row. Everything else stays cached.
- What is incremental computation often confused with?
- It is not a nightly full refresh, and it is not a cache you remember to invalidate.