Evaluation
What is dataset curation?
Dataset curation selects, cleans, and versions the rows that train or evaluate a model.
Updated · Part of What is active learning?
How it works
- Gather candidate rows.
- Drop exact and near duplicates, fix labels, and hold out a test slice.
- Pin the version the run used.
What it is not
It is not an unversioned scrape, and it is not only storing weights.
dataset curation: this, and the thing it is confused with
| This | Not this | |
|---|---|---|
| Output | A versioned set of rows | A dump |
| Includes | Labels and media | Weights |
| Tools | Filters, embeddings, people | A spreadsheet with paths |
Where Pixeltable fits
Pixeltable curation is queries and columns on the media table: dedup, labels, and a version you can name.
Questions
How does dataset curation work?
Gather candidate rows. Drop exact and near duplicates, fix labels, and hold out a test slice. Pin the version the run used.
What is dataset curation often confused with?
It is not an unversioned scrape, and it is not only storing weights.
Next step
Install the Pixeltable skill, or sign up for Cloud and run the same file on a hosted database.