Agile was designed for software development. Data science, analytics, and data engineering have different uncertainty profiles, different output types, and different feedback loops. Applying standard Scrum to data work without adaptation produces exactly the kind of mismatches that give Agile a bad reputation in data teams.
Why Standard Scrum Struggles With Data Work
| Scrum assumption | Data science reality |
|---|---|
| Stories are estimable | Research questions are not estimable — you find out what you need when you look at the data |
| Sprint output is a working increment | Data exploration may produce "we learned this doesn't work" — valuable but not a shippable increment |
| Definition of Done is fixed | Model accuracy targets may shift as data quality is understood |
| Backlog items are independent | Data pipeline work is highly sequential — downstream models depend on upstream features |
| Velocity is stable | Data cleaning and preparation complexity is highly variable |
Adapted Agile Approaches for Data Teams
The Two-Track Model
Separate research/exploration work from engineering/productionisation work into two tracks with different operating models:
- Research track: Time-boxed exploration (1–2 weeks), output is a recommendation or finding, not a deliverable. No story points — time-box only.
- Engineering track: Standard Scrum sprints building data pipelines, dashboards, and model serving infrastructure. Normal velocity tracking applies.
The Data Product Team Model
Organise data teams around data products — persistent, versioned, documented datasets and models that other teams consume — rather than around analytical projects. Each data product team owns their product end-to-end (ingestion, transformation, quality, serving, documentation) and applies Agile to product iteration rather than project delivery.
Adapting the Definition of Done for Data Work
Standard DoD (code complete, tests passing, deployed to prod) needs extension for data work:
- Data quality checks passing (null rates, referential integrity, freshness)
- Data dictionary updated with new fields/tables
- Lineage documented (what sources feed this output)
- Performance benchmarked (query time for dashboards, inference latency for models)
- Consumers notified of new/changed datasets
Kanban for Data Engineering
For data engineering teams with high variability in task complexity and frequent external dependencies (data provider SLAs, upstream system changes), Kanban is often more appropriate than Scrum. WIP limits on pipeline stages prevent the team from starting more pipelines than they can maintain. Cycle time per pipeline type becomes the key metric. Scrum Masters working with data engineering teams should evaluate whether the sprint cadence adds value or just adds ceremony overhead.
ML Delivery in Sprints
Machine learning model development has a natural sprint structure if you apply it to experiments rather than features. Each sprint: define the experiment hypothesis (e.g. "adding transaction history features will improve fraud detection F1 score by 5%"), run the experiment, evaluate results, ship to production if successful. This creates a regular cadence of incremental model improvement with measurable outcomes per sprint.