The DIRTY Method
Develop - Ingest - Refine - Test - Yield
DIRTY: a method built from the experience of having dirtied our hands with data. Transition disparate, unhomegenous data into viable insights underpinning the expectations of modern business. Let us unlock your data's potential.
Develop data awareness
-
Identify data sources (APIs, databases, files, sensors)
-
Understand business context and intended use
-
Perform initial data profiling
-
Map data lineage and dependencies
-
Assess risks: quality, bias, privacy, security
Ingest data in a controlled, reproducible method
-
Extract from source systems
-
Validate schema and apply structural checks (types, formats, completeness)
-
Store raw data in a landing zone or data lake
-
Log ingestion events for traceability
Refine raw data into a reliable, analysis ready asset
-
Cleanse (missing values, outliers, duplicates)
-
Standardise formats and conventions (dates, units, categories)
-
Normalize or denormalize tables
-
Apply business rules and domain logic
-
Enrich with external datasets
- Automated data quality tests (constraints, ranges, uniqueness)
- Validate data meets quality, performance and expectations
- Data drift detection
- Business rule validation
- Performance and scalability checks
- Alerting and monitoring for failures​
Test data expectations
Yield
-
Publish data to stores or repositories
-
Deliver trusted data products
-
Provide dashbards and reports to consumers
-
Ensure data remainsusable and maintainable
-
Track metrics
-
Communicate changes and version updates