Data Cleaning
The process of detecting and correcting errors, inconsistencies, and inaccuracies in datasets to improve data quality.
In-Depth Explanation
Data cleaning (or data cleansing) is the process of identifying and correcting errors in data. It's a critical but often underestimated part of any data or AI project.
Common data quality issues:
- Missing values: Null or empty fields
- Duplicates: Same record multiple times
- Inconsistent formats: Dates, phone numbers, addresses
- Invalid values: Out of range, wrong type
- Outdated information: No longer accurate
- Typos and errors: Human entry mistakes
Cleaning techniques:
- Deduplication
- Standardisation (format consistency)
- Validation (check against rules)
- Imputation (fill missing values)
- Outlier detection and handling
- Cross-reference verification
Tools for data cleaning:
- Python (pandas, great_expectations)
- OpenRefine
- Data quality platforms
- Database constraints
- ETL tools with cleaning capabilities
The 80/20 rule:
- Data scientists spend ~80% of time on data preparation
- Only ~20% on actual modelling
- Clean data dramatically improves model quality
Business Context
"Garbage in, garbage out" - AI model quality depends on data quality. Investing in data cleaning pays dividends in model performance.
How Clever Ops Uses This
We emphasise data cleaning for Australian business AI projects, knowing that quality data is the foundation of reliable AI systems.
Example Use Case
"Cleaning customer database: deduplicating merged accounts, standardising addresses to postal format, filling missing postcodes from suburb lookup."