The Role of Data Cleaning in ML: Why It Matters More Than You Think

When most people think about machine learning, they imagine sophisticated algorithms, neural networks, and cutting edge AI models. But let’s get real for a second: the actual success of your ML project is determined way before you select an algorithm.
It’s determined by the quality of your data.

The Inconvenient Truth About ML Projects
There’s a popular truism among data science: data scientists spend 80% of their time cleaning and preparing the data and only 20% actually building and training models. Some studies suggest it’s even worse, up to 90% data preparation, and just 10% modeling.
This isn’t just inefficiency. It is necessity. Because no matter how sophisticated your model may be, it can’t overcome fundamentally flawed data. As the old computer science adage goes: “Garbage in, garbage out.”

What Exactly is Data Cleaning?

Data cleaning (or data cleansing or data scrubbing) refers to the process of detecting and correcting corrupt, inaccurate, or irrelevant records from a dataset. It includes the following:

Identifying and Handling Missing Values
Removing or correcting outliers
Correcting structural mistakes and discrepancies
Handling Duplicate Records
Standardize data formats
Data Entry Error Correction

Dealing with irrelevant data Think of it as quality control for your data. You don’t want to build a house on a cracked foundation any more than you would want to train a model on dirty data.

The Common Culprits: Data Quality Issues

1. Missing Values

Perhaps the most common problem. Data are missing for a multitude of reasons—sensor failures, user skipping of form fields, data not being collected at some time periods, or errors during data transfer.

Example: In a customer dataset, 30% of entries may have missing income information because users chose not to disclose their income. Ignoring these records could introduce bias if people who share income data systematically differ from the ones that do not.

2. Duplicate Records

Duplicates could be the result of data entry errors, multiple sources of data, or system glitches. They can dramatically bias your analysis and model training.

Example: If one of your customers appears three times in your dataset, then your model will overweigh their characteristics and make biased predictions.

3. Outliers and Anomalies

Not all outliers are errors. Many are valid extreme values. The task is determining which are which.

Example: In a house price data set, a $50 million mansion is a valid outlier. But a $5 entry is probably a data entry error where someone forgot zeros.

4. Inconsistent Formatting

Different formats for the same type of data can wreak havoc on your models.

Example: Dates “01/12/2024”, “January 12, 2024”, “12-Jan-24”, and “2024-01-12” are the same date, but without standardization, they will be considered different values.

5. Label Noise

With supervised learning, erroneous labels are often worse than missing data. Your model is actually learning from these labels; hence, the mistakes directly propagate into predictions.

Example: In an image classification dataset, if 5% of cat images are mislabeled as dogs, your model will learn from these incorrect patterns.

The Real Cost of Dirty Data

Performance Degradation

Clean data can give a simple model much better performance than a complex model on messy data. Research has shown that data quality improvement can give improvements in model performance of 10 to 30% or more; far more than switching from one algorithm to another.

Amplified Bias

Bias and error in your data will become the patterns your ML models learn and amplify. This has already led to some very real outcomes: Facial recognition systems don’t work for some demographics, hiring algorithms discriminate, and credit-scoring models perpetuate inequity.

Production Failures

A model often performs well during testing but fails in production due to data quality issues. The training data didn’t truly represent real-world conditions, or the characteristics of the production data are different than anticipated.

Business Impact According to Gartner research, poor data quality costs organizations an average of $12.9 million annually. Incorrect predictions lead to bad business decisions, lost revenue, and damaged reputation.