Data Preprocessing

If you’ve ever tried to cook using ingredients straight from a garden — unwashed vegetables, whole spices, a chicken that still needs plucking — you know there’s a lot of prep work before actual cooking can start. Real-world data arrives in an equally raw state: messy, inconsistent, full of gaps and errors. Data preprocessing is that prep work, applied to data instead of ingredients.

The simple definition

Data preprocessing is the process of cleaning, organizing, and transforming raw data into a form a model can actually use. Recall from the Data article that data is the raw material AI learns from — but “raw” is the key word there. Real-world data almost never arrives ready to feed directly into a learning Algorithm; it needs to be cleaned up first, and that cleanup process is what this article covers.

Cleaning one messy row

Suppose raw house data arrives like this:

area = "1,200 sq ft"
bedrooms = "Three"
age = missing
city = " bengaluru "

A model needs consistent values. Preprocessing might produce:

area_sq_ft = 1200
bedrooms = 3
age = 8        # filled using an approved strategy
city = "bengaluru"

Every transformation needs a reason. Filling missing age with 8 is not automatically correct; the strategy must be learned from appropriate training data and documented.

The preprocessing pipeline

flowchart LR
    A[Raw data] --> B[Validate]
    B --> C[Clean]
    C --> D[Handle missing values]
    D --> E[Encode and scale]
    E --> F[Model-ready features]

Typical steps include removing impossible values, standardizing units, handling missing data, converting categories, parsing text or images, normalizing numbers, and selecting usable fields.

Fit on training data, apply everywhere

Suppose missing ages are filled with the median age.

Training ages: 10, 12, 14
Training median: 12

Use the training median 12 for the validation, test, and production transformations. Calculating a new median from the test set would allow test information to influence the pipeline.

Training data → learn preprocessing settings
Validation/test/new input → reuse those settings

Production consistency

The same input must receive the same transformations during training and inference. Otherwise the model learns one representation and receives another after deployment.

A production pipeline should version preprocessing code and settings together with the model, validate unexpected categories and units, log failures safely, and test the transformation with known examples.

Why this step exists at all

Real-world data is messy in ways that would surprise a beginner who’s only ever seen clean textbook examples. Spreadsheets have missing values where someone forgot to fill in a field. Sensor readings have occasional wild, obviously-wrong spikes from equipment glitches. Text data has typos, inconsistent formatting, and duplicate entries. Different data sources record the same thing differently — one system logs dates as “01/02/2026,” another as “2026-02-01,” and a naive model would treat these as completely unrelated pieces of information rather than the same date.

If you fed a model this raw mess directly, several things would go wrong: missing values might crash the training process entirely, wildly inconsistent scales between features could make training unstable (a problem covered in depth in the next article, Normalization), and inconsistent formatting could hide real patterns the model would otherwise be able to find. Preprocessing exists to prevent all of this — it’s the difference between handing a model clean, usable material and handing it chaos.

What preprocessing actually involves

Preprocessing isn’t one single step — it’s a collection of related cleanup tasks, applied as needed depending on what’s wrong with a given dataset:

• Handling missing values. An engineer has to decide what to do with gaps: remove the incomplete records entirely, fill them in with a reasonable estimate (like the average value for that feature), or use a value that explicitly signals “this was missing,” depending on how much data would be lost and how important that missing information is.
• Removing duplicates. Repeated or near-identical records can quietly bias a model toward whatever happens to be duplicated most, and can also cause the data leakage problems introduced in the Train-Test Split article if duplicates end up split across training and test sets.
• Fixing inconsistent formats. Standardizing dates, units of measurement, capitalization in text, and category naming (e.g., making sure “USA,” “U.S.A.,” and “United States” are treated as the same value) so the model doesn’t mistakenly treat identical information as different.
• Handling outliers. Deciding what to do with values that are technically real but extreme enough to distort training — like a single sensor reading of 9,999 degrees in an otherwise normal temperature dataset, almost certainly a malfunction rather than a real measurement.
• Encoding categorical data. Converting non-numeric categories — like “red,” “blue,” “green” — into a numeric form a model can actually process, since, as covered in the Input article, models only compute on numbers.

flowchart LR
    A[Raw, messy data] --> B[Handle missing values]
    B --> C[Remove duplicates]
    C --> D[Fix inconsistent formats]
    D --> E[Handle outliers]
    E --> F[Encode categories numerically]
    F --> G[Clean data, ready for training]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a librarian receiving a huge donated box of books with no consistent organization — some missing covers, some duplicated, some catalogued under inconsistent author-name spellings. Before those books can go on the shelves and actually be useful to patrons, the librarian has to sort, deduplicate, and standardize everything.

Where this breaks down: A librarian applies judgment and context to each individual book. Preprocessing decisions in ML are usually made systematically, at scale, across an entire dataset at once, using consistent rules rather than case-by-case human review — simply because most real datasets are far too large for anyone to inspect record by record. This means preprocessing choices, once made, get applied uniformly, for better or worse, across the whole dataset.

A concrete example, layered

Take the hospital readmission dataset used throughout this glossary. Raw patient records might have some fields blank (a lab test that wasn’t ordered for a particular patient), some dates recorded in inconsistent formats across different hospital departments, and the occasional obviously wrong entry, like a patient age of 200. Preprocessing would mean deciding how to handle the missing lab values, standardizing every date into one consistent format, and either correcting or removing the clearly erroneous age entries — all before that data ever reaches the training process described in the Training article.

Why this eats so much real project time

It’s genuinely common, in real ML projects, for data preprocessing to take up the majority of a project’s total time and effort — often cited by practicing engineers as 60–80% of the work, well more than the time spent on model training itself. This surprises a lot of beginners who assume the “real work” is choosing and tuning a sophisticated algorithm.

In practice, a simple algorithm trained on well-preprocessed data will usually beat a sophisticated algorithm trained on messy, poorly-cleaned data — which is exactly why experienced engineers treat preprocessing as a first-class part of the job, not a chore to rush through on the way to the “interesting” part.

Key terms

• Raw data: Information before task-specific preparation.
• Cleaning: Correcting or handling invalid and inconsistent values.
• Imputation: Filling missing values using an explicit method.
• Encoding: Converting information into a machine-usable representation.
• Pipeline: An ordered, repeatable sequence of transformations.

Check your understanding

Should preprocessing be chosen using the test set? No. Settings should normally be learned from training data.

Is preprocessing merely cosmetic cleanup? No. It changes the exact numerical information the model receives.

Common misconception

Beginners sometimes assume preprocessing is a purely mechanical, low-skill task — something to automate quickly and move past.

In reality, many preprocessing decisions require real domain judgment: how to handle a missing value depends on why it’s likely missing (a lab test not ordered for a healthy patient is different from a lab test lost due to a data-entry error); which outliers are genuine anomalies worth removing versus genuine (if unusual) real-world cases worth keeping requires actual understanding of the data’s context.

Treating preprocessing as thoughtless busywork is one of the more common — and more damaging — mistakes in early-career ML work.

Prevent train-serving skew

Train-serving skew happens when training and production prepare equivalent data differently. For example, training may replace a missing age with the training median while the live application replaces it with zero, causing the model to receive a pattern it was not trained to interpret.

The fitted preprocessing rules should therefore be saved, versioned, tested, and deployed with the model. “Unknown” must not automatically become zero: zero can be a real measurement, while unknown means the measurement is absent.

Where this fits in what comes next

Data preprocessing is the broad umbrella; the next article, Normalization, zooms into one specific and especially important preprocessing technique — putting numeric features on a consistent scale — followed by Feature Engineering, which goes a step further than cleanup and looks at actively creating new, more useful features from the cleaned data.

In one sentence

Data preprocessing is the essential, often time-consuming work of turning messy, inconsistent, real-world data into something clean and consistent enough for a model to actually learn a genuine pattern from — and skipping or rushing it quietly undermines everything built on top of it.

Related Terms

• Data
• Dataset
• Normalization
• Feature Engineering
• Feature