Data, Features and Labels | TechByteByByte

Begin with the central question

Before choosing an algorithm, how do we turn a real problem into learnable examples?

This question explains why Data, Features and Labels deserves its own lesson. Each definition, calculation, and code example below will answer a specific part of it.

real event → row of data → input features + target label

Before you continue: three tools for this module

Data point: one example or row.
Feature: information available when making a prediction.
Label: the answer the model should learn to predict.

You do not need to memorize these yet. Return to this small map whenever a term reappears.

What You Will Understand

Data Fundamentals: Learn how to reason about features, sample density, numerical vs. categorical data, and techniques to handle missing values and outliers.
Data Leakage: Understand this critical, silent failure mode where future information leaks into training data, making validation results deceptively high.
AI Integration: Map these data cleaning and preprocessing concepts directly to RAG datasets, document chunking pipelines, and agent evaluation setups.

Picture one training example as one row:

house size | bedrooms | location  → price
  feature  | feature  | feature   → label

many rows together → dataset
dataset quality     → limits what the model can learn

Features are the information available when making a prediction. The label is the answer used during supervised training and evaluation. A field that will not exist at prediction time must not secretly become a feature.

Why Data Quality Controls Model Quality

Module 1 established that ML learns patterns from data instead of hand-written rules. That immediately raises the real engineering question: what happens when the data itself is flawed? A model doesn’t know the difference between a genuine pattern and noise, bias, or a mistake in your dataset — it will happily learn whatever pattern is actually present, including ones you never intended it to learn.

Understanding data quality isn’t a “nice to have” alongside ML — it is one of the major factors in whether a model works. Task definition, coverage of important cases, model choice, evaluation quality, and production conditions also matter.

The Literal Student Analogy

A model is like an extremely literal, extremely diligent student who has no common sense to filter what they’re taught. If you accidentally teach them a wrong fact, a biased pattern, or a shortcut that only happens to work in your training material, they will learn it exactly as faithfully as anything true and useful — and repeat it with total confidence.

4. Core Concept

Term	Definition
Dataset	The full collection of samples used for training/evaluation
Sample	One row / one example in the dataset
Numerical feature	A feature that’s a meaningful number (age, price, word count)
Categorical feature	A feature representing a category from a limited set (country, product type)
Text feature	Raw or processed natural language text used as input
Missing value	A feature that’s absent/unrecorded for a given sample
Outlier	A sample with feature values far outside the typical range
Data quality	How accurate, complete, consistent, and representative the dataset is
Feature engineering	Creating new, more useful features from raw data
Data leakage	Information from outside the legitimate training data accidentally influencing the model, producing misleadingly good results

Feature types, concretely

Numerical:    age = 34, price = 199.99, word_count = 512
Categorical:  country = "IN", plan_type = "premium"
Text:         review_text = "This product exceeded my expectations."

Different feature types often need different preprocessing before a model can use them (Module 5 covers this in depth) — a model can’t directly do math on the string "premium" the way it can on the number 34.

5. How It Works — Step by Step

1. Collect raw data                →  from logs, databases, forms, scraped sources
2. Inspect data quality             →  check for missing values, outliers, inconsistencies
3. Clean data                       →  fix, drop, or impute problematic values
4. Engineer features                →  create features more useful than the raw data alone
5. Separate legitimate signal        →  ensure no leakage from the label into the features
   from the label
6. Use the resulting dataset for     →  Module 4's train/validation/test process
   training

Missing values — the three practical options

Option A: Drop the sample entirely      — safe if very few rows affected
Option B: Drop the feature entirely     — if a feature is missing too often to be useful
Option C: Impute a reasonable value      — fill with mean/median/mode, or a
                                           model-based estimate

🧠 Intuition: There’s no universally “correct” choice — it depends on why the data is missing. Missing because a field simply wasn’t applicable (e.g., “spouse’s name” for an unmarried customer) behaves very differently from missing because of a broken data pipeline, and should usually be handled differently.

Outliers — investigate before you delete

# Build a small, inspectable example of Data, Features and Labels.
# Follow the data, learned values, predictions, and evaluation in order.
values = [45, 47, 46, 44, 5000, 48]
# is 5000 a genuine rare case (e.g., a legitimate high-value transaction)?
# or a data entry error (e.g., a misplaced decimal point, 500.0 -> 5000)?

An outlier is a signal to investigate, not an automatic candidate for deletion — sometimes it’s exactly the case your model most needs to learn about (e.g., in fraud detection, the outliers often are the fraud).

6. Mathematical Intuition

Read the mathematics as a story

real event → row of data → input features + target label

First identify the input, the operation, and the output. Then read the symbols as a shorter way to describe that same journey; do not begin by memorizing the formula.

No new formulas needed here — but one useful, concrete idea:

Mean vs. median for imputing missing numerical values:

mean   = sum of all values / number of values
median = the middle value when all values are sorted

Mean is sensitive to outliers (a single 5000 in a list of 40s and 50s drags the mean far upward); median is not. This is why median imputation is often the safer default choice for skewed real-world data.

# Build a small, inspectable example of Data, Features and Labels.
# Follow the data, learned values, predictions, and evaluation in order.
import numpy as np
values = [44, 45, 46, 47, 5000]
print("Mean:", np.mean(values))     # heavily skewed by the outlier
print("Median:", np.median(values)) # robust to the outlier

Expected Output:

Mean: 1036.4
Median: 46.0

7. Small Worked Example — Data Leakage, by hand

Walk through the example

Identify what each input number represents.
Follow one operation at a time and keep the units or class meanings attached.
Translate the result back into an ordinary sentence about the original problem.

The goal is not merely to obtain the answer; it is to expose the model’s decision process.

Imagine building a model to predict whether a hospital patient will be readmitted within 30 days, using a feature called total_hospital_charges_this_visit.

Sounds reasonable — until you realize that charge amount is only fully known after the visit concludes, which correlates strongly with readmission (longer/more complex visits cost more and have higher readmission rates) — but at actual prediction time (when a patient is first admitted), you don’t have that number yet.

The model would show excellent accuracy during training/testing, then perform far worse in production, where that feature simply isn’t available in time. This is data leakage: information that wouldn’t realistically be available at real prediction time is leaking into training.

8. Python Example

What the code will demonstrate

The following Data, Features and Labels code turns the worked example into an experiment you can repeat. First predict the result; then prepare the small dataset, apply the technique, inspect the important intermediate values, and compare the actual output with your prediction.

Python and library symbols used below

NumPy (np) stores and calculates with numeric arrays.
pandas (pd) represents table-shaped data when it is used.
scikit-learn provides tested implementations with a consistent .fit(...) and .predict(...) workflow.

# Build a small, inspectable example of Data, Features and Labels.
# Follow the data, learned values, predictions, and evaluation in order.
import pandas as pd
import numpy as np

# Simulated raw customer data with realistic messiness
data = {
    "age": [25, 30, np.nan, 45, 5000],          # missing value + likely data-entry outlier
    "country": ["IN", "US", "IN", None, "UK"],   # missing categorical value
    "purchase_amount": [120.5, 85.0, 200.0, 60.0, 150.0],
}
df = pd.DataFrame(data)
print("Raw data:\n", df)

# 1. Inspect data quality
print("\nMissing values per column:\n", df.isna().sum())

# 2. Handle the likely data-entry outlier in 'age' (5000 is not a plausible age)
df.loc[df["age"] > 120, "age"] = np.nan   # treat impossible ages as missing instead

# 3. Impute missing numerical values with the median (robust to outliers)
df["age"] = df["age"].fillna(df["age"].median())

# 4. Impute missing categorical values with the most common category
df["country"] = df["country"].fillna(df["country"].mode()[0])

print("\nCleaned data:\n", df)

Expected Output (approximate):

Raw data:
      age country  purchase_amount
0    25.0      IN            120.5
1    30.0      US             85.0
2     NaN      IN            200.0
3    45.0    None             60.0
4  5000.0      UK            150.0

Missing values per column:
 age                1
country             1
purchase_amount     0
dtype: int64

Cleaned data:
     age country  purchase_amount
0  25.0      IN            120.5
1  30.0      US             85.0
2  30.0      IN            200.0
3  45.0      IN             60.0
4  30.0      UK            150.0

How It Works

We first treat the implausible 5000 as a data-quality problem (an impossible age), not a legitimate outlier — converting it to a missing value rather than blindly trusting it.
.fillna(median) and .fillna(mode) are the imputation choices from Section 5, applied concretely to numerical and categorical columns respectively.

9. Real-World Example

A ride-sharing company builds a model to predict trip duration.

Numerical features: distance, historical average speed on this route, time of day (as a number, e.g., hour 0–23)
Categorical features: city, vehicle type, day of week
Text feature (less common but possible): free-text notes from the driver about traffic conditions
Missing values: GPS occasionally drops out mid-trip, producing incomplete distance data for some trips
Outliers: a trip logged as 8 hours long in a city where trips normally take under an hour — almost certainly a logging error, not a genuine long trip, and worth investigating rather than blindly including

Every real production ML system deals with exactly this kind of messiness — clean, textbook datasets are the exception, not the norm.

10. How This Is Used in AI

From mechanism to product

Features and labels define the task more strongly than the algorithm name. In LLM training, token context acts as input and the next token supplies the self-supervised target.

How this connects to LLMs

request → data or context preparation → model computation → evaluated output

An LLM may use this idea during training, or an AI application may use a separate ML component around the LLM. Those are different locations in the system, and the explanation below identifies which one applies.

🤖 How Is This Used in AI?

ML Concept	AI Equivalent
Dataset	An LLM’s pretraining corpus; a RAG system’s document collection
Sample	One document, one chunk, one training example
Features	For a reranking model: query-document similarity signals; for a classifier: extracted text signals
Labels	Human-written “ideal” responses for supervised fine-tuning; human preference rankings for RLHF
Missing values	Incomplete or corrupted documents in a RAG corpus
Outliers	An unusually long or malformed document that could produce a poor-quality chunk/embedding
Data leakage	An evaluation benchmark whose answers were accidentally included in a model’s training data — a real, well-known problem called benchmark contamination
Data quality	Directly determines RAG retrieval quality — messy, duplicated, or poorly-chunked documents (Module 16 of the Python course) produce poor retrieval no matter how good the LLM is

🧠 The RAG-specific version of “garbage in, garbage out”: if your document collection contains outdated, duplicated, or poorly formatted content, your RAG system will confidently retrieve and cite that bad content — the LLM has no independent way to know your source documents were flawed.

11. How This Is Used in Agentic AI

Trace one agent step

goal + state → model proposes → runtime validates → tool or response → evaluation

The model produces a prediction or proposal. The agent runtime is ordinary software that manages tools, permissions, state, retries, and execution; it may use this ML concept directly, indirectly through an LLM, or not at all.

🤖 Agent systems are frequently trained, evaluated, or fine-tuned using agent traces — recorded sequences of an agent’s reasoning steps, tool calls, and outcomes. These traces are, structurally, exactly the “dataset of samples with features and labels” from this module:

Sample: one full agent run (a task attempt)
Features: the tools available, the user’s request, the conversation history
Label: whether the task ultimately succeeded, and possibly a human quality rating of the agent’s approach

Feedback data (thumbs up/down, human corrections) collected from real agent usage becomes training/evaluation data for improving the agent over time — and is subject to exactly the same data-quality concerns covered in this module: biased feedback, missing context, or leakage (e.g., accidentally evaluating an agent on a task it was already fine-tuned on) can all silently corrupt your understanding of how well the agent actually performs.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: Deleting outliers reflexively

Why it is incorrect: Outliers are sometimes the most important signal in the dataset (fraud, rare disease diagnosis, high-value customers). Always ask why a value is unusual before removing it.

⚠️ Mistake

Incorrect idea: Imputing missing values using statistics computed from the entire dataset, including data that will later become the test set

Why it is incorrect: This is a subtle form of data leakage — see Module 4 for why this specifically corrupts evaluation results.

⚠️ Mistake

Incorrect idea: Not distinguishing “missing because irrelevant” from “missing because of a data problem.”

Why it is incorrect: These often need completely different handling (e.g., leaving irrelevant fields as a distinct “not applicable” category vs. imputing a numeric estimate for a genuinely broken data pipeline).

13. Important Distinctions

Missing Value	Outlier
A feature has no recorded value	A feature has an extreme, unusual value
Handled via drop or imputation	Handled via investigation, then decide whether to keep, cap, or remove
Often a data collection problem	Can be either a data error OR genuinely important information

Training Data	Production Data
Data used to build/evaluate the model	Real-world data the deployed model encounters
Assumed representative of reality	Can drift away from training data’s patterns over time (Module 21)

14. When Should You Use This?

Invest heavily in data quality work whenever:

Model performance in production matters (which is almost always)
Your data comes from messy real-world sources (logs, user input, scraped content, sensors)
You’re building anything that will be evaluated or compared against a benchmark — leakage there silently invalidates your results

15. When Should You NOT Use This?

This section is less about “don’t clean data” (you almost always should) and more about proportionality:

Don’t over-engineer elaborate cleaning pipelines for a small, one-off exploratory analysis where perfect data quality isn’t worth the engineering time.
Don’t blindly automate outlier removal without human review on high-stakes datasets (medical, financial, safety-critical) — automated rules can silently discard exactly the cases that matter most.

16. Production Considerations

Data validation pipelines — production ML systems typically run automated checks (schema validation, range checks, missing-value alerts) on incoming data before it ever reaches a model, catching pipeline bugs early.
Monitoring for drift — even well-cleaned training data can become unrepresentative over time as real-world patterns shift (Module 21).
Reproducibility — document exactly how missing values and outliers were handled, so results can be reproduced and audited later.
Bias awareness — data quality issues aren’t only technical; skewed or unrepresentative data can encode and amplify real-world bias into a model’s predictions, a serious production and ethical concern.

17. AI Engineer Takeaway

🎯 AI Engineer Takeaway: A model can only ever be as good as the data it learns from — no amount of clever modeling later fixes a fundamentally flawed dataset. The specific failure mode to watch for most carefully is data leakage, because it’s silent: your model will look great during evaluation and then quietly fail in production, precisely because the evaluation itself was compromised.

This exact failure mode reappears, almost unchanged, as benchmark contamination in LLM evaluation and as inflated RAG evaluation results when eval questions overlap with the retrieval corpus in unrealistic ways.

18. Interview Questions

Basic Questions

Q: What is the difference between a numerical and a categorical feature?

A: A numerical feature is a meaningful number a model can do arithmetic on directly (age, price). A categorical feature represents membership in one of a limited set of categories (country, product type) and typically needs to be encoded (Module 5) into a numerical form before most models can use it.

Q: What is data leakage?

A: Data leakage happens when information that wouldn’t realistically be available at real prediction time accidentally influences training or evaluation — making a model’s measured performance look better than it will actually be in production. A classic example: including a feature only known after the outcome you’re trying to predict.

Intermediate Questions

Q: Why is “garbage in, garbage out” especially unforgiving in machine learning, compared to traditional software?

A: In traditional software, a bug in the input data typically causes an obviously wrong or crashing output — visible and quick to catch. In ML, a model trained on flawed data doesn’t crash; it confidently learns whatever pattern is actually present in the flawed data, including biases, spurious correlations, or leaked information — producing plausible-looking, confidently wrong predictions that can be much harder to detect than an outright crash.

Q: You’re given a dataset with a feature that’s missing for 60% of samples. What are your options, and how would you decide between them?

A: Options: drop the feature entirely (if it’s rarely useful and mostly missing), drop only the affected samples (risky at 60% — you’d lose most of your data), or impute values (mean/median/mode, or a more sophisticated model-based estimate) if the feature is valuable enough to keep. The deciding factors: how important the feature is predictively, why it’s missing (a systematic reason may bias any imputation), and how much data you can afford to lose versus how much noise imputation might introduce.

Scenario-Based Questions

Q: Your fraud detection model achieves 98% accuracy in testing, but performs poorly once deployed. Investigation reveals one of the model’s features was "was_this_transaction_flagged_by_a_human_reviewer". What’s going on, and how would you fix it?

A: Thought process: A feature this strongly correlated with the label should immediately raise suspicion — this is a classic data leakage pattern.

Investigation: That feature is essentially a proxy for the label itself — transactions flagged by human reviewers were very likely already judged fraudulent by the time this feature was recorded. The model isn’t learning to detect fraud from legitimate transaction patterns; it’s learning to echo an existing human decision that happens to already be in the dataset — a decision that, critically, wouldn’t exist yet at the real moment a new transaction needs to be scored.

Correct answer: Remove this feature entirely, and audit every other feature in the dataset for the same issue: “would this value actually be known and available at the real moment of prediction, before any human or downstream process has already made a judgment about this transaction?” Retrain and re-evaluate without it.

Production consideration: This kind of leakage is exactly why realistic accuracy numbers often drop significantly once leakage is fixed — that’s not a regression, it’s the model finally being evaluated honestly. Build leakage checks into your standard data pipeline review process so this doesn’t silently reappear in future model versions.

Next: Module 04 — Train, Validation and Test Sets — why a model can perform extremely well during training but poorly in production, and how this connects directly to LLM, RAG, and agent evaluation.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026