Dimensionality Reduction | TechByteByByte

Begin with the central question

How can thousands of measurements be compressed without losing the most useful structure?

This question explains why Dimensionality Reduction deserves its own lesson. Each definition, calculation, and code example below will answer a specific part of it.

many features → compact directions/components → fewer features

Before you continue: three tools for this module

Dimension: one measurable coordinate or feature.
Component: a new direction combining original features.
Variance: how widely values spread along a direction.

You do not need to memorize these yet. Return to this small map whenever a term reappears.

What You Will Understand

Dimensionality Reduction: Understand the Curse of Dimensionality and why high-dimensional vector spaces are challenging to search and calculate.
Principal Component Analysis (PCA): Master the math of projecting data onto principal axes to maximize variance while reducing dimension size.
t-SNE and UMAP: Learn how modern non-linear techniques map high-dimensional embeddings down to 2D/3D visual spaces for analysis and debugging.

Dimensionality reduction compresses many features into fewer coordinates:

100 related measurements
          ↓ preserve selected structure
10 compressed features
          ↓
faster model, visualization, or noise reduction

Compression always chooses what structure to preserve. PCA preserves directions of high variance; a two-dimensional visualization can be useful while still distorting some distances from the original space.

Why Fewer Dimensions Can Reveal Useful Structure

Real datasets — and especially embeddings — often have hundreds or thousands of dimensions (features). Humans can visualize at most 2 or 3 dimensions directly. Beyond the visualization problem, very high-dimensional data can also genuinely hurt some models’ performance and dramatically increase computational cost.

Dimensionality reduction exists to compress high-dimensional data down into far fewer dimensions, while preserving as much of the meaningful structure as possible.

A Shadow of a Higher-Dimensional Object

Imagine trying to describe a person using 500 different measurements (height, weight, shoe size, arm span, and 496 more obscure ones). Many of these measurements are highly correlated with each other (height and arm span, for instance, move together) — so most of the genuinely independent information could actually be captured with far fewer numbers, without losing much.

Dimensionality reduction is the mathematical process of finding that smaller, more efficient representation.

4. Core Concept

Term	Definition
High-dimensional data	Data with many features/dimensions per sample (embeddings often have hundreds to thousands)
PCA (Principal Component Analysis)	A technique that finds new, uncorrelated “directions” (principal components) capturing the most variance in the data, using far fewer dimensions
Principal component	One of these new directions — a combination of the original features
Variance	How spread out the data is along a given direction — PCA prioritizes directions with the most variance, since these carry the most information
Projection	Mapping high-dimensional data points onto the lower-dimensional space defined by the principal components
t-SNE	A non-linear technique specifically designed for visualizing high-dimensional data in 2D/3D, preserving local similarity structure
UMAP	A more modern alternative to t-SNE, often faster and better at preserving both local and some global structure

5. How It Works — Step by Step

PCA

1. Start with high-dimensional data (many correlated features)
2. Find the direction (principal component) along which the
   data varies the MOST — this captures the most information
3. Find the next direction, perpendicular to the first, that
   captures the next-most variance
4. Repeat until you have as many components as you want to keep
5. PROJECT the original data onto just these top components,
   discarding the rest — this is the dimensionality reduction

Original 2D data (correlated):        After PCA (1D projection):

  y                                     Principal Component 1
  │    * *                              (captures most of the
  │  *   *  *                            original variance,
  │*   *   *                             now just one number
  │  *    *                              per point)
  └──────────── x
  (points roughly along a diagonal —
   PCA finds that diagonal direction
   as its first principal component)

🧠 Intuition: If your data mostly varies along one diagonal direction (as in the picture above), PCA recognizes that most of the “real information” is captured by position along that diagonal — the perpendicular spread is comparatively minor and can often be discarded with minimal information loss.

t-SNE and UMAP, conceptually

Unlike PCA (which finds simple, linear directions), t-SNE and UMAP use more complex, non-linear techniques specifically optimized for visualization — their explicit goal is “make points that are close in high-dimensional space stay close in the 2D/3D picture,” even if that means the resulting axes have no simple, interpretable meaning (unlike PCA’s principal components, which are literally weighted combinations of the original features).

High-dimensional embeddings (e.g., 768 dimensions)
        ↓
   t-SNE / UMAP
        ↓
2D scatter plot where nearby points in the
plot genuinely represent semantically similar
items in the original high-dimensional space

6. Mathematical Intuition

Read the mathematics as a story

many features → compact directions/components → fewer features

First identify the input, the operation, and the output. Then read the symbols as a shorter way to describe that same journey; do not begin by memorizing the formula.

You don’t need the full linear algebra behind PCA (eigenvectors and eigenvalues) to use it effectively — the practical intuition:

Each principal component "explains" some percentage of the
total variance in the data.

explained_variance_ratio = [0.65, 0.20, 0.08, 0.04, 0.03]
                             ↑
                        PC1 alone explains 65% of all
                        the variance in the original data

Cumulative sum: 0.65 + 0.20 = 0.85 → the first two components together capture 85% of the original information, even though you might have started with, say, 50 original dimensions. This is the practical number you check to decide how many components to keep.

7. Small Worked Example

Walk through the example

Identify what each input number represents.
Follow one operation at a time and keep the units or class meanings attached.
Translate the result back into an ordinary sentence about the original problem.

The goal is not merely to obtain the answer; it is to expose the model’s decision process.

Two highly correlated features: “total purchase amount” and “number of items purchased” (customers who buy more items tend to spend more, naturally). PCA would likely find that a single principal component — a weighted combination of both — captures the vast majority of the real information in these two features, since they’re not really two independent pieces of information; they largely move together.

8. Python Example

What the code will demonstrate

The following Dimensionality Reduction code turns the worked example into an experiment you can repeat. First predict the result; then prepare the small dataset, apply the technique, inspect the important intermediate values, and compare the actual output with your prediction.

Python and library symbols used below

NumPy (np) stores and calculates with numeric arrays.
pandas (pd) represents table-shaped data when it is used.
scikit-learn provides tested implementations with a consistent .fit(...) and .predict(...) workflow.

# Build a small, inspectable example of Dimensionality Reduction.
# Follow the data, learned values, predictions, and evaluation in order.
import numpy as np
from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt

# load_digits: 1797 images of handwritten digits, each 64 dimensions (8x8 pixels)
digits = load_digits()
X, y = digits.data, digits.target
print("Original shape:", X.shape)   # (1797, 64) — 64-dimensional data

# --- PCA: reduce to 2 dimensions for visualization ---
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)
print("Reduced shape:", X_pca.shape)   # (1797, 2)
print("Variance explained by each component:", pca.explained_variance_ratio_)
print("Total variance explained by 2 components:", sum(pca.explained_variance_ratio_))

# --- How many components would we need to explain 95% of the variance? ---
pca_full = PCA().fit(X)
cumulative_variance = np.cumsum(pca_full.explained_variance_ratio_)
n_components_95 = np.argmax(cumulative_variance >= 0.95) + 1
print(f"\nComponents needed for 95% variance: {n_components_95} (down from {X.shape[1]})")

Expected Output (approximate):

Original shape: (1797, 64)
Reduced shape: (1797, 2)
Variance explained by each component: [0.1489 0.1361]
Total variance explained by 2 components: 0.285

Components needed for 95% variance: 29 (down from 64)

How It Works

Compressing to just 2 dimensions only preserves about 28.5% of the original variance — enough for a rough visualization, but a genuinely lossy compression, exactly as expected when going from 64 dimensions down to 2.
The 95%-variance check shows a more realistic, useful trade-off: 29 components (less than half the original 64) still retain 95% of the real information — this is the kind of practical dimensionality reduction used to speed up downstream models, not just for visualization.

9. Real-World Example

A company has customer data with 40 highly-correlated financial features (income, credit limit, average balance, various derived ratios).

Before feeding this into a downstream model, they apply PCA to reduce it to, say, 10 components that still capture 95% of the original variance — speeding up training, reducing overfitting risk from having too many correlated features relative to the dataset size, and often improving distance-based methods like KNN (Module 10), which can degrade in very high-dimensional spaces.

10. How This Is Used in AI

From mechanism to product

Dimensionality reduction supports visualization, compression, denoising, and exploration of embeddings. A two-dimensional plot is a projection and can hide relationships that existed in the original space.

How this connects to LLMs

request → data or context preparation → model computation → evaluated output

An LLM may use this idea during training, or an AI application may use a separate ML component around the LLM. Those are different locations in the system, and the explanation below identifies which one applies.

🤖 How Is This Used in AI?

Direct relevance to Agentic AI: Moderate, primarily through embeddings.

🧠 Why embeddings are high-dimensional, and how dimensionality reduction helps analyze them: a typical text embedding might have 384, 768, or 1536+ dimensions — far too many to visualize or intuitively inspect directly. Dimensionality reduction (especially t-SNE or UMAP) is the standard practical tool for actually looking at what your embeddings are doing:

1536-dimensional document embeddings
              ↓
        t-SNE or UMAP → 2D
              ↓
   A scatter plot where clusters of
   points visually reveal semantically
   similar documents grouping together

Use case	Why dimensionality reduction helps
Visualizing embedding clusters	Human eyes need 2D/3D, not 768D
Sanity-checking a RAG document corpus	Visually spot unexpected clusters, outliers, or duplicate content
Speeding up downstream models	Fewer dimensions can mean faster training/inference for auxiliary classifiers built on top of embeddings
Noise reduction	Dropping low-variance components can sometimes filter out embedding noise, occasionally improving downstream task performance
Compression	Storing/searching lower-dimensional vectors is cheaper, though at some cost to retrieval precision

🤖 A genuinely common real workflow: after building a RAG system, an AI engineer runs UMAP on all document embeddings and visually inspects the resulting plot to sanity-check that documents are clustering the way they’d expect — catching data quality issues (unexpected duplicate clusters, mislabeled content, outlier documents) that would be very hard to spot by scanning raw text or raw high-dimensional vectors directly.

11. How This Is Used in Agentic AI

Trace one agent step

goal + state → model proposes → runtime validates → tool or response → evaluation

The model produces a prediction or proposal. The agent runtime is ordinary software that manages tools, permissions, state, retries, and execution; it may use this ML concept directly, indirectly through an LLM, or not at all.

🤖 When debugging why an agent’s retrieval step keeps surfacing unexpected or seemingly irrelevant documents, visualizing the relevant embeddings with t-SNE/UMAP is a genuinely practical diagnostic step — it can reveal, for example, that two conceptually different document types are embedding suspiciously close together (indicating a chunking or embedding model issue), which would be extremely difficult to notice just by reading raw retrieval scores or document text directly.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: Interpreting t-SNE/UMAP axes as having real, direct meaning

Why it is incorrect: Unlike PCA’s principal components (which are literal weighted combinations of original features), t-SNE and UMAP’s output axes are not directly interpretable — only relative closeness between points in the plot is meaningful, not the specific axis values or directions themselves.

⚠️ Mistake

Incorrect idea: Using PCA-reduced or t-SNE-reduced data as if no information was lost

Why it is incorrect: Dimensionality reduction is inherently lossy — always check the explained variance (for PCA) to understand how much real information you’ve kept versus discarded, especially before using reduced dimensions as direct input to a downstream model.

⚠️ Mistake

Incorrect idea: Forgetting to scale features before PCA

Why it is incorrect: PCA is sensitive to feature scale, exactly like KNN and K-Means (Modules 5, 10, 11) — an unscaled feature with a larger numeric range will disproportionately dominate the principal components.

13. Important Distinctions

PCA	t-SNE / UMAP
Linear technique	Non-linear technique
Components are interpretable (combinations of original features)	Output axes are not directly interpretable
Fast, scalable to large datasets	Generally slower, especially t-SNE, though UMAP is notably faster
Good for general dimensionality reduction (compression, noise reduction, speeding up models)	Primarily used for visualization, less suited as general-purpose preprocessing
Preserves global variance structure	t-SNE prioritizes local structure; UMAP better balances local and global structure

14. When Should You Use This?

You need to visualize high-dimensional data (especially embeddings) — use t-SNE or UMAP.
You want to compress features for a downstream model while retaining most of the real information, or reduce noise/redundancy from highly correlated features — use PCA.
You’re diagnosing or sanity-checking a RAG/embedding-based system and want an intuitive, visual way to inspect whether embeddings are behaving as expected.
Distance-based methods (KNN, K-Means) are struggling with very high-dimensional data — reducing dimensions first can sometimes help (the “curse of dimensionality,” briefly mentioned in Module 10).

15. When Should You NOT Use This?

Don’t use t-SNE/UMAP output as direct input to a downstream predictive model — they’re optimized for visualization quality, not for preserving the kind of structure a downstream model actually needs; PCA (or no reduction at all) is generally more appropriate for that purpose.
Don’t apply PCA blindly without checking explained variance — if the top components only capture a small fraction of total variance, you may be discarding genuinely important information.
For modern embedding models specifically, don’t assume dimensionality reduction is needed before using embeddings for their primary purpose (similarity search) — vector databases are specifically built to handle full-dimensional embeddings efficiently; reduction is mainly valuable for visualization/analysis, not as a required preprocessing step for retrieval itself.

16. Production Considerations

Fit dimensionality reduction on training data only — exactly the same leakage discipline from Module 4/5 applies: fit PCA on training data, then apply the same fitted transformation to new data, never refitting separately on validation/test/production data.
t-SNE/UMAP results can vary between runs (depending on random initialization and hyperparameters) — don’t over-interpret minor differences between two separately-generated visualizations of the same data.
Storage/compute trade-offs — reduced-dimension embeddings are cheaper to store and search, but retrieval accuracy can degrade; this is a genuine engineering trade-off decision, not a free win, when considered for production retrieval systems (as opposed to pure visualization/analysis use cases).

17. AI Engineer Takeaway

🎯 AI Engineer Takeaway: Dimensionality reduction exists to make high-dimensional data tractable — either computationally (PCA, for compression and noise reduction) or visually (t-SNE/UMAP, for genuinely seeing what’s happening in your data).

For an AI engineer, the single most valuable practical use is visualizing embeddings: taking a RAG corpus’s 768+-dimensional document embeddings and reducing them to 2D lets you literally see whether your retrieval system’s semantic space looks the way you’d expect — a genuinely useful, hands-on diagnostic tool that’s easy to underuse.

18. Interview Questions

Basic Questions

Q: Why is dimensionality reduction needed for high-dimensional data like embeddings?

A: High-dimensional data (often hundreds or thousands of dimensions) is impossible for humans to directly visualize or intuitively inspect, and can also increase computational cost and sometimes hurt certain models’ performance. Dimensionality reduction compresses this data into far fewer dimensions while trying to preserve as much of the meaningful structure as possible — making it possible to visualize, analyze, or more efficiently process.

Q: What is a principal component in PCA?

A: A principal component is a new direction in the data, computed as a weighted combination of the original features, chosen specifically to capture as much of the data’s variance (spread/information) as possible. The first principal component captures the most variance, the second captures the next-most (while being uncorrelated with the first), and so on.

Intermediate Questions

Q: What’s the key difference between PCA and t-SNE/UMAP, and when would you choose one over the other?

A: PCA is a linear technique whose output components are directly interpretable (literal combinations of original features), and it’s well-suited for general-purpose compression or noise reduction where you want to preserve overall variance structure. t-SNE and UMAP are non-linear techniques specifically optimized for visualization — they excel at preserving local similarity (making genuinely similar points appear close together in a 2D/3D plot), but their output axes have no direct interpretable meaning, and they’re not generally suitable as preprocessing for downstream predictive models. Use PCA for compression and general preprocessing; use t-SNE/UMAP specifically when you need to visually inspect high-dimensional data.

Q: If PCA’s first two principal components only explain 30% of the total variance in a dataset, what does that tell you, and what would you do next?

A: It means a 2D visualization or reduction using just those two components is discarding roughly 70% of the real information in the data — a significant amount of structure is being lost. For visualization purposes, this might still be acceptable as a rough, approximate picture, but for any downstream modeling use, you’d want to check the cumulative explained variance across more components (e.g., how many components are needed to reach 90-95%) before deciding how aggressively to reduce dimensionality, rather than arbitrarily settling on 2 components just because that’s easy to plot.

Scenario-Based Questions

Q: After building a RAG system, you visualize your document embeddings with UMAP and notice two clusters that seem to overlap significantly, even though the documents are about clearly distinct topics (e.g., “billing policies” and “technical troubleshooting”). What would you investigate, and what could this indicate?

A: Thought process: Overlapping clusters for topics that should be semantically distinct is a genuine red flag worth investigating rather than dismissing as a visualization artifact — though ruling out the visualization itself as the cause is also a reasonable first check.

Investigation: First, consider whether this could simply be a UMAP visualization artifact — rerun with different hyperparameters (n_neighbors, min_dist) to see if the overlap persists, since t-SNE/UMAP results can shift somewhat with different settings. If the overlap persists, investigate the actual documents in the overlapping region directly — this could reveal genuinely mixed-topic documents (a single document discussing both billing and technical issues), a chunking strategy that’s splitting documents in a way that loses distinguishing context, or an embedding model that isn’t capturing the specific semantic distinction your application cares about as clearly as expected.

Correct answer: Investigate the specific documents landing in the overlapping region directly, since visualization alone only flags where to look, not why — the root cause is very likely one of: poor chunking (losing distinguishing context), genuinely mixed-topic source documents, or an embedding model that doesn’t separate this particular topic distinction as clearly as your application needs.

Production consideration: This kind of embedding-space visual inspection is a genuinely underused but valuable RAG debugging technique — catching this kind of issue before it manifests as confusing, seemingly-irrelevant retrieval results in production is far cheaper than debugging it after the fact from user complaints alone.

Next: Module 13 — Loss Functions — how a model actually knows its prediction was wrong, connecting directly to neural network and LLM training.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026