CNNs and Computer Vision | TechByteByByte

Begin with the central question

How can a network recognize an object wherever it appears in an image?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

pixels → sliding filters → feature maps → pooled features → prediction

Before you continue: three tools for this module

Pixel: one tiny colored location in an image.
Kernel or filter: a small grid of learned numbers moved across an image.
Feature map: the grid of responses produced by applying a filter at many locations.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

Why plain fully-connected networks (Module 2) handle images poorly, and how Convolutional Neural Networks (CNNs) solve this using convolution, filters, feature maps, and pooling — covered at the depth an AI engineer needs, with a real, verified edge-detection example, not a full computer vision specialization.

A convolution reuses one small filter across local regions:

image pixels → sliding learned kernel → feature map
                  same weights reused
early maps: local patterns → later maps: combinations of patterns

This weight sharing makes CNNs far more efficient than connecting every image pixel independently to every neuron. Learned filters are not guaranteed to have one neat human meaning, even when some respond strongly to edges or textures.

Why Images Need Local Patterns and Shared Weights

A fully-connected layer (Module 2) treats every input independently, with its own separate weight — for a modest 224×224 color image, that’s over 150,000 input values, and a single hidden layer of reasonable size would need tens of millions of weights just for that one layer, with no built-in understanding that nearby pixels are related.

CNNs exist to exploit a structural fact about images that fully-connected layers ignore entirely: spatial structure matters — nearby pixels are meaningfully related, and the same simple pattern (an edge, a corner) can appear anywhere in the image.

Sliding One Detector Across an Image

instead of a giant layer connecting every pixel to every neuron, a CNN slides a small pattern-detector (a “filter” or “kernel”) across the image, checking “does this small local pattern appear here?” at every position. The same small filter is reused across the entire image — an edge detector that works in the top-left corner works identically in the bottom-right, dramatically reducing the number of parameters needed while directly encoding the assumption that spatial patterns matter.

Analogy: The Detective’s Sliding Magnifying Glass Imagine you are a detective analyzing a giant map of a city to find specific structures (like stadiums or parking lots):

Fully Connected approach (Overwhelming): You try to take in the entire map simultaneously. You connect a string from every single building to your brain at once. It causes sensory overload, and you get lost in the noise.

Convolutional approach (The Magnifying Glass): You hold a small 3x3 magnifying glass (the filter or kernel) with a target outline drawn on it (weights). You slide this glass sequentially across the map, step-by-step (stride of 1). At each position, you look through the glass and count how many structures match your target outline (computes a weighted sum).

Feature Map: You draw a small mark on a notepad for each coordinate on the map showing how strongly the target outline matched. This notepad is the feature map.

Hierarchical Magnifying Glasses: You stack detectives. The first detective uses a tiny glass to find simple horizontal lines. The second detective looks at the first detective’s notepad through a larger glass, combining those simple lines to detect rectangles. The third detective combines those rectangles to locate the final stadium.

📊 Visual Diagram: The Convolution Operation (Sliding Kernel)

Here is how a 3x3 kernel slides across a 5x5 input matrix to calculate a single element of a 3x3 output feature map:

graph TD
    subgraph ConvOp ["Convolution Matrix Operation (3x3 Kernel on 5x5 Input)"]
        InputMatrix["5x5 Input Matrix (Pixels)<br>[ 1  0  1  0  0 ]<br>[ 0  1  1  1  0 ]<br>[ 0  0  1  0  1 ]<br>[ 0  1  0  1  0 ]<br>[ 1  1  0  0  1 ]"]

Kernel["3x3 Kernel (Weights)<br>[  1  0 -1 ]<br>[  1  0 -1 ]<br>[  1  0 -1 ]"]

InputSubmatrix["3x3 Local Receptive Field (Top-Left)<br>[ 1  0  1 ]<br>[ 0  1  1 ]<br>[ 0  0  1 ]"]

Summation["Multiply Element-wise & Sum:<br>(1*1) + (0*0) + (1*-1) +<br>(0*1) + (1*0) + (1*-1) +<br>(0*1) + (0*0) + (1*-1)<br>= 1 + 0 - 1 + 0 + 0 - 1 + 0 + 0 - 1 = -2"]

OutputMap["3x3 Output Feature Map<br>[ -2  .  . ]<br>[  .  .  . ]<br>[  .  .  . ]"]

InputMatrix -->|1. Extract Local 3x3| InputSubmatrix
        InputSubmatrix -->|2. Dot Product with Kernel| Summation
        Kernel -->|2. Dot Product| Summation
        Summation -->|3. Place in top-left cell| OutputMap
    end

4. Core Concept

Term	Definition
Convolution	Sliding a small filter across an image, computing a weighted sum at each position
Filter / kernel	A small matrix of learnable weights, detecting a specific local pattern
Feature map	The output of applying one filter across an entire image — where that pattern was detected, and how strongly
Stride	How many pixels the filter moves between positions
Padding	Adding extra border pixels so the filter can process edge regions properly
Pooling	Downsampling a feature map, keeping only the most important information from local regions

Hierarchical features

image
  ↓
edges           (early layers: simple, local patterns)
  ↓
textures/shapes  (middle layers: combinations of edges)
  ↓
object parts     (later layers: combinations of shapes)
  ↓
objects          (final layers: combinations of parts)

⚠️ This is an intuitive abstraction, not a guarantee. Real trained CNNs often do show something like this hierarchy in early research visualizations, but individual filters don’t always map cleanly to a single human-nameable concept — much like Module 2’s caution about neurons not always corresponding to one interpretable feature.

5. How It Works — Step by Step

1. A small filter (e.g., 3x3 weights) starts at the top-left of
   the image
2. At each position, compute the weighted sum of the filter's
   weights times the pixel values currently under it (exactly
   Module 2's weighted sum, applied to a local image region)
3. Slide the filter over by the STRIDE, repeat
4. The collected outputs across all positions form ONE feature map
5. Multiple DIFFERENT filters are applied in parallel, each
   producing its own feature map, each potentially detecting a
   different pattern
6. POOLING then downsamples each feature map, keeping the
   strongest signal from each local region
7. This entire block (convolution + activation + pooling) can be
   stacked -- later layers operate on the FEATURE MAPS from
   earlier layers, building increasingly complex representations

6. Mathematical Intuition

First, use only small numbers

A 2 × 2 filter examines four nearby pixels at a time. It multiplies matching positions and adds them, producing one output number; sliding one position repeats the same test elsewhere in the image.

Read the mathematics as a story

A convolution reuses one small filter across the image. This parameter sharing lets the same detector find a pattern in many locations without learning a separate detector for every pixel.

pixels → sliding filters → feature maps → pooled features → prediction

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. Convolution, worked on a tiny 6×6 image with a vertical edge (left half 0, right half 1), using a classic vertical-edge-detecting kernel:

kernel = [[-1, 0, 1],
          [-1, 0, 1],
          [-1, 0, 1]]

At a position straddling the edge (3 columns of 0s and the first column of 1s not yet included, sliding across), the weighted sum computes (-1×0) + (0×0) + (1×1), repeated down each row of the 3×3 kernel — producing a strong positive response exactly where the edge is, and near-zero everywhere flat/uniform (both all-0 and all-1 regions average out to nothing, since the kernel’s positive and negative sides cancel identically-valued neighbors).

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. A single 3×3 vertical-edge kernel, applied across an entire image, lights up wherever a vertical edge exists — regardless of whether that edge is near the top, middle, or bottom of the image.

This is the concrete meaning of “the same filter works anywhere in the image” — one small set of learned weights, reused everywhere, rather than a separate weight for every pixel position.

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of CNNs and Computer Vision.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

# Tiny 6x6 "image" with a vertical edge in the middle
image = np.array([
    [0, 0, 0, 1, 1, 1],
    [0, 0, 0, 1, 1, 1],
    [0, 0, 0, 1, 1, 1],
    [0, 0, 0, 1, 1, 1],
    [0, 0, 0, 1, 1, 1],
    [0, 0, 0, 1, 1, 1],
], dtype=float)

kernel = np.array([
    [-1, 0, 1],
    [-1, 0, 1],
    [-1, 0, 1],
], dtype=float)

def convolve2d(img, kernel, stride=1):
    kh, kw = kernel.shape
    ih, iw = img.shape
    oh = (ih - kh) // stride + 1
    ow = (iw - kw) // stride + 1
    output = np.zeros((oh, ow))
    for i in range(oh):
        for j in range(ow):
            region = img[i*stride:i*stride+kh, j*stride:j*stride+kw]
            output[i, j] = np.sum(region * kernel)
    return output

feature_map = convolve2d(image, kernel)
print("Feature map (after convolution):\n", feature_map)

def max_pool2d(fmap, pool_size=2, stride=2):
    ph, pw = pool_size, pool_size
    fh, fw = fmap.shape
    oh = (fh - ph) // stride + 1
    ow = (fw - pw) // stride + 1
    output = np.zeros((oh, ow))
    for i in range(oh):
        for j in range(ow):
            region = fmap[i*stride:i*stride+ph, j*stride:j*stride+pw]
            output[i, j] = np.max(region)
    return output

pooled = max_pool2d(feature_map)
print("\nAfter 2x2 max pooling:\n", pooled)

print("\nOriginal image shape:", image.shape)
print("Feature map shape:", feature_map.shape)
print("Pooled shape:", pooled.shape)

Expected Output:

Feature map (after convolution):
 [[0. 3. 3. 0.]
 [0. 3. 3. 0.]
 [0. 3. 3. 0.]
 [0. 3. 3. 0.]]

After 2x2 max pooling:
 [[3. 3.]
 [3. 3.]]

Original image shape: (6, 6)
Feature map shape: (4, 4)
Pooled shape: (2, 2)

9. How It Works

The feature map shows 0 everywhere flat (uniform 0s or uniform 1s under the kernel cancel out) and 3 exactly where the vertical edge sits — the kernel genuinely detected the edge’s location, purely through the weighted-sum-and-slide mechanism, with no explicit “edge-finding” logic ever written.
Max pooling reduced the 4×4 feature map to 2×2, keeping only the strongest detected signal (3) from each 2×2 region — this is downsampling in action: the pooled output still clearly indicates “an edge was detected here,” using a quarter of the spatial resolution.
Shape shrinks at each stage: 6×6 image → 4×4 feature map (a 3×3 kernel with no padding loses 2 pixels per dimension) → 2×2 pooled output — exactly the “hierarchy compresses spatial detail while building up feature information” pattern described in Section 4.

10. Real-World Example

A real CNN for image classification stacks many convolution+pooling blocks: early layers with filters like this module’s edge detector, later layers with filters that respond to increasingly complex, learned combinations of earlier feature maps — eventually feeding into a fully-connected layer (Module 2) that makes a final classification decision, using the accumulated hierarchy of features rather than raw pixels directly.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

CNNs remain useful for image classification, detection, medical imaging, and efficient vision backbones. Many current multimodal systems also use Vision Transformers, so ‘computer vision’ does not mean ‘CNN only.’

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

CNNs remain the standard choice for many practical computer vision tasks (image classification, object detection) — though Transformer- based vision architectures (Vision Transformers) have also become competitive for some tasks in recent years. Multimodal LLMs (models that accept image input alongside text) typically use a vision encoder — historically CNN-based, increasingly Transformer-based — to convert images into embeddings (Module 12) the language model can process alongside text tokens.

Concept	AI application
Convolution	Image classification, object detection, still widely used
Feature maps	The learned, hierarchical visual representations these models build
Vision encoders in multimodal LLMs	Converting images into embeddings for a language model to reason over

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Low-to-Moderate. Directly relevant when an agent needs to process image input (e.g., analyzing a screenshot, reading a document photo) via a multimodal model’s vision component — otherwise, CNNs aren’t a core part of most text-based agent architectures. Worth knowing conceptually for exactly this multimodal case, and for recognizing it in architecture diagrams.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: each filter always detects one clean, human-nameable concept

Why it is incorrect: (like “always detects cat ears”). As Section 4 cautions, this is an intuitive simplification — real trained filters can respond to combinations of patterns that don’t map cleanly to any single human concept.

⚠️ Mistake

Incorrect idea: CNNs are now obsolete, replaced entirely by Transformers.

Why it is incorrect: CNNs remain a strong, efficient, widely-used choice for many vision tasks — Vision Transformers are competitive in some settings but haven’t uniformly replaced CNNs the way Transformers have largely replaced RNNs for text (Module 14 explains why that replacement happened for sequences specifically).

⚠️ Mistake

Incorrect idea: a CNN “sees” an image the way a human does.

Why it is incorrect: A CNN processes raw pixel-intensity patterns through learned filters — it has no innate concept of objects, physics, or meaning; whatever useful structure it captures emerged entirely from training data and the architecture’s spatial-locality assumptions.

14. Important Distinctions

Convolution	Fully-Connected Layer
Small filter, REUSED across the entire image (few parameters)	Every input connects to every neuron (many parameters)
Exploits spatial locality assumption	No inherent spatial assumption

Convolutional Layer	Pooling Layer
Has learnable weights (the filter)	Has NO learnable weights — a fixed downsampling operation
Detects patterns	Reduces spatial resolution, keeping the strongest signal

15. When to Use

Use CNNs (or a vision-specific architecture) for image or spatially- structured data. Their parameter-sharing (the same filter reused everywhere) is specifically valuable when the same local pattern can meaningfully appear anywhere in the input.

16. When Not to Use

Don’t use CNNs for data with no meaningful spatial/local structure — a tabular dataset (Module 5 of the ML course) or a bag of unrelated features gains nothing from convolution’s spatial-locality assumption and is better served by classical ML or plain fully-connected layers.

17. Interview Questions

Beginner

Q: Why don’t fully-connected networks work well for raw images?

Ans: A fully-connected layer treats every pixel as an independent input with its own separate weight, requiring an enormous number of parameters for even modest-sized images, and has no built-in understanding that nearby pixels are related. This ignores images’ spatial structure entirely and scales very poorly with image size.

Intermediate

Q: What does a convolutional filter actually compute?

Ans: It’s a small matrix of learnable weights that slides across the image; at each position, it computes a weighted sum of the filter’s weights times the pixel values currently under it — the same weighted- sum operation as a neuron (Module 2), applied to a small local region, with the same filter weights reused at every position across the entire image.

Advanced

Q: Why does convolution use dramatically fewer parameters than a fully-connected layer processing the same image, and why does that matter beyond just efficiency?

Ans: A fully-connected layer needs a separate weight for every input-pixel- to-neuron connection. A convolutional filter has a fixed, small number of weights (e.g., 9 for a 3×3 filter), reused identically at every spatial position across the entire image.

Beyond efficiency, this parameter sharing directly encodes a meaningful assumption: that a useful local pattern (like an edge) is equally useful to detect regardless of where in the image it appears — this assumption (translation invariance) is exactly why CNNs generalize well for vision tasks with comparatively less training data than a fully-connected approach would need.

Scenario

Q: You apply a single 3×3 convolutional filter to a 6×6 image and get a 4×4 output. Why did the spatial dimensions shrink, and how would you prevent this?

Ans: The filter can’t be centered on pixels too close to the image’s border without extending past the edge — a 3×3 filter can only be placed at (6−3+1)=4 positions along each dimension without padding, producing a 4×4 output instead of 6×6, exactly as demonstrated in this module.

To preserve the original spatial size, you’d add padding — extra border pixels (commonly zeros) around the image — giving the filter room to be centered on every original pixel, including those near the edges.

AI Engineering

Q: Where do CNNs (or CNN-like components) typically appear in a modern multimodal LLM?

Ans: In the vision encoder — the component that converts raw image input into a sequence of embeddings the language model portion can process alongside text tokens. Historically this vision encoder was CNN-based; increasingly, Vision Transformer-style encoders are also used.

Either way, its job is the same: transform raw pixels into a meaningful, lower-dimensional representation the rest of the (typically Transformer- based) model can reason over.

18. What You Should Remember

Convolution slides a small, learnable filter across an image, reusing the same weights at every position — exploiting spatial structure that fully-connected layers ignore.
Feature maps show where a pattern was detected; pooling downsamples them, keeping the strongest local signal.
Verified concretely: a 3×3 edge-detecting kernel correctly located a vertical edge in a tiny image, producing 0 in flat regions and a strong response exactly at the edge.
CNNs remain a strong, standard choice for vision tasks and appear inside multimodal LLMs’ vision encoders — worth recognizing, not reimplementing from scratch.

19. How This Helps Me Build AI Systems

If you ever work with a multimodal agent that processes images, this module is what’s happening inside its vision encoder before that image ever becomes something the language model can reason about — raw pixels, transformed through exactly this convolution-and-pooling mechanism, into a representation ready to be embedded (Module 12) alongside text.

Next: Module 14 — RNNs, LSTM and GRU — how networks process sequences, and the specific limitations that directly motivate attention.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026