The Vision-Language Model (VLM) article referenced a “vision encoder converting the image into vectors” without explaining how that conversion actually happens. This article covers exactly that: the vision encoder.
The simple definition
A vision encoder is the component that converts a raw image into a sequence of vectors — a numeric representation a Transformer-based model can actually process, using the same underlying attention mechanism covered throughout the Transformers phase. Recall from the Encoder article’s general definition — a component that processes an input and builds a rich internal representation of it. A vision encoder is that same idea, applied specifically to images rather than text.
Why images can’t just be fed into a Transformer directly
Recall from the Token article that text has a natural way to break into discrete units — words and word-pieces. An image, as a grid of continuous pixel values, has no equivalent natural vocabulary at all.
Recall from the Attention article that self-attention compares every token against every other token — directly comparing millions of individual pixels this way would be computationally impossible, especially given the O(n²) cost of self-attention covered throughout the Context Window article’s discussion of computational scaling. A vision encoder exists specifically to solve both problems at once: converting continuous pixels into a manageable number of discrete, token-like units.
flowchart LR
A[Raw image: e.g., 224x224 pixels] --> B[Split into fixed-size patches, e.g., 16x16 each]
B --> C[Each patch flattened and converted into a vector]
C --> D["Sequence of 'visual tokens,' processed like text tokens"]How this actually works: the Vision Transformer approach
This is worth being precise about, since it’s a real, specific, widely adopted technique. The dominant modern approach, called the Vision Transformer (ViT), published by Google researchers in 2020, solves the “no natural vocabulary” problem directly: it chops an image into a grid of fixed-size square patches — commonly 16 pixels by 16 pixels — and treats each patch as if it were a single token, exactly the unit covered in the Token article.
Each patch gets flattened into a single vector and passed through a linear transformation, and — critically — positional encoding, covered in its own article back in the Transformers phase, gets added so the model knows which patch came from which part of the image, the visual equivalent of knowing word order in a sentence.
What vision encoders replaced, and why the switch happened
Recall from the Computer Vision article’s discussion of AlexNet’s 2012 breakthrough — that system, and nearly everything that followed it for close to a decade, used convolutional neural networks (CNNs) as their image-processing backbone, not the patch-and-attention approach this article describes.
A CNN extracts visual features by sliding small filters across an image, building up progressively larger, more abstract patterns layer by layer — a genuinely different mechanism from ViT’s global self-attention across patches, which lets every patch directly relate to every other patch from the very first layer, rather than only gradually widening its view of the image the way a CNN’s layered filters do.
Published research comparing the two found that ViT-style vision encoders tend to need more training data to reach their full potential, but scale more effectively at very large sizes — part of why CNNs remained the practical default for smaller datasets and resource-constrained settings even after ViT was published, rather than being replaced outright everywhere at once.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of cutting a large photograph into a grid of small square tiles, like a jigsaw puzzle, and numbering each tile according to its position in the original image. Once tiled and numbered, you could hand the pieces to someone in any order, and they could still reconstruct the full picture’s layout using the numbers alone — exactly the role positional encoding plays for image patches.
Where this breaks down: A jigsaw puzzle’s pieces are physically cut along the picture’s actual content. A Vision Transformer’s patches are cut along a rigid, predetermined grid, with no regard for where an object’s actual edges happen to fall — a patch might contain half a cat’s ear and half background, split arbitrarily by the fixed grid rather than by any meaningful visual boundary, and the model has to learn to work with this arbitrary splitting rather than a human-drawn, sensible outline.
Why this specific technique unlocked vision-language models
Recall directly from the Multimodal Model article’s discussion of CLIP. Once an image can be converted into a sequence of patch-vectors that look structurally just like a sequence of text tokens, the exact same Transformer machinery — attention, positional encoding, everything covered throughout that entire phase — can process both modalities using compatible, comparable representations. This is precisely the technical breakthrough that made building genuine, unified vision-language models practical, rather than requiring two entirely separate, incompatible architectures bolted together.
A concrete example, layered
For a simple beginner example: a 224×224 pixel image split into 16×16 patches produces 196 individual patches (14 patches across, 14 down), each converted into its own vector — 196 “visual tokens” processed the same way a 196-word sentence’s tokens would be.
For a production example: OpenAI’s CLIP model, referenced in the Multimodal Model article, uses exactly this Vision Transformer-style patch encoding as its image-processing half, trained jointly with a text encoder on 400 million real image-caption pairs scraped from the internet, producing the shared vector space that much of today’s vision-language work, including the VLMs covered in the previous article, builds directly on.
Why patch size involves a genuine, real trade-off
It’s worth being direct about a real, practical engineering decision here, echoing the chunk-size trade-off covered throughout the RAG & Retrieval phase. Smaller patches capture finer visual detail but produce many more tokens, directly increasing computational cost, since self-attention’s cost scales with sequence length as covered in the Context Window article. Larger patches are cheaper to process but risk losing fine detail — small text within an image, or a small but important object, might get blurred together within one patch and become genuinely harder for the model to recognize accurately.
A real-sized patch calculation
Suppose a Vision Transformer receives a 224 × 224 pixel image and divides it into 16 × 16 pixel patches:
224 ÷ 16 = 14 patches across
224 ÷ 16 = 14 patches down
14 × 14 = 196 image patches
196 patches → 196 visual token vectorsA smaller patch captures finer local details but creates more visual tokens and more computation. A larger patch creates fewer tokens and costs less, but a small object or tiny writing may be compressed into too little detail.
Visual tokens are vectors of learned numbers, not miniature JPEG files. They represent patterns found in patches plus position information describing where those patches appeared.
Frozen or jointly trained?
Some VLMs keep a pretrained vision encoder frozen, meaning its weights do not change while a connector learns to communicate with the language model. Other systems fine-tune part or all of the vision encoder jointly. Freezing is cheaper and preserves existing visual knowledge; joint training can improve task alignment but requires more data and compute.
Turn one image into visual tokens
Imagine a tiny image divided into four patches:
┌────────┬────────┐
│ sky │ bird │
├────────┼────────┤
│ tree │ grass │
└────────┴────────┘
↓ vision encoder
[visual vector 1, visual vector 2, visual vector 3, visual vector 4]Each vector contains learned numerical features for one region. Position information helps the model retain where the patch appeared. A connector or projection layer can then reshape these visual vectors into the representation expected by the language model.
The vision encoder does not normally produce the complete sentence “A bird is above a tree.” It produces useful visual representations. The language component uses those representations with the prompt to generate the sentence.
Hugging Face’s VLM documentation shows this pattern through image-text-to-text models. Google’s Gemini documentation demonstrates the production result: an application can submit an image and a question without training a separate task-specific classifier. Sources: Hugging Face’s VLM guide and Google’s image-understanding guide.
Common misconception
A frequent beginner assumption: that a vision encoder “looks” at an image the way a camera lens or a human eye does, processing the whole scene holistically at once. As the patch-based mechanism above explained, this isn’t accurate — a vision encoder processes an image as a rigid grid of independent, fixed-size pieces, only relating them to each other afterward through the same attention mechanism used for text, a fundamentally different, more mechanical process than holistic visual perception.
Where this fits in what comes next
You now understand the specific technique that converts pixels into a Transformer-compatible format. The next article, Computer Vision, zooms out to the much older, broader field this specific technique belongs to — decades of research into machine visual understanding that long predates Transformers and vision encoders entirely.
In one sentence
A vision encoder converts a raw image into a sequence of patch-based vectors — most commonly using the Vision Transformer approach of chopping an image into a fixed grid and treating each patch like a token — and this specific technique is the concrete engineering breakthrough that let the same Transformer architecture built for text handle images too.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed