Pretraining

Every mention throughout this glossary of “training a model from scratch”. The GPT-3 example used across the Training Mechanics and Neural Networks phases, the 300-billion-token training run, the 96-layer architecture — has actually been describing one specific phase of a model’s life: pretraining.

The learning journey in this phase

flowchart LR
    A[Raw text, code, images, audio] --> B[Self-supervised pretraining]
    B --> C[General foundation model]
    C --> D[Instruction tuning]
    C --> E[Task or domain fine-tuning]
    E --> F[Full fine-tuning]
    E --> G[PEFT]
    G --> H[LoRA]
    G --> I[QLoRA]

Think of pretraining as completing a very large general education. Transfer learning reuses that education. Fine-tuning provides extra practice for a particular job. PEFT, LoRA, and QLoRA make that extra practice cheaper by changing only a small part of the trainable system.

The simple definition

Pretraining is the initial, large-scale training process that teaches a model general language patterns, facts, and structure, using a broad, massive dataset, before it’s ever adapted to any specific task.

Recall from the Foundation Model article’s Stanford-coined definition: a foundation model is “trained on broad data… that can be adapted to a wide range of downstream tasks.”

Pretraining is precisely the process that produces that broad, adaptable base — it’s the “training on broad data” half of that definition, given its own name.

Why this needs to be named as a distinct phase at all

Recall from the Foundation Model article that most companies don’t train models entirely from scratch for every new application — they adapt an existing, broadly capable base instead.

Pretraining is the name for the process that creates that base in the first place: one enormous, expensive training run, using a general dataset covering a huge range of topics and styles, with no specific downstream task in mind yet.

Everything covered later in this phase — Fine-Tuning, Instruction Tuning, LoRA — describes what happens after pretraining, adapting an already-pretrained model rather than building general capability from nothing.

flowchart LR
    A[Massive, general dataset] --> B[Pretraining: one large, expensive training run]
    B --> C[Pretrained foundation model: broad, general capability]
    C --> D[Later: adapted for specific tasks]

What actually happens during pretraining, mechanically

This connects directly to mechanics already covered throughout this glossary — pretraining isn’t a different kind of training, just a specific, enormous application of the exact process covered throughout the Training Mechanics phase.

Recall from the Next-Token Prediction article: a model is trained to predict the next token, over and over, across its training data.

During pretraining, that training data is deliberately broad and general — recall from the Training Data article’s discussion of Common Crawl and similarly vast text sources — rather than narrowly focused on any one task, so the model’s learned patterns end up covering an enormous range of language, facts, and reasoning styles, rather than specializing in any one of them.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of the years someone spends getting a broad, general education — reading widely, studying many subjects, absorbing a huge range of facts and ways of thinking — before ever starting a specific job.

That broad education doesn’t prepare them for any one particular career task directly, but it gives them the general foundation that makes learning any specific job, later, dramatically faster than starting from nothing.

Where this breaks down: A person’s general education involves genuine understanding built up over years of conscious learning.

Pretraining is the mechanical, repeated gradient descent and backpropagation process covered throughout the Training Mechanics phase, applied at enormous scale. The “broad education” analogy captures the purpose well, but the actual mechanism is statistical pattern absorption across billions of tokens, not anything resembling conscious study.

Why pretraining is, by far, the most expensive part of a model’s life

This is worth grounding concretely, tying directly back to real numbers already established elsewhere in this glossary. Recall from the Training and Gradient articles: OpenAI’s GPT-3 was pretrained on roughly 300 billion tokens, reportedly using a cluster of around 10,000 GPUs running for weeks.

This single, massive pretraining run is what produces the base model’s broad capability, and it’s overwhelmingly more expensive, in both time and compute, than any of the adaptation techniques covered later in this phase, which typically work with a small fraction of that data and compute, applied to an already-pretrained starting point.

A concrete example, layered

For a simple beginner example: pretraining a small model on a broad mix of children’s stories, news articles, and simple non-fiction gives it a general sense of grammar, common facts, and everyday language patterns. None of it aimed at any specific task, but useful as a starting point for almost anything built afterward.

For a production example: Meta’s Llama models, GPT-3, and virtually every major LLM covered throughout this glossary all begin with a pretraining phase on a broad, general corpus, before any of the task-specific adaptation techniques covered in the rest of this phase — Fine-Tuning, Instruction Tuning, LoRA — are ever applied.

Why pretraining alone usually isn’t enough for a usable product

It’s worth being direct about a genuine limitation here, one that motivates the rest of this entire phase.

A purely pretrained model is very good at predicting statistically likely continuations of text, but recall from the OpenAI 3.5/InstructGPT case study, well-documented in public reporting: a raw pretrained model, given an instruction like “Explain quantum computing,” might just as easily continue with more similar-sounding instructions rather than actually answering, since its training only taught it to predict plausible next text, not specifically to follow instructions helpfully.

This exact gap — broad capability without task-following behavior — is precisely what the rest of this phase’s techniques exist to close.

Real pretrained models across different fields

Beyond general-purpose models like GPT and Llama, several real, published models have been pretrained specifically on domain-focused text from the ground up, illustrating how pretraining itself gets specialized by field:

• Finance — BloombergGPT, a 50-billion-parameter model developed by Bloomberg, was pretrained on a mix of general text and a large collection of financial documents (roughly 363 billion tokens of financial data, according to its published research), giving it a foundation specifically shaped by financial language and concepts before any further adaptation.
• Biomedical research — BioGPT, a 1.5-billion-parameter model, was pretrained specifically on PubMed biomedical literature, giving it a foundation in medical and scientific terminology that a general web-text model wouldn’t have to the same degree.
• Protein engineering — ProtGPT2, a 738-million-parameter model, was pretrained on protein sequence data rather than natural language at all, illustrating that the same underlying next-token-prediction pretraining approach generalizes well beyond human language to other kinds of sequential data entirely.
• Legal research — models like InternLM-Law have combined continual domain pretraining on statutes and case law with further instruction tuning, giving the model a foundation specifically shaped by legal language and structure.
• General-purpose comparison — GPT-3, Llama, and Gemini remain pretrained on broad, general web text rather than any single domain, which is precisely why domain-focused models like the ones above often outperform general models on their specific field’s benchmarks, despite being far smaller.

Three simple pretraining exercises

Text

Input:  Water freezes at 0 degrees ___
Target: Celsius

The target comes from the original text itself. Nobody needs to label the sentence as “a fact about freezing.”

Code

Input:  for item in items:
Target:     print(item)

Repeated prediction across large code collections helps a model learn syntax and common programming patterns.

Images and other media

A multimodal model can learn from image-text pairs, image regions, audio segments, or video frames using objectives suited to those representations. It learns statistical relationships between modalities, not a human-style lived understanding of the scene.

How GPT and Gemini use pretraining

OpenAI’s GPT-4 research page says its base model was trained to predict the next word using web-scale public and licensed data. OpenAI explains that most capabilities come from this pretraining stage, while later post-training mainly steers behavior toward user intent.

Google’s Gemini 1.0 technical report describes Gemini as multimodal from the beginning, trained to accept interleaved text, image, audio, and video information. The precise datasets and every training detail are not public, but the documented architecture shows why multimodal pretraining is broader than reading text alone.

Curiosity question: does the model memorize the internet?

The weights store learned numeric patterns, not a neat library where each training document can be opened by title.

Models can sometimes reproduce memorized fragments, especially duplicated or unusual text, but they also combine patterns to produce new sequences. This is why training-data quality, privacy filtering, deduplication, and evaluation matter.

From raw data to a pretrained checkpoint

Pretraining is more than sending the internet directly into a model:

flowchart LR
    A[Collect licensed and permitted data] --> B[Parse and clean]
    B --> C[Remove duplicates and unsafe or private material]
    C --> D[Create train, validation, and test partitions]
    D --> E[Tokenize or encode media]
    E --> F[Self-supervised training]
    F --> G[Periodic evaluation and checkpoints]
    G --> H[Pretrained base model]

Filtering is imperfect, but it matters. Duplicate examples can encourage memorization. Data leakage can make evaluations look better than real performance. Low-quality, biased, copyrighted, private, or malicious material can create legal, safety, and reliability problems.

One end-to-end customer-support story

This example connects the entire phase:

1. A foundation model is pretrained on broad language and code using self-supervised targets.
2. Transfer learning reuses those general language abilities for customer support.
3. Instruction tuning teaches the model to respond to requests rather than merely continue text.
4. The company prepares reviewed support conversations for fine-tuning.
5. Limited hardware leads the team to choose PEFT instead of changing every parameter.
6. LoRA trains small update matrices while the base model remains frozen.
7. If the normal base model does not fit in memory, QLoRA stores its frozen weights in quantized form during adaptation.
8. Validation checks support accuracy, general ability, safety, and memorization before deployment.

broad model → instruction follower → company specialist → evaluated deployment

When should a team pretrain from scratch?

Pretraining makes sense when a team needs a genuinely new foundation model, has distinctive large-scale data or modality needs, and can afford extensive infrastructure and evaluation. Most application teams should begin with an existing suitable model and adapt or retrieve information rather than repeat foundation-model pretraining.

Common misconception

A frequent beginner assumption: that pretraining and “training” are simply two words for the same thing, with no meaningful distinction.

As this article has shown, pretraining specifically refers to the initial, broad, general-purpose phase. Everything covered in the rest of this phase (fine-tuning, instruction tuning, and the parameter-efficient techniques that follow) is also technically “training,” in the sense of updating a model’s weights. However, it’s training applied to an already-pretrained model, for a narrower, more specific purpose, using dramatically less data and compute than pretraining itself required.

Where this fits in what comes next

You now understand pretraining as the massive, general foundation everything else in a model’s life builds on.

The next article, Transfer Learning, covers the broader concept this foundation makes possible — reusing knowledge learned on one task or dataset to help with a completely different one, the core idea underlying every adaptation technique covered for the rest of this phase.

In one sentence

Pretraining is the massive, expensive, general-purpose training run that builds a model’s broad foundation, using a huge and diverse dataset with no specific task in mind, and it’s overwhelmingly the most costly phase of a model’s life, which is exactly why the rest of this phase focuses on cheaper ways to adapt that foundation rather than repeating it from scratch.

Related Terms

• Foundation Model
• Training
• Transfer Learning
• Training Data
• Self-Supervised Learning