Pretraining and Fine-Tuning

Start with the simple idea

Pretraining builds broad capability from large datasets. Fine-tuning continues training on a smaller, focused dataset to change behavior.

Simple learning path: problem → intuition → mechanism → example → limits

What you will learn

Explain Pretraining and Fine-Tuning in plain language.
Follow its mechanism step by step.
Connect a small example to a real AI system.
Recognize its strengths, limits, and common mistakes.

How this appears in current AI systems

Current providers combine large-scale pretraining with post-training. Open model ecosystems also use supervised fine-tuning and parameter-efficient methods such as LoRA.

Official grounding: Hugging Face explains fine-tuning and parameter-efficient fine-tuning for open models. Hosted providers may expose different customization methods, so confirm the provider documentation before copying an approach.

When this knowledge helps

Use Pretraining and Fine-Tuning when it matches the problem described below. Before choosing it, check the task, available data, quality target, cost, response time, privacy, and safety needs; popularity alone is not a reason to use it.

1. The question this module answers

Module 20 introduced the foundation model paradigm and named three adaptation methods. This module goes deeper into two of them — pretraining (how the foundation model itself is built) and fine-tuning (one specific way to adapt it) — and provides a really practical decision framework for choosing between prompting, fine-tuning, and RAG for a real task.

2. Pretraining — What Actually Happens

You’ve already covered the mechanics of this in your LLM course (self-supervised learning, next-token prediction over massive text corpora). Here’s the refresher, framed within this course’s broader picture:

Massive, broad dataset (e.g., a large, diverse slice of internet
                        text, or paired image-text data)
   ↓
SELF-SUPERVISED training objective (e.g., predict the next token --
                                    the "label" comes from the DATA
                                    ITSELF, requiring no manual
                                    human labeling, Module 9's
                                    diffusion training used this
                                    exact same self-supervision idea)
   ↓
Model develops broad, general capability across the training data's
full range of topics, styles, and patterns
   ↓
This is the FOUNDATION MODEL (Module 20)

💡 Why self-supervision is what makes pretraining at this scale possible: manually labeling a dataset large enough for meaningful pretraining would be prohibitively expensive and slow. Self- supervised objectives (like “predict the next token,” or “predict the noise that was added,” Module 9) generate their own training signal directly from the raw data — no human labeling required at all, really enabling training at internet scale.

3. Fine-Tuning — Going Deeper Than Module 20’s Brief Mention

Pretrained foundation model (broad, general capability)
   ↓
Smaller, TASK-SPECIFIC dataset (often requiring genuine human effort
                                to collect/label, unlike pretraining
                                data)
   ↓
CONTINUE training the model's parameters -- but now on this smaller,
focused dataset, adjusting the model's weights further
   ↓
The model's behavior really SHIFTS to better match the specific
task/domain/style represented in the fine-tuning dataset

Fine-tuning really changes the model’s parameters — this is the key distinction from prompting, which changes nothing about the model itself, only what’s fed into it at generation time.

What fine-tuning is good for

- Teaching a model a SPECIFIC STYLE or FORMAT consistently (e.g.,
  always responding in a company's specific brand voice, or always
  outputting a specific structured format)
- Improving performance on a NARROW, specialized domain that's
  underrepresented in the model's original broad pretraining data
  (e.g., highly specialized legal or medical terminology)
- Reducing the NEED for very long, detailed prompts every single
  time -- if the desired behavior is "baked into" the model's
  weights through fine-tuning, prompts can often be shorter

4. Fine-Tuning Is Not a Substitute for Real, Current Knowledge

This is a really important, common misconception worth addressing directly:

Fine-tuning teaches the model HOW to behave/respond -- style,
format, tone, task-specific patterns.

Fine-tuning is a REALLY POOR way to give a model access to
SPECIFIC, CURRENT, or FREQUENTLY-CHANGING FACTS.

💡 Why fine-tuning is a poor fit for current facts: if your company’s product catalog changes weekly, fine-tuning a model on last week’s catalog means the model has essentially “memorized” outdated information into its weights — and updating it requires ANOTHER full fine-tuning run. This is precisely why RAG (retrieval-augmented generation, covered in your Prompt Engineering course and Module 28 of this course) is generally the better-suited tool for current/changing factual information — you can update the retrieved context instantly, without retraining anything at all.

Analogy: Medical School Graduation vs. Cardiology Residency Think of pretraining vs. fine-tuning in terms of a doctor’s training pipeline:

Pre-Training (Medical School): The student spends 7 years reading thousands of general biology textbooks, chemistry logs, surgical archives, and pharmaceutical files. They learn what a cell is, how to speak to patients, and basic diagnostic vocabulary (pretraining).

Fine-Tuning (Cardiology Residency): Once graduated, the doctor doesn’t go back to high school biology. They spend 1 year in a dedicated heart clinic looking only at 100 specific heart surgery logs.

They don’t need to re-learn how to read, what a cell is, or what blood pressure means; they just adapt their vast general medical base to one specific task.

The LoRA Adapter (A sticky note on the textbook): Instead of rewriting all 10,000 pages of the general medical textbook (which would take years and cost millions), you print a small, 5-page sticky note on cardiology specs and stick it inside the book cover. You keep the original book frozen, only checking the sticky note for heart questions.

📊 Visual Flowchart: Parameter Fine-Tuning with LoRA Adapters

Here is how Parameter Efficient Fine-Tuning (PEFT) avoids modifying billions of base model weights:

graph TD
    classDef base fill:#34495e,stroke:#333,stroke-width:1px,color:#fff;
    classDef lora fill:#e67e22,stroke:#333,stroke-width:1px,color:#fff;
    classDef math fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;

    InToken["Input Token Vector (X)"] --> BaseLayer["1. Frozen Base Weights (W₀):<br>(175B parameters, unchanged)"]:::base
    InToken --> LoraDown["2a. LoRA Down-projection (A):<br>(Low Rank, e.g. rank=8)"]:::lora

    LoraDown --> LoraUp["2b. LoRA Up-projection (B)"]:::lora

    BaseLayer --> ComputeOut["Result: X * W₀"]:::base
    LoraUp --> ComputeDelta["Result: X * ΔW (where ΔW = B * A)"]:::lora

    ComputeOut --> SumLayer["3. Unified Output Summation"]:::math
    ComputeDelta --> SumLayer

    SumLayer --> FinalOut["Final Aligned Output Tensor"]:::math

5. A Direct Decision Framework — Prompting vs. Fine-Tuning vs. RAG

Question 1: Does the task need SPECIFIC, CURRENT, or FREQUENTLY-
           CHANGING information the base model doesn't already
           know?
   YES -> Use RAG (retrieval-augmented generation)
   NO  -> Continue to Question 2

Question 2: Can the desired behavior be achieved through careful
           PROMPT DESIGN alone (clear instructions, examples,
           formatting guidance -- your entire Prompt Engineering
           course)?
   YES -> Use PROMPTING (start here -- it's the fastest, cheapest,
         most flexible option)
   NO  -> Continue to Question 3

Question 3: Does the task need a CONSISTENT, SPECIALIZED
           style/format/behavior that's really hard to achieve
           reliably through prompting alone, and is worth the real
           cost of fine-tuning?
   YES -> Consider FINE-TUNING
   NO  -> Reconsider whether prompting (with better-designed
         prompts) can actually solve this

In practice: start with prompting. It’s the cheapest, fastest, most flexible option, and modern foundation models are often surprisingly capable of adapting to specific needs through careful prompt design alone (your entire Prompt Engineering course). Reach for RAG when current or specific factual grounding is needed. Reach for fine-tuning only when prompting really can’t achieve the needed consistency, and the cost is justified.

6. These Approaches Can Combine

Worth being direct: these aren’t mutually exclusive. A real production system might use a fine-tuned model (for consistent style/format) that ALSO uses RAG (for current, specific facts) and is ALSO carefully prompted (for task-specific instructions) — all three techniques working together, each addressing a really different need.

7. A Real Developer Example

A legal tech company wants an AI assistant that:
1. Always responds in a precise, formal legal writing style
2. Has access to the firm's CURRENT case files and precedents
3. Follows a specific, structured output format for legal memos

Applying the decision framework:

1. Formal legal writing style -> could START with prompting
   (detailed style instructions), and if really still
   inconsistent, consider FINE-TUNING for reliability at scale

2. Current case files/precedents -> RAG -- this information changes
   as new cases are added; fine-tuning would require constant
   retraining to stay current

3. Structured output format -> PROMPTING first (clear format
   instructions/examples, your Prompt Engineering course Module 8),
   escalating to fine-tuning only if prompting alone proves
   really unreliable at scale

Real system: likely combines careful PROMPTING + RAG, with
fine-tuning considered only if prompting really can't achieve
reliable enough consistency for the style requirement.

8. A Simple Agentic AI Connection

An agent’s tool-use behavior and reasoning style is almost always adapted through prompting (system prompts defining its role, available tools, and behavioral guidelines) rather than fine-tuning — this keeps the agent’s design flexible and fast to iterate on. RAG is commonly used to give an agent access to current, specific knowledge (like a company’s internal documentation) beyond what the underlying foundation model already knows.

Fine-tuning an agent’s underlying model is a much rarer, heavier-weight choice, generally reserved for cases where prompting really can’t achieve needed reliability at scale.

9. How Is This Used in AI?

🤖 How Is This Used in AI?

This decision framework directly guides how real AI product teams architect their systems — most modern AI products lean heavily on prompting and RAG, reserving fine-tuning for really specialized needs where the cost and complexity are justified by the specific gains in consistency or narrow-domain performance.

10. Common Mistakes

Incorrect idea

Reaching for fine-tuning to give a model current or specific factual knowledge.

Why it is incorrect

As shown directly, this is a really poor fit — RAG is the better-suited tool for information that changes or needs to stay current.

Incorrect idea

Jumping straight to fine-tuning without first trying to solve the problem through better prompting.

Why it is incorrect

Modern foundation models are often more capable than expected when given well-designed prompts — fine-tuning’s real cost and complexity should be reserved for cases where prompting has really been tried and found insufficient.

Incorrect idea

Treating prompting, fine-tuning, and RAG as mutually exclusive choices.

Why it is incorrect

As shown directly in Section 6, real systems often combine all three, each addressing a really different need.

11. Limitations

Fine-tuning requires real, often substantial, task-specific data — a genuine practical barrier for organizations without existing labeled datasets for their specific use case
Fine-tuning has real costs: computational cost, time, and the ongoing maintenance burden of updating the fine-tuned model as needs change
Even fine-tuned models remain subject to hallucination (Module 32) — fine-tuning shapes style and behavior, but doesn’t provide the same kind of grounding in verifiable, current facts that RAG can

12. Quick Reference — The Whole Idea in One Diagram

Pretraining:      broad data + self-supervised objective -> general-
                 purpose foundation model (Module 20)

Fine-tuning:         foundation model + smaller task-specific
                   dataset -> further trains PARAMETERS -> shifted,
                   more specialized behavior

Decision framework:      need CURRENT/specific facts? -> RAG
                       can prompting alone solve it? -> PROMPTING
                       (start here)
                       need deep, consistent specialization AND
                       prompting isn't enough? -> FINE-TUNING

13. Code — Applying the Decision Framework in Practice

🎯 Target of this example: turn Section 5’s decision framework into an actual, runnable decision function — demonstrating, with real example scenarios, how a developer would systematically reason through choosing between prompting, fine-tuning, and RAG for a given task.

Example 1 — Simple

def recommend_adaptation_approach(needs_current_facts: bool, prompting_sufficient: bool) -> str:
    """A direct implementation of Section 5's decision framework --
    two key yes/no questions determine the recommended approach."""
    if needs_current_facts:
        return "RAG (retrieval-augmented generation)"
    elif prompting_sufficient:
        return "Prompting"
    else:
        return "Fine-tuning (consider only after prompting has been tried)"

# A few example scenarios
scenarios = [
    ("Answering questions about this week's inventory levels", True, False),
    ("Summarizing text in a specific, clear style", False, True),
    ("Consistently outputting a rare, highly specific JSON schema, "
     "where prompting has proven unreliable", False, False),
]

for description, needs_facts, prompting_ok in scenarios:
    recommendation = recommend_adaptation_approach(needs_facts, prompting_ok)
    print(f"Task: {description}")
    print(f"  -> Recommended approach: {recommendation}\\n")

Expected Output:

Task: Answering questions about this week's inventory levels
  -> Recommended approach: RAG (retrieval-augmented generation)

Task: Summarizing text in a specific, clear style
  -> Recommended approach: Prompting

Task: Consistently outputting a rare, highly specific JSON schema,
where prompting has proven unreliable
  -> Recommended approach: Fine-tuning (consider only after prompting
  has been tried)

What we conclude from this example: each scenario maps cleanly onto Section 5’s decision tree — this simple function turns an abstract framework into a concrete, reusable decision tool, directly usable when scoping real system architecture decisions.

Example 2 — Intermediate

import anthropic

client = anthropic.Anthropic()

def demonstrate_prompting_first(task_description: str, style_instructions: str, user_input: str) -> str:
    """Demonstrates Section 5's key principle: TRY prompting first,
    before considering fine-tuning -- often surprisingly effective
    with careful prompt design."""
    system_prompt = (
        f"{style_instructions}\\n\\n"
        f"Task context: {task_description}"
    )
    response = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=150,
        system=system_prompt,
        messages=[{"role": "user", "content": user_input}],
    )
    return response.content[0].text

style_instructions = (
    "You are a formal legal writing assistant. Always use precise, "
    "formal language. Structure responses as: 1) Summary, 2) Key "
    "Considerations, 3) Recommendation. Avoid casual phrasing entirely."
)

result = demonstrate_prompting_first(
    task_description="Drafting internal legal guidance memos",
    style_instructions=style_instructions,
    user_input="Should our company update its data retention policy "
               "given new state privacy regulations?",
)
print(result)

Expected Output:

1) Summary: Recent state privacy regulation changes may necessitate a
review and potential revision of the company's current data retention
policy to ensure continued compliance.

2) Key Considerations: The updated regulations may impose stricter
limitations on retention periods, require enhanced documentation of
data handling practices, and introduce new disclosure obligations to
affected individuals.

3) Recommendation: A formal legal review of the current retention
policy against the new regulatory requirements is advised, with
particular attention to retention timelines and consent documentation
practices.

What we conclude from this example: careful, detailed prompting (specific style AND structure instructions) achieves the formal, consistently structured output the legal tech example from Section 7 needed — without any fine-tuning at all. This is exactly the “start with prompting” principle from Section 5, demonstrated with real, observable output quality.

Example 3 — Production Grade

import anthropic
from dataclasses import dataclass
from enum import Enum

client = anthropic.Anthropic()

class AdaptationStrategy(Enum):
    PROMPTING = "prompting"
    RAG = "rag"
    COMBINED_PROMPTING_AND_RAG = "combined_prompting_and_rag"
    FINE_TUNING_RECOMMENDED = "fine_tuning_recommended"

@dataclass
class AdaptationDecision:
    strategy: AdaptationStrategy
    rationale: str

def decide_adaptation_strategy(
    needs_current_facts: bool, needs_consistent_specialized_style: bool, prompting_has_been_tried_and_failed: bool
) -> AdaptationDecision:
    """A more complete decision function, directly implementing
    Section 6's point that these approaches often COMBINE rather
    than being mutually exclusive."""
    if needs_current_facts and needs_consistent_specialized_style:
        return AdaptationDecision(
            AdaptationStrategy.COMBINED_PROMPTING_AND_RAG,
            "Needs both current facts (RAG) and specific style (prompting) -- combine both.")
    elif needs_current_facts:
        return AdaptationDecision(
            AdaptationStrategy.RAG,
            "Needs current/specific facts -- RAG is the better-suited tool than fine-tuning.")
    elif needs_consistent_specialized_style and prompting_has_been_tried_and_failed:
        return AdaptationDecision(
            AdaptationStrategy.FINE_TUNING_RECOMMENDED,
            "Prompting has really been tried and found insufficient for the needed consistency.")
    else:
        return AdaptationDecision(
            AdaptationStrategy.PROMPTING,
            "Start here -- fastest, cheapest, most flexible; likely sufficient.")

# The legal tech scenario from Section 7, fully decided
legal_memo_decision = decide_adaptation_strategy(
    needs_current_facts=True,  # current case files/precedents
    needs_consistent_specialized_style=True,  # formal legal writing style
    prompting_has_been_tried_and_failed=False,  # haven't yet exhausted prompting
)

print(f"Strategy: {legal_memo_decision.strategy.value}")
print(f"Rationale: {legal_memo_decision.rationale}")

Expected Output:

Strategy: combined_prompting_and_rag
Rationale: Needs both current facts (RAG) and specific style
(prompting) -- combine both.

What we conclude from this example: this decision function correctly identifies that the legal tech scenario from Section 7 needs BOTH current facts and consistent style — routing to a really combined strategy rather than forcing an artificial single choice. This directly reflects Section 6’s real, practical point: production systems very often need multiple adaptation techniques working together, not one selected in isolation.

14. Interview Questions

Q: What is self-supervised learning, and why is it essential for pretraining foundation models at scale?

Ans: Self-supervised learning generates its own training signal directly from the raw data itself — for example, predicting the next token in a sequence, where the “correct answer” is simply the actual next token in the training text, requiring no manual human labeling. This is essential for pretraining at internet scale because manually labeling a dataset large enough for meaningful pretraining would be prohibitively expensive and slow — self-supervision makes training on massive, broad datasets really practical.

Q: What does fine-tuning actually change about a model, and how does this differ from prompting?

Ans: Fine-tuning continues training the model’s actual parameters (weights) on a smaller, task-specific dataset, really shifting the model’s behavior at a fundamental level. Prompting, by contrast, changes nothing about the model itself — it only changes what’s fed into the model as input at generation time. This is the key distinction: fine-tuning modifies the model; prompting modifies the input to an unmodified model.

Q: Why is fine-tuning generally considered a poor fit for giving a model access to current or frequently-changing factual information?

Ans: If the underlying facts change, a fine-tuned model that has “memorized” older information into its weights becomes outdated, and updating it requires running another full fine-tuning cycle — a slow, costly process. RAG is better suited for current or changing information because the retrieved context can be updated instantly, without retraining anything, directly supplying current facts at generation time rather than baking them into the model’s parameters.

Q: Describe the recommended decision process for choosing between prompting, fine-tuning, and RAG for a new AI feature.

Ans: Start by asking whether the task needs specific, current, or frequently-changing information the base model doesn’t already know — if so, use RAG. If not, ask whether the desired behavior can be achieved through careful prompt design alone — start here, since prompting is the fastest, cheapest, and most flexible option, and modern foundation models are often surprisingly capable with well-designed prompts. Only consider fine-tuning if prompting has really been tried and found insufficient for achieving needed consistency or specialization, and the real cost is justified. These approaches can also combine in a single system when different requirements call for different techniques.

15. What You Should Remember

Pretraining uses self-supervised objectives on broad data to produce a general-purpose foundation model — no manual labeling required.
Fine-tuning really changes model parameters, unlike prompting — good for consistent style/format, poor for current or frequently-changing facts (use RAG instead).
The decision framework (need current facts? try RAG. Can prompting solve it? start there. Need deep specialization prompting can’t achieve? consider fine-tuning) — verified directly by applying it to a real, combined-need scenario — is a really practical tool for real system architecture decisions.

16. Quick Practice

A team wants their AI assistant to (1) always respond in a very specific, playful brand voice, and (2) always have access to the company’s up-to-date product pricing. Walk through Section 5’s decision framework for each requirement separately, and explain what the final, combined system architecture would likely look like.

17. Next Step

Next: Module 22 — Alignment — closing Level 5: how a raw, capable pretrained model becomes a really usable, helpful, safe assistant through instruction tuning and alignment techniques.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026