Begin with the problem
Prompting, RAG, and fine-tuning solve different problems. The correct choice depends on whether you need better instructions, current knowledge, consistent behavior, or a combination.
problem type → prompt first → add retrieval for knowledge → fine-tune for repeated behavior → evaluateWhat you will learn
- Match prompting, RAG, fine-tuning, and tools to the problem they solve.
- Explain why fine-tuning is not a dependable database for changing facts.
- Choose the least costly intervention that passes evaluation.
Current production grounding: Google’s Gemini tools documentation distinguishes managed built-in tools from custom functions executed by the application.
These links document current public features and practices. They do not reveal a provider’s private implementation or guarantee that every product uses identical defaults.
1. The Engineering Problem
Module 20 closed with a decision hierarchy for diagnosing an underperforming system. This module addresses the upfront architectural decision: when designing a new capability, which of prompting, RAG, or fine-tuning should you reach for — and, honestly, when should the answer be “none of these, write deterministic code instead”?
2. The Core Distinction
PROMPTING ALONE: the model's EXISTING training knowledge,
guided by instructions -- no
external data, no weight changes
RAG: the model's REASONING, GROUNDED
in external, retrieved knowledge -- the
model itself is UNCHANGED
FINE-TUNING: the model's WEIGHTS are adjusted -- changing HOW it behaves or
reasons, not what facts it retrievesThe single most important distinction: RAG changes WHAT information the model has access to. Fine-tuning changes HOW the model behaves. These solve different problems, and conflating them is precisely why teams often fine-tune when they actually needed retrieval, or vice versa.
3. When Prompting Alone Suffices
appropriate when:
- The needed knowledge is small and static enough to
fit directly in a prompt
- The task is well within the model's existing training
knowledge
- No PRIVATE or frequently-changing data is involved4. When RAG Is the Right Choice
appropriate when:
- The task needs CURRENT information the model wasn't trained on
- The task needs PRIVATE, organization-specific knowledge
- The underlying knowledge CHANGES frequently -- re-indexing is
cheaper than re-trainingYour entire RAG course — this module doesn’t re-teach RAG, it clarifies exactly when to reach for it versus the alternatives.
5. When Fine-Tuning Is the Right Choice
appropriate when:
- The issue is CONSISTENT STYLE or BEHAVIOR, not missing knowledge
(Module 20, Section 6's second scenario)
- The task needs a specific output format or reasoning
pattern that prompting alone can't reliably achieve
- You have, sufficient training examples of the desired
behaviorIncorrect idea: A common misconception: fine-tuning “teaches the model new facts.” It doesn’t do this reliably — fine-tuning shapes HOW the model behaves and responds, not WHAT current facts it knows. For knowledge, RAG is the right tool; for behavior/style, fine-tuning is.
Why it is incorrect: This shortcut removes a validation or control boundary, allowing an error to pass into later stages where it becomes harder and more expensive to detect.
6. When NONE of These — Traditional Code Wins
Your RAG course's Module 30: a task needing a PRECISE,
COMPUTED value (an exact revenue figure, a mathematical calculation)
is more reliably solved by a direct SQL query or
deterministic code than ANY LLM-based approach, including RAG.7. A Real-World Analogy — The Hospital, Once More
Module 10 and 20's doctor analogy: PROMPTING is like
consulting a well-trained doctor's existing knowledge.
RAG is like handing that doctor the PATIENT'S actual, current chart
before the consultation. FINE-TUNING is like the doctor completing additional residency training that changes HOW they
approach diagnoses generally, not just for one patient.
None of these REPLACES a lab test that gives an EXACT number
(TRADITIONAL CODE) -- you don't ask a doctor to "estimate" a blood
test result when a lab instrument can measure it precisely.8. A worked developer example
TechCorp evaluates three different capability requests using this module’s framework:
| Request | Analysis | Chosen Approach |
|---|---|---|
| “Answer questions using our current, private policy documents” | Needs external, current, private knowledge | RAG |
| “What was our exact Q3 revenue?” | Needs a precise, computed value | Traditional code (direct query) |
| “Our responses need consistent tone/style across all customer interactions” | A style/behavior issue, not a knowledge gap | Fine-tuning |
9. How Is This Used in the Industry?
🤖 How Is This Used in the Industry?
Production teams apply this exact decision framework before committing engineering effort to any of these three approaches — reaching for RAG’s infrastructure investment or fine-tuning’s data/training investment only when the actual problem’s characteristics warrant it, rather than defaulting to whichever approach is currently most discussed.
10. Common Mistakes
Incorrect idea: Fine-tuning to “teach the model new facts.”
Why it is incorrect: As shown directly in Section 5, fine-tuning doesn’t reliably serve this purpose — RAG does.
Incorrect idea: Using RAG for a task needing a precise, computed value.
Why it is incorrect: As shown directly in Section 6, direct code is more reliable here.
Incorrect idea: Reaching for RAG by default even when knowledge fits in a single, static prompt.
Why it is incorrect: As shown directly in Section 3, this adds unnecessary infrastructure.
11. Code — An Approach Recommendation Function
What this shows: implementing this module’s decision framework directly — a function that walks through Sections 3-6’s conditions in the correct order, exactly Section 8’s real developer example made into working, repeatable decision logic.
from dataclasses import dataclass
from enum import Enum
class Approach(Enum):
PROMPTING = "prompting_alone"
RAG = "retrieval_augmented_generation"
FINE_TUNE = "fine_tuning"
TRADITIONAL_CODE = "traditional_deterministic_code"
@dataclass
class DecisionResult:
approach: Approach
rationale: str
def recommend_approach(needs_current_knowledge: bool, needs_private_data: bool,
needs_precise_computed_value: bool, needs_consistent_style_not_knowledge: bool,
knowledge_fits_in_one_prompt: bool) -> DecisionResult:
"""Directly implements Sections 3-6's decision framework -- each
condition checked in an order that correctly prioritizes the
most SPECIFIC, need over a more general default."""
if needs_precise_computed_value:
return DecisionResult(Approach.TRADITIONAL_CODE,
"A precise, computed value is more reliably obtained via direct query/code than any LLM-based approach.")
if needs_current_knowledge or needs_private_data:
return DecisionResult(Approach.RAG,
"needs external, current, or private knowledge beyond the model's training data.")
if needs_consistent_style_not_knowledge:
return DecisionResult(Approach.FINE_TUNE,
"This is about consistent STYLE/behavior, not missing knowledge -- fine-tuning fits.")
if knowledge_fits_in_one_prompt:
return DecisionResult(Approach.PROMPTING,
"Knowledge fits in a single prompt and doesn't change -- no retrieval infrastructure needed.")
return DecisionResult(Approach.PROMPTING, "No external knowledge or style-consistency need identified.")
# Exactly Section 8's three real developer requests
r1 = recommend_approach(needs_current_knowledge=True, needs_private_data=True,
needs_precise_computed_value=False, needs_consistent_style_not_knowledge=False,
knowledge_fits_in_one_prompt=False)
print(f"[{r1.approach.value}] {r1.rationale}")
r2 = recommend_approach(needs_current_knowledge=False, needs_private_data=False,
needs_precise_computed_value=True, needs_consistent_style_not_knowledge=False,
knowledge_fits_in_one_prompt=False)
print(f"[{r2.approach.value}] {r2.rationale}")
r3 = recommend_approach(needs_current_knowledge=False, needs_private_data=False,
needs_precise_computed_value=False, needs_consistent_style_not_knowledge=True,
knowledge_fits_in_one_prompt=True)
print(f"[{r3.approach.value}] {r3.rationale}")Expected Output:
[retrieval_augmented_generation] needs external, current,
or private knowledge beyond the model's training data.
[traditional_deterministic_code] A precise, computed value is more
reliably obtained via direct query/code than any LLM-based approach.
[fine_tuning] This is about consistent STYLE/behavior, not
missing knowledge -- fine-tuning fits.What this confirms: all three of Section 8’s distinct requests correctly route to their appropriate approach — RAG for external/private knowledge, traditional code for a precise computed value, and fine-tuning for a style-consistency issue — exactly this module’s decision framework, made into working, repeatable logic rather than a case-by-case debate.
12. Production Considerations
- These approaches are NOT mutually exclusive — a real system might use RAG for knowledge AND fine-tuning for consistent output style, combined
- Revisit this decision periodically — a task’s requirements can shift (Module 20’s feedback loop) in ways that change which approach is actually correct over time
13. Trade-offs
- Fine-tuning requires training data and infrastructure investment RAG doesn’t — a real, higher upfront cost for the right kind of problem
- RAG’s infrastructure (Modules 7) is unnecessary overhead for knowledge that fits in a static prompt
14. Chapter Summary
Prompting, RAG, and fine-tuning solve different problems — prompting leverages existing model knowledge, RAG changes what information the model has access to and fine-tuning changes how the model behaves. A precise, computed value is often more reliably solved by traditional deterministic code than any of the three.
The most common mistake is conflating these — especially fine-tuning to “teach new facts,” which is RAG’s job, not fine-tuning’s.
15. Visual Cheat Sheet
Precise computed value? --> Traditional Code
Current/private knowledge? --> RAG
Consistent style/behavior? --> Fine-Tuning
Small, static, known knowledge --> Prompting alone
RAG changes WHAT the model knows.
Fine-tuning changes HOW the model behaves.16. Top Takeaways
- RAG changes what information a model has access to; fine-tuning changes how the model behaves — different problems.
- Fine-tuning does not reliably “teach new facts” — that’s RAG’s job.
- A precise, computed value is often more reliably solved by direct, traditional code than any LLM-based approach.
- These approaches are not mutually exclusive — a real system may combine RAG and fine-tuning.
- Prompting alone suffices when knowledge is small, static, and already within the model’s training data.
17. Interview Questions
Q: 1. A team wants to fine-tune a model to “teach it about our company’s current product catalog.” What would you push back on?**
Ans: Fine-tuning doesn’t reliably serve this purpose — it shapes how a model behaves and reasons, not what specific, current facts it retrieves. A product catalog is exactly the kind of current, frequently-changing, private knowledge RAG is designed for — retrieval lets the system stay current as the catalog changes, without needing to re-train every time.
- Why it matters: This is one of the most common misconceptions in this space, leading teams to invest in expensive fine-tuning when RAG would solve the actual problem more reliably and cheaply.
- Real-world example: Section 8’s first row — current, private knowledge maps to RAG, not fine-tuning.
- Common mistake: Assuming fine-tuning is a general-purpose way to “add knowledge” to a model.
- Interviewer is testing: Whether the candidate understands the structural difference between changing model behavior and changing model knowledge access.
- Likely follow-up: “When WOULD fine-tuning be appropriate here?” → If the team also needed the model to consistently follow a specific response format or tone across every product-related answer — a style/behavior need, not a knowledge one.
Q: 2. Why might a task requiring an exact, computed business metric be better solved by direct code than by RAG, even though RAG could technically retrieve relevant documents?**
Ans: RAG is a similarity-search and generation mechanism — well-suited to finding and grounding on relevant text, but not a computation engine.
A precise value like “exact Q3 revenue” is more reliably obtained via a direct SQL query or deterministic calculation against the actual data source, which guarantees exactness in a way that retrieval-then-generation from text documents cannot.
- Why it matters: Defaulting to RAG for every knowledge-adjacent need, including precise computed values, introduces, unnecessary unreliability.
- Real-world example: Section 8’s second row, and your RAG course’s Module 30 text-to-SQL discussion.
- Common mistake: Building a RAG pipeline over structured, computable data that would be better served by a direct query.
- Interviewer is testing: Whether the candidate recognizes RAG’s scope and limitations, not treating it as a universal solution.
- Likely follow-up: “How would you route between these approaches in one system handling both types of questions?” → Query classification/routing logic (directly your RAG course’s Module 30 pattern) in the orchestration layer (Module 3).
18. Scenario-Based Question
Scenario: TechCorp’s team spends three months and significant budget fine-tuning a model on their internal documentation, hoping it will “know” the company’s current policies. After deployment, the model still gives outdated answers whenever a policy changes, since the fine-tuned knowledge is frozen at training time.
- Problem Analysis: Section 5’s warning realized in practice — fine-tuning was used to attempt what RAG is designed for: current, changeable knowledge.
- How to Think: This is a structural mismatch between the chosen approach and the actual problem — no amount of additional fine-tuning effort would fix this since the fundamental limitation (frozen knowledge at training time) is inherent to the approach, not a matter of degree.
- Investigation: Confirm that policy updates require a new fine-tuning run to reflect, and that this update cycle is too slow for how often policies actually change.
- Root Cause: Fine-tuning was chosen for a RAG-shaped problem — current, frequently-changing, retrievable knowledge.
- Solution: Migrate to a RAG-based architecture (Module 7) for policy knowledge specifically — current by design, since retrieval happens against a live, updatable knowledge base rather than baked-in model weights.
- Trade-offs: This requires building RAG infrastructure the team hadn’t originally invested in — a real, additional cost, though ultimately the correct one for this specific problem.
- Production Considerations: This scenario directly demonstrates Section 2’s core distinction — the team needed to change WHAT the model knows (RAG’s job), but invested in changing HOW it behaves (fine-tuning), a costly architectural mismatch this module’s framework is designed to prevent.
19. Next Step
Next: Module 22 — AI Workflow vs. AI Agent — comparing deterministic workflows, LLM workflows, and agents across reliability, cost, latency, and control, directly extending your Agents course’s “least autonomous architecture” principle.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed