Reinforcement Learning (RL)

Every training technique covered throughout this glossary so far — supervised learning on labeled data, self-supervised learning on raw text — has shared one thing in common: a known correct answer to learn from. This phase opens with the one major training paradigm that doesn’t require that at all: reinforcement learning.

How this phase connects

flowchart LR
    A[Human preference data] --> B[Reward model or direct preference objective]
    B --> C[RLHF or DPO changes model behavior]
    C --> D[Better alignment with intended behavior]
    D --> E[Safety evaluation]
    E --> F[Application guardrails]
    F --> G[Monitored real-world system]

This phase moves from learning through consequences to the broader question of how an AI system should behave around people. Training can improve model behavior, but deployed applications still require safety testing, permissions, filters, monitoring, and human oversight.

The simple definition

Reinforcement learning is a training approach where a system, called an agent, learns by taking actions in an environment and receiving rewards or penalties, gradually learning which actions lead to better outcomes. Recall from the Machine Learning article’s brief introduction of this as the third major learning type, alongside supervised and unsupervised learning.

Unlike supervised learning, covered throughout the Data Handling phase, there’s no dataset of “correct answers” to learn from directly — instead, the agent discovers good behavior through repeated trial, observing which actions tend to produce better rewards over time.

Why this is a genuinely different kind of learning

Recall from the Label and Ground Truth articles that supervised learning needs a known correct answer for every training example. Many real problems don’t have this — there’s no single “correct” sequence of moves in a chess game, only outcomes (win or lose) that only become clear after a long sequence of decisions.

Reinforcement learning is built specifically for this kind of problem: instead of being told the right answer for each individual step, an agent learns from the consequences of its actions, adjusting its behavior to make higher-reward outcomes more likely over time.

flowchart LR
    A[Agent takes an action] --> B[Environment responds]
    B --> C[Agent receives a reward or penalty]
    C --> D[Agent adjusts future behavior]
    D --> A

How this actually works, step by step

An RL system has a few core components worth naming precisely, since they show up throughout the rest of this phase. The agent is the system being trained — the thing making decisions.

The environment is whatever the agent interacts with — a game, a simulation, or, as covered in the very next article, a conversation with a person. The action is a specific choice the agent makes at a given moment. The reward is a number indicating how good or bad that action turned out to be.

Training proceeds by repeating this loop many times, using the accumulated reward signal to adjust the agent’s decision-making — its policy — in whatever direction tends to produce higher rewards, using update mechanics that echo the gradient-based optimization covered throughout the Training Mechanics phase, even though the underlying goal (maximize long-term reward) is different from the loss-minimization goal covered there.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of training a dog with treats. You don’t hand the dog a manual explaining exactly how to sit — you reward it with a treat whenever it happens to do something close to sitting, and withhold the treat otherwise. Over many repetitions, the dog learns, purely from this pattern of rewards, which specific behavior reliably produces treats.

Where this breaks down: A dog’s learning involves genuine biological reward processing and memory. An RL agent’s “learning” is a mathematical optimization process — adjusting a policy’s parameters, exactly as covered throughout the Weights and Gradient Descent articles, to make higher-reward actions statistically more likely in similar future situations, with no biological reward system involved, just numbers being nudged in a direction that historically correlated with better outcomes.

Real, well-documented examples of RL in action

This is worth grounding concretely, since RL has a genuine, celebrated track record outside of language models entirely. DeepMind’s AlphaGo and its successor AlphaZero, real, published systems, used reinforcement learning — playing millions of games against themselves — to reach superhuman performance at Go and chess, discovering strategies that surprised even expert human players.

Robotics research has used RL to teach simulated and real robots to walk, grasp objects, and navigate, learning through repeated physical trial and reward rather than being explicitly programmed with movement rules.

A concrete example, layered

For a simple beginner example: an RL agent learning to navigate a simple maze receives a positive reward for reaching the exit and a small penalty for every step taken, gradually learning, through many attempted runs, which paths lead to the exit fastest.

For a production example: as covered in detail in the very next article, OpenAI’s InstructGPT and the broader RLHF process apply this exact reinforcement learning framework to language models — treating the model as the agent, a generated response as an action, and human judgments of response quality as the reward signal, adapting a technique originally developed for games and robotics to the very different domain of conversational AI.

Why reinforcement learning has genuine, honest limitations

It’s worth being direct about a real, well-known challenge in RL, not presenting it as a universally superior training method.

RL can require an enormous number of trial-and-error attempts to learn effectively, since useful reward signal is often sparse — an agent might take hundreds of actions before ever receiving a single meaningful reward, especially in tasks with long delays between an action and its consequence.

This is part of why RL is typically applied on top of an already-pretrained model, as covered throughout the Pretraining article, rather than being used to teach a system everything from complete scratch.

Learn RL through a delivery robot

Imagine a robot learning to deliver a parcel through a small hallway:

State:   robot is at the entrance
Action:  move left
Result:  bumps into a wall
Reward:  -2

State:   robot is at the entrance
Action:  move forward
Result:  gets closer to the destination
Reward:  +1

Final delivery completed
Reward:  +10

The robot tries actions, observes consequences, and updates its policy so actions associated with better long-term reward become more likely. It must sometimes explore unfamiliar actions, but it must also exploit actions already known to work.

The five pieces of an RL problem

Piece	Delivery-robot meaning	Language-model meaning during RLHF
Agent	Robot	Language model or policy
Environment	Hallway	Prompt, training system, and evaluation process
State or observation	Robot’s current situation	Prompt and tokens generated so far
Action	A movement	Selecting the next token or response
Reward	Score for the result	Learned or rule-based score for response quality

Language-model RL differs from a physical robot because a generated response is usually evaluated after many token actions, and the reward often represents a learned estimate of human preference rather than a directly measurable destination.

Real model connection

OpenAI’s InstructGPT work used PPO, a reinforcement-learning algorithm, to update a GPT-3 policy using scores from a learned reward model. Google’s Gemini 1.0 report also documents safety-focused post-training that includes supervised fine-tuning and RLHF using a reward model.

Use RL when actions influence later outcomes and a useful reward signal exists. Ordinary supervised learning is simpler when every input already has a clear correct target and no sequential consequence needs to be optimized.

Another example: reducing energy use in a building

Imagine an RL system controlling heating and cooling:

State:   room temperatures, outside weather, time, and occupancy
Action:  increase, decrease, or maintain heating and cooling
Reward:  comfort score − energy cost − penalty for unsafe temperatures

Turning every heater off would reduce energy use, but it would make people uncomfortable and might allow pipes to freeze. The reward must represent the complete goal rather than one easy-to-measure number.

Over many simulated or carefully controlled trials, the policy can learn when to warm a room before people arrive and when to reduce energy in unused spaces. Engineers still impose hard temperature limits because an imperfect learned policy should not control safety boundaries by itself.

Common misconception

A frequent beginner assumption: that reinforcement learning is simply a more advanced or more powerful version of supervised learning.

As this article has explained, they’re solving genuinely different kinds of problems — supervised learning needs a known correct answer for each example; reinforcement learning works precisely in situations where that kind of direct labeling isn’t available or doesn’t make sense, learning instead from the downstream consequences of a whole sequence of decisions.

Where this fits in what comes next

You now understand the general framework of learning through trial, action, and reward. The next article, RLHF, covers the specific, historically significant application of this framework to language models — using real human judgments as the reward signal, exactly the technique behind the shift from raw GPT-3 to genuinely helpful assistants like ChatGPT.

In one sentence

Reinforcement learning trains an agent through repeated trial, action, and reward rather than labeled examples, and while it has a genuine, celebrated track record in games and robotics, its most consequential recent application, covered throughout the rest of this phase, has been teaching language models to behave helpfully based on real human judgment.

Related Terms

• Machine Learning (ML)
• RLHF
• Reward Model
• Weights
• Gradient Descent