Annotation

Every article so far in this phase has referenced Ground Truth and Labels as if they simply exist, ready and waiting, once a dataset is assembled. They don’t. Someone — usually a person — has to actually sit down and mark up the raw data with the correct answer, one example at a time, before any of it can be used. That marking-up work is called annotation.

The simple definition

Annotation is the process of adding meaningful tags, labels, or markings to raw data so it can be used to train a supervised Machine Learning model. A photo of a dog isn’t useful as training data for an image classifier until someone marks it “dog.” A sentence isn’t useful for training a sentiment model until someone marks it “positive” or “negative.” Annotation is that marking-up act — the practical, hands-on bridge between raw data and the labeled training data described in the Training Data article.

Adding meaning to raw data

Suppose we have a street photograph. The raw image contains pixels, but a computer does not automatically know which pixels belong to a car or pedestrian.

An annotator may add:

Image
├── box around car → label: car
├── box around person → label: pedestrian
└── box around traffic light → label: traffic_light

The boxes, categories, and other added information are annotations.

Annotation depends on the task

Data	Possible annotation
Photograph	Category, bounding box, or pixel mask
Sentence	Sentiment or named entities
Audio	Transcript and speaker timestamps
Video	Object tracks across frames
LLM response	Correctness, safety, relevance, or preference judgment

One item can contain several annotations. A sentence may have a sentiment label plus marked names and locations.

A text-annotation example

Sentence:

Anita works at OpenAI in San Francisco.

Annotated entities:

Anita         → PERSON
OpenAI        → ORGANIZATION
San Francisco → LOCATION

These annotations can teach a model to identify similar entities in new sentences.

The annotation workflow

flowchart LR
    A[Define task] --> B[Write guidelines]
    B --> C[Annotators label examples]
    C --> D[Measure agreement]
    D --> E[Review disagreements]
    E --> F[Approve and version dataset]

Good guidelines explain ambiguous cases with examples. Without them, two careful people may apply different interpretations.

Quality controls

• Train annotators with practice examples.
• Include known quality-check examples.
• Assign difficult items to multiple reviewers.
• Measure agreement instead of assuming consistency.
• Use expert adjudication for high-impact domains.
• Record uncertainty rather than forcing false certainty.
• Protect annotators from private, disturbing, or harmful content.

Why this deserves its own article, distinct from Ground Truth and Label

You’ve already met the concept of a correct answer (Ground Truth) and the technical form it takes inside a dataset (Label). Annotation is neither of those things directly — it’s the process that actually produces them. Think of it as the verb where the other two are nouns: annotation is the act of doing the work; the label is what results from having done it.

What annotation actually looks like, depending on the data

The specific work involved varies a lot depending on the type of data and task:

• Image annotation. Drawing a box around every car in a photo (called bounding box annotation, common in self-driving car systems), or tracing the precise outline of an object pixel by pixel (segmentation), or simply tagging a whole photo with a single category.
• Text annotation. Marking a sentence’s sentiment, tagging named entities like people and places within a paragraph, or — increasingly common with large language models — comparing two AI-generated responses and marking which one is better, a task directly tied to the human preference data introduced in the Ground Truth article.
• Audio annotation. Transcribing spoken words into text, or marking where one speaker stops and another starts in a recorded conversation.

flowchart LR
    A[Raw, unlabeled data] --> B[Human annotator reviews it]
    B --> C[Applies label/markup based on guidelines]
    C --> D[Labeled data, ready for training]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a museum receiving a huge collection of unsorted artifacts with no descriptions attached. Before the collection is useful to researchers, someone has to go through each item, research it, and attach an accurate description — this piece is Roman pottery from 200 AD, that one is a medieval coin. Annotation is exactly this cataloguing work, applied to training data instead of artifacts.

Where this breaks down: A museum curator brings deep specialized expertise to each individual item, often spending significant time researching a single piece. Most large-scale ML annotation, by contrast, involves many annotators working through enormous volumes of relatively quick, standardized judgments, guided by detailed written instructions rather than deep individual expertise — a trade-off between depth and scale that shapes a lot of the quality issues covered later in this article.

A real, substantial industry behind this work

This isn’t a niche or minor part of the AI pipeline — it’s a genuine, large-scale industry.

Companies like Scale AI and Surge AI specialize specifically in providing annotation and RLHF (reinforcement learning from human feedback) services to major AI labs; Surge AI’s publicly listed customers include OpenAI, Microsoft, Meta, and Anthropic, and the broader data annotation market was valued at roughly $11.8 billion in 2022 by one industry report, with continued strong growth projected since.

Meta made a significant investment in Scale AI in 2025. Meta’s subsequent Form 10-K filing records the minority investment at $13.80 billion, while Scale AI’s announcement confirms the transaction without presenting it as a full acquisition. This is different from Scale AI’s company valuation, so those two numbers should not be treated as interchangeable. The investment nevertheless shows how strategically important data and human feedback became to frontier-model development.

Where annotation quality actually breaks down

Annotation quality problems are one of the more common, underappreciated sources of trouble in real ML projects:

• Inconsistent guidelines. If instructions for annotators are ambiguous — “mark this review as positive or negative” doesn’t specify what to do with a genuinely mixed review — different annotators will make different calls on similar examples, introducing noise directly into the ground truth.
• Annotator disagreement. Even with clear guidelines, reasonable people can disagree, especially on subjective tasks like judging which of two AI responses is “better.” This is exactly why the Ground Truth article recommended collecting judgments from multiple independent annotators rather than trusting a single person’s call.
• Fatigue and scale pressure. Annotators working through thousands of repetitive examples can make more mistakes later in a long session than earlier — a genuine, measurable quality concern in large-scale annotation projects, and part of why quality-control processes (like having a second annotator review a sample of the first annotator’s work) are standard practice at serious annotation operations.

A concrete example, layered

For a simple beginner example: annotating a set of fruit photos means someone looking at each photo and typing “ripe” or “unripe.” For a more advanced, production-scale example: building a self-driving car’s perception system requires annotators to draw precise bounding boxes around every pedestrian, vehicle, and traffic sign across millions of frames of driving footage — exactly the kind of large-scale visual annotation work companies like Scale AI built their early business around.

Key terms

• Annotation: Information added to raw data.
• Bounding box: Rectangle locating an object in an image.
• Segmentation mask: Pixel-level marking of an object or region.
• Guideline: Rules and examples defining how to annotate.
• Agreement: Consistency among independent annotators.

Check your understanding

Is annotation only attaching one category? No. It can include locations, spans, masks, transcripts, rankings, and explanations.

Does human annotation guarantee ground truth? No. Humans can disagree or make mistakes, so quality processes matter.

Common misconception

People sometimes imagine annotation as a fully automated, invisible process — that a company simply “has data” ready to train on. In reality, for most supervised learning tasks, that labeled data represents a substantial amount of deliberate human labor, often outsourced to large annotation workforces, governed by detailed guidelines, and subject to real quality-control challenges. The apparent “magic” of a well-trained AI model often rests on a large amount of unglamorous, careful human annotation work happening well before any training ever begins.

Keep annotation decisions reproducible

An annotation project needs written guidelines, examples of difficult boundary cases, agreement checks between annotators, reviewer decisions, and version history. Without these records, two teams may attach different meanings to the same data while believing they created one consistent ground truth.

Annotation is the broader act of adding meaning. Labeling is the act of assigning the target category or value. A reviewed annotation may become accepted ground truth, but those terms describe different roles in the process.

Where this fits in what comes next

Annotation is the general process; the next article, Labeling, looks specifically at the practical mechanics and specific techniques of assigning labels within that broader annotation process — including approaches like crowdsourcing and semi-automated labeling that help this work scale to the enormous datasets modern AI systems require.

In one sentence

Annotation is the deliberate, often human-powered process of marking up raw data with correct answers, and it’s the real, substantial, and imperfect labor that quietly makes every supervised Machine Learning system’s “learning” possible in the first place.

Related Terms

• Ground Truth
• Label
• Training Data
• Labeling
• Dataset