Where Does an LLM Store ‘Paris Is the Capital of France’?

Following one fact from a text file into model weights—and back into an answer

Suppose an LLM is trained on this tiny page:

France is a country in Western Europe. Paris is the capital of France. In other words, the capital of France is Paris. The city is known for the Eiffel Tower and the River Seine. France uses the euro, and French is its official language.

Later, a user types:

The capital of France

The model continues:

is Paris.

Where did Paris come from?

Did the model save the original page somewhere inside itself?

Did it convert the sentence into one vector and store that vector in a hidden database?

Is there a row that looks like this?

France.capital = Paris

Usually, no.

During pretraining, a language model repeatedly tries to predict missing future tokens. Its errors produce gradients. Those gradients make tiny changes across many numerical parameters called weights. After enormous numbers of such updates, patterns from the training data are distributed through those weights.

At inference time, the original training page is normally absent. The model uses its tokenizer, learned parameters and the user’s current prompt to produce new probabilities.

flowchart TD
    Page["France page in training corpus"] --> Tokens["Token IDs"]
    Tokens --> Training["Prediction, loss and updates"]
    Training --> Checkpoint["Learned weight tensors"]
    Prompt["The capital of France"] --> Loaded["Model loaded from checkpoint"]
    Checkpoint --> Loaded
    Loaded --> Answer["is Paris"]

That is the complete story in one diagram. But every arrow hides an important transformation. Let us open them one at a time.

---

First, a warning about our tiny example

A real LLM is not usefully trained from one page.

It learns language, facts, writing styles, code patterns and reasoning-like behaviors from very large collections of text and other data. The statement about Paris may appear in many forms:

Paris is the capital of France.
France's capital is Paris.
The French capital, Paris, ...
What is the capital of France? Paris.
Paris, the nation's capital, ...

Our one-page experiment is a teaching model. We will make its vocabulary, vectors and output layer extremely small so that we can inspect every number.

The mechanics—tokenization, forward pass, loss, backpropagation, weight update, checkpoint and inference—match the broad training process. The size and certainty do not.

One page shown once would make only a tiny contribution to a large model. If we repeat it until our toy model becomes highly confident, the toy can overfit or memorize that example. We will do a few repetitions only to make the direction of learning visible, not to claim this is how a production LLM gets all its knowledge.

---

Stage 1: the page exists as ordinary data

Before training begins, the France page may exist in a text file, dataset shard or object-storage system.

At this stage, it is training data, not model knowledge.

Storage before training
└── dataset shard
    └── "Paris is the capital of France ..."

A data pipeline may perform operations such as:

• collecting permitted source documents;
• extracting readable text;
• removing corrupt records;
• filtering unwanted content;
• detecting duplicates or near-duplicates;
• recording metadata;
• splitting long streams into training sequences;
• shuffling sequences into batches.

The model does not learn merely because a page was downloaded. Learning occurs only when tokenized examples participate in forward passes and their losses contribute to parameter updates.

---

Stage 2: text becomes tokens

Neural networks calculate with numbers. A tokenizer converts text into a sequence of tokens, then maps those tokens to integer IDs.

For illustration, pretend our tokenizer produces whole-word tokens:

Text token	Token ID
Paris	41
is	17
the	8
capital	93
of	12
France	55
.	4

The sentence:

Paris is the capital of France.

becomes:

[41, 17, 8, 93, 12, 55, 4]

Real tokenizers often use subwords or byte-based pieces, so words need not map one-to-one with tokens. France might be one token in one tokenizer and several pieces in another.

Is tokenization inside the model?

It depends on what boundary we are drawing.

In an application request flow, tokenization is commonly performed by software around the neural network. The Transformer receives token IDs or tensors, not raw Unicode text.

But the tokenizer is still part of the complete model package needed to use the LLM correctly. Its vocabulary and rules must match the weights used during training.

flowchart LR
    Text["Raw text"] --> Tokenizer["Tokenizer"]
    Tokenizer --> IDs["Token IDs"]
    IDs --> NeuralModel["Neural network"]

---

Stage 3: one sentence creates several learning opportunities

A causal language model learns by predicting the next token from earlier tokens.

From one sequence, it can learn several input–target relationships:

Input available at a position	Target token
Paris	is
Paris is	the
Paris is the	capital
Paris is the capital	of
Paris is the capital of	France
Paris is the capital of France	.

The page also expresses the same fact using the reverse wording:

The capital of France is Paris.

One useful position is:

input context: The capital of France is
target token:  Paris

In real causal-LM training, many positions in a sequence are evaluated in parallel using a causal mask. The model is not necessarily called separately for every row shown above. The table expresses the prediction task, while tensor operations perform many of those tasks together.

This training method is often called teacher forcing: the model receives the real earlier tokens from the training sequence, even if its own prediction at a previous position would have been wrong.

---

Stage 4: token IDs retrieve embedding vectors

An integer such as token ID 55 is only an address. It has no useful magnitude: 55 is not more French than 41.

The model contains an embedding matrix:

$$E\in\mathbb{R}^{vocabulary\ size\times model\ width}$$

If the vocabulary has 50,000 tokens and the model width is 768:

embedding matrix shape = [50,000, 768]

Token ID 55 selects row 55. That row is a learned vector for the token.

Our toy vectors might look like:

The     → [ 0.2,  0.1, -0.3]
capital → [ 0.7, -0.2,  0.5]
of      → [-0.1,  0.4,  0.2]
France  → [ 0.9,  0.3,  0.6]
is      → [ 0.1,  0.5, -0.2]

These values are parameters learned during training. They are not dictionary definitions written in decimal form.

The model also needs position information so that:

Paris is the capital of France

is distinguishable from a different ordering of the same tokens.

The resulting sequence of vectors enters the Transformer blocks.

---

Stage 5: context is transformed through the network

Inside each Transformer block, attention allows token positions to gather relevant information from allowed earlier positions. Feed-forward networks transform each position further. Residual connections and normalization help information travel through many layers.

flowchart TD
    Emb["Token and position representations"] --> Attn["Causal attention"]
    Attn --> FFN["Feed-forward transformation"]
    FFN --> More["More Transformer blocks"]
    More --> Hidden["Final contextual vector"]

For the final position in:

The capital of France is

suppose our tiny model produces this final contextual vector:

$$h=[0.8,-0.4,0.6]$$

We are deliberately treating the earlier embedding and Transformer operations as a black box in this numerical calculation. We will fully calculate what happens after this vector: logits, probabilities, loss, gradients and one optimizer update. A real training run also sends gradients backward through the Transformer operations that produced this vector, so those earlier weights and relevant embedding rows can learn too.

This vector is not a human-readable sentence or a database record. It is the current numerical representation of what the network needs for predicting the next token at this position.

In a real model, $h$ would contain far more than three values and would depend on all the learned operations in earlier layers.

---

Stage 6: the output layer scores every possible next token

The model must convert the final hidden vector into one score per vocabulary token.

It uses an output weight matrix:

$$W_{out}\in\mathbb{R}^{model\ width\times vocabulary\ size}$$

Our teaching vocabulary has only three candidate tokens:

Paris, Lyon, Berlin

Where did the initial weights come from?

Before the model reads our France page, its trainable parameters already exist as numbers. At the beginning of training, most weight matrices are initialized using carefully chosen random values. They are usually small enough to keep signals stable, but not all identical. If every weight began with exactly the same value, many neurons could behave identically and learn the same thing.

This distinction matters:

The France page does not directly calculate the final weights.

Initialization creates starting numbers.
The forward pass uses those numbers to make a prediction.
The loss measures the prediction error.
Backpropagation calculates how each involved number affected that error.
The optimizer changes the numbers slightly.
Repeated updates turn starting weights into learned weights.

Our small matrix below is therefore a set of illustrative starting weights. We chose simple values so that every calculation remains visible. In a real model, an initialization method creates millions or billions of starting values before the first training batch arrives.

Use this tiny output matrix:

$$W_{out}= \begin{bmatrix} 0.2 & 0.1 & -0.1\\ 0.1 & 0.2 & 0.1\\ -0.1 & 0.1 & 0.2 \end{bmatrix}$$

The columns correspond to:

column 1 → Paris
column 2 → Lyon
column 3 → Berlin

Biases begin at zero:

$$b=[0,0,0]$$

We show the input embedding matrix and output matrix as separate structures. That is true for some language models. Other architectures use weight tying, where the input embedding table and the final output projection share parameters. The forward-pass idea remains the same: the model must turn its final contextual representation into one score for every candidate token.

Calculate the logits:

$$o=hW_{out}+b$$

Score for Paris

$$(0.8\times0.2)+(-0.4\times0.1)+(0.6\times-0.1)$$ $$=0.16-0.04-0.06=0.06$$

Score for Lyon

$$(0.8\times0.1)+(-0.4\times0.2)+(0.6\times0.1)$$ $$=0.08-0.08+0.06=0.06$$

Score for Berlin

$$(0.8\times-0.1)+(-0.4\times0.1)+(0.6\times0.2)$$ $$=-0.08-0.04+0.12=0$$

The logits are:

$$[0.06,0.06,0.00]$$

Softmax produces probabilities:

Candidate	Probability before learning from this example
Paris	33.99%
Lyon	33.99%
Berlin	32.01%

The model is almost equally uncertain between the three candidates.

---

Stage 7: the correct token creates a loss

The training data tells us the correct next token is Paris.

Represent the target as:

$$y=[1,0,0]$$

Cross-entropy loss for the correct token is:

$$L=-\ln(P(Paris))$$ $$=-\ln(0.3399)\approx1.079$$

The loss says the model needs correction. It does not directly say which weights must change.

Backpropagation answers that question.

---

Stage 8: backpropagation assigns responsibility

For softmax followed by cross-entropy, the gradient with respect to the logits is:

$$g_o=p-y$$ $$=[0.3399,0.3399,0.3201]-[1,0,0]$$ $$\boxed{g_o=[-0.6601,0.3399,0.3201]}$$

Interpret the signs:

• the negative Paris gradient means raising its logit would reduce loss;
• the positive Lyon and Berlin gradients mean raising those logits would increase loss.

The output-weight gradient is:

$$\frac{\partial L}{\partial W_{out}}=h^Tg_o$$ $$= \begin{bmatrix} 0.8\\ -0.4\\ 0.6 \end{bmatrix} [-0.6601,0.3399,0.3201]$$ $$\boxed{ \frac{\partial L}{\partial W_{out}}= \begin{bmatrix} -0.5281 & 0.2719 & 0.2561\\ 0.2640 & -0.1360 & -0.1281\\ -0.3960 & 0.2040 & 0.1921 \end{bmatrix}}$$

The bias gradient is:

$$\frac{\partial L}{\partial b}=g_o$$

Backpropagation would not stop here in a real LLM. The gradient would continue through the output projection, Transformer blocks, attention matrices, feed-forward weights and embedding rows that contributed to the prediction.

Our numerical example freezes the hidden vector so that we can focus on one visible output-layer update. This is a simplification, not a claim that only the output layer learns facts.

---

Stage 9: the optimizer updates the weights

Use basic gradient descent with learning rate:

$$\eta=0.1$$

For every parameter:

$$\theta_{new}=\theta_{old}-\eta\nabla_\theta L$$

Take the first weight in the Paris column:

old weight =  0.2000
gradient   = -0.5281

$$0.2-(0.1\times-0.5281)=0.2528$$

Take the first weight in the Lyon column:

old weight = 0.1000
gradient   = 0.2719

$$0.1-(0.1\times0.2719)=0.0728$$

After updating the complete output matrix:

$$W_{out}^{new}\approx \begin{bmatrix} 0.2528 & 0.0728 & -0.1256\\ 0.0736 & 0.2136 & 0.1128\\ -0.0604 & 0.0796 & 0.1808 \end{bmatrix}$$

The new biases are:

$$b^{new}\approx[0.0660,-0.0340,-0.0320]$$

These numbers—not the sentence itself—are what changed inside this small trainable layer.

---

Stage 10: run the same context again

Using the same hidden vector and updated output parameters, the new logits are:

$$[0.2026,-0.0134,-0.0691]$$

Softmax now gives:

Candidate	Before update	After one update
Paris	33.99%	38.94%
Lyon	33.99%	31.38%
Berlin	32.01%	29.68%

The loss falls:

$$1.079\rightarrow0.943$$

One example caused one small movement in the desired direction.

If the model encounters this relationship in varied, compatible contexts, their gradient contributions can reinforce a reusable pattern. If it sees conflicting or noisy statements, gradients may compete. Training is the combined effect of enormous numbers of examples, not a single assignment.

---

Where exactly is the fact stored now?

This question has two answers: a physical answer and a representational answer.

Physical answer: parameters are stored as tensors

The model’s weights are arrays of numbers called tensors. A training checkpoint may contain files storing:

• embedding weights;
• attention projection weights;
• feed-forward weights;
• normalization parameters;
• output projection weights;
• sometimes optimizer state and training metadata.

On persistent storage, these tensors live in checkpoint files on SSDs or object storage. When the model runs, the required weights are loaded into machine memory—often accelerator memory such as GPU HBM.

flowchart LR
    Checkpoint["Checkpoint files on storage"] --> RAM["Host memory"]
    RAM --> VRAM["GPU accelerator memory"]
    VRAM --> Math["Matrix operations"]

The exact file format and loading path depend on the framework and deployment system.

Representational answer: the fact is distributed

There is usually no single “France-to-Paris weight.”

Producing Paris can depend on a distributed combination of:

• token and subword embeddings;
• attention projections that connect relevant context;
• feed-forward transformations;
• normalization and residual pathways;
• the output direction associated with Paris;
• patterns shared with countries, capitals, geography and sentence forms.

Change one weight and the fact will not normally disappear cleanly. The same weight may participate in many unrelated predictions. The same relationship may be supported by many parameters and training examples.

So “the model stores the fact in its weights” is broadly correct, but it does not mean the weights form a readable fact table.

---

Are facts stored as vectors?

This question needs careful wording.

Many things inside an LLM are vectors or matrices:

• an embedding row is a stored parameter vector;
• a token’s hidden state is a temporary vector;
• a Key or Value in attention is a temporary vector derived from parameters and the current input;
• model weights are stored as tensors, including matrices and higher-dimensional arrays.

But saying “the sentence is stored as one vector” is usually misleading.

The page affects many parameter tensors through training. At inference time, the prompt creates temporary activations that interact with those parameters. The answer emerges from this computation.

Item	Persistent?	What it represents
Training page	Separate during training	Source text
Tokenizer vocabulary	Yes, as model-package data	Mapping between token pieces and IDs
Model weights	Yes	Learned numerical parameters
Hidden-state vector	No, normally request-time	Current contextual representation
Attention KV cache	No, normally request-time	Reusable Keys and Values for current generation
Output probabilities	No	Distribution for one prediction step

---

Saving a checkpoint is not saving the training corpus

After training, the system writes the current parameters to a checkpoint.

Conceptually:

checkpoint = {
    "embedding_weights": embedding_matrix,
    "attention_weights": attention_parameters,
    "feed_forward_weights": ffn_parameters,
    "output_weights": output_matrix,
    "normalization_parameters": norm_parameters,
}

The original France page is part of the training corpus, not normally copied verbatim into this checkpoint as a searchable document.

However, models can sometimes memorize and reproduce training passages, especially when data is duplicated, distinctive or overemphasized. Therefore, “the corpus is not stored as a document database” does not guarantee that no training text can ever be reconstructed or regurgitated.

Distributed learning and memorization can coexist.

---

Training storage, optimizer storage and inference storage differ

During training, memory may hold more than the model weights:

Stored item	Needed for training?	Usually needed for basic inference?
Model parameters	Yes	Yes
Gradients	Yes	No
Optimizer state	Yes	No
Saved forward activations	Yes, for backpropagation	Not in the same way
Training batches	Yes	No
KV cache	Sometimes, depending on operation	Yes during autoregressive generation

An optimizer such as Adam may keep additional running statistics for each trainable parameter. This is one reason a training checkpoint can require much more storage than the raw deployed weights.

Inference servers often load only what they need to make predictions.

---

Now the user asks: “The capital of France”

Training is over. Gradients and optimizer updates are disabled for this normal request.

The prompt begins as raw text:

The capital of France

Step 1: tokenize the prompt

Our teaching tokenizer produces:

[The, capital, of, France]

and then token IDs.

Step 2: retrieve embeddings

Each ID selects an embedding vector. Position information is included so the model knows the order.

Step 3: run the Transformer forward pass

Attention lets later positions use relevant earlier context. Feed-forward layers transform the representations. Learned parameters influence every step.

Step 4: calculate next-token logits

The model first predicts what follows the exact prompt.

Because the user stopped at France, a natural next token may be is:

The capital of France → is

The model does not jump directly to Paris unless its token probabilities and prompt form make that the immediate next token.

Step 5: append the selected token

The sequence becomes:

The capital of France is

Step 6: perform the next decoding step

For this context, the learned transformations produce a hidden representation. The output layer creates logits for the vocabulary. Paris receives a strong score relative to alternatives.

The capital of France is → Paris

Step 7: continue until stopping

The model may next produce punctuation:

The capital of France is Paris.

flowchart TD
    Prompt["The capital of France"] --> Is["Predict: is"]
    Is --> Extended["The capital of France is"]
    Extended --> Paris["Predict: Paris"]
    Paris --> Period["Predict punctuation"]

Every generated token requires another probability distribution. Generation is iterative, not a single database response.

---

Is the model searching its weights?

“Searching” is an intuitive word, but it can suggest the wrong mechanism.

A database search might:

find row where country = France
return capital column

A standard forward pass does something different:

token IDs
→ embeddings
→ attention and feed-forward transformations
→ final hidden state
→ vocabulary logits
→ probabilities
→ selected token

The model applies all relevant numerical transformations. It does not iterate through its weights looking for a matching sentence.

The weights shape the computation itself.

---

How does Paris beat Lyon if both are French cities?

Lyon may receive non-zero probability because it is a plausible city token in a France-related context. The model has learned many statistical relationships, not a rule that only exact training sentences may continue.

The prompt contains a stronger pattern:

capital + of + France + is

Across layers, the current hidden state represents that complete context—not just the last word is. The output projection measures how that state aligns with learned vocabulary directions.

Possible probabilities might be:

Next token	Probability
Paris	94%
Lyon	2%
France	1%
All other tokens combined	3%

These are illustrative. A real model’s distribution depends on its tokenizer, weights, prompt, decoding settings and surrounding context.

The model did not need to see the false sentence “The capital of France is Lyon” to assign Lyon a small probability. Softmax distributes probability across the entire vocabulary according to the logits. Related or generally plausible tokens may receive some mass even when the target relationship points strongly to Paris.

---

The complete toy training loop in Python

This code reproduces our visible output-layer training example. It keeps the hidden vector fixed and repeatedly trains on Paris as the target.

import numpy as np


# Contextual representation for: "The capital of France is"
hidden = np.array([0.8, -0.4, 0.6])

# Columns correspond to: Paris, Lyon, Berlin
output_weights = np.array([
    [0.2, 0.1, -0.1],
    [0.1, 0.2,  0.1],
    [-0.1, 0.1, 0.2],
])
output_bias = np.array([0.0, 0.0, 0.0])

target = np.array([1.0, 0.0, 0.0])
learning_rate = 0.1


def softmax(logits):
    shifted = logits - np.max(logits)
    exponentials = np.exp(shifted)
    return exponentials / exponentials.sum()


def forward(hidden, weights, bias):
    logits = hidden @ weights + bias
    probabilities = softmax(logits)
    loss = -np.log(probabilities[0])
    return logits, probabilities, loss


for step in range(101):
    logits, probabilities, loss = forward(
        hidden,
        output_weights,
        output_bias,
    )

    if step in {0, 1, 2, 5, 10, 20, 50, 100}:
        print(
            f"step={step:3d}  "
            f"P(Paris)={probabilities[0]:.4f}  "
            f"loss={loss:.4f}"
        )

    # Backpropagation for softmax + cross-entropy.
    gradient_logits = probabilities - target
    gradient_weights = np.outer(hidden, gradient_logits)
    gradient_bias = gradient_logits

    # Gradient-descent update.
    output_weights -= learning_rate * gradient_weights
    output_bias -= learning_rate * gradient_bias

Expected output:

step=  0  P(Paris)=0.3399  loss=1.0790
step=  1  P(Paris)=0.3894  loss=0.9430
step=  2  P(Paris)=0.4374  loss=0.8270
step=  5  P(Paris)=0.5627  loss=0.5749
step= 10  P(Paris)=0.7027  loss=0.3529
step= 20  P(Paris)=0.8321  loss=0.1838
step= 50  P(Paris)=0.9324  loss=0.0699
step=100  P(Paris)=0.9671  loss=0.0335

The model becomes increasingly confident because we repeat the same hidden context and target.

That is also the limitation of this demonstration. It shows parameter updates, but not broad learning. A production model must learn from varied contexts and retain useful performance across many tasks. Repeating one example 100 times can create memorization and damage other behavior.

---

What changes in a real training run?

Our toy example updated only a 3×3 output matrix and three biases.

A real training step may update parameters across:

• a vocabulary embedding table;
• Query, Key, Value and output projections in many attention layers;
• feed-forward or mixture-of-experts layers;
• normalization components;
• the final vocabulary projection.

Many sequences are grouped into batches. Losses from many token positions contribute gradients. Distributed training may split computation and parameter storage across many accelerators.

The useful pattern emerges from repeated, overlapping evidence:

France ↔ Paris
country ↔ capital
capital-of questions ↔ city answers
geography wording ↔ related continuations

No single gradient update fully explains the final behavior.

---

What if the model has never seen the exact sentence?

It may still answer correctly if training connected the underlying patterns through other phrasings.

For example, it may have seen:

Paris, France's capital, hosted ...
The French government is based in Paris ...
France — Capital: Paris

The model can use shared representations and contextual patterns to respond to:

Which city is France's capital?

This is closer to generalization than exact string matching.

But we should not turn that into a guarantee. LLMs can produce wrong or outdated answers because next-token prediction generates plausible text; it does not perform a mandatory fact verification step.

---

How this differs from RAG

In ordinary parameter-based generation:

prompt + learned weights → answer

In retrieval-augmented generation:

user question
→ search an external knowledge source
→ retrieve a France passage
→ place it in the prompt
→ model generates using retrieved evidence and learned weights

RAG keeps the page outside the model as retrievable data. The model can receive the current page during the request.

This makes RAG useful for private, frequently changing or source-cited knowledge. Updating a document index can be easier than retraining model weights.

Parameter memory	RAG retrieval
Knowledge influences learned weights	Documents remain externally stored
No guaranteed source record	Retrieved passage can be cited
Updating may require more training	Update the external knowledge base
Fast access through normal forward pass	Adds retrieval latency and failure modes

RAG does not replace model knowledge. The model still needs learned language and reasoning capabilities to understand the retrieved passage and formulate an answer.

---

Common misunderstandings

“The model stores every page as a hidden text file”

Training pages influence numerical parameters. They are not normally preserved inside the checkpoint as a browsable document collection.

“The fact must be stored in one weight”

Knowledge is typically distributed across many parameters and computations.

“Token embeddings contain all facts about a word”

An embedding row is only the starting representation. Contextual behavior also depends on attention, feed-forward layers and other parameters.

“A weight is calculated once from one sentence”

Weights receive many updates from many batches. Each gradient makes a small contribution to their final values.

“The tokenizer learns the fact”

The tokenizer converts text to token IDs. The neural-network parameters learn predictive patterns.

“If the page was in training, the model will answer correctly”

Not necessarily. Exposure strength, wording, conflicts, capacity, optimization and later training all influence recall.

“Non-zero probability for Lyon proves a false sentence was in training”

No. Softmax assigns probability across all vocabulary tokens based on current logits. A token can receive probability without the exact false sentence ever appearing.

“The KV cache contains permanent learned knowledge”

KV cache entries are temporary request-time activations used to avoid repeated computation. They are not trained model weights.

“The model updates itself when I ask a question”

A normal inference request runs forward passes. It does not ordinarily perform backpropagation or change the base model’s parameters.

---

The one idea to remember

The France page exists in two very different forms at two different times.

During training:

page on storage
→ tokens
→ token IDs
→ vectors
→ next-token predictions
→ loss
→ gradients
→ changed weights
→ saved checkpoint

During inference:

new prompt
+ tokenizer
+ learned checkpoint weights
→ forward pass
→ next-token probabilities
→ "is"
→ next forward pass
→ "Paris"

The original page is normally not opened during that inference request.

The model does not retrieve a row called France.capital. Its learned tensors shape the transformations that make Paris a strong continuation after:

The capital of France is

So where does an LLM store the fact?

The most accurate beginner answer is:

The training text changes many numerical parameters. The resulting knowledge is distributed through those stored weight tensors and becomes visible only when a new prompt activates the learned pattern during a forward pass.

That is less tidy than a database row—but it is much closer to how a language model actually works.

---

Sources and further reading

• Vaswani et al.: Attention Is All You Need
• PyTorch: Embedding
• PyTorch: Saving and Loading Models
• PyTorch: CrossEntropyLoss
• Hugging Face: Causal Language Modeling
• Petroni et al.: Language Models as Knowledge Bases?