The Engineering Behind Large Language Models: Architecture, Training, and Decoding

Large Language Models (LLMs) such as ChatGPT have fundamentally transformed modern software systems. While prompting LLMs has become common practice, understanding the underlying systems architecture—spanning tokenization, neural network layers, reinforcement learning, decoding strategies, and grounding—is essential for any engineer building scalable AI-powered products.

---

1. What is an LLM at Its Core?

At its mathematical foundation, a large language model is an autoregressive next-token predictor. Given a context sequence of tokens, the model computes a categorical probability distribution over its entire vocabulary to predict the single most plausible subsequent token.

$$P(w_t \mid w_1, w_2, \dots, w_{t-1})$$

LLMs do not conceptualize entire sentences or paragraphs as monolithic entities. Instead, generation is an iterative fill-in-the-blank operation executed one token at a time at the tail end of the sequence.

Input:  "Hi, my name is Ram" 
Step 1: Predict -> "sree" (Context becomes "Hi, my name is Ram sree")
Step 2: Predict -> "."    (Context becomes "Hi, my name is Ram sree.")

The emergence of complex reasoning, translation, and code generation is a byproduct of scale: optimizing next-token cross-entropy loss over billions of text tokens scraped across the internet forces the neural network to encode syntax, semantic facts, world knowledge, and logic directly into its parameter weights.

---

2. Tokenization: Subwords and Byte Pair Encoding (BPE)

Human language is arbitrary and practically infinite due to compound words, names, and neologisms. Storing every word in a static dictionary would result in unbounded vocabulary sizes and high out-of-vocabulary (OOV) error rates.

Subwords and Fixed Vocabularies

Instead of whole words or raw characters, LLMs tokenize input into subword units (typically 20,000 to 50,000 tokens in size):

• Portmanteaus such as “Chizza” split naturally into subwords: ["Ch", "izza"] or ["chicken", "pizza"].
• Unseen proper nouns like “Ramsri” break down into subwords that already exist in the vocabulary: ["Ram", "sri"].

Byte Pair Encoding (BPE)

Byte Pair Encoding is a data compression algorithm repurposed for tokenization:

1. Initialize vocabulary with base tokens (individual bytes/characters).
2. Count all adjacent token pairs in a training corpus.
3. Iteratively merge the most frequently occurring pair into a single new subword token.
4. Repeat until reaching the target vocabulary limit $V$.

Original Corpus: "low lower newest widest"
Initial Units:   'l', 'o', 'w', 'e', 'r', 'n', 's', 't', 'i', 'd'
Frequent pair:   ('l', 'o') -> 'lo'
Frequent pair:   ('lo', 'w') -> 'low'
New Token:       'low'

The Multilingual Latency and Cost Tax

Because base tokenizers for models like GPT-3 were predominantly trained on English corpora, English words typically map to single tokens or clean 2-token pairs.

In contrast, non-Latin scripts (e.g., Devanagari, Telugu, Mandarin) often cannot map to pre-existing subwords. The tokenizer is forced to decompose a single grapheme into raw UTF-8 byte sequences. Generating a single Hindi character might require 4 to 6 byte-tokens. As a result:

• Context Window Exhaustion: Non-English text fills the context window up to $15\times$ faster.
• Inference Latency: Because generation is autoregressive ($1\text{ token} = 1\text{ forward pass}$), generation latency increases proportionally.
• Economic Cost: APIs billing per token impose an implicit cost penalty on non-English workloads.

---

3. Neural Architecture and Parameters

When a model is described as having 40 billion parameters, those parameters are the scalar weights ($W$) and biases ($b$) across every artificial neuron in the transformer layers.

flowchart LR
    In[Input Tokens] --> Emb[Embedding Layer] 
    Emb --> L1[Layer 1: Linear Transform + Non-Linear Activation]
    L1 --> L2[Layer 2: Hidden Expansion / Attention]
    L2 --> Ln[Layer N: Linear Contraction]
    Ln --> Logits[Output Logits: Vocab Size V]
    Logits --> Softmax[Softmax Probability Distribution]

Inside the Neuron

A single neuron performs a weighted sum followed by a non-linear activation:

$$z = \sum_{i=1}^{n} w_i x_i + b$$ $$a = \sigma(z) \quad \text{or} \quad \text{GELU}(z)$$

• Non-Linearity: If every transformation were linear, an $N$-layer deep neural network would mathematically collapse into a single matrix multiplication ($W_2(W_1 x) = W_{combined} x$). Non-linear activation functions (like ReLU or GELU) allow the network to model complex high-dimensional manifolds.
• Dimension Expansion and Contraction: Transformers project input embeddings across wider internal projection layers (e.g., expanding from hidden dimension $d_{model} = 4096$ up to feed-forward dimension $d_{ff} = 16384$) before compressing them back to output logits across vocabulary size $|V|$.

During pre-training, backpropagation computes the gradient of the loss with respect to all parameters via gradient descent, adjusting these weights to minimize prediction error.

---

4. The 3-Stage Training Pipeline: From Pre-Training to RLHF

Pre-training produces a raw base model—an “athlete” with raw agility and strength. However, an undirected base model will simply complete text rather than follow instructions (e.g., prompted with “Write a poem about dogs”, it might continue with “Write a poem about cats” instead of writing the poem).

Converting an undirected model into an instruction-following assistant involves a rigorous multi-stage pipeline.

flowchart TD
    A[Unsupervised Web Corpus] -->|Self-Supervised Next-Token Loss| B(Pre-Trained Base LLM)
    B -->|Instruction Prompt-Response Pairs| C(Supervised Fine-Tuning: SFT)
    C --> D(SFT Policy Model)
    D -->|Generates Multiple Outputs| E[Human Labeler Rankings]
    E -->|Train| F(Reward Model: The Coach)
    D --> G[PPO / Reinforcement Learning]
    F -->|Reward Signal Scores 1-10| G
    G --> H(Instruction-Aligned Model: ChatGPT)

Stage 1: Self-Supervised Pre-Training

• Objective: Predict next tokens across hundreds of billions of web pages, books, and code repositories.
• Advantage: Requires zero manual labeling. The raw text supplies its own labels ($x_t$ is the target for $x_{1...t-1}$).
• Result: Encodes broad linguistic and factual representations, but suffers from bias, hallucinations, and conversational incoherence.

Stage 2: Supervised Fine-Tuning (SFT)

• Objective: Warm-up the model on curated instruction-output pairs (Instruction, Ideal Output).
• Process: Human labelers author high-quality responses for tasks such as translation, copywriting, and code generation.
• Result: The model learns the conversational syntax and responds directly to imperatives.

Stage 3: Reinforcement Learning from Human Feedback (RLHF)

To scale beyond human labeling throughput and fine-tune nuanced objectives (truthfulness, safety, tone), RLHF separates generation from evaluation.

1. Train the Reward Model (“The Coach”):
   • The SFT model generates multiple candidate responses for a given prompt.
   • Human evaluators rank responses from best to worst based on helpfulness, accuracy, and safety.
   • A separate neural network (the Reward Model) is trained on this comparative preference data to assign a scalar reward value (e.g., 1 to 10) to any generated text.
2. Optimize Policy via PPO (Proximal Policy Optimization):
   • The generative LLM acts as the actor/agent.
   • For any prompt, the LLM generates an output; the Reward Model scores it.
   • If an output scores poorly, policy gradient descent updates the generative model’s weights to shift probability mass away from that trajectory.
   • Guardrails: The reward model penalizes hate speech, dangerous instructions (e.g., chemical synthesis), and encourages the model to acknowledge uncertainty rather than hallucinate.

---

5. Decoding Strategies: Controlling Creativity vs. Determinism

Once the model outputs a probability distribution over the vocabulary for the next token, how should that token be selected? Picking purely deterministically creates repetitive and sterile text, while unbounded sampling yields incoherence.

Vocabulary Logits: ["The": 0.45, "A": 0.25, "One": 0.15, "Elephant": 0.0001, ...]

Decoding Strategy	Mechanism	Trade-Offs
Greedy Search	Always select the token with maximum probability: $\text{argmax}_w P(w \mid \text{context})$.	Highly deterministic; frequently falls into infinite repetitive loops; lacks creative variation.
Temperature Scaling	Rescale logits prior to softmax: $P(w_i) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$.	Low $T$ ($< 0.3$) concentrates probability on top tokens. High $T$ ($> 1.0$) flattens the distribution, increasing randomness and risk of nonsensical outputs.
Top-K Sampling	Sort tokens by probability and restrict selection exclusively to the top $K$ candidates (e.g., $K=40$).	Eliminates the long-tail of improbable tokens while preserving controlled randomness.
Top-P (Nucleus) Sampling	Dynamically select the smallest set of top tokens whose cumulative probability exceeds threshold $P$ (e.g., $P=0.90$).	Adapts dynamically to certainty: if one token has 95% probability, only that token is considered; if distribution is flat, candidate pool expands.
Beam Search	Track the top $B$ paths with highest cumulative joint probability: $\prod_{t=1}^N P(w_t \mid w_{<t})$.	Optimizes global sequence likelihood over local greediness. Excellent for translation and summarization, but computationally intensive.

Engineering generative products requires selecting the right decoding regime:

• Deterministic Tasks (Code, Extraction, SQL): Low temperature ($T \approx 0.0$), Greedy or Top-P ($P=0.95$, $K$ small).
• Creative Tasks (Brainstorming, Copywriting): Moderate temperature ($T \approx 0.7 - 0.9$), combined with Nucleus sampling.

---

6. Eliminating Hallucinations: Retrieval-Augmented Generation (RAG)

An LLM stores knowledge in parametric memory—the fixed static weights established during training. Parametric memory has three critical weaknesses:

1. Knowledge Cutoffs: It cannot answer questions about events occurring after the training run.
2. Private Data Gaps: It has no access to proprietary company documentation, databases, or personalized user context.
3. Hallucination: When parametric probabilities degrade, the model invents plausible-sounding fictions.

To build production-grade enterprise software, developers decouple the reasoning engine from the information store using Retrieval-Augmented Generation (RAG).

sequenceDiagram
    autonumber
    actor User
    participant App as Application Server
    participant DB as Vector Database
    participant LLM as Language Model

    User->>App: "What is our company's refund policy?"
    App->>DB: Semantic Vector Search (Query Embedding)
    DB-->>App: Top-K Context Chunks (e.g., "Refund within 14 days")
    App->>LLM: Injected Prompt: [Context + User Query]
    Note over LLM: Evaluates ground truth context
    LLM-->>App: Generates dynamic, factual response
    App-->>User: "You can request a refund within 14 days..."

Generative vs. Extractive Answers

• Extractive QA: Finds the exact substring slice in the documentation and returns only that span. Often clumsy and disjointed.
• RAG (Generative Grounding): The model synthesizes the ground-truth context into natural dialogue. If an FAQ states “Refunds are limited to 14 days” and the user asks “Can I get my money back after 21 days?”, the LLM calculates the 7-day delta and generates an empathetic, contextually aware refusal.

---

7. Context Windows and Conversational Memory

LLMs are stateless functions. They do not retain state between discrete API calls. To maintain the illusion of ongoing memory in a chat session, applications maintain a running context window.

[
  {"role": "system", "content": "You are an enterprise support assistant."},
  {"role": "user", "content": "Can you create a 3-day Paris itinerary?"},
  {"role": "assistant", "content": "Day 1: Louvre. Day 2: Eiffel Tower. Day 3: Montmartre."},
  {"role": "user", "content": "Make Day 2 child-friendly."}
]

When the follow-up prompt arrives, the client sends the entire history back to the model. The self-attention mechanism attends over previous turns, enabling the model to rewrite only Day 2 while leaving Days 1 and 3 untouched.

---

8. Summary of Key Architectural Concepts

1. Next-Token Extender: LLMs are autoregressive models trained on tokenized subwords, predicting sequence continuations using cross-entropy loss.
2. BPE Tokenization: Balances vocabulary size and character coverage. Suboptimal tokenization for non-English languages degrades inference latency and increases API costs.
3. RLHF Alignment: Pre-training builds linguistic capacity; SFT teaches task instruction; RLHF uses a reward model (the coach) to steer the policy toward helpfulness, safety, and conciseness.
4. Decoding Controls: Temperature, Top-K, and Top-P govern the exploration-exploitation trade-off during text generation.
5. RAG Architecture: Grounding LLMs with external vector retrieval circumvents parameter staleness and eliminates factual hallucinations.