Quantization Fundamentals: How LLMs Run Efficiently with Reduced Memory and Fast Arithmetic

Modern Large Language Models (LLMs) push modern hardware to its physical limits. Flagship open-weights models like DeepSeek-R1 (671 billion parameters) require over 720 GB of raw VRAM just to fit into GPU memory—before accounting for the dynamic key-value (KV) cache, context window overhead, and runtime activations.

Even a modest 1-billion-parameter model stored in standard 32-bit floating-point (FP32) precision requires:

$\text{Memory} = 1\text{ billion parameters} \times 4\text{ bytes} = 4\text{ GB}$

As model parameters scale from billions to trillions, managing GPU memory footprint and compute efficiency becomes the defining engineering bottleneck. Quantization is the fundamental technique used in inference engineering to overcome both the memory wall and compute saturation.

---

The Two Core Bottlenecks of Floating-Point Formats

Why do high-precision floating-point formats degrade inference performance?

1. Memory Bandwidth and Capacity

Each weight and bias in a neural network is an individual number. At 32 bits (4 bytes) or 16 bits (2 bytes, such as FP16 or BF16), large parameter counts consume tens or hundreds of gigabytes across GPU High Bandwidth Memory (HBM). LLM token generation is typically memory-bandwidth bound: the GPU cores often sit idle waiting for weights to stream across the memory bus from HBM to on-chip SRAM/registers.

2. The Arithmetic Penalty: Floats vs. Integers

CPUs and GPUs execute integer operations far faster and with greater energy efficiency than floating-point operations.

• Integer Representation (Two’s Complement): Signed integers are stored in standard two’s complement. Hardware addition simply requires bitwise logic and ripple/lookahead carry chains, completing in 1 CPU/GPU clock cycle.
• Floating-Point Representation (Mantissa and Exponent): Floats represent numbers using scientific notation ($(-1)^s \times M \times 2^{E}$). Performing a single addition requires:
  1. Subtracting exponents to compute alignment shift.
  2. Shifting the smaller mantissa.
  3. Adding the aligned mantissas.
  4. Normalizing the resulting mantissa and updating the exponent.
  5. Rounding and checking for overflow/underflow.

Because of this multi-stage pipeline, standard floating-point addition demands 3 to 4 clock cycles and significantly higher silicon real estate and energy consumption.

Quantization resolves both problems by mapping continuous floating-point weights into compact, discrete integers (such as INT8 or INT4).

---

Why Does Precision Loss Not Break Inference?

If we compress a 16-bit float (BF16) into a 4-bit integer (INT4, which only has 16 distinct discrete levels: $0$ through $15$), there is substantial precision loss. Why does the model still produce coherent language?

Continuous Domain (BF16):  [-1.0 ----------------- 0.0 ----------------- +1.0]
                                      │                     │
                              (Bucketing / Mapping)         │
                                      ▼                     ▼
Discrete Domain (INT4):    [  0   1   2   3   4   5   6   7   8 ... 14  15 ]

Training Requires High Precision

During backpropagation and gradient descent, parameter updates ($\\Delta W = -\eta \nabla L$) are tiny fractions. To minimize the loss function effectively, gradients and optimizer states must retain high fidelity (typically FP32 or mixed-precision BF16/FP16). Rounding errors during training would stall convergence.

Inference Is Robust to Small Perturbations

During inference, the network executes massive general matrix multiplications (GEMM) representing dot products:

$y = \sum_{i=1}^{n} w_i \cdot x_i$

Because dot products aggregate thousands of multiplications over high-dimensional vectors, individual rounding errors are zero-mean and tend to cancel each other out across the summation. As a result, the final layer activations and output token probabilities (logits) remain remarkably stable.

---

Quantization Lifecycles: Where Quantization Kicks In

Quantization can be integrated into the machine learning pipeline at three distinct stages:

┌─────────────────────────────────────────────────────────────────────────────┐
│                     Quantization Lifecycles                                 │
├──────────────────────────┬───────────────────────┬──────────────────────────┤
│ 1. Quantization-Aware    │ 2. Post-Training       │ 3. Runtime / Inference   │
│    Training (QAT)        │    Quantization (PTQ) │    Quantization          │
├──────────────────────────┼───────────────────────┼──────────────────────────┤
│ Injects simulated round- │ Compresses weights    │ Weights are already      │
│ ing noise during the     │ after training (e.g.  │ quantized; KV cache      │
│ forward pass so weights  │ unsloth, llama.cpp,   │ tensors are dynamically  │
│ adapt prior to export.   │ AWQ, GPTQ).           │ quantized on the fly.    │
└──────────────────────────┴───────────────────────┴──────────────────────────┘

Why Not Quantize Weights Dynamically During Inference?

Quantizing full-precision model weights on the fly during inference provides zero benefit. To quantize weights at runtime, you would still need to stream the full FP16 parameters over the memory bus into GPU SRAM before converting them. This completely defeats the primary objective: reducing memory bandwidth saturation.

Weights are quantized ahead of time (PTQ or QAT) so that only low-bit integers are read from GPU memory. However, dynamic activations—most notably the KV cache vectors generated dynamically per token—can be quantized on the fly during generation to preserve VRAM.

---

Symmetric vs. Asymmetric Quantization

When mapping continuous floating-point values ($x \in [x_{min}, x_{max}]$) into discrete integers ($q \in [q_{min}, q_{max}]$), two primary mapping schemes are used:

1. Symmetric Quantization

In symmetric quantization, the real continuous value 0.0 maps strictly to the integer 0. The clipping range is centered around zero:

$q = \text{round}\left(\frac{x}{S}\right)$

$\text{where } S = \frac{\max(|x_{min}|, |x_{max}|)}{q_{max}}$

• Pros: Simpler math; zero-point offset operations are completely eliminated from the matrix multiplication pipeline.
• Cons: Inefficient bit utilization when the weight or activation distribution is heavily skewed (e.g., following a ReLU activation where all values are $\ge 0$). If only positive values exist, half of the available discrete bit range (the negative integers) is wasted.

2. Asymmetric Quantization

Asymmetric quantization maps the arbitrary continuous interval $[x_{min}, x_{max}]$ across the entire discrete integer interval $[q_{min}, q_{max}]$ by introducing an explicit integer zero-point ($Z$):

$q = \text{round}\left(\frac{x}{S}\right) + Z$

$S = \frac{x_{max} - x_{min}}{q_{max} - q_{min}}$

$Z = \text{round}\left(- \frac{x_{min}}{S}\right) + q_{min}$

• Pros: Full utilization of available discrete bit depth, even when distributions are strictly non-negative or skewed.
• Cons: Hardware implementations must track the zero-point offset during accumulation.

---

The Mathematical Trick: Factoring the Scale Factor

A common misconception is that because the scale factor $S$ is a floating-point number, multiplying weights by $S$ destroys the speedup of integer arithmetic.

Consider an unoptimized layer computing the dot product of quantized weights $w_i$ and inputs $x_i$:

$\text{Continuous Weight: } \hat{w}_i = S \cdot w_i$

$\text{Dot Product: } y = \sum_{i=1}^{n} (S \cdot w_i) \cdot x_i$

If the GPU scales each weight before multiplying it with the activation, the operation degrades back into floating-point arithmetic on every step, discarding hardware acceleration.

The Factoring Optimization

Because multiplication is associative and distributive, the scale factor $S$ can be factored out of the accumulation sum completely:

$y = S \cdot \left( \sum_{i=1}^{n} w_i \cdot x_i \right)$

┌────────────────────────────────────────────────────────┐
│ Integer Hardware Pipeline                              │
│                                                        │
│   w₁ ──┐                                              │
│   x₁ ──┴──> [ INT Multiply ] ──┐                       │
│   w₂ ──┐                       ├──> [ INT Accumulate ] │
│   x₂ ──┴──> [ INT Multiply ] ──┘         (INT32)       │
│                                              │         │
└──────────────────────────────────────────────┼─────────┘
                                               │ Raw Integer Sum
                                               ▼
                                     [ × Float Scale Factor S ]  (Only 1 Float Op per Row)
                                               │
                                               ▼
                                      Normalized Float Output

1. Integer Matrix Multiplication: The inner loop performs exclusively low-bit integer multiplications and additions (e.g., INT4 or INT8 inputs accumulating into an INT32 accumulator). This leverages high-throughput integer tensor cores at maximum clock efficiency.
2. Deferred Scaling: The continuous scale factor $S$ is multiplied only once per output dot product/row computation.

This single algebraic rearrangement preserves hardware integer execution efficiency across 99.9% of arithmetic operations while maintaining continuous output ranges for downstream layers.

---

Key Takeaways

• Parameters Define Memory Footprint: At 32-bit floating point, every 1 billion parameters requires 4 GB of pure weight storage. Quantizing to 4-bit cuts this demand to roughly 0.5 GB per billion parameters.
• ALU Clock Cycles Matter: Integer arithmetic completes in a single cycle using standard logic, whereas floating-point math requires multi-stage mantissa shifting, normalization, and exponent reconciliation (3–4 cycles).
• Matrix Cancellation Prevents Drift: High precision is mandatory for gradient calculation during training, but inference dot products aggregate errors in a way that preserves overall output fidelity.
• PTQ vs. Dynamic Quantization: Weights are quantized post-training to prevent memory bus saturation; dynamic quantization is reserved for runtime activations such as the KV cache.
• Scale Factor Factoring: Hardware implementations avoid intermediate floats by accumulating raw integer products and applying the floating-point scale factor once at the end of each dot product.