Primers • DeepSeek-R1

• Introduction
  • Architectural Foundations
  • Mixture of Experts (MoE)
    • Overview
    • Key Features
    • Evolution from DeepSeek-V2 to DeepSeek-R1
      • Background: MoE in DeepSeek-V2
      • Basic Architecture
      • Device-Limited Routing (DLR)
      • Load Balancing Mechanisms
      • Enhancements in DeepSeek-R1
    • Mathematical Formulation
      • Load Balancing Loss
    • Inference Efficiency
  • Multihead Latent Attention (MLA)
    • Overview
    • Key Features
    • Evolution from DeepSeek-V2 to DeepSeek-R1
      • MLA in DeepSeek-V2
      • Enhancements in DeepSeek-R1
    • Comparative Performance Analysis
    • Implementation
      • Standard Multi-Head Attention (MHA) Background
      • Low-Rank Key-Value Joint Compression
      • Decoupled Rotary Position Embedding (RoPE)
      • Hierarchical Caching Mechanism
      • Adaptive Attention Scaling
    • Comparative Efficiency Analysis
  • FP8 Quantization
    • Overview
    • Key Features
    • Evolution from DeepSeek-V3 to DeepSeek-R1
      • Background: DeepSeek-V3
      • Enhancements in DeepSeek-R1
      • Mathematical Representation
  • Multi-Token Prediction (MTP)
    • Overview
    • Key Features
    • Evolution from DeepSeek-V3 to DeepSeek-R1
      • Background: DeepSeek-V3
      • Enhancements in DeepSeek-R1
    • Implementation Details
    • Mathematical Formulation
• Training Pipeline: From Pre-Training to Reasoning
  • Stage 1: Cold Start with Supervised Fine-Tuning (SFT)
    • Fine-Tuning with High-Quality Chain-of-Thought (CoT) Examples
    • Structured Output Format
    • Loss Function for SFT
  • Stage 2: Reinforcement Learning
    • DeepSeek’s RL Methodology: A Conceptual Overview
    • RL Algorithm: Group Relative Policy Optimization (GRPO)
      • Evolution of GRPO: From DeepSeekMath to DeepSeek-R1
        • Phase 1: GRPO in DeepSeekMath (Mathematical RL)
        • Phase 2: GRPO in DeepSeek-R1-Zero (Self-Evolving Reasoning)
        • Phase 3: GRPO in DeepSeek-R1 (Refined Reasoning & Cold Start)
      • How GRPO Works
        • GRPO Intuition
        • Mathematical Formulation
      • Step-by-Step Breakdown
        • Likelihood Ratio \(\rho_i\)
        • Advantage Function \(A_i\)
        • Clipping Mechanism
        • KL Divergence Penalty
      • Implementation Details
        • Training Setup
        • Reward Function Design
        • Optimization Process
        • Efficiency Considerations
      • Applications
        • DeepSeek-R1-Zero: Reinforcement Learning from Scratch
        • DeepSeek-R1: Multi-Stage RL with Cold Start
      • PPO vs. DPO vs. KTO vs. APO vs. GRPO
        • Tabular Comparison
    • Reward Functions
  • Stage 3: Rejection Sampling & Expanded Supervised Fine-Tuning
  • Stage 4: Secondary Reinforcement Learning for Alignment & Generalization
• Emergent Reasoning Behaviors
  • Implementation Details
  • Example: Quadratic Equation Solving
• Distillation: Reasoning in Compact Models
  • Implementation Details
• Results
• Open Questions
  • Open-R1
    • Objectives of Open-R1
    • Impact on the Community
• Reasoning Datasets
• References

Introduction

• DeepSeek-R1 represents a landmark in reasoning-capable Large Language Models (LLMs). Released under an MIT license, this model rivals closed-source giants like OpenAI’s o1 and o3 series while pioneering a reinforcement learning (RL)-driven framework for reasoning tasks.
• DeepSeek-R1 leverages Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath, which replaces traditional methods like PPO, making training both efficient and scalable. DeepSeek-R1 also utilizes Multihead Latent Attention (MLA), introduced in DeepSeek-V2, which reduces computational and memory inefficiencies particularly for long-context processing by projecting Key-Query-Value (KQV) matrices into a lower-dimensional latent space.
• DeepSeek-R1 demonstrates how reasoning capabilities emerge naturally through RL alone without relying on massive Supervised Fine-Tuning (SFT). Through innovations like GRPO, FP8 quantization, and emergent CoT reasoning, it rivals closed-source models while fostering transparency and accessibility. As the research community builds upon these innovations, DeepSeek-R1 signals a shift towards efficient, reasoning-driven AI accessible to all.
• This primer explores its architecture, multi-stage training pipeline, GRPO mechanics, and emergent reasoning behaviors, alongside how distillation propagates reasoning capabilities to smaller models.

Architectural Foundations

• DeepSeek-R1 builds upon the foundational advancements introduced in DeepSeek-V2 — specifically, Mixture of Experts (MoE) and Multihead Latent Attention (MLA) — and DeepSeek-V3 — specifically, FP8 Quantization and Multi-Token Prediction (MTP) — integrating cutting-edge architectural innovations that optimize both training efficiency and inference performance.
• This section provides a detailed breakdown of the architectural components that evolved from DeepSeek-V2 and DeepSeek-V3 to DeepSeek-R1, highlighting improvements that make DeepSeek-R1 a leading open-source model, capable of rivaling proprietary alternatives in reasoning efficiency and performance.

Mixture of Experts (MoE)

Overview

• The Mixture of Experts (MoE) mechanism selectively activates a subset of the total model parameters at each inference step, achieving computational savings while maintaining model quality. This approach enables scaling up model parameters without a proportional increase in computational cost.
• DeepSeek-R1 refines DeepSeek-V2’s MoE framework, introducing dynamic expert routing, reinforcement learning-based load balancing, and enhanced sparsity constraints. These innovations make DeepSeek-R1 one of the most efficient and scalable open-source MoE models available.

Key Features

• Dynamic/Adaptive Expert Activation: Dynamically adjusts the number of active experts per token based on sequence complexity, improving efficiency by scaling computation according to context needs by using a RL-based approach, ensuring computational efficiency while maximizing reasoning performance.

• Device-Limited Routing (DLR): Selects experts based on device constraints to minimize cross-device communication, reducing synchronization overhead and improving training and inference speeds.

• Sparse Activation with Hierarchical Gating: Implements top-\(K\) expert selection at multiple levels to enforce sparsity constraints, preventing over-specialization by adjusting entropy-based token distribution.

• Load Balancing Optimization: Integrates expert-level, device-level, and communication-level loss functions to ensure uniform load distribution, leveraging reinforcement learning to dynamically adjust gating mechanisms.

Evolution from DeepSeek-V2 to DeepSeek-R1

Background: MoE in DeepSeek-V2

• DeepSeek-V2 employs the DeepSeekMoE architecture, which is designed to optimize training costs and inference efficiency while maintaining strong model performance. Unlike traditional dense transformer architectures, DeepSeekMoE introduces sparse activation of experts, significantly reducing the computational burden per token while allowing for a high overall parameter count. The key innovations in DeepSeekMoE include:

Basic Architecture

• DeepSeekMoE follows the general Mixture of Experts (MoE) paradigm, where each token is dynamically routed to a subset of specialized feed-forward network (FFN) experts rather than passing through a monolithic dense FFN.
• The model consists of 236B total parameters, but only 21B parameters are activated per token, striking a balance between model scalability and computational efficiency.
• Token Routing: Each token is assigned to a subset of top-\(K\) experts based on learned affinity scores, ensuring effective specialization while preventing unnecessary activation of experts.

Device-Limited Routing (DLR)

• To optimize efficiency, DeepSeek-V2 introduces a Device-Limited Routing (DLR) mechanism:
  • Constraint-Based Routing: Tokens are assigned only to a subset \(M\) of available devices, reducing communication overhead.
  • Affinity-Based Device Selection: The top \(M\) devices with the highest token-expert affinity scores are selected before choosing the top-\(K\) experts within them.
  • Optimized GPU Communication: By capping communication between GPUs, DeepSeek-V2 reduces MoE-related synchronization costs, leading to faster training convergence.

Load Balancing Mechanisms

• DeepSeek-V2 employs three auxiliary loss functions to ensure balanced expert utilization and reduce computational bottlenecks:

  1. Expert-Level Balance Loss (\(\mathcal{L}_{\text{ExpBal}}\))
     • Ensures uniform expert usage across different training batches.
     • Defined as:
       \(\mathcal{L}_{\text{ExpBal}} = \alpha_1 \sum_{i=1}^{N_r} f_i P_i\)
       • where \(f_i\) represents the fraction of tokens assigned to expert \(i\).
  2. Device-Level Balance Loss (\(\mathcal{L}_{\text{DevBal}}\))
     • Ensures equal computational load distribution across GPUs.
     • Defined as:
       \(\mathcal{L}_{\text{DevBal}} = \alpha_2 \sum_{i=1}^{D} f'_i P'_i\)
       • where \(D\) is the number of devices.
  3. Communication Balance Loss (\(\mathcal{L}_{\text{CommBal}}\))
     • Ensures balanced information flow between GPUs.
     • Defined as:
       \(\mathcal{L}_{\text{CommBal}} = \alpha_3 \sum_{i=1}^{D} f''_i P''_i\)

Enhancements in DeepSeek-R1

• DeepSeek-R1 refines the MoE framework by incorporating:

  • Dynamic Expert Assignment:
    • Experts are dynamically allocated based on contextual embeddings.
    • Softmax temperature scaling prevents expert over-specialization.
  • Reinforcement Learning-Guided Routing:
    • Introduces policy-based optimization to guide expert selection.
    • Feedback loop optimizes computational load balancing.
  • Sparse Activation Constraints:
    • Implements hierarchical top-\(K\) gating to enforce sparsity constraints.
    • Adjusts token-level entropy metrics to reduce unnecessary activations.

Mathematical Formulation

• The expert selection process in DeepSeek-R1 follows a gating function:

  \[G(x) = \text{softmax}(W_g x)\]
  • where \(W_g\) is a trainable weight matrix.

• The final output is computed as:

  \[y = \sum_{k \in K} G_k(x) E_k(x)\]
  • where:
    • \(K\) represents the top-K selected experts.
    • \(E_k(x)\) is the computation performed by expert \(k\).
    • \(G_k(x)\) is the gating probability.

Load Balancing Loss

• To ensure equal utilization of experts, DeepSeek-R1 applies a load balancing loss:

  \[\mathcal{L}_{\text{balance}} = \lambda \sum_k \left(\frac{n_k}{N} - \frac{1}{K}\right)^2\]
  • where:
    • \(n_k\) is the number of tokens assigned to expert \(k\).
    • \(N\) is the total number of tokens in a batch.
    • \(K\) is the number of active experts per token.

• Additionally, an entropy regularization term prevents expert over-reliance:

  \[\mathcal{L}_{\text{entropy}} = -\gamma \sum_k G_k(x) \log G_k(x)\]
  • where \(\gamma\) controls entropy strength.

Inference Efficiency

• To enhance inference efficiency, DeepSeek-R1 implements:

  1. FP8 Quantization:
     • Reduces memory overhead while maintaining precision.
  2. KV Cache Optimization:
     • Multi-Head Latent Attention (MLA) compresses KV-cache size.
     • Allows for larger batch sizes at inference time.
  3. Expert Parallelism and Communication Optimization:
     • 8-way expert parallelism ensures even GPU workload distribution.
     • Pipeline parallelism (16-way zero-bubble) minimizes idle compute time.
  4. Adaptive Expert Activation:
     • Adjusts active experts per token based on sequence complexity.

Multihead Latent Attention (MLA)

Overview

• Multihead Latent Attention (MLA) enhances efficiency by projecting Key-Query-Value (KQV) matrices into a lower-dimensional latent space, significantly reducing computational and memory costs.
• By utilizing low-rank compression techniques, MLA minimizes the storage overhead of the key-value (KV) cache, ensuring faster inference and supporting longer context lengths or larger batch sizes.
• With these refinements, DeepSeek-R1 significantly enhances MLA’s efficiency, achieving state-of-the-art performance in long-context tasks while maintaining extremely low memory overhead.

Key Features

• Dynamic Hybrid Latent Projection: Adjusts the latent space compression dynamically based on token complexity and sequence length, ensuring optimal memory usage by reducing compression for simple tokens while preserving higher-dimensional information for complex ones, leading to efficient inference with minimal performance loss.
• Hierarchical Caching Mechanism: Implements multi-level caching to reuse previously computed latent KV projections across similar tokens, minimizing redundant calculations and significantly reducing memory and computational overhead in long-context processing.
• Decoupled Rotary Position Embedding (RoPE): Separates RoPE application from the compressed latent KV pairs, allowing efficient positional encoding without interfering with key-value compression, eliminating redundant KV recomputation during inference and improving processing speed.
• Adaptive Attention Scaling: Introduces a self-adjusting attention weight mechanism that dynamically modulates attention scores based on token entropy and positional significance, enhancing long-context retention and recall accuracy while keeping computational demands low.
• Optimized Low-Rank Key-Value Compression: Extends the compression efficiency of DeepSeek-V2 by further reducing KV cache size beyond 93.3%, significantly decreasing memory requirements and boosting inference speed by 7.2× compared to DeepSeek 67B.
• Latency Reduction via Prefetching & Parallelization: Implements predictive KV caching and parallelized computation, precomputing frequently accessed KV projections to reduce on-demand inference latency, enabling faster batch processing for long sequences.
• Extended Context Length Support: Expands supported context length from 128K to 160K tokens, leveraging MLA’s compression and caching optimizations to maintain high efficiency and responsiveness across extended sequences, ensuring scalable performance in long-context tasks.

Evolution from DeepSeek-V2 to DeepSeek-R1

MLA in DeepSeek-V2

• DeepSeek-V2 introduced MLA as a key innovation to optimize the memory and compute requirements of the attention mechanism while maintaining the expressive power of standard Multi-Head Attention (MHA). The improvements included:

1. Low-Rank Key-Value Joint Compression:
   • Instead of storing full-dimensional key-value (KV) caches for each token, MLA projects them into a compact latent space.
   • The compression process involves two steps:
     1. Down-projection of full-dimensional K and V matrices into a latent representation: \(C_{KV} = W_{D_{KV}} X\) where:
        • \(C_{KV}\) is the compressed KV representation.
        • \(W_{D_{KV}}\) is the down-projection matrix that maps high-dimensional keys/values to a smaller space.
     2. Reconstruction via up-projection at the attention computation step: \(K_L = W_{U_K} C_{KV}, \quad V_L = W_{U_V} C_{KV}\) where \(W_{U_K}\) and \(W_{U_V}\) are learned transformation matrices that recover effective representations from the latent space.
   • This significantly reduces the number of stored elements per token, allowing a 93.3% reduction in memory consumption compared to standard MHA.
2. Inference Speed-Up:
   • Traditional MHA requires caching all KV pairs, leading to an inference-time bottleneck, especially for long sequences.
   • MLA compresses KV pairs before caching, drastically lowering memory overhead.
   • This results in a 5.76× improvement in inference speed over DeepSeek 67B.
3. Improved Model Scalability:
   • By reducing the memory footprint of KV caches, DeepSeek-V2 supports larger context lengths (up to 128K tokens) without performance degradation.
   • The efficiency gains allow larger batch sizes during inference, enabling higher throughput in real-world applications.
4. Mathematical Efficiency:
   • The standard MHA complexity is \(O(N^2 d_h)\), where \(N\) is the sequence length, and \(d_h\) is the head dimension.
   • With MLA, this complexity is reduced to \(O(N d_L)\), where \(d_L\) (latent dimension) is much smaller than \(d_h\).
   • This enables a much lighter computational cost while retaining the effectiveness of the original attention mechanism.

Enhancements in DeepSeek-R1

• DeepSeek-R1 builds upon the MLA framework of DeepSeek-V2, introducing several key enhancements that further improve memory efficiency, dynamic adaptability, and retrieval speed.

1. Hybrid Latent Projection with Adaptive Dimensioning:
   • While DeepSeek-V2 used a fixed-dimensional latent space, DeepSeek-R1 dynamically scales the latent projection based on token complexity.
   • This is achieved by adaptive compression factors that optimize memory use: \(d_L = f(N, C)\) where:
     • \(f(N, C)\) is a function of sequence length (N) and token complexity (C).
     • Simple tokens (e.g., stopwords) have lower-rank compression, while complex tokens (e.g., reasoning-intensive words) retain higher-rank features.
   • This reduces memory overhead without sacrificing retrieval accuracy.
2. Hierarchical Caching for Efficient Retrieval:
   • Traditional MLA stored compressed latent vectors but did not optimize retrieval across long sequences.
   • DeepSeek-R1 introduces Hierarchical Caching, which:
     • Reuses stored latent projections for similar tokens to avoid redundant recomputation.
     • Implements cache locality optimizations, reducing the need to compute new KV representations if similar tokens have already been processed.
     • Reduces unnecessary memory accesses, improving efficiency in long-context inference.
3. Decoupled Rotary Position Embedding (RoPE) Enhancement:
   • Standard RoPE couples positional encoding with both query and key matrices, making compression challenging.
   • DeepSeek-R1 solves this by decoupling RoPE from compressed latent KV pairs: \(Q_R = \text{RoPE}(W_{Q_R} C_Q), \quad K_R = \text{RoPE}(W_{K_R} X)\)
     • This allows RoPE to operate on separate query-key pairs without affecting KV compression.
     • Ensures faster inference by avoiding unnecessary recomputations.
4. Adaptive Attention Scaling for Long-Context Retention:
   • Standard MLA compressed all KV pairs uniformly, which could lose information for long sequences.
   • DeepSeek-R1 introduces adaptive attention scaling, where attention weights dynamically adjust based on:
     • Token entropy (importance of a token in a given context).
     • Positional information (distance from query token).
   • This results in improved recall for long-context understanding while keeping computation minimal: \(A' = \sigma \left( \frac{Q_L K_L^T}{\sqrt{d_L}} \right) \cdot S_{\text{adaptive}}\) where \(S_{\text{adaptive}}\) dynamically adjusts token influence.
5. Latency Reduction via Prefetching and Parallelization:
   • DeepSeek-R1 predicts future KV queries based on prior activations and preloads relevant caches.
   • Parallelized low-rank attention computation reduces bottlenecks in the attention mechanism.
   • These optimizations allow DeepSeek-R1 to handle longer sequences while maintaining low inference latency.

Comparative Performance Analysis

Implementation

Standard Multi-Head Attention (MHA) Background

• For a standard multi-head attention (MHA) mechanism, the Key (K), Query (Q), and Value (V) matrices are computed as follows:

  \[K, Q, V = W_k X, W_q X, W_v X\]
  • where \(W_k, W_q, W_v\) are weight matrices for key, query, and value projections.

• The attention weights are computed as:

• and the output is given by:

• This requires storing the full key-value cache during inference, leading to significant memory overhead.

Low-Rank Key-Value Joint Compression

• MLA optimizes MHA by jointly compressing the keys and values into a lower-dimensional latent space:

1. Compression of Key-Value Representations:

   \[C_{KV} = W_{D_{KV}} X\] \[K_L = W_{U_K} C_{KV}, \quad V_L = W_{U_V} C_{KV}\]
   • where:
     • \(C_{KV}\) is the compressed latent representation.
     • \(W_{D_{KV}}\) is the down-projection matrix.
     • \(W_{U_K}, W_{U_V}\) are up-projection matrices for reconstructing keys and values.

2. Compression of Query Representations (for training efficiency):

   \[C_Q = W_{D_Q} X\] \[Q_L = W_{U_Q} C_Q\]
   • This step ensures that memory footprint is minimized during training, although it does not contribute to inference efficiency.

3. Final Attention Computation:

   • The attention scores are computed using the low-rank compressed matrices:
   \[A_L = \text{Softmax} \left( \frac{Q_L K_L^T}{\sqrt{d_L}} \right)\] \[O_L = A_L V_L\]
   • This reduces attention complexity from \(O(N^2)\) to \(O(N d_L)\), where \(d_L\) is the latent space dimension, leading to significant computational savings.

Decoupled Rotary Position Embedding (RoPE)

• A challenge with low-rank KV compression is the incompatibility with position encoding. RoPE typically couples positional embeddings with keys and queries, making it difficult to compress the key representations without losing positional information.

• To solve this, DeepSeek-V2 and DeepSeek-R1 introduce a decoupled RoPE mechanism:

  1. A secondary set of auxiliary query and key vectors is created specifically for positional encoding:

     \[Q_R = \text{RoPE}(W_{Q_R} C_Q), \quad K_R = \text{RoPE}(W_{K_R} X)\]

  2. The final queries and keys are concatenated before attention computation:

     \[Q = [Q_L; Q_R], \quad K = [K_L; K_R]\]

  3. This ensures that positional information is retained without interfering with the compressed KV projections, eliminating redundant recomputations during inference.

Hierarchical Caching Mechanism

• Unlike standard KV caching, which requires storing all key-value pairs, DeepSeek-R1 introduces a hierarchical caching approach:

  1. Primary Latent KV Cache:
     • Stores compressed latent vectors \(C_{KV}\) to minimize memory footprint.
     • Uses a hierarchical lookup table for fast retrieval.
  2. Context-Aware Retrieval:
     • Tokens that appear in similar contexts reuse cached latent vectors.
     • A similarity-based indexing system dynamically retrieves the most relevant cache entries.
  3. Latency Reduction via Prefetching:
     • Predicts future attention patterns and precomputes required latent keys and values.
     • Reduces on-demand computation overhead.

Adaptive Attention Scaling

• DeepSeek-R1 further refines MLA by introducing adaptive attention scaling, which dynamically adjusts the importance of different tokens in long-context scenarios:

  \[A' = \sigma \left( \frac{Q_L K_L^T}{\sqrt{d_L}} \right) \cdot S_{\text{adaptive}}\]
  • where \(S_{\text{adaptive}}\) is a scaling factor that adjusts based on token importance.

• Adaptive attention scaling ensures:

  • Critical tokens receive more attention.
  • Redundant information is downweighted.
  • Improved recall in long-context scenarios.

Comparative Efficiency Analysis

FP8 Quantization

Overview

• DeepSeek-R1 leverages 8-bit floating-point (FP8) quantization to enhance efficiency during both training and inference. This optimization significantly reduces the memory footprint and computational complexity without sacrificing numerical stability. By using FP8 precision, DeepSeek-R1 achieves faster training speeds, reduced GPU memory consumption, and lower inference latency, making it highly scalable for real-world applications with lower computational costs.

Key Features

• Dynamic Scaling for Stability: Implements a learned dynamic scaling factor ($S$) that adapts per layer, reducing quantization errors and improving numerical stability.
• Extended FP8 for Inference: Expands FP8 quantization to inference, reducing latency and optimizing GPU memory usage, enabling more efficient deployment.
• Adaptive Clipping Mechanism: Introduces per-token dynamic clipping thresholds, ensuring values remain within FP8 representable limits (-127 to 127), preventing numerical overflow.
• Optimized RL Quantization: Enhances RL stability by dynamically adjusting precision based on gradient fluctuations, preventing instability during policy optimization.
• Low-Precision MoE Efficiency: Utilizes block-wise FP8 quantization for MoE layers, reducing computational and memory overhead for large-scale models.
• Hardware-Aware Optimization: Designed to fully leverage NVIDIA H800 Tensor Cores, ensuring efficient low-precision computation without sacrificing accuracy.

Evolution from DeepSeek-V3 to DeepSeek-R1

Background: DeepSeek-V3

• Early Adoption of FP8: DeepSeek-V3 was among the pioneering models to adopt FP8 mixed precision training to achieve both computational efficiency and reduced memory usage.
• Mixed Precision Framework: The model integrated FP8 with other numerical formats such as BF16 in a hybrid precision approach to maintain training stability while benefiting from the lower precision for faster computations.
• Improved Precision in Multiplication: DeepSeek-V3 employed adaptive quantization scaling techniques to minimize precision loss during matrix multiplications and weight updates. By leveraging per-tensor scaling factors, it ensured that numerical stability was maintained even under extreme compression.
• Low-Precision Storage & Communication: DeepSeek-V3 optimized training efficiency by quantizing activations and gradients before transmission across distributed computing nodes. This approach, coupled with block-wise quantization, significantly reduced memory bandwidth requirements and accelerated training by decreasing inter-GPU communication overhead.
• Efficient Hardware Utilization: The FP8 implementation was hardware-aware, designed to leverage NVIDIA H800 Tensor Cores, ensuring compatibility with modern AI accelerators. This allowed for near full computation-communication overlap, reducing the overall latency of large-scale model training.
• Challenges in Quantization Stability: Despite these advancements, DeepSeek-V3 faced challenges in ensuring numerical stability at extreme precision reductions, particularly during RL and fine-tuning phases. Additional refinements to FP8 scaling and adaptive precision handling were required to prevent degradation in model accuracy.

Enhancements in DeepSeek-R1

• Dynamic Scaling Factor: DeepSeek-R1 refines the scaling-based transformation used for numerical stability. Unlike DeepSeek-V3, which relied on static scaling factors, DeepSeek-R1 incorporates a learned dynamic scaling factor (\(S\)) optimized based on loss gradients. This allows per-layer adaptive precision adjustments, significantly reducing quantization-induced errors and improving numerical stability.
• Extended Use in Inference: While FP8 in DeepSeek-V3 was primarily applied during training, DeepSeek-R1 extends FP8 quantization to inference, resulting in lower latency and reduced GPU memory consumption. The implementation involves selective quantization of attention layers, feedforward layers, and key-value cache storage, enabling efficient deployment of the model in real-world applications.
• Clipping for Numerical Stability: DeepSeek-R1 introduces an advanced clipping mechanism for FP8 quantization, ensuring values remain within the representable range (-127 to 127). This prevents numerical overflow and maintains precision during gradient accumulation and weight updates. Additionally, per-token dynamic clipping thresholds are applied in transformer layers to improve training robustness.
• RL Integration: A major enhancement in DeepSeek-R1 is its improved handling of FP8 quantization within RL processes. The model dynamically adjusts per-layer precision needs by adapting scaling factors throughout RL-based optimization. This reduces the risk of gradient vanishing or exploding, ensuring more stable policy learning.
• Optimized Memory Efficiency: DeepSeek-R1 further optimizes low-precision storage and computation by implementing block-wise quantization strategies for MoE models, ensuring that expert activations and gradients are efficiently compressed. This results in more efficient expert selection and routing, reducing MoE-related computational overhead.
• Overall Improvements: The transition from DeepSeek-V3 to DeepSeek-R1 marks a significant leap in FP8 quantization by improving training stability, expanding FP8’s role to inference, introducing per-layer adaptive scaling, and enhancing RL robustness. These enhancements collectively make DeepSeek-R1 more efficient and scalable while maintaining high model accuracy.

Mathematical Representation

• FP8 quantization in DeepSeek-R1 follows a scaling-based transformation:

  \[x_q = \text{clip} \left( \text{round} \left( \frac{x}{S} \right), -127, 127 \right)\]
  • where:
    • \(S\) is a learned dynamic scaling factor optimized based on loss gradients.
    • Clipping ensures values remain within the FP8 representable range (-127 to 127), preventing numerical overflow.
    • The scaling factor is updated dynamically to adapt to per-layer precision needs.

Multi-Token Prediction (MTP)

Overview

• Multi-Token Prediction (MTP) allows DeepSeek-R1 to predict multiple tokens in parallel, significantly improving inference speed.

Key Features

• Parallel Decoding: Extends the autoregressive framework by allowing multiple token predictions within the same context window.
• Dynamic Prediction Horizon: Adjusts the number of tokens predicted per step based on model confidence, improving efficiency and coherence.
• Token Sampling and Re-ranking: Multi-token outputs are sampled from a probabilistic distribution and re-ranked for coherence.
• Speculative Decoding with Verification: Generates multiple token sequences in parallel and validates their correctness before finalizing outputs.
• RL-Based Token Selection: Uses RL to prioritize high-quality token sequences based on fluency, coherence, and factual accuracy.
• Efficient Memory and Computation: Reduces memory overhead by selectively caching representations instead of storing full causal chains.
• Optimized Training with RL: Enhances standard cross-entropy loss with reinforcement-driven rewards to improve prediction reliability.
• Higher Throughput and Reduced Latency: Achieves faster inference by reducing computational complexity from ( O(T) ) to ( O(T/k) ), where ( k ) is the adaptive prediction depth.

Evolution from DeepSeek-V3 to DeepSeek-R1

Background: DeepSeek-V3

• DeepSeek-V3 introduced MTP as an advanced mechanism to improve model efficiency and inference speed. Instead of predicting tokens one at a time in a strict autoregressive manner, DeepSeek-V3 extended its capability to predict multiple future tokens in a structured fashion. The implementation of MTP in DeepSeek-V3 leveraged a sequential module-based architecture, where each token prediction maintained a complete causal chain for contextual accuracy. Specifics below:

  • Sequential Multi-Token Generation: Unlike other parallel decoding approaches, DeepSeek-V3 used a sequence of transformer modules, each predicting subsequent tokens while preserving context from previous steps. The MTP framework consisted of \(D\) sequential modules, each responsible for predicting the next token in the sequence while retaining dependencies through hierarchical representations.
  • Token Embedding Sharing: The MTP modules utilized a shared embedding layer to ensure consistent representations across different prediction depths. This was achieved by applying a linear projection to combine the hidden state of the previous depth with the embedding of the next token, allowing the transformer block to process a well-structured representation.
  • High Acceptance Rate: A key metric for evaluating MTP effectiveness in DeepSeek-V3 was the acceptance rate of predicted tokens, which reached 85-90% in various contexts. This high rate demonstrated the effectiveness of the prediction model in minimizing unnecessary recomputations, leading to faster inference.
  • Training Objective: The training framework for MTP in DeepSeek-V3 involved a cross-entropy loss function applied across multiple predicted tokens. The overall MTP loss was computed as:
    \(L_{MTP} = \frac{\lambda}{D} \sum_{k=1}^{D} L_{CE}(P_k, T_k)\)
    • where \(D\) is the number of tokens predicted per step, and \(L_{CE}\) represents the cross-entropy loss.
  • Speculative Decoding Integration: MTP was designed to work seamlessly with speculative decoding methods, leveraging the precomputed token sequences to accelerate inference. By pre-generating and verifying multiple candidate tokens before committing to a single output, the system reduced computational overhead, resulting in an approximate 1.8× increase in Tokens Per Second (TPS) compared to standard autoregressive models.

• Despite these improvements, DeepSeek-V3’s MTP implementation remained limited by its fixed depth, as the model did not dynamically adjust the number of predicted tokens based on confidence. This constraint led to inefficiencies in cases where greater flexibility was required to maintain coherence in complex linguistic structures.

Enhancements in DeepSeek-R1

• DeepSeek-R1 introduced a refined MTP mechanism, addressing the limitations of DeepSeek-V3 by incorporating dynamic prediction strategies, RL optimization, and enhanced speculative decoding.
• These advancements enabled DeepSeek-R1 to achieve superior performance in real-world applications that required fast and coherent text generation. The transition from DeepSeek-V3’s static multi-token framework to DeepSeek-R1’s dynamic, RL-enhanced approach marked a major step forward in optimizing multi-token prediction efficiency. Specifics below:
  • Dynamic Prediction Horizon: Unlike DeepSeek-V3, which relied on a fixed number of predicted tokens per step, DeepSeek-R1 introduced an adaptive mechanism to determine the number of tokens predicted in each step. This was achieved using confidence-based dynamic adjustment, where the model estimated the uncertainty of its predictions and adjusted the prediction depth accordingly, reducing unnecessary token reevaluations.
  • Speculative Decoding with Verification: DeepSeek-R1 improved upon DeepSeek-V3’s speculative decoding approach by introducing a hierarchical token verification process. This method involved generating multiple candidate token sequences and evaluating their coherence before finalizing predictions. If a low-confidence token was identified, the model dynamically re-evaluated and adjusted its predictions.
  • RL-Based Token Selection: DeepSeek-R1 leveraged RL to optimize its MTP framework. By training the model to prioritize sequences with higher coherence and fluency, RL-based token selection enabled improved accuracy. The reward function was structured to favor outputs with high linguistic alignment, reducing error propagation during multi-token predictions.
  • Improved Memory and Computational Efficiency: The MTP module in DeepSeek-R1 was further optimized for efficient memory usage. Unlike DeepSeek-V3, which stored complete causal chains for all predicted tokens, DeepSeek-R1 employed selective caching and retrieval mechanisms. This optimization reduced redundant computations and improved inference latency while maintaining high-quality predictions.
  • Training and Optimization: DeepSeek-R1 maintained a similar cross-entropy-based training loss but incorporated additional RL objectives. The loss function was augmented with a reinforcement-driven term: \(L_{MTP}^{RL} = L_{MTP} + \alpha \sum_{i=1}^{N} R(y_i)\)
    • where \(R(y_i)\) represents the reinforcement reward for selecting token \(y_i\), and \(\alpha\) controls the trade-off between standard cross-entropy loss and RL-based optimization.
  • Higher Throughput and Reduced Latency: Through its enhanced verification and adaptive prediction mechanisms, DeepSeek-R1 achieved an even greater improvement in inference speed compared to DeepSeek-V3. The model further reduced the computational complexity from \(O(T)\) to \(O(T/k)\), where \(k\) represented the dynamically adjusted prediction depth per step.

Implementation Details

• DeepSeek-R1 incorporates an advanced MTP strategy to boost decoding efficiency and reduce latency. Unlike traditional autoregressive decoding, where each token is predicted sequentially, MTP allows multiple tokens to be predicted per decoding step. This is achieved through a hierarchical approach that balances performance improvements with the risk of error propagation. Specifics below:

  1. Multi-Layer Representation Propagation:
     • DeepSeek-R1’s transformer architecture is enhanced to support simultaneous token prediction across multiple layers.
     • Each layer in the model computes token probabilities independently while maintaining consistency across the sequence.
  2. Speculative Decoding with Verification:
     • During inference, DeepSeek-R1 generates speculative multi-token sequences and verifies their coherence through a hierarchical token verification mechanism.
     • This approach dynamically adjusts the number of tokens predicted in each step based on confidence scores, ensuring that low-confidence tokens are reevaluated before finalizing outputs.
  3. Training Objective:
     • The model is trained with a combination of standard cross-entropy loss for next-token prediction and an auxiliary loss that encourages parallel token prediction.
     • The loss function is formulated as:
       \(L_{MTP} = \lambda \sum_{k=1}^{D} L_{CE}(P_k, T_k)\)
       • where \(D\) is the number of parallel tokens predicted per step, and \(L_{CE}\) represents the cross-entropy loss for each predicted token.
  4. Adaptive Token Selection with RL:
     • DeepSeek-R1 employs an RL-based approach to refine multi-token predictions, ensuring that higher-quality token sequences are prioritized.
     • The RL framework assigns rewards based on coherence, fluency, and alignment with ground-truth data.
     • This RL-driven strategy effectively reduces hallucinations and improves long-range coherence in generated text.
  5. Memory and Compute Efficiency:
     • The MTP module is optimized to minimize additional memory overhead, leveraging weight-sharing mechanisms within transformer layers.
     • The speculative decoding mechanism integrates efficiently with DeepSeek-R1’s caching strategy, ensuring that redundant computations are avoided.

Mathematical Formulation

• The prediction function follows an autoregressive formulation:

• By introducing parallel decoding, DeepSeek-R1 reduces inference complexity from \(O(T)\) to \(O(\frac{T}{k})\), where \(k\) is the number of tokens predicted per step.

Training Pipeline: From Pre-Training to Reasoning

• DeepSeek-R1 employs a multi-stage training pipeline designed to enhance reasoning capabilities while maintaining efficiency. This process includes distinct phases, each guided by task-specific loss functions and reward mechanisms, ensuring progressive refinement in performance. The key stages are Supervised Fine-Tuning (SFT), RL, Rejection Sampling, and an additional RL phase for generalization. Together, these steps improve DeepSeek-R1’s ability to tackle complex reasoning tasks while ensuring clarity and coherence in its outputs.
• DeepSeek-R1’s training process unfolds in four key phases, each progressively refining its reasoning ability while expanding generalization and alignment:
  1. Cold Start with SFT
     • Fine-tuning on thousands of high-quality Chain-of-Thought (CoT) examples to establish structured reasoning.
     • Uses a structured output format for improved readability.
     • Employs a cross-entropy-based loss function for optimization.
  2. RL with GRPO
     • Policy optimization via Group-based Reward Normalization (GRPO).
     • Rewards assigned based on accuracy, format consistency, and language alignment.
     • Prevents reward hacking by avoiding neural reward models.
  3. Rejection Sampling & Expanded SFT
     • Filters high-quality RL outputs to enhance supervised fine-tuning.
     • Expands training data to include non-reasoning tasks, ensuring broader applicability.
  4. Final RL Phase for Generalization
     • Integrates diverse task distributions, extending beyond structured reasoning.
     • Ensures alignment with human feedback, particularly in conversational settings.
• Through this multi-stage refinement process, DeepSeek-R1 surpasses previous models in accuracy, coherence, and real-world usability, setting a new benchmark for AI reasoning capabilities.

Stage 1: Cold Start with Supervised Fine-Tuning (SFT)

Fine-Tuning with High-Quality Chain-of-Thought (CoT) Examples

• DeepSeek-R1 begins its journey by fine-tuning the DeepSeek-V3-Base model with a carefully curated dataset of high-quality Chain-of-Thought (CoT) examples. These examples are obtained through a combination of:
  1. Few-shot prompting: Generating detailed reasoning paths using large-scale pre-trained models.
  2. Manual annotation and refinement: Filtering and refining reasoning steps through human reviewers.
  3. Post-processing DeepSeek-R1-Zero outputs: Extracting well-structured reasoning paths from the RL-trained precursor model.
• The fine-tuning step ensures that DeepSeek-R1 has a structured reasoning framework before entering RL. Unlike DeepSeek-R1-Zero, which learned reasoning solely from RL, DeepSeek-R1 leverages cold-start fine-tuning to avoid the chaotic early stages of RL training.

Structured Output Format

• One of the key issues encountered in DeepSeek-R1-Zero was language mixing and poor readability. To address this, the fine-tuning phase enforces a structured reasoning format:

<reasoning_process> Step-by-step explanation of the problem-solving approach </reasoning_process>
<summary> Final Answer </summary>

This format ensures readability and helps align the model’s outputs with human expectations.

Loss Function for SFT

• The model is optimized using a supervised cross-entropy loss:

• where:
  • \(o_i\) is the \(i^{th}\) token in the output sequence,
  • \(q\) is the input query,
  • \(o_1, ..., o_{i-1}\) are previously generated tokens.
• This step helps DeepSeek-R1 establish a strong foundation for structured reasoning before RL.

Stage 2: Reinforcement Learning

• RL is the backbone of DeepSeek-R1’s reasoning evolution. The model learns to optimize its reasoning trajectories based on reward-driven feedback mechanisms, leading to significant improvements in accuracy and coherence.

DeepSeek’s RL Methodology: A Conceptual Overview

• DeepSeek’s RL methodology is fundamentally inspired by self-play paradigms, akin to training AI models in games like chess. Traditionally, AI models trained for complex reasoning tasks leverage large datasets composed of human-annotated examples. However, such datasets often lack comprehensive coverage and may not contain optimal solutions. RL circumvents this limitation by allowing AI models to explore solutions autonomously, refining their strategies based on reward-driven feedback mechanisms.
• Consider an AI model trained to play chess. Instead of learning from a fixed dataset of historical games, the AI is programmed with only the fundamental rules of chess. It then engages in self-play, continuously experimenting with various moves. Initially, the model executes suboptimal actions, leading to losses. However, through iterative play, it identifies effective strategies and reinforces moves that contribute to victories while discarding ineffective ones. This trial-and-error process, governed by RL principles, enables the AI to develop strategies surpassing human intuition.
• DeepSeek applies this RL-based approach to reasoning-intensive domains, such as mathematical problem-solving. Rather than training on explicit mathematical derivations, the AI is provided with fundamental mathematical rules and tasked with solving problems autonomously. The model systematically explores various solution paths, reinforcing those that yield correct answers while discarding ineffective methodologies. Over time, this process enhances the AI’s mathematical reasoning abilities beyond traditional supervised learning approaches. The self-improving nature of RL fosters the discovery of novel problem-solving strategies, resulting in superior performance in mathematical reasoning and logic-based tasks.

RL Algorithm: Group Relative Policy Optimization (GRPO)

• Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath, is a RL method that has played a pivotal role in the development of DeepSeek-R1. It is a simplified and cost-efficient alternative to traditional policy optimization techniques like Proximal Policy Optimization (PPO), since it does not require a separate critic model. Instead, it estimates the baseline from a group of generated outputs, reducing computational overhead while maintaining sample efficiency. This group-based approach ensures that each update step improves on previous iterations without overfitting to individual trajectories.
• GRPO has evolved from a mathematical reasoning optimizer in DeepSeekMath to a core optimization technique in DeepSeek-R1, driving advanced reasoning capabilities across diverse tasks. By eliminating the critic model, leveraging group-based advantages, and incorporating multi-stage RL refinements, GRPO has made DeepSeek-R1 a powerful open-source reasoning models.
• GRPO is central to DeepSeek-R1’s RL pipeline, providing a lightweight yet powerful optimization mechanism. Its key innovations include:
  • Removing the critic model, which significantly reduces memory overhead.
  • Stabilizing policy updates through group-based advantage estimation.
  • Efficient training while maintaining strong performance compared to PPO-based methods.
• From its inception in DeepSeekMath to its refined implementation in DeepSeek-R1, GRPO has undergone several enhancements, including multi-stage RL, improved reward modeling, and refined optimization strategies. This section details GRPO’s mathematical formulation, its implementation, and its role in DeepSeek-R1.

Evolution of GRPO: From DeepSeekMath to DeepSeek-R1

Phase 1: GRPO in DeepSeekMath (Mathematical RL)

• GRPO was originally introduced in DeepSeekMath to optimize models for mathematical reasoning.
• It replaced PPO’s critic model with a group-based reward normalization technique, making training more efficient while maintaining stability.
• The reward function primarily evaluated mathematical correctness, using structured evaluation metrics.

Phase 2: GRPO in DeepSeek-R1-Zero (Self-Evolving Reasoning)

• With DeepSeek-R1-Zero, GRPO was applied without any supervised fine-tuning (SFT)—pure RL was used to shape reasoning behaviors from scratch.
• The model self-learned reasoning skills such as step-by-step problem-solving and self-verification.
• However, DeepSeek-R1-Zero exhibited readability issues (e.g., unstructured reasoning outputs, language mixing).

Phase 3: GRPO in DeepSeek-R1 (Refined Reasoning & Cold Start)

• DeepSeek-R1 introduced a multi-stage RL pipeline incorporating a small amount of cold-start fine-tuning before applying GRPO.
• The reward model was expanded beyond mathematics to include general reasoning tasks.
• A language consistency reward was added to improve coherence and readability.

How GRPO Works

• GRPO modifies traditional policy optimization by leveraging group-based normalization instead of a critic model. This enables efficient and stable policy updates while reducing computational overhead.

GRPO Intuition

• To understand GRPO, it is useful to analyze its mathematical formulation from a reverse-engineering perspective. The complexity of the equations can be misleading; in reality, GRPO consists of three main components:

  \[J_{GRPO} = \min([\text{Block 1}], [\text{Block 2}]) - [\text{Block 3}]\]
  • where:
    • Block 1 corresponds to the first term inside the summation of the GRPO objective function: \(\rho_i A_i = \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)} A_i.\) This represents the primary objective of policy optimization: ensuring the updated policy \(\pi_\theta\) improves upon the previous policy \(\pi_{\theta_{old}}\). The core principle is straightforward: the new policy should outperform the old one in expectation.
    • Block 2 corresponds to the clipped version of \(\rho_i A_i\), i.e., \(\text{clip}(\rho_i, 1 - \epsilon, 1 + \epsilon) A_i.\) This originates from PPO and serves as a safeguard to prevent excessive updates. By taking the minimum between Block 1 and this clipped value, GRPO ensures training stability and prevents over-exaggerated policy updates.
    • Block 3 corresponds to the KL-divergence regularization term in the GRPO equation: \(\beta D_{KL}(\pi_\theta || \pi_{ref}).\) This term enforces similarity between the new policy and a reference policy, preventing the optimization process from deviating too far from the original distribution and ensuring controlled updates.
• One of the most notable aspects of GRPO’s success is its redesigned approach to advantage computation. Traditional PPO computes advantages using a learned value network combined with temporal difference learning, requiring additional memory and computation to maintain a separate critic model. In contrast, GRPO fundamentally simplifies this by directly comparing sampled actions within a group and leveraging statistical normalization to compute advantages. This group-based methodology eliminates the need for a value network, significantly reducing memory overhead—by approximately half—while simultaneously aligning with the core principle of evaluating mathematical solutions relative to other approaches to the same problem.
• This design choice has proven especially effective for mathematical reasoning tasks. By using a direct group-based comparison, GRPO enhances the model’s ability to develop structured reasoning strategies. Empirical results demonstrate that this method not only improves performance on mathematical reasoning benchmarks but also maintains training stability and computational efficiency. The elimination of the critic network removes potential biases from learned value functions, making GRPO particularly well-suited for domains requiring objective evaluation of multiple solution paths.
• Additionally, the “Group” aspect in GRPO refers to computing the expectation over a set of sampled outputs, which are then averaged to stabilize training. The presence of normalization within \(A\) (mean and standard deviation) may initially appear complex, but it simply follows conventional normalization techniques used in machine learning.
• Thus, when stripped of indices, subscripts, and hyperparameters, GRPO reduces to a simple balance between policy improvement and control mechanisms, reinforcing why it is regarded as an efficient and intuitive optimization method.

Mathematical Formulation

• The GRPO objective function is:

  \[J_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^G \min\left(\rho_i A_i, \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon) A_i\right) - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}) \right]\]
  • where:
    • \(\rho_i\) is the likelihood ratio, indicating how much the new policy diverges from the old one: \(\rho_i = \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}\)
    • \(A_i\) is the group-based advantage function, which normalizes rewards across sampled outputs: \(A_i = \frac{r_i - \text{mean}(r_1, ..., r_G)}{\text{std}(r_1, ..., r_G)}\)
    • \(D_{\text{KL}}(\pi_\theta \| \pi_{ref})\) is a KL regularization term that constrains updates within a stable range.
    • \(G\) is the group size (number of sampled outputs per query).
    • \(\epsilon\) controls clipping to prevent overly aggressive updates.
    • \(\beta\) controls the strength of KL regularization.

• The expanded form of the GRPO objective function can be written as:

  \[J_{\text{GRPO}}(\theta) = \mathbb{E} \left[ \sum_{i=1}^{G} \min \left( \frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} A_i, \text{clip} \left(\frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}, 1-\epsilon, 1+\epsilon \right) A_i \right) - \beta D_{\text{KL}}(\pi_{\theta} || \pi_{\text{ref}}) \right]\]
  • where:
    • \(\epsilon\) is the trust region clipping parameter to stabilize training,
    • \(A_i\) is the advantage function, computed from group-based reward normalization.

Step-by-Step Breakdown

Likelihood Ratio \(\rho_i\)

• Measures how much the probability of generating output \(o_i\) has changed under the new policy compared to the old policy: \(\rho_i = \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}\)

Advantage Function \(A_i\)

• Instead of relying on a separate value network (critic), GRPO estimates the advantage function using a group of sampled outputs: \(A_i = \frac{r_i - \text{mean}(r_1, ..., r_G)}{\text{std}(r_1, ..., r_G)}\)
• This reduces training instability and enhances efficiency.

Clipping Mechanism

• Prevents drastic policy updates that could destabilize training: \(\text{clip}(\rho_i, 1-\epsilon, 1+\epsilon)\)

KL Divergence Penalty

• Ensures the policy remains close to a reference distribution: \(\beta D_{\text{KL}}\bigl(\pi_\theta \;\|\; \pi_{\text{ref}}\bigr)\)
• Prevents mode collapse and excessive policy drift.

Implementation Details

Training Setup

• GRPO is implemented by sampling multiple outputs per query and computing rewards over the group.
• The mean and standard deviation of rewards provide a normalized baseline for training.

Reward Function Design

• In DeepSeekMath: The reward was primarily based on mathematical correctness.
• In DeepSeek-R1: The reward function expanded to include:
  • Accuracy Rewards: Evaluating correctness for general reasoning tasks (e.g., coding, science, logic).
  • Format Rewards: Ensuring structured reasoning using <think> and <answer> tags.

Optimization Process

• The model samples multiple outputs per query, computes likelihood ratios and advantage estimates, and updates its policy using the clipped objective function.

Efficiency Considerations

• Removes critic model, reducing memory consumption.
• Batch computation for group sampling, improving efficiency.
• Iterative RL refinement, enabling continual improvement.

Applications

DeepSeek-R1-Zero: Reinforcement Learning from Scratch

• DeepSeek-R1-Zero applied GRPO without any pretraining, allowing the model to self-learn reasoning.
• The model naturally developed skills like self-verification and reflection.
• However, poor readability and language mixing emerged as challenges.

DeepSeek-R1: Multi-Stage RL with Cold Start

• To refine DeepSeek-R1-Zero, DeepSeek-R1 introduced:
  1. Cold Start Fine-Tuning:
     • The model was first fine-tuned on high-quality Chain-of-Thought (CoT) examples.
     • This ensured structured reasoning and better readability.
  2. RL with GRPO:
     • GRPO was used to refine reasoning skills in math, logic, and general problem-solving.
     • A language consistency reward was added to prevent language mixing.
  3. Final RL Optimization:
     • After RL, a rejection sampling step generated better training data.
     • A final GRPO optimization phase was conducted with diverse prompts.

PPO vs. DPO vs. KTO vs. APO vs. GRPO

1. PPO:
   • Function: An RL algorithm that optimizes the language model by limiting how far it can drift from a previous version of the model.
   • Implementation: Involves sampling generations from the current model, judging them with a reward model, and using this feedback for updates.
   • Practical Challenges: Can be slow and unstable, especially in distributed settings.
2. DPO:
   • Function: Minimizes the negative log-likelihood of observed human preferences to align the language model with human feedback.
   • Data Requirement: Requires paired preference data.
   • Comparison with KTO: While DPO has been effective, KTO offers competitive or superior performance without the need for paired preferences.
3. KTO:
   • Function: Adapts the Kahneman-Tversky human value function to the language model setting. It uses this adapted function to directly maximize the utility of model outputs.
   • Data Requirement: Does not need paired preference data, only knowledge of whether an output is desirable or undesirable for a given input.
   • Practicality: Easier to deploy in real-world scenarios where desirable/undesirable outcome data is more abundant.
   • Model Comparison: Matches or exceeds the performance of direct preference optimization methods across various model sizes (from 1B to 30B).
4. APO:
   • Function: Introduces a family of contrastive objectives explicitly accounting for the relationship between the model and the preference dataset. This includes APO-zero, which increases desirable outputs while decreasing undesirable ones, and APO-down, which fine-tunes models based on specific quality thresholds.
   • Data Requirement: Works effectively with paired preference datasets created through controlled methods like CLAIR and supports stable alignment even for challenging datasets.
   • Practicality: Excels at aligning strong models with minimally contrasting preferences, enhancing performance on challenging metrics like MixEval-Hard while providing stable, interpretable training dynamics.
   • Model Comparison: Outperformed conventional alignment objectives across multiple benchmarks, closing a 45% performance gap with GPT4-turbo when trained with CLAIR preferences.
5. GRPO:
   • Function: A variant of PPO that removes the need for a critic model by estimating the baseline using group scores, improving memory and computational efficiency while enhancing the mathematical reasoning of models.
   • Data Requirement: Utilizes group-based rewards computed from multiple outputs for each query, normalizing these scores to guide optimization.
   • Practicality: Focuses on reducing training resource consumption compared to PPO and improving RL stability.
   • Model Comparison: Demonstrated superior performance on tasks like GSM8K and MATH benchmarks, outperforming other models of similar scale while improving both in-domain and out-of-domain reasoning tasks.

Tabular Comparison

Reward Functions

• DeepSeek-R1 employs two primary reward functions to guide the RL process:

1. Accuracy Reward
   • Used for deterministic tasks like mathematics and coding.
   • The model’s final output is compared against ground-truth values.
   • In code-generation tasks, unit tests verify correctness.
2. Format Reward
   • Ensures consistency in reasoning structure.
   • The model is rewarded for maintaining the XML-style format:

     <reasoning_process> Step-by-step breakdown </reasoning_process>
     <answer> Final Output </answer>
     

   • A language consistency reward is also introduced to mitigate language mixing.

• These rewards encourage both correctness and structured reasoning, helping the model align with human expectations.

Stage 3: Rejection Sampling & Expanded Supervised Fine-Tuning

• After RL convergence, DeepSeek-R1 undergoes an additional fine-tuning step based on rejection sampling. This stage refines the reasoning process by incorporating:
  • Reasoning Trajectories: Selecting correct and well-structured CoT explanations from RL outputs.
  • Expanded Task Coverage: Augmenting the dataset with non-reasoning tasks like:
    • Writing & Summarization
    • Fact-based Question Answering
    • Self-cognition and safety-related responses
• The rejection sampling process filters out low-quality reasoning paths and ensures that the model maintains clarity, readability, and logical consistency.

Stage 4: Secondary Reinforcement Learning for Alignment & Generalization

• The final stage involves another round of RL, but this time with a broader task distribution. Unlike the first RL stage, which focused primarily on reasoning-intensive tasks, this stage incorporates general user interactions such as:
  • Conversational depth (multi-turn dialogues)
  • Complex instructions & role-playing scenarios
  • Ensuring helpfulness & harmlessness in responses

• For general tasks, a reward model is used to align outputs with human preferences. For reasoning tasks, the original rule-based rewards (accuracy & format) are retained.

• This final RL phase optimizes DeepSeek-R1 for real-world deployment, ensuring that it remains robust across a variety of domains beyond structured problem-solving.

  Emergent Reasoning Behaviors

• DeepSeek-R1 demonstrated remarkable emergent reasoning behaviors during its training process, particularly due to the RL approach that guided its self-evolution. These behaviors include:

  • Reflection: The model exhibits the ability to revisit and revise its intermediate steps. By analyzing prior outputs and reconsidering logical pathways, it refines its reasoning, ensuring a higher probability of correctness. This reflection is especially visible in long Chain-of-Thought (CoT) processes where multiple reasoning paths are explored.

  • Self-Correction: DeepSeek-R1 can detect errors in its own logical steps and apply corrective adjustments. This behavior is incentivized by reward modeling, where the model is trained to recognize inconsistencies and rerun calculations when necessary. This prevents incorrect conclusions from being solidified.

  • Aha Moments: Perhaps the most striking emergent behavior is the spontaneous “aha moment,” where DeepSeek-R1 halts its current reasoning trajectory, reevaluates the problem from a new angle, and finds a more optimal solution. This is often triggered by a discrepancy between expected and derived results, prompting the model to explore alternative pathways.

Implementation Details

• DeepSeek-R1’s reasoning behaviors emerged through a structured RL framework that included:

  1. Reward-Based Training: The model was incentivized to provide correct and structured solutions through accuracy and format rewards. This helped shape behaviors like reflection and self-correction.
  2. Policy Optimization: Using Group Relative Policy Optimization (GRPO), the model iteratively refined its reasoning processes based on feedback from sampled responses.
  3. Rejection Sampling: Intermediate outputs were filtered based on correctness, ensuring that only accurate and well-structured reasoning chains were reinforced.
  4. Cold Start Data: Unlike its predecessor, DeepSeek-R1-Zero, which purely relied on RL, DeepSeek-R1 was trained on curated long-form reasoning examples as a base, significantly improving its ability to structure logical steps coherently.

Example: Quadratic Equation Solving

• Consider the problem:

  \[x^2 - 5x + 6 = 0\]
  1. The model initially proposes an incorrect factorization.
  2. It pauses to reevaluate and notices an inconsistency in the calculated roots.
  3. Upon reflection, it correctly factors the equation and derives \(x = 2, x = 3\).

• This self-correcting behavior is illustrated in the table from the original paper:

Distillation: Reasoning in Compact Models

• DeepSeek-R1’s advanced reasoning capabilities were distilled into smaller models, including Qwen-7B and Llama-8B, through an optimized training pipeline designed to preserve reasoning depth while reducing computational complexity.

Implementation Details

1. Teacher-Student Paradigm:
   • DeepSeek-R1 was used as the “teacher” model.
   • The distilled models (e.g., Qwen-7B, Llama-8B) were fine-tuned on 800K reasoning-related samples generated by DeepSeek-R1.
2. Training Process:
   • Unlike RL-based training for DeepSeek-R1, distilled models were trained primarily using Supervised Fine-Tuning (SFT).
   • The dataset included:
     • 600K reasoning-based samples covering math, logical reasoning, and coding.
     • 200K general-purpose samples to ensure well-rounded performance.
3. Comparison Against RL Training:
   • Experiments showed that distilling reasoning behaviors from DeepSeek-R1 was significantly more effective than training smaller models from scratch using RL.
   • A direct RL-trained Qwen-32B model underperformed compared to the distilled DeepSeek-R1-Distill-Qwen-32B, highlighting the efficiency of distillation in preserving complex reasoning patterns.
4. Performance Metrics:
   • The table below showcases how distilled DeepSeek-R1 models compare against non-reasoning models like GPT-4o and larger models like OpenAI o1-mini.

Results

• The figure below from the original paper illustrates the performance of DeepSeek-R1 across multiple benchmarks, showing it is on par with or even surpassing OpenAI’s models in several areas:

  

  • Mathematical Reasoning: Achieved a 97.3% pass rate on MATH-500, outperforming previous open-source models.
  • Code Competitions: Placed in the 96.3rd percentile on Codeforces, equivalent to expert-level human competitors.
  • General Knowledge: Scored 90.8% on MMLU, demonstrating strong performance in broad knowledge domains.

• DeepSeek-R1 represents a major leap in the ability of LLMs to develop, refine, and transfer complex reasoning skills. Its RL-based self-evolution and highly effective distillation pipeline set a new standard for reasoning models, enabling smaller models to achieve state-of-the-art performance with minimal computational overhead.

Open Questions

• As shown in the figure below (source), making a powerful reasoning model is now very simple if you have access to a capable base model and a high-quality data mixture:

• Despite DeepSeek-R1’s advances, several open questions remain regarding its development and optimal implementation:

  • Data Collection: How were the reasoning-specific datasets curated? Understanding the sources and selection criteria for data is crucial for replicating and improving the model’s performance.
  • Model Training: No training code was released by DeepSeek, leaving uncertainty about which hyperparameters work best and how they differ across model families and scales.
  • Scaling Laws: What are the compute and data trade-offs in training reasoning models? Identifying these relationships is critical for optimizing future models.

Open-R1

• While DeepSeek-R1 provides open weights, the datasets and code used in training remain proprietary. The aforementioned questions have driven the Open-R1 project, an initiative to systematically reconstruct DeepSeek-R1’s data and training pipeline as open-source, validate its claims, and push the boundaries of open reasoning models.
• The motivation behind building Open-R1 is to provide transparency on how RL can enhance reasoning, share reproducible insights with the open-source community, and create a foundation for future models to leverage these techniques.

Objectives of Open-R1

1. Reproducing R1-Distill Models: By distilling a high-quality reasoning dataset from DeepSeek-R1, Open-R1 aims to replicate the R1-Distill models faithfully.
2. Replicating the RL Training Pipeline: A critical component of DeepSeek-R1 is its RL-based training methodology. Open-R1 will curate large-scale datasets for mathematics, reasoning, and code to enable this training process.
3. Advancing Multi-Stage Training: Demonstrating the full transition from a base model through SFT to RL will be a key milestone, ensuring a reproducible and scalable methodology.

• As shown in the figure below (source), here’s the Open-R1 plan:

Impact on the Community

• Accessible Reasoning Models: Open-R1’s synthetic datasets will allow anyone to fine-tune existing or new LLMs for reasoning tasks simply by leveraging these datasets.
• Open RL Recipes: The initiative will provide well-documented RL methodologies that can serve as a foundation for future research and experimentation.
• Exploring Beyond Math: While mathematical reasoning is a primary focus, Open-R1 will explore extensions into other domains, including programming and scientific applications such as medicine, where reasoning models can make a significant impact.

Reasoning Datasets

1. OpenThoughts: 114k samples distilled from R1 on math, code, and science.
2. R1-Distill-SFT: 1.7M samples distilled from R1-32B on NuminaMath and Allen AI’s Tulu.

References

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeek-R1: A Pure RL-based Reasoning Model
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• DeepSeek-V3 Technical Report
• Open-R1: a fully open reproduction of DeepSeek-R1