Overview

  • Gemma 4 is an open-weight, multimodal model family built around four deployment tiers: E2B, E4B, 31B dense, and 26B A4B MoE. It supports text and image input across the family, audio on E2B and E4B, text output, pre-trained and instruction-tuned checkpoints, 128K context on smaller models, and 256K context on larger models.

  • At a high level, Gemma 4 combines a mostly standard decoder-only Transformer lineage from Attention Is All You Need by Vaswani et al. (2017), which introduced self-attention as the core sequence modeling primitive, with long-context and deployment-oriented modifications: alternating local and global attention, RoPE-derived positional encoding, shared KV caching, Per-Layer Embeddings in small models, and an MoE variant.

Model family and deployment shape

  • Gemma 4 is organized as a practical deployment ladder rather than a single flagship model. E2B and E4B target ultra-mobile, edge, browser, and laptop-style use; 31B dense targets stronger local or workstation inference; 26B A4B uses sparse activation so only about 4B parameters are active per token while the full parameter set remains loaded.

  • The dense variants are E2B with 35 layers, E4B with 42 layers, and 31B with 60 layers. The smaller dense models use 512-token sliding windows and 128K context, while 31B uses a 1024-token sliding window and 256K context. The 26B A4B MoE has 30 layers, a 1024-token sliding window, 256K context, 128 total experts, 8 active experts, and 1 shared expert.

  • The following tables shows the Gemma 4 model variants including their parameter sizes and context window lengths.

Model Parameter Size Context Window
Gemma 4 E2B 2.3 B effective, 5.1 B with embeddings 128k
Gemma 4 E4B 4.5 B effective, 8B with embeddings 128k
Gemma 4 31B 31B dense model 256K
Gemma 4 26B A4B mixture-of-experts with 4B activated/26B total parameters 256 K
  • The following figure (source) shows a plot of Gemma 4 model performance v/s size for Gemma 4 31B Thinking and Gemma 4 26B A4B Thinking.

Capability surface

  • The family is designed for reasoning, coding, function calling, long-context processing, multilingual text, image understanding, video-through-frame processing, and audio understanding on small variants. Its image capabilities include OCR, document and PDF parsing, screen and UI understanding, chart comprehension, handwriting recognition, object detection, and pointing.

  • The multimodal stack is not just an add-on: all models accept image/video plus text inputs, while E2B and E4B additionally support audio. The image encoder preserves variable aspect ratios and lets implementations choose among image token budgets such as 70, 140, 280, 560, and 1120, which creates an explicit speed-memory-quality knob.

Architectural through-line

  • Gemma 4’s central design goal is high capability per deployed byte. It uses local sliding-window attention to avoid making every layer pay full-context attention cost, then inserts global attention layers to maintain long-range awareness. This follows the general efficiency motivation behind Longformer by Beltagy et al. (2020), which showed how local windowed attention plus global attention can reduce long-sequence attention cost.

  • For positional encoding, Gemma 4 uses dual RoPE-style configurations, with standard RoPE in sliding layers and pruned or proportional RoPE in global layers to reduce positional noise over long context. This builds on RoFormer by Su et al. (2021), which introduced Rotary Position Embedding as a way to encode relative position through rotations applied to query and key vectors.

  • For sparse scaling, the 26B A4B model uses a Mixture-of-Experts design where routing activates only a subset of expert feed-forward blocks per token, following the broad conditional-computation idea from Outrageously Large Neural Networks by Shazeer et al. (2017), which introduced sparsely gated expert layers to increase capacity without activating all parameters every step.

  • A useful mental model is:
\[\text{Gemma 4 efficiency} \approx \text{local attention} + \text{periodic global attention} + \text{KV reuse} + \text{sparse or effective parameters}\]
  • and:
\[\text{Gemma 4 capability} \approx \text{long context} + \text{multimodal encoders} + \text{reasoning tuning} + \text{agentic interfaces}\]

Relationship to Gemma 3

  • Gemma 4 extends ideas already visible in Gemma 3 Technical Report by Gemma Team et al. (2025), which emphasized multimodality, longer context, and reduced KV-cache memory through a higher ratio of local to global attention layers. Gemma 4 keeps that long-context efficiency direction while adding a broader model ladder, shared KV cache, Per-Layer Embeddings for smaller models, stronger multimodal deployment support, and a 26B sparse MoE option.

Core Architecture

  • Gemma 4 largely follows the decoder-only Transformer paradigm, but modifies nearly every subsystem to optimize for long context, multimodality, and deployment efficiency. The resulting architecture is best understood as a layered combination of attention innovations, positional encoding strategies, parameter efficiency mechanisms, and modality-specific encoders.

Transformer backbone and residual structure

  • At its core, Gemma 4 uses a standard causal decoder stack derived from the Transformer introduced in Attention Is All You Need by Vaswani et al. (2017), where each layer consists of:

    • Multi-head self-attention
    • Feed-forward network (FFN)
    • Residual connections
    • Normalization layers

  • The forward pass for a single layer can be abstracted as:

\[h^{(l+1)} = h^{(l)} + \text{FFN}\Big(h^{(l)} + \text{Attn}(h^{(l)})\Big)\]

  • This residual stacking allows the model to iteratively refine token representations while maintaining gradient stability across deep networks.

  • Modern LLMs, including Gemma, typically use RMSNorm instead of LayerNorm, following trends discussed in Root Mean Square Layer Normalization by Zhang and Sennrich (2019), which reduces parameter overhead and improves efficiency.

Hybrid attention: local + global

  • One of the defining architectural features of Gemma 4 is its hybrid attention mechanism, which interleaves:

    • Sliding window (local) attention
    • Full (global) attention

  • This design is explicitly stated in the Gemma 4 model card, where attention layers alternate, ensuring the final layer is always global.

  • The following figure (source) shows Gemma 3 27B and Gemma 4 31B side by side, highlighting Gemma 4’s hybrid local/global attention, GQA, p-RoPE, and KV-sharing changes.

Local attention

  • Local attention restricts each token to attend only to nearby tokens within a window of size \(w\). Formally:
\[\text{Attn}(q_i) = \sum_{j=i-w}^{i} \alpha_{ij} v_j\]

  • This reduces complexity from:

    \[O(n^2) \rightarrow O(n \cdot w)\]
    • which is critical for long-context efficiency.

  • This idea builds directly on Longformer by Beltagy et al. (2020), where sparse attention patterns enable scaling to long documents.

Global attention

  • Global attention layers restore full sequence awareness:
\[\text{Attn}(q_i) = \sum_{j=1}^{n} \alpha_{ij} v_j\]

  • By interleaving these two types, Gemma 4 balances:

    • Efficiency from local attention
    • Expressivity from global attention

  • The ratio differs by model size. Smaller models use tighter windows (512 tokens), while larger models expand to 1024 tokens.

Positional encoding: dual RoPE variants

  • Gemma 4 uses Rotary Position Embeddings (RoPE), introduced in RoFormer by Su et al. (2021), which encode positional information via rotation in query and key space:

    \[\text{RoPE}(x) = x \cdot e^{i \theta}\]
    • where \(\theta\) depends on position.

  • Gemma 4 extends this with two configurations:

    • Standard RoPE for local layers
    • Modified (pruned or proportional) RoPE for global layers

  • The motivation is subtle but important:

    • Local layers benefit from precise relative position encoding
    • Global layers must avoid positional drift over long contexts

  • This dual setup allows stable attention over sequences as long as 256K tokens.

Feed-forward networks and scaling behavior

  • Each Transformer block includes a feed-forward network (FFN), typically implemented as:

    \[\text{FFN}(x) = \text{SwiGLU}(xW_1) W_2\]

  • In Gemma 4:

    • Dense models use standard FFNs
    • MoE models replace FFNs with expert routing

Mixture-of-Experts (MoE) variant

  • The 26B A4B model introduces sparse computation via Mixture-of-Experts.

  • Instead of a single FFN, each layer contains multiple experts:

    \[\text{FFN}(x) = \sum_{k \in \text{Top-K}(x)} g_k(x) \cdot \text{Expert}_k(x)\]

    • where:

      • \(g_k(x)\) is the routing weight
      • Only a subset (e.g., 8 experts) is active per token

  • This follows the paradigm introduced in Outrageously Large Neural Networks by Shazeer et al. (2017), enabling:

    • Large total parameter count (26B)
    • Small active parameter count (~4B per token)

  • The result is near-dense performance at much lower inference cost.

KV cache and long-context scaling

  • Like most autoregressive transformers, Gemma 4 uses a KV cache:
\[\text{Cache} = \{K_1, V_1, ..., K_t, V_t\}\]

  • This avoids recomputing past attention states during generation.

  • However, KV cache size scales as:

    \[O(n \cdot d \cdot L)\]

    • where:

      • \(n\) = sequence length
      • \(d\) = hidden dimension
      • \(L\) = number of layers

  • Gemma 4 addresses this scaling problem through:

    • Sliding window attention (limits KV growth)
    • Shared KV cache (discussed later in architectural tricks section)

Multimodal encoders

  • Gemma 4 integrates modality-specific encoders into the token stream.

Vision encoder

  • ~150M parameters (small models), ~550M (large)
  • Supports variable resolution and aspect ratios

  • Uses 2D positional encoding + RoPE variants

  • This follows trends from Vision Transformer (ViT) by Dosovitskiy et al. (2020), which treats image patches as tokens.

Audio encoder

  • Conformer-based architecture
  • Inspired by speech models like Conformer by Gulati et al. (2020), which combines convolution and attention for audio

Gemma 4 pipeline

  • A simplified view of the Gemma 4 forward pipeline:

    • Input tokens (text, image patches, audio features)
    • Token embeddings
    • Optional Per-Layer Embeddings (small models)
    • Transformer stack with:
    • Alternating local/global attention
    • RoPE positional encoding
    • Dense or MoE FFN layers
    • Output projection to vocabulary logits

Key takeaways

  • Gemma 4 is not a radical departure from Transformers but a carefully engineered evolution
  • Its strength comes from combining multiple efficiency techniques:
    • Hybrid attention
    • KV cache optimizations
    • Sparse MoE scaling
  • The architecture is explicitly designed for long context and real-world deployment constraints

Unique Architectural Tricks in Gemma 4

  • Among modern open-weight language models, Gemma 4 stands out not because it replaces the Transformer, but because it introduces two unusually practical architectural innovations aimed at maximizing quality under tight deployment constraints:

    • Per-Layer Embeddings (PLE)
    • Shared KV Cache

  • These mechanisms are especially important for on-device inference, long-context generation, improved parameter efficiency, and reduced KV-cache memory usage. Both techniques first appeared in earlier Google research and Gemma-3n, but Gemma 4 integrates them into the production-ready multimodal model family described in

Per-Layer Embeddings (PLE)

  • Per-Layer Embeddings are a parameter-efficiency mechanism used in the E2B and E4B models, but not in the 31B dense or 26B A4B MoE variants.

  • The central idea is simple: instead of giving each token only one embedding vector at the input layer, Gemma 4 provides each decoder layer with its own small token-specific embedding.

Why standard embeddings are limiting

  • In a conventional Transformer, every token receives one embedding:
\[e_t \in \mathbb{R}^{d_{\text{model}}}\]

  • This same representation is reused throughout all layers. As a result, the initial embedding must encode all information that may be needed by every layer.

  • PLE relaxes this constraint by introducing layer-specific side information:

\[p_t^{(l)} \in \mathbb{R}^{d_{\text{PLE}}}\]
  • for every token \(t\) and layer \(l\), where:
\[d_{\text{PLE}} \ll d_{\text{model}}\]

Constructing the PLE vectors

  • Each PLE vector combines two components.

  • Token identity component: A learned embedding lookup specific to each layer:

    \[u_t^{(l)} = E_{\text{PLE}}^{(l)}[\text{token}_t]\]
    • This acts like a miniature embedding table for every layer.

  • Context-aware component: The original token embedding is projected into a tensor containing layer-specific slices:

    \[c_t^{(l)} = W_{\text{ctx}} e_t\]

    • reshaped into:

      \[[T, L, d_{\text{PLE}}]\]
      • where \(T\) is sequence length and \(L\) is the number of layers.

    • The combined PLE representation is:

    \[p_t^{(l)} = u_t^{(l)} + c_t^{(l)}\]
    • This creates a dedicated low-dimensional conditioning vector for each token at every layer.

Gated injection into the residual stream

  • PLE is not simply added directly. Each layer computes a gate based on the current hidden state \(h_t^{(l)}\):
\[g_t^{(l)} = \text{GELU}(W_{\text{gate}} h_t^{(l)})\]
  • The gated PLE signal is then:
\[r_t^{(l)} = W_{\text{up}}\left(g_t^{(l)} \odot p_t^{(l)}\right)\]
  • and injected into the residual stream:
\[h_t^{(l)} \leftarrow h_t^{(l)} + r_t^{(l)}\]
  • This lets the model decide dynamically whether to use layer-specific token information, how strongly to apply it, and which dimensions are relevant.

Why PLE is effective

  • PLE increases model capacity without proportionally increasing compute.

  • Capacity gain through lookup-heavy parameters: The additional parameters mostly reside in embedding tables, which require indexed lookups rather than expensive matrix multiplications.

  • Layer-specific token conditioning: Each layer can receive token information specialized for that layer instead of relying only on the original input embedding.

  • Effective-parameter accounting: This is why E2B is described as having 2.3B effective parameters despite requiring memory equivalent to roughly 5.1B parameters with embeddings.

Shared KV Cache

  • The second major innovation is Shared KV Cache, which reduces memory and compute in later Transformer layers.

Standard KV projection

  • Normally, every attention layer computes its own:
\[K^{(l)} = X W_K^{(l)}, \qquad V^{(l)} = X W_V^{(l)}\]

  • These tensors are stored for every token and every layer.

  • Total KV memory scales approximately as:

    \[O(L \cdot n \cdot d)\]
    • where \(L\) is the number of layers, \(n\) is context length, and \(d\) is KV dimension.

KV reuse across layers

  • Gemma 4 introduces a hyperparameter: num_kv_shared_layers

  • The final \(N\) layers skip their own KV projections and instead reuse the cache from earlier layers of the same attention type, local or global.

  • Formally, if layer \(l\) shares KV with reference layer \(r\):

\[K^{(l)} = K^{(r)}, \qquad V^{(l)} = V^{(r)}\]
  • Only queries remain unique:
\[Q^{(l)} = X W_Q^{(l)}\]
  • Thus attention becomes:
\[\text{Attn}^{(l)}(Q^{(l)}, K^{(r)}, V^{(r)})\]

Attention-type-aware sharing

  • Because Gemma 4 alternates between sliding-window attention and global attention, KV reuse occurs only within the same type.

  • Global-layer reuse: Global attention layers reuse KV tensors from the most recent suitable global layer.

  • Local-layer reuse: Sliding-window attention layers reuse KV tensors from the most recent suitable local layer.

  • This preserves compatibility with differing attention masks and positional schemes.

Benefits of Shared KV Cache

  • Lower memory usage for long contexts: KV cache can dominate memory at 128K-256K contexts, so avoiding redundant KV storage is especially valuable.

  • Fewer projection operations: Later shared layers do not need their own key and value projection matrix multiplications.

  • Faster autoregressive decoding: Reusing KV tensors reduces repeated per-token work during generation.

  • Better edge-device feasibility: Lower runtime memory and compute pressure make long-context inference more realistic on constrained hardware.

Why quality is preserved

  • Adjacent upper layers frequently learn similar key-value representations. By allowing only query projections to differ, later layers can still interpret the same memory differently.

  • This resembles a shared memory bank with multiple independent readers:

  • Shared \(K\) and \(V\): Common memory contents.

  • Unique \(Q\): Layer-specific retrieval strategy.

Combined impact

  • PLE and Shared KV Cache address opposite sides of the efficiency equation.

  • Per-Layer Embeddings (PLE): PLE increases representational capacity by giving each decoder layer its own small token-specific conditioning signal. It does not directly reduce memory, but it can improve parameter efficiency because many added parameters are embedding lookups rather than expensive per-token matrix operations.

  • Shared KV Cache: Shared KV Cache does not primarily increase model capacity. Instead, it reduces inference memory and compute by letting later layers reuse previously computed key and value tensors from earlier layers of the same attention type.

  • Together they allow Gemma 4 to:

    • Support 128K-256K context more efficiently.
    • Improve smaller-model quality without simply adding more dense layers.
    • Reduce redundant KV projection work during generation.
    • Run more practically on consumer, edge, and mobile-class hardware.

Architectural significance

  • Most LLM improvements focus on larger datasets or more compute. Gemma 4 instead introduces structural mechanisms that directly improve deployment economics.

  • PLE changes how information enters each layer, while Shared KV Cache changes how layers access historical context. These are orthogonal innovations that can be integrated into many Transformer-based systems.

  • For practitioners, they represent some of the most practically useful architectural ideas introduced in open models in recent years.

Long-Context Mechanisms and Hybrid Attention

  • One of Gemma 4’s defining characteristics is its ability to process extremely long sequences while remaining deployable on commodity hardware. The E2B and E4B models support context windows of up to 128K tokens, while the 31B dense and 26B A4B MoE models extend this to 256K tokens. Gemma 4 model card. These context lengths place Gemma 4 among the longest-context open-weight models available, while maintaining strong reasoning and coding performance.

  • Achieving this scale requires several coordinated architectural choices:

    • Alternating sliding-window and global attention layers.
    • Dual RoPE configurations specialized for local and global attention.
    • Shared KV Cache to reduce memory overhead.
    • Large vocabulary and optimized tokenization.
    • Careful attention scheduling so the final layer is always global.

  • Together, these mechanisms allow the model to approximate full-context reasoning without paying the full quadratic cost in every layer.

Why long context is difficult

  • In a standard Transformer, self-attention over a sequence of length \(n\) requires computing an \(n \times n\) attention matrix:
\[A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)\]

  • The computational complexity is \(O(n^2 d)\) and the memory complexity is \(O(n^2)\) during training, while autoregressive inference additionally requires storing key and value tensors for every token and layer.

  • For long contexts such as 256K tokens, naive full attention becomes prohibitively expensive. This challenge motivated a large body of research, including Longformer by Beltagy et al. (2020), which introduced sparse local-global attention patterns, and Transformer-XL by Dai et al. (2019), which extended effective context using recurrence.

Sliding-window attention

  • Gemma 4 uses sliding-window attention in most layers.

  • In sliding-window attention, token \(i\) attends only to nearby tokens within a fixed window of size \(w\):

\[j \in [i-w, i]\]

  • The attention computation becomes:

    \[\text{Attn}(q_i) = \sum_{j=i-w}^{i} \alpha_{ij} v_j\]

    • where:

      \[\alpha_{ij} = \text{softmax}\left( \frac{q_i^\top k_j}{\sqrt{d_k}} \right)\]

  • This reduces complexity from:

    \[O(n^2) \rightarrow O(nw)\]

  • When \(w \ll n\), the savings are dramatic.

Window sizes by model family

  • E2B and E4B: 512-token sliding window.

  • 31B dense and 26B A4B: 1024-token sliding window.

  • These windows are large enough to capture sentence- and paragraph-level dependencies while keeping memory usage manageable.

Global attention layers

  • Local attention alone cannot directly connect distant tokens. To preserve long-range reasoning, Gemma 4 periodically inserts full global attention layers.

  • In these layers, every token attends to the entire context:

\[j \in [1, n]\]

  • Global layers act as synchronization points that propagate information gathered through local layers across the entire sequence.

  • This design mirrors multigrid or message-passing systems:

    • Local layers perform dense short-range processing.
    • Global layers aggregate and redistribute long-range information.

  • The model card notes that the final Transformer layer is always global, ensuring that output logits are computed from representations with full-context visibility.

Information propagation intuition

  • Suppose there are five local layers between two global layers and the local window size is \(w\).

  • A token’s information can influence approximately \(5w\) tokens away before the next global layer.

  • After the global layer, this information becomes available to all tokens. Repeating this pattern allows efficient diffusion of information across very long contexts.

Dual RoPE configurations

  • Gemma 4 uses two variants of Rotary Position Embeddings (RoPE), introduced in RoFormer by Su et al. (2021).

  • Standard RoPE applies a position-dependent rotation to each two-dimensional subspace of the query and key vectors:

\[R(\theta_p) = \begin{bmatrix} \cos \theta_p & -\sin \theta_p \\ \sin \theta_p & \cos \theta_p \end{bmatrix}\]

  • This encodes relative positions directly into dot products.

  • Gemma 4 uses:

    • Standard RoPE in sliding-window layers.

    • Proportional or pruned RoPE (p-RoPE) in global layers.

  • The goal is to maintain stable positional behavior over 128K-256K tokens, where standard extrapolation can degrade.

Proportional RoPE intuition

  • In standard RoPE, angular frequency grows linearly with position. Over very long sequences, high-frequency components can rotate excessively, making distant tokens harder to compare.

  • p-RoPE rescales or selectively prunes frequencies so that effective rotation grows more slowly:

\[\theta_p' = \alpha \theta_p \qquad \text{with } 0 < \alpha < 1\]
  • This preserves relative ordering while reducing phase distortion at long range.

KV cache scaling and long-context memory

  • During autoregressive decoding, each layer stores keys and values for all prior tokens:
\[\{K_1, V_1, \ldots, K_n, V_n\}\]

  • Approximate memory scales as:

    \[O(Lnd_{\text{kv}})\]

    • where:

      • \(L\) is the number of layers.
      • \(n\) is sequence length.
      • \(d_{\text{kv}}\) is the key/value dimension.

  • At 256K tokens, the KV cache can consume more memory than the model weights themselves.

  • Gemma 4 mitigates this with:

    • Sliding-window attention, which bounds active KV usage for local layers.
    • Shared KV Cache, which reduces duplicated KV tensors across upper layers.

Attention schedule and final global layer

  • Gemma 4’s attention pattern is carefully structured.

    • Most layers are local: This minimizes per-layer cost.

    • Periodic global layers: These restore full-sequence connectivity.

    • Last layer is global: Final representations incorporate information from the entire context.

  • This final global layer is especially important for tasks such as:

    • Retrieval-augmented generation.
    • Long document summarization.
    • Multi-hop reasoning.
    • Codebase understanding.

Practical implications

  • The long-context design enables workflows that would be impractical in shorter-context models. Some of them are listed below:

    • Whole-repository reasoning: Large codebases can be processed in a single prompt.

    • Book-length analysis: Entire books, contracts, or technical manuals fit in context.

    • Multimodal conversations: Large sets of images and text can be jointly analyzed.

    • Agent memory: Autonomous systems can maintain long interaction histories.

Architectural significance

  • Gemma 4 demonstrates that long-context capability is not solely a matter of increasing context limits in training. It requires an integrated design that coordinates sparse attention, positional encoding, and memory optimizations.

  • The central insight is that most reasoning is local most of the time. By reserving expensive global computation for strategically placed layers, Gemma 4 achieves near-full-context awareness at a fraction of the cost of a fully global Transformer.

Multimodal Architecture

  • Gemma 4 is designed as a natively multimodal model family rather than a text model with auxiliary encoders bolted on afterward. Every model in the family supports text and image inputs, while the E2B and E4B variants additionally support audio. Video is handled by converting frames, and optionally audio, into the same multimodal token stream. Gemma 4 model card and

  • The overall design follows a now-standard pattern:

    • Specialized encoders convert raw modalities into dense feature vectors.
    • Feature vectors are projected into the language model embedding space.
    • Placeholder tokens in the text sequence are replaced with these projected vectors.
    • The decoder processes a single unified sequence.
    • Text tokens are generated autoregressively.

  • This architectural template builds on work such as Flamingo by Alayrac et al. (2022), which introduced interleaved multimodal conditioning, and PaLI by Chen et al. (2022), which unified multilingual vision-language modeling at scale.

Unified multimodal token stream

  • Once modality-specific features are projected into the decoder hidden dimension, all inputs are treated as ordinary sequence elements:

    \[X = [x_1, x_2, \ldots, x_T] \in \mathbb{R}^{T \times d_{\text{model}}}\]

    • where each \(x_i\) may correspond to:

      • A text token embedding.
      • An image patch embedding.
      • An audio frame embedding.
      • A video frame embedding.
      • A special control token.

  • After this point, the decoder is modality-agnostic. Attention layers operate on all elements uniformly, allowing the model to reason jointly across text, visual, and auditory information.

Vision encoder

  • All Gemma 4 models include a vision encoder, with the vision encoder sizes as follows:

    • E2B and E4B: Approximately 150M parameters.

    • 31B and 26B A4B: Approximately 550M parameters.

  • The encoder preserves original image aspect ratios and supports variable input resolutions, avoiding the distortions introduced by fixed-size resizing.

Architectural foundations

  • The encoder is similar to the one used in Gemma 3 and follows the general patch-based paradigm introduced in Vision Transformer (ViT) by Dosovitskiy et al. (2020), which processes images as token sequences rather than convolutional feature maps.

  • Gemma 4 augments this design with:

    • Learned two-dimensional positional embeddings.
    • Multidimensional RoPE.
    • Variable aspect-ratio handling.
    • Configurable token budgets.

Image token budgets

  • Implementations can select among several token budgets:

    • 70 tokens: Minimal latency and memory use.

    • 140 tokens: Lightweight visual understanding.

    • 280 tokens: Balanced quality and speed.

    • 560 tokens: Stronger fine-grained perception.

    • 1120 tokens: Maximum fidelity.

  • This creates a practical trade-off between:

    • Visual detail retention.
    • Context consumption.
    • Compute cost.
    • Inference speed.

Image token projection

  • If the vision encoder produces feature matrix \(V \in \mathbb{R}^{m \times d_v}\), these features are projected into decoder space \(Z = V W_{\text{proj}}\):

    • where \(W_{\text{proj}} \in \mathbb{R}^{d_v \times d_{\text{model}}}\) and \(Z \in \mathbb{R}^{m \times d_{\text{model}}}\).

  • The resulting tokens replace placeholder positions in the text sequence.

Audio encoder

  • The E2B and E4B models include a dedicated audio encoder based on a USM-style Conformer architecture.

  • This design follows Conformer by Gulati et al. (2020), which combines self-attention with convolution to capture both local acoustic structure and long-range dependencies.

  • The audio encoder contains approximately 300M parameters.

  • It enables:

    • Speech transcription.
    • Audio understanding.
    • Spoken question answering.
    • Video with synchronized audio.

Conformer intuition

  • A Conformer layer integrates three major components:

    • Self-attention for long-range temporal context.
    • Convolution for local spectral patterns.
    • Feed-forward blocks for nonlinear transformation.

  • For acoustic input sequence \(A\):

\[H = \text{Conformer}(A)\]
  • The resulting feature vectors are projected into the language model embedding space in the same way as image tokens.

Video understanding

  • Video inputs are represented as ordered frame sequences, with optional audio features on smaller models.

  • The model is not described as explicitly post-trained on video-specific tasks, but because frames and audio are processed in temporal order, the decoder can reason over:

    • Actions and events.
    • Temporal progression.
    • Scene changes.
    • Speech and visual context.

  • This capability emerges naturally from the unified token stream.

  • The following figure (source) shows Gemma 4 video understanding examples across model sizes, including audio support on the smaller variants.

Multimodal placeholder replacement

  • A typical prompt may contain a special placeholder token such as: <image>

  • At preprocessing time:

    • The placeholder token is identified.
    • The corresponding encoder produces feature vectors.
    • The placeholder is replaced by projected modality tokens.
    • The resulting sequence is passed to the decoder.

  • For example: Describe the chart in <image>. becomes Describe the chart in [v_1, v_2, ..., v_m].

Interaction with Per-Layer Embeddings

  • For models using PLE, the per-layer signals are computed before multimodal token replacement because PLE relies on token IDs.

  • Once modality features replace placeholder tokens, there is no direct token identity. Multimodal positions therefore use the pad token ID, producing effectively neutral per-layer embeddings.

Cross-modal reasoning through self-attention

  • Because all modalities share the same sequence, standard self-attention enables direct interactions.

  • A text token can attend to visual regions:

    \[\alpha_{ij} = \text{softmax} \left( \frac{q_i^\top k_j}{\sqrt{d_k}} \right)\]
    • where \(j\) may correspond to an image or audio token.

  • This unified attention mechanism supports tasks such as:

    • OCR and document extraction.
    • Chart and diagram interpretation.
    • GUI understanding.
    • Object detection and pointing.
    • Audio transcription.
    • Multimodal chain-of-thought reasoning.

Function calling and structured outputs

  • Gemma 4 instruction-tuned models natively support function calling and structured outputs across both text-only and multimodal prompts.

  • This allows applications such as:

    • Returning bounding boxes in JSON.
    • Producing API call arguments.
    • Extracting structured document fields.
    • Triggering downstream tools.

  • The following figure (source) shows Gemma 4 performing GUI element detection and returning a structured bounding box for the “view recipe” element. An image is fed and the following text prompt is used: “What’s the bounding box for the “view recipe” element in the image?”

Architectural significance

  • Gemma 4’s multimodal design emphasizes simplicity and compatibility.

  • Instead of using modality-specific reasoning modules, it converts all modalities into a common embedding space and relies on the decoder’s existing self-attention and reasoning abilities. This preserves the elegance of the Transformer while extending it to images, audio, and video with minimal architectural fragmentation.

Mixture-of-Experts Architecture

  • The 26B A4B variant of Gemma 4 is the family’s sparse model. It contains approximately 25.2B total parameters, but activates only about 3.8B to 4B parameters per token during inference. Gemma 4 model card. The name “26B A4B” reflects this distinction: “26B” denotes total parameters, while “A4B” denotes the number of active parameters per token.

  • This architecture enables the model to approach the quality of much larger dense models while maintaining throughput closer to a 4B-parameter model. In practice, the 26B A4B model performs near the 31B dense model on reasoning, coding, and multimodal benchmarks, but at substantially lower per-token compute cost.

Why sparse models matter

  • In dense Transformers, every parameter participates in every forward pass. If the feed-forward network contains most of the model’s parameters, all of those weights are used for every token regardless of whether they are relevant.

  • Let \(y = \text{FFN}(x)\) where the same network is applied to all tokens. Mixture-of-Experts (MoE) replaces this single FFN with a collection of specialized feed-forward networks and a router that selects only a subset for each token.

  • This idea was introduced in Outrageously Large Neural Networks by Shazeer et al. (2017), which showed that conditional computation can dramatically increase model capacity without proportional inference cost.

Expert structure

  • Each expert is an independent feed-forward block, typically using a SwiGLU-based architecture similar to the dense FFNs used elsewhere in the model.

  • If there are \(E\) experts, then:

\[\{\text{Expert}_1, \text{Expert}_2, \ldots, \text{Expert}_E\}\]

  • In Gemma 4 26B A4B:

    • 128 total experts are available in each MoE layer.

    • 8 routed experts are selected for each token.

    • 1 shared expert is always active.

  • Thus, every token is processed by nine experts in total, though only eight are chosen dynamically. Gemma 4 model card.

Router mechanism

  • The router computes a score for each expert:

    \[s = W_r x\]

    • where:

      \[s \in \mathbb{R}^{E}\]

  • The top-\(k\) experts are selected:

    \[\mathcal{S}(x) = \text{TopK}(s, k)\]
    • with \(k = 8\) in Gemma 4.

  • Routing weights are normalized and used to combine expert outputs:

    \[y = \sum_{e \in \mathcal{S}(x)} g_e(x)\,\text{Expert}_e(x) + \text{Expert}_{\text{shared}}(x)\]
    • where \(g_e(x)\) are learned token-dependent mixture coefficients.

Shared expert

  • Gemma 4 includes one shared expert that is active for every token.

  • This design follows the same general idea used in DeepSpeed-MoE by Rajbhandari et al. (2022) and DeepSeekMoE by Dai et al. (2024), where a common expert captures broadly useful transformations while routed experts specialize.

  • The shared expert serves several purposes.

    • Common pattern handling: Frequently needed computations do not need to be duplicated across many experts.

    • Improved specialization: Routed experts can focus on niche behaviors.

    • Training stability: The model retains a consistent transformation path for all tokens.

Active versus total parameters

  • A key distinction in MoE systems is between total and active parameters.

    • Total parameters: All weights stored in memory.

    • Active parameters: Only the subset used for a specific token.

  • For Gemma 4 26B A4B:

    • Total parameters: Approximately 25.2B.

    • Active parameters: Approximately 3.8B to 4B.

  • This implies:

    • Model capacity scales with total parameters.
    • Per-token compute scales with active parameters.

  • A useful approximation is \(\text{Compute Cost} \propto \text{Active Parameters}\) rather than total parameters.

Memory implications

  • Although only a subset of experts is used per token, all expert weights must remain loaded in memory to allow fast routing and execution.

  • As a result:

    • Inference compute resembles a 4B model.
    • Weight memory resembles a 26B model.

  • This is why the model requires substantially more VRAM than a true 4B dense model even though its token throughput is much closer to that size.

Expert specialization

  • During training, different experts naturally specialize to distinct patterns, domains, or reasoning styles.

  • Examples of emergent specialization may include:

    • Programming languages.
    • Mathematical reasoning.
    • Multilingual processing.
    • Factual recall.
    • Structured outputs.

  • The router learns to direct tokens to the most relevant experts.

Load balancing

  • MoE models require auxiliary training objectives to prevent a few experts from receiving all traffic.

  • A typical load-balancing loss has the form:

    \[\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \mathcal{L}_{\text{balance}}\]

    • where:

      • \(\mathcal{L}_{\text{task}}\) is the standard language-model objective.
      • \(\mathcal{L}_{\text{balance}}\) encourages uniform expert utilization.

  • Although Gemma 4 training details are not fully disclosed, some variant of this mechanism is almost certainly used.

Comparison with the 31B dense model

  • The 31B dense model and 26B A4B MoE represent two different scaling philosophies:

    • 31B dense model: All parameters are used for every token, providing uniform computation and simpler execution.

    • 26B A4B MoE model: A much larger parameter pool is available, but only a small subset is activated per token.

  • The dense model offers implementation simplicity, while the MoE model delivers a superior capacity-to-compute ratio.

Why MoE is attractive for deployment

  • For organizations running large-scale inference, MoE offers several advantages.

    • Higher capacity at fixed compute budget.

    • Improved throughput relative to dense models of similar quality.

    • Better specialization and reasoning diversity.

    • Competitive performance on coding and agentic tasks.

  • This makes the 26B A4B model especially appealing when sufficient memory is available but per-token latency and cost remain important.

Architectural significance

  • The 26B A4B model demonstrates how Gemma 4 scales beyond dense architectures without abandoning deployment efficiency.

  • By combining hybrid attention, long-context optimizations, and sparse expert routing, it achieves near-31B dense quality while activating only a small fraction of its total parameters per token. This places it among the most practical open-weight sparse models for advanced reasoning and multimodal applications.

Reasoning and Thinking Modes

  • One of the most important shifts in Gemma 4 relative to earlier generations is that the entire model family is designed as a reasoning-first system. Rather than treating chain-of-thought and multi-step problem solving as emergent side effects, Gemma 4 is explicitly trained and instruction-tuned to support configurable “thinking modes” that improve performance on mathematics, coding, planning, and tool use. Gemma 4 model overview.

  • This design aligns with a broader trend in modern LLM development, where models are optimized not only to predict the next token accurately, but also to allocate additional computation to intermediate reasoning. This paradigm was popularized by Chain-of-Thought Prompting by Wei et al. (2022), which demonstrated that explicitly generating intermediate steps can dramatically improve complex reasoning performance.

What thinking mode means

  • In standard inference, the model directly generates a final answer:

    \[p(y \mid x)\]

    • where:

      • \(x\) is the input prompt.
      • \(y\) is the output sequence.

  • In thinking mode, the model first produces latent or visible intermediate reasoning tokens \(z\):

    \[p(y \mid x) = \sum_z p(z \mid x)\, p(y \mid x, z)\]

    • where:

      • \(z\) represents multi-step reasoning traces.
      • \(y\) is conditioned on the generated reasoning.

  • In practice, the model autoregressively emits reasoning tokens before the final answer, effectively allocating more test-time compute to difficult tasks.

Test-time compute as adaptive inference

  • Thinking mode converts inference into a variable-compute process, as shown below:

    • Easy questions: Require few reasoning tokens.

    • Hard problems: Generate long intermediate traces.

    • Tool-oriented tasks: Use reasoning to decide whether and how to call functions.

  • This is analogous to iterative optimization, where more compute is devoted to more complex inputs.

Training signals for reasoning

  • Although Google has not disclosed the complete reasoning-training recipe, state-of-the-art approaches typically combine several techniques as listed below:

    • Supervised chain-of-thought fine-tuning: Models learn from step-by-step demonstrations.

    • Preference optimization: Outputs with stronger reasoning are favored.

    • Reinforcement learning: Multi-step trajectories are rewarded for correctness.

    • Self-consistency and filtering: High-quality reasoning traces are selected for training.

  • Relevant foundations include Self-Consistency Improves Chain of Thought Reasoning by Wang et al. (2022), which showed that multiple reasoning paths improve reliability, and DeepSeek-R1 by Guo et al. (2025), which demonstrated large-scale reasoning-focused post-training.

Benchmark evidence

  • The model card shows substantial gains on reasoning-intensive benchmarks.

    • AIME 2026 (31B): 89.2% without tools.

    • GPQA Diamond (31B): 84.3%.

    • BigBench Extra Hard (31B): 74.4%.

    • Humanity’s Last Exam (31B): 19.5% without tools, 26.5% with search.

  • These results indicate that Gemma 4 is trained to excel at tasks requiring sustained deliberation rather than shallow pattern matching.

  • The Gemma 4 model card offers a detailed set of Gemma 4 benchmark results across reasoning, coding, multilingual, and vision tasks.

Mathematical perspective

  • Reasoning can be viewed as latent search over intermediate states.

  • Let \(h_0 = f_{\text{encode}}(x)\) and iteratively \(h_{t+1} = F(h_t)\) until a stopping condition produces the final answer \(y = g(h_T)\).

  • Longer reasoning traces correspond to larger effective inference depth, even though the model parameters remain fixed.

Reasoning and tool use

  • Thinking mode is especially useful when paired with function calling and external tools.

  • The model can:

    • Interpret the task.
    • Plan required steps.
    • Decide whether a tool is needed.
    • Generate structured function arguments.
    • Integrate returned results.
    • Produce a final response.

  • This planning capability is central to agentic systems.

Reasoning and multimodal inputs

  • Because all modalities are represented in a shared token sequence, reasoning applies equally to:

    • Images.
    • Audio.
    • Video.
    • Documents.
    • GUI screenshots.

  • The model can therefore perform multimodal chain-of-thought, for example:

    • Reading a chart.
    • Extracting numerical values.
    • Performing calculations.
    • Producing structured conclusions.

Inference trade-offs

  • Thinking mode introduces several operational considerations.

  • Higher token usage: More intermediate tokens increase cost.

  • Longer latency: Generation takes more time.

  • Better reliability: Difficult tasks often become substantially more accurate.

  • Adaptive compute: Users can selectively enable reasoning only when needed.

  • This creates a practical quality-latency trade-off.

Conceptual interpretation

  • Thinking mode can be interpreted as allowing the model to externalize hidden computation into the token stream.

  • Instead of relying solely on fixed-depth internal activations, the model performs additional sequential computation:

\[\text{Computation} = \text{Parameters} \times \text{Generated Reasoning Tokens}\]
  • This is one of the most important developments in modern LLM design because it increases effective capability without increasing model size.

Architectural significance

  • Gemma 4’s configurable thinking modes make the model family more than a static predictor. They transform it into a variable-compute reasoning system capable of allocating additional inference effort when tasks require sustained deliberation.

  • Combined with long context, multimodal inputs, and function calling, this reasoning-centric design makes Gemma 4 particularly well suited for mathematics, coding, scientific analysis, and autonomous agent workflows.

Training Objectives and Post-Training

  • Gemma 4’s performance comes from the interaction between architecture, multimodal training, instruction tuning, reasoning post-training, and deployment-oriented alignment.

  • Training pipeline: Gemma 4 likely follows the modern frontier-model pattern: large-scale multimodal pretraining, supervised instruction tuning, reasoning-oriented post-training, preference optimization, safety alignment, and function-calling tuning. This follows the broader direction established by InstructGPT by Ouyang et al. (2022), which showed that preference-based alignment improves instruction following.

Autoregressive pretraining objective

  • The foundational objective is next-token prediction:
\[\mathcal{L}_{\text{LM}} = - \sum_{t=1}^{T} \log p(x_t \mid x_{<t})\]
  • What this trains: The model learns to predict the next text token given previous tokens, building broad statistical knowledge over language, code, reasoning traces, structured data, and multimodal sequences.

Multimodal pretraining

  • For multimodal inputs, encoder outputs are projected into the decoder embedding space and inserted into the token stream:
\[p(y \mid m, x) = \prod_{t=1}^{T} p(y_t \mid y_{<t}, m, x)\]

  • What this trains: The model learns to generate text conditioned on images, audio, video frames, documents, charts, screenshots, and other non-text inputs.

  • Relevant data types: Training likely includes image-caption pairs, OCR examples, document understanding data, speech-text pairs, video-transcript examples, and multimodal instruction data.

Instruction tuning

  • Instruction-tuned checkpoints: Gemma 4 is released in both pre-trained and instruction-tuned variants, with the instruction-tuned models intended for dialogue, reasoning, function calling, and structured assistant behavior.

  • Native system prompt support: Gemma 4 introduces built-in support for the system role, which lets applications provide higher-priority behavioral instructions for more structured conversations.

Preference optimization

  • A representative preference objective is Direct Preference Optimization from Direct Preference Optimization by Rafailov et al. (2023):

    \[\mathcal{L}_{\text{DPO}} = - \log \sigma \left( \beta \left[ \log \pi_\theta(y^+ \mid x) - \log \pi_\theta(y^- \mid x) \right] \right)\]

    • where:

      • \(y^+\) is the preferred response.
      • \(y^-\) is the dispreferred response.
      • \(\beta\) controls preference sharpness.

  • What this optimizes: The model is pushed toward preferred responses and away from dispreferred responses, improving helpfulness, clarity, correctness, refusal behavior, and formatting reliability.

Reasoning-focused post-training

  • Thinking behavior: Gemma 4 models are described as reasoning-capable systems with configurable thinking modes.

  • Likely training signal: Reasoning post-training probably uses math solutions, coding trajectories, multi-hop reasoning examples, tool-use traces, and filtered chain-of-thought style demonstrations.

  • Why it matters: This teaches the model to spend more inference tokens on difficult problems rather than jumping directly to a final answer.

Function-calling and agentic tuning

  • Function-calling support: Gemma 4 includes built-in function-calling support, making it useful for agents that need to call tools, produce structured arguments, and continue after tool outputs.

  • Training target: The model learns when to call a function, how to fill the function schema, how to produce JSON-like structured outputs, and how to integrate tool results into a final answer.

Safety alignment

  • Responsible deployment: Gemma 4 is released with open weights, Apache 2.0 licensing, model documentation, and responsible use guidance.

  • Likely alignment methods: Safety tuning likely includes red-teaming, refusal examples, preference data, policy filtering, and adversarial instruction tuning.

Base versus instruction-tuned checkpoints

  • Base models: These are best suited for continuation, research, adaptation, and downstream fine-tuning.

  • Instruction-tuned models: These are better suited for chat, agents, reasoning, structured outputs, multimodal QA, and production applications.

Architectural significance

  • Main takeaway: Gemma 4’s architecture provides the substrate, but training and post-training determine how that substrate behaves. Hybrid attention, PLE, Shared KV Cache, MoE routing, and multimodal encoders create the capacity and efficiency profile; instruction tuning, reasoning post-training, function-calling data, and safety alignment turn it into a usable assistant model.

Evaluation and Benchmark Profile

  • Gemma 4 is evaluated across reasoning, coding, multilingual understanding, multimodal perception, and agentic tasks. The benchmark pattern suggests that the family is optimized not just for general text completion, but for frontier-style workloads where long-context reasoning, code generation, tool use, and multimodal understanding matter.

Reasoning benchmarks

  • AIME 2026: The 31B model reaches 89.2% without tools, while the 26B A4B model reaches 88.3%, showing that the MoE model nearly matches the dense model despite activating far fewer parameters per token.

  • GPQA Diamond: The 31B model reaches 84.3%, and the 26B A4B model reaches 82.3%, indicating strong performance on difficult graduate-level science questions.

  • BigBench Extra Hard: The 31B model scores 74.4%, while the 26B A4B model scores 64.8%, showing that larger dense compute still helps on broad hard reasoning tasks.

Coding benchmarks

  • LiveCodeBench v6: The 31B model reaches 80.0%, the 26B A4B model reaches 77.1%, E4B reaches 52.0%, and E2B reaches 44.0%, showing a strong scaling curve across the family.

  • Codeforces ELO: The 31B model reaches 2150, while the 26B A4B model reaches 1718, suggesting that dense capacity still has an advantage on competitive programming despite the MoE model’s efficiency.

Agentic and tool-oriented benchmarks

  • Tau2: Gemma 4 reports strong gains over Gemma 3, with the 31B model at 76.9% and 26B A4B at 68.2%, indicating better performance on agentic tasks that require instruction following, planning, and tool-style behavior.

  • Function-calling relevance: The benchmark gains align with the model overview’s emphasis on built-in function calling and agentic capabilities.

Multilingual benchmarks

  • MMMLU: The 31B model reaches 88.4%, the 26B A4B model reaches 86.3%, E4B reaches 76.6%, and E2B reaches 67.4%, reflecting the family’s large 262K vocabulary and support for over 140 languages.

Evaluation interpretation

  • Dense 31B: Best overall when maximum quality matters and sufficient memory is available.

  • 26B A4B MoE: Best quality-to-compute trade-off among larger models, since it approaches 31B performance while activating only about 4B parameters per token.

  • E4B: Strong compact option for on-device and laptop-class applications where reasoning and multimodal capability are still needed.

  • E2B: Smallest practical deployment target, especially relevant for mobile, browser, and edge use cases.

Inference, Deployment, and Quantization

  • Gemma 4 is designed to run across a wide deployment range, from mobile and browser environments to laptops, consumer GPUs, workstations, and servers. The model family is available through Google’s developer ecosystem and common open-model distribution channels such as Gemma 4 on Hugging Face and Google Gemma on GitHub.

Deployment targets

  • E2B: This is the smallest deployment-oriented model, built for ultra-mobile, edge, browser, and local-device use. It supports text, image, and audio inputs, has a 128K context window, and uses Per-Layer Embeddings to increase effective capacity without making every parameter equally expensive at inference time.

  • E4B: This is the stronger small-model option for laptops, local assistants, browser applications, and mobile-class workflows that need better reasoning and multimodal performance than E2B while remaining much lighter than the 31B and 26B A4B models.

  • 31B dense: This model targets higher-quality local or workstation inference where memory is available and predictable dense execution is preferred. It supports a 256K context window and text/image inputs.

  • 26B A4B MoE: This model targets high-throughput advanced reasoning. It stores roughly 26B parameters but activates only about 4B parameters per token, making it faster than a similarly sized dense model while still requiring memory for the full expert pool.

Precision and memory trade-offs

  • Gemma 4 can be loaded in multiple precision formats, including BF16, SFP8, and 4-bit quantized formats. Lower precision reduces memory consumption and can improve latency, but it may reduce model quality depending on the task, quantization method, backend, and hardware.

  • E2B memory profile: The E2B model requires approximately 9.6 GB in BF16, 4.6 GB in SFP8, and 3.2 GB in Q4_0. This makes it the most practical model for constrained local deployments.

  • E4B memory profile: The E4B model requires approximately 15 GB in BF16, 7.5 GB in SFP8, and 5 GB in Q4_0. It is a strong candidate for laptop-class use when quantized.

  • 31B memory profile: The 31B dense model requires approximately 58.3 GB in BF16, 30.4 GB in SFP8, and 17.4 GB in Q4_0. This places it in workstation or high-memory consumer GPU territory.

  • 26B A4B memory profile: The 26B A4B MoE model requires approximately 48 GB in BF16, 25 GB in SFP8, and 15.6 GB in Q4_0. Even though only about 4B parameters are active per token, the full expert set must remain loaded for fast routing.

Why effective parameters differ from loaded parameters

  • Effective parameters in E2B and E4B: The “E” notation means effective parameters, not total parameters. The smaller models include Per-Layer Embeddings, which add large embedding tables that are cheap to query but still contribute to static model weight memory.

  • Loaded memory implication: E2B behaves computationally more like a 2.3B-parameter model, but its loaded memory includes embeddings that bring the total closer to 5.1B parameters. E4B behaves computationally more like a 4.5B-parameter model, but its loaded memory includes embeddings that bring the total closer to 8B parameters.

KV cache memory during inference

  • The model-weight memory estimates do not include the full runtime cost of long-context inference. The KV cache grows with prompt length, generated length, number of layers, and KV dimensionality.
\[\text{KV Memory} \propto L \cdot n \cdot d_{\text{kv}}\]

  • Layer count effect: More layers increase KV storage because each layer normally maintains its own cached keys and values.

  • Context length effect: Moving from short prompts to 128K or 256K context can make KV memory a dominant part of total inference memory.

  • Gemma 4 mitigation: Sliding-window attention bounds local-layer context usage, while Shared KV Cache reduces duplicated key-value storage in later layers.

Quantization

  • Quantization reduces the number of bits used to store weights:
\[\text{Memory} \approx \frac{\text{Parameters} \times \text{Bits per Parameter}}{8}\]

  • BF16: This is the highest-memory listed format and is usually preferred when quality and numerical stability matter most.

  • SFP8: This offers a middle ground, reducing memory roughly by half relative to BF16 while keeping more precision than 4-bit formats.

  • Q4_0: This is the most memory-efficient listed format, making larger models accessible on smaller hardware, although quality can become more task-sensitive.

Backend support

  • Gemma 4 is designed to work across common local and production inference stacks. The Hugging Face launch notes emphasize availability across Transformers, llama.cpp, MLX, WebGPU, Rust-based runtimes, and other local-agent tooling.

  • Transformers: Useful for research, prototyping, fine-tuning, multimodal pipelines, and Python-based serving.

  • llama.cpp: Useful for CPU and quantized local inference, especially where low operational overhead matters.

  • MLX: Useful for Apple Silicon local deployment.

  • WebGPU and browser runtimes: Useful for private, client-side AI applications that avoid server round trips.

  • Rust and edge runtimes: Useful for production systems where memory control, portability, and low-latency execution matter.

Fine-tuning considerations

  • Fine-tuning requires substantially more memory than inference because training stores activations, optimizer states, gradients, and sometimes full-precision master weights. The model overview explicitly notes that fine-tuning memory depends heavily on framework, batch size, and whether full-precision tuning or parameter-efficient methods such as LoRA are used.

  • Full fine-tuning: This updates all model weights and generally requires the most memory.

  • LoRA-style fine-tuning: LoRA by Hu et al. (2021) freezes the base model and trains low-rank adapter matrices, making adaptation much cheaper.

  • QLoRA-style fine-tuning: QLoRA by Dettmers et al. (2023) combines quantized base weights with trainable adapters, enabling memory-efficient fine-tuning of large models.

Deployment decision guide

  • Choose E2B: Best when memory, latency, battery, and local execution are the primary constraints.

  • Choose E4B: Best when a compact model still needs stronger reasoning, coding, multimodal understanding, and audio support.

  • Choose 31B dense: Best when maximum quality and predictable dense execution matter more than throughput efficiency.

  • Choose 26B A4B: Best when high reasoning quality and throughput matter, and the deployment environment can load the full MoE parameter set.

Architectural significance

  • Main takeaway: Gemma 4 treats deployment as an architectural problem, not merely an inference-engine problem. PLE reduces the compute burden of added capacity, Shared KV Cache reduces long-context runtime overhead, hybrid attention reduces attention cost, quantization reduces static memory, and MoE routing reduces active compute for the 26B A4B model.

Comparison with Prior Gemma Generations

  • Gemma 4 builds upon ideas introduced in earlier members of the Gemma family rather than replacing them. Successive releases have progressively shifted the design toward longer context windows, stronger multimodal capabilities, better deployment efficiency, and more capable reasoning while retaining the same underlying decoder-only Transformer foundation.

  • The evolution can be viewed as four major themes:

    • Scaling context length
    • Expanding multimodal capability
    • Improving inference efficiency
    • Strengthening reasoning and agentic behavior

Architectural evolution

  • The overall decoder architecture remains recognizable across Gemma releases. Each generation continues to use causal self-attention, residual connections, feed-forward networks, and autoregressive decoding derived from the Transformer introduced in Attention Is All You Need by Vaswani et al. (2017).

  • The major architectural changes instead focus on improving efficiency and deployment rather than replacing the Transformer itself.

Evolution of attention mechanisms

  • Attention has evolved substantially across Gemma generations, as delinated below:

    • Gemma 1: Primarily followed a conventional decoder-only Transformer architecture with global attention.

    • Gemma 2: Introduced hybrid attention by combining sliding-window attention with global attention. This was inspired by sparse-attention ideas such as Longformer by Beltagy et al. (2020), allowing substantially lower KV-cache memory while preserving long-range information flow.

    • Gemma 3: Increased the proportion of local attention relative to global attention, making the architecture significantly more efficient for long-context inference. According to the Gemma 3 technical report, this change produced large reductions in KV-cache memory with minimal impact on perplexity. Gemma 3 Technical Report by the Gemma Team et al. (2025).

    • Gemma 4: Retains the hybrid attention strategy while pairing it with Shared KV Cache and dual RoPE configurations, allowing context windows of up to 256K tokens with improved deployment efficiency.

Evolution of context length

  • Long-context capability has steadily expanded, as delinated below:

    • Gemma 1: Designed primarily for conventional LLM context lengths.

    • Gemma 2: Extended usable context while introducing hybrid attention to reduce inference cost.

    • Gemma 3: Further increased context capacity and optimized long-context processing through a higher ratio of sliding-window attention.

    • Gemma 4: Supports 128K context on the E2B and E4B models and 256K context on the 31B dense and 26B A4B models.

  • This progression reflects an industry-wide shift toward treating entire repositories, books, technical documentation, and long conversations as single inference contexts.

Evolution of multimodal capabilities

  • Multimodal support has expanded significantly, as delinated below:

    • Gemma 1: Text-only.

    • Gemma 2: Primarily focused on text generation and reasoning.

    • Gemma 3: Added multimodal understanding through an integrated vision encoder.

    • Gemma 4: Extends multimodality by supporting text and image across the full family, audio on the E2B and E4B models, and video understanding through frame-based processing. The vision encoder now supports variable image resolutions, preserved aspect ratios, and configurable image token budgets.

  • Rather than simply accepting more modalities, Gemma 4 also improves how efficiently those modalities are represented inside the decoder.

Evolution of parameter efficiency

  • One of the most significant changes occurs in how model capacity is increased, as delinated below:

    • Gemma 1 and Gemma 2: Increased capability primarily through larger dense models.

    • Gemma 3n: Introduced Per-Layer Embeddings (PLE), allowing small models to gain additional representational capacity without proportionally increasing computation.

    • Gemma 4: Continues using PLE in the E2B and E4B models, while reserving conventional dense embeddings for the larger 31B model and sparse MoE scaling for the 26B A4B model.

  • This reflects two complementary scaling strategies:

    • Improving parameter efficiency in small models.
    • Improving compute efficiency in large models.

Evolution of KV-cache optimization

  • Memory efficiency has become progressively more important as context windows have grown, as delinated below:

    • Gemma 2: Reduced KV-cache requirements through hybrid attention.

    • Gemma 3: Increased the fraction of sliding-window layers, further reducing KV growth.

    • Gemma 4: Introduces Shared KV Cache, allowing later layers to reuse key-value tensors from earlier layers of the same attention type instead of recomputing them. This reduces both compute and memory during autoregressive decoding.

  • This optimization is especially valuable at 128K and 256K context lengths, where KV-cache memory can otherwise dominate total inference memory.

Evolution of reasoning capability

  • Reasoning has also become progressively more central, as delinated below:

    • Gemma 1: Focused primarily on strong general language modeling.

    • Gemma 2: Improved reasoning through larger-scale pretraining and instruction tuning.

    • Gemma 3: Introduced stronger reasoning-oriented post-training and multimodal reasoning.

    • Gemma 4: The entire model family is explicitly designed around configurable thinking modes, stronger mathematical reasoning, coding, planning, and agentic workflows.

  • This mirrors a broader industry trend in which additional inference-time computation has become a major driver of reasoning performance.

Evolution of deployment philosophy

  • The deployment strategy has also broadened considerably, as delinated below:

    • Earlier Gemma releases: Primarily targeted GPUs and traditional local inference.

    • Gemma 4: Explicitly targets mobile devices, browsers, Apple Silicon, edge hardware, workstations, servers, and production inference frameworks such as Transformers, llama.cpp, MLX, WebGPU, and Rust runtimes.

  • Deployment efficiency is now treated as a first-class architectural objective rather than solely an implementation concern.

Overall design trajectory

  • Across the Gemma family, the dominant trend has not been radical architectural replacement but systematic refinement, as delinated below:

    • Gemma 1 established a strong decoder-only Transformer foundation.

    • Gemma 2 introduced hybrid attention and longer-context efficiency.

    • Gemma 3 significantly expanded multimodal capability while increasing the emphasis on sliding-window attention.

    • Gemma 4 integrates those earlier innovations with Shared KV Cache, Per-Layer Embeddings, configurable reasoning modes, sparse Mixture-of-Experts scaling, longer context windows, native function calling, and deployment-oriented optimizations.

Architectural significance

  • Main takeaway: Gemma 4 represents the culmination of Google’s iterative design philosophy for open-weight language models. Rather than pursuing entirely new architectures, each generation introduces targeted improvements that address specific bottlenecks, whether memory consumption, long-context inference, multimodal processing, reasoning quality, or deployment efficiency. The result is a model family that remains architecturally familiar while substantially advancing practical capability across nearly every dimension.

Conclusion and Practical Takeaways

  • Gemma 4 is best understood as a deployment-first, multimodal, long-context Transformer family. Its main contribution is not one isolated architectural breakthrough, but the way several efficiency mechanisms are combined into a coherent system.

What makes Gemma 4 important

  • Long-context by design: Gemma 4 supports 128K context on E2B and E4B, and 256K context on 31B and 26B A4B, using hybrid local-global attention rather than full attention everywhere.

  • Efficient memory use: Shared KV Cache reduces duplicated key-value storage and redundant KV projections in later layers, which matters especially at long context lengths.

  • Smaller-model capacity: Per-Layer Embeddings let E2B and E4B gain layer-specific token information without making every added parameter expensive to use.

  • Sparse large-model scaling: The 26B A4B MoE model keeps a large parameter pool while activating only about 4B parameters per token.

  • Practical multimodality: Text, image, video, and audio support make Gemma 4 useful for document analysis, OCR, GUI understanding, coding workflows, and local assistants.

When to use each model

  • Use E2B: Best for mobile, browser, edge, and privacy-preserving local applications where memory and latency are tight.

  • Use E4B: Best when you still need compact deployment but want stronger reasoning, coding, audio, and multimodal capability.

  • Use 31B dense: Best when quality, stability, and predictable dense inference matter more than throughput.

  • Use 26B A4B: Best when you want near-large-model capability with lower active compute, and can afford loading the full MoE parameter set.

Final takeaway

  • Gemma 4 shows where open-weight LLM architecture is heading: not simply larger dense models, but carefully engineered systems that combine long-context attention, KV-cache efficiency, multimodal encoders, sparse routing, quantization compatibility, and reasoning-oriented post-training.

  • In practical terms, Gemma 4 is a strong example of a modern model family optimized for real deployment constraints rather than benchmark performance alone.

References

Gemma model family

Transformer foundations

Long-context Transformers and positional encoding

Efficient attention and KV-cache optimization

Mixture-of-Experts

Normalization and feed-forward architectures

Multimodal language models

Vision and audio encoders

Reasoning and inference-time scaling

Instruction tuning, preference optimization, and alignment

Fine-tuning and deployment

Benchmarks and evaluation

Citation

If you found our work useful, please cite it as:

@article{Chadha2020DistilledGemma4,
  title   = {Gemma 4},
  author  = {Chadha, Aman and Jain, Vinija},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}