Interactively Visualizing the Qwen3 MoE Architecture
Beneath the visualizations are explanations of key concepts that might be unfamiliar to readers: Grouped Query Attention, QK-Norm, and Rotary Positional Embeddings (RoPE). Note that a full explanation of transformers is out of scope for this post. For that, I recommend the Wikipedia article on Transformers. I would also recommend checking out Sebastian Raschka's LLM Architecture Comparison, which is what inspired this post.
Qwen3 (Dense)
Hover over each module to explore how the Qwen3 architecture works. The diagrams show a single block; full models stack many, many copies according to size.
Architectural Concepts
Queries are partitioned into groups that share a key/value set, so each group computes \(\mathrm{softmax}\!\left(\frac{Q_g K_g^{\top}}{\sqrt{d}}\right) V_g\). Reusing keys and values cuts memory bandwidth compared with standard multi-head attention. The per-group outputs are concatenated (or projected) back into the full hidden dimension before the residual connection.
Before attention logits are formed, queries and keys are rescaled to fixed RMS magnitude: \(\hat{Q} = Q / \mathrm{RMS}(Q)\) and \(\hat{K} = K / \mathrm{RMS}(K)\). (Note: the RMS normalization of Q/K is performed across the feature dimension - i.e. it is per-token.) The normalization keeps logits balanced across layers so that softmax outputs stay well-conditioned even for very long contexts.
RoPE rotates each query/key pair in a complex plane by an angle proportional to the token index. For a 2-D head slice \([\mathbf{u}, \mathbf{v}]\), the rotated version is given by $$[\mathbf{u}', \mathbf{v}'] = [\mathbf{u}, \mathbf{v}] R(\theta)$$ where \(R(\theta)\) is the standard 2-by-2 rotation matrix that turns the head slice by angle \(\theta\). with \(\theta = n / \omega\). The resulting phase difference between any two tokens depends only on their relative distance. This is important because it lets the model generalize to sequence lengths beyond those seen in training.