Hugging Face 模型 · ByteDance-Seed ↗ · 2026-11-05
Qwen3-30B-A3B-MoE-SAE
Model release Hugging Face 模型 · ByteDance-Seed Source language: Chinese
官方机构模型仓库创建于 2026-11-05。仓库创建时间不等于正式发布日期。任务类型:feature-extraction
Full text
Qwen3-30B-A3B MoE-Output Sparse Autoencoders
TopK sparse autoencoders (k = 100, 2^20 latents) trained on the MoE block output of Qwen3-30B-A3B at layers 4, 30 and 46, released with the paper *Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization* (project page, code, paper).
Files
Each SAE reads the output of model.model.layers[L].mlp (the Qwen3MoeSparseMoeBlock, before the residual addition, layers 0-indexed) as is, without any normalization. sae.safetensors (about 17.2 GB) holds four float32 tensors:
Tensor Shape W_enc [2048, 1048576] b_enc [1048576] W_dec [1048576, 2048], rows have unit norm b_dec [2048]
Latent i is column i of W_enc and row i of W_dec; the paper writes it as L-F , e.g. 30-F97783. config.json lists the same settings.
Usage
The dense pre-activation takes 4 MiB per token, so encode long inputs in chunks.
Training
Each SAE was trained independently on MoE block outputs collected from the Olmo 3 training corpus: 5/8 Dolma 3 pretraining data, 1/4 Dolci-Think-SFT (reasoning chains of thought) and 1/8 Dolci-Instruct-SFT (instruction following). Training used a TopK activation with k decaying from 1000 to 100 over the first 10,000 steps, an auxiliary loss, AdamW and about 3.2B tokens per SAE; the full recipe is given in the appendix "SAE training details" of the paper.
Evaluation
Fraction of variance unexplained (FVU) of the reconstruction, as reported in the paper:
Layer FVU 4 12.6% 30 25.1% 46 7.6%
License
Apache License 2.0. The base model Qwen3-30B-A3B is released under the Apache License 2.0. The SAEs were trained on activations computed from Dolma 3, Dolci-Think-SFT and Dolci-Instruct-SFT, which Ai2 makes available under the ODC-BY v1.0 license.