Qwen3-30B-A3B-MoE-SAE

模型发布 Hugging Face 模型 · ByteDance-Seed 原文语言:中文

官方机构模型仓库创建于 2026-11-05。仓库创建时间不等于正式发布日期。任务类型:feature-extraction

全文

Qwen3-30B-A3B MoE-Output Sparse Autoencoders

TopK sparse autoencoders (k = 100, 2^20 latents) trained on the MoE block output of Qwen3-30B-A3B at layers 4, 30 and 46, released with the paper *Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization* (project page, code, paper).

Files

Each SAE reads the output of model.model.layers[L].mlp (the Qwen3MoeSparseMoeBlock, before the residual addition, layers 0-indexed) as is, without any normalization. sae.safetensors (about 17.2 GB) holds four float32 tensors:

Tensor Shape W_enc [2048, 1048576] b_enc [1048576] W_dec [1048576, 2048], rows have unit norm b_dec [2048]

Latent i is column i of W_enc and row i of W_dec; the paper writes it as L-F , e.g. 30-F97783. config.json lists the same settings.

Usage

The dense pre-activation takes 4 MiB per token, so encode long inputs in chunks.

Training

Each SAE was trained independently on MoE block outputs collected from the Olmo 3 training corpus: 5/8 Dolma 3 pretraining data, 1/4 Dolci-Think-SFT (reasoning chains of thought) and 1/8 Dolci-Instruct-SFT (instruction following). Training used a TopK activation with k decaying from 1000 to 100 over the first 10,000 steps, an auxiliary loss, AdamW and about 3.2B tokens per SAE; the full recipe is given in the appendix "SAE training details" of the paper.

Evaluation

Fraction of variance unexplained (FVU) of the reconstruction, as reported in the paper:

Layer FVU 4 12.6% 30 25.1% 46 7.6%

License

Apache License 2.0. The base model Qwen3-30B-A3B is released under the Apache License 2.0. The SAEs were trained on activations computed from Dolma 3, Dolci-Think-SFT and Dolci-Instruct-SFT, which Ai2 makes available under the ODC-BY v1.0 license.

正文以来源语言显示。

参考来源

← 返回每日简报