Hugging Face 模型 · ByteDance-Seed ↗ · 2026-11-05
Cross-Layer-TopK-CLT
Model release Hugging Face 模型 · ByteDance-Seed Source language: Chinese
官方机构模型仓库创建于 2026-11-05。仓库创建时间不等于正式发布日期。任务类型:feature-extraction
Full text
Cross-Layer Transcoder for Qwen3-8B-Base
This model is described in the paper Cross-Layer TopK: Dynamically Allocating Sparsity Across Layers in Transcoders.
A cross-layer transcoder (CLT) for Qwen3-8B-Base: 36 layers × 32,768 latents. Each layer's latents read the residual stream right before that layer's MLP (resid_mid) and write to the MLP outputs (mlp_out) of that layer and every later layer. 94.2B parameters, fp32, 377 GB in total.
Format
File Contents W_enc_{l}.safetensors (l = 0..35) W_enc_{l}: [32768, 4096], encoder of layer l W_dec_{l}.safetensors (l = 0..35) W_dec_{l}: [32768, 36 - l, 4096]; W_dec_norm_{l}: [32768] (357 GB, decoding only) W_dec_norm.safetensors W_dec_norm: [36, 32768], decoder-column norms for scoring factors.safetensors resid_mid_l2, mlp_out_l2: [36], mean activation norms config.json d_feature (32768), k (10), base model and dimensions
Hooks on the base model: resid_mid is the input of model.layers[l].post_attention_layernorm; mlp_out is the output of model.layers[l].mlp.
Encode: a_l = ReLU(W_enc_l · x_l) (no bias). Score each latent as a_l[i] · W_dec_norm[l, i] and keep the 36 · k = 360 highest-scoring latents across all layers jointly. Decode: mlp_out_hat[m] = Σ_{l ≤ m} a_l @ W_dec_{l}[:, m - l, :]. The activation normalization used in training is already folded into the weights: the CLT takes raw resid_mid and reconstructs raw mlp_out. Do not apply the factors in factors.safetensors again. Tokens whose resid_mid or mlp_out norm on any layer reaches 20× that layer's mean norm are dropped, as in training (in practice the first token). The CLT was trained on fp32 activations; use fp32 for the base model too.
Demo
Download the checkpoint, save the code below as hf_encode_demo.py next to the downloaded clt/ directory, and run it:
The demo encodes a hardcoded sentence, prints the active latents of every token, and decodes them back to mlp_out. Latents are split evenly across GPUs (any count dividing 32,768). Output (latent_id:activation, strongest first, ids are per layer):
In code, the result is activations[pos][layer] = [[latent_id, activation], ...] (None for outliers) and the reconstruction is mlp_out_hat_nxlxh.
To only encode, set DECODE = False in the code and skip the decoder files; one 80 GB GPU (--nproc_per_node 1) is then enough:
Training data and license
Trained on Dolma 3, Dolci-Think and Dolci-Instruct, mixed 5 : 2 : 1 by tokens (≈3.4B tokens). These datasets are licensed under ODC-BY.
The weights and code are released under the Apache License 2.0, Copyright 2026 ByteDance and/or its affiliates.