Hugging Face 模型 · tencent ↗ · 2026-09-14
Simple-Attention-Sparsification
模型发布 Hugging Face 模型 · tencent 原文语言:中文
官方机构模型仓库创建于 2026-09-14。仓库创建时间不等于正式发布日期。任务类型:text-generation
全文
Simple Attention Sparsification (SAS)
SAS learns to rank and select KV blocks for each query. Unlike methods that train a sparse-attention selector by distilling dense attention scores, SAS adds continuous gates to the selected blocks so that the language-modeling loss can optimize context ranking end to end.
Overview of SAS. Continuous gates provide a differentiable path through discrete Top-K block selection, enabling end-to-end context-ranking optimization with the language modeling loss.
Released checkpoints
Directory Base model Gate parameters Gate checkpoint size Qwen3-4B-AttnGates Qwen3-4B 33.0M 64 MiB Qwen3-8B-AttnGates Qwen3-8B 33.0M 64 MiB Qwen3-14B-AttnGates Qwen3-14B 42.0M 81 MiB
Each directory is an SGLang-compatible AttnGates package containing:
attn_gate_weights.pth: the learned gate-router weights; config.json: the base-model reference and seerattn_* configuration; tokenizer and chat-template files copied from the corresponding Qwen3 model.
[!IMPORTANT] These are router-only checkpoints, not standalone language models. The Qwen3 backbone was frozen during training and is not included. Standard transformers.AutoModelForCausalLM.from_pretrained() cannot directly run these directories. Inference requires the corresponding Qwen3 base model and the seer_attn backend in our sglang-blocksparse fork.
Checkpoint configuration
All three checkpoints use the same sparse-attention setup:
Setting Value KV block size 64 tokens Training Top-K 31 historical blocks Gate hidden size 128 Query projection Qproj Key block pooling max + min + average Q/K normalization enabled Gate RoPE enabled Training sequence length 32,768 tokens Training data OpenR1-Math-220k
Only the gate routers were optimized; all base-model parameters remained frozen. The released setup uses a 2,048-token sparse decode budget by default. The same checkpoints can be evaluated with 1,024-, 2,048-, or 4,096-token budgets without retraining.
Inference
1. Set up SAS and the sparse SGLang backend
Download this Hugging Face repository and point GATES to the directory that matches the desired base-model size.
2. Launch a server directly
The following example serves the Qwen3-4B gate checkpoint:
config.json points to the corresponding public Qwen3 repository, so the base model is downloaded separately when needed. For offline use, download the base model in advance and replace the base_model value in the gate checkpoint's config.json with its local path.
To use another checkpoint, change GATES and the served model name:
The server exposes an OpenAI-compatible API at http://127.0.0.1:30000/v1.
3. Run the official evaluation wrappers
The code repository also provides wrappers that start the server, run a benchmark, and stop the server automatically:
Use MODE=seer_8b or MODE=seer_14b with the matching gate directory for the other model sizes.
[!NOTE] Evaluating on BFCL and VitaBench requires additional benchmark-specific environments and dependencies. Please follow the setup instructions in the code repository before running scripts/eval/run_bfcl.sh or scripts/eval/run_vitabench.sh.
正文以来源语言显示。