pplx-qwen-3-8-27b-dflash2-20260819

Model release Hugging Face 模型 · perplexity-ai Source language: Chinese

官方机构模型仓库创建于 2026-08-20。仓库创建时间不等于正式发布日期。任务类型:未提供

Full text

pplx-computer-qwen-3-8-27b-dspark2-20260819

Qwen3.8-27B (arch class Qwen3_5ForConditionalGeneration) quantized to NVFP4 (ModelOpt mixed-precision mirroring RadixArk/Qwen3.8-27B-NVFP4's recipe, FP8 KV cache) with one deliberate change: an unquantized bf16 lm_head (DFlash2 refuses a quantized one — every public NVFP4 quant packs it to FP4), plus baked-in dspark2 MTP draft tensors (mtp.* in model-mtp.safetensors, referenced by model.safetensors.index.json).

Snapshot of the checkpoint serving on the pplx-computer DGX Spark fleet as of 2026-08-19. Verified perf on one GB10: ~30–35 tok/s single-stream decode, 84 tok/s aggregate @ 4 streams (2.9× over no-spec NVFP4).

How to run (DGX Spark / GB10)

Everything needed is in serving/. Stock vLLM can not serve this with DFlash2 — it needs the pure-Python vLLM PR #52816 overlay plus a one-line lm_head fix (see notes below), packaged here as two Dockerfiles.

Tunables (env vars on docker compose up): GPU_UTIL (default 0.75 — raise to ~0.9 on a dedicated box), MAX_LEN (262144), MAX_SEQS (4), SERVED_NAME (the OpenAI-API model id; defaults to this repo's name — override if your client expects a different id).

Serve flags

Notes / gotchas

The DFlash2 draft is vendored at draft/ (a mirror of incoai/Qwen3.8-27B-DFlash2, 2026-08-19) and referenced by local path in the compose — the repo is fully self-contained, no external downloads at serve time. The draft is separate from the target weights: the in-checkpoint mtp.* tensors are the dspark2 MTP module — MTP k=7 reaches similar acceptance but decodes ~2× slower (sequential draft passes), so DFlash2 is the recommended path. bf16 lm_head is load-bearing. DFlash2 requires an unquantized target lm_head; it is excluded via exclude_modules in both hf_quant_config.json and config.json (vLLM's modelopt loader keys on the exclude wildcards, not on absence from quantized_layers). Why Dockerfile.lmheadfix: vLLM gives an excluded lm_head an UnquantizedLinearMethod, which PR #52816's isinstance check wrongly rejects (it only accepts UnquantizedEmbeddingMethod). The second image layer relaxes that check. --max-num-batched-tokens 8192 is required: with spec decode enabled, vLLM otherwise clamps scheduling to 2048 and prefill collapses.

Sources

← Back to the daily brief