Hugging Face Papers ↗ · 2026-11-19
Accelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining
Media reports Hugging Face Papers Source language: Chinese
Mixture of Block Attention reduces long-context computation but faces bottlenecks from fixed budgets and rigid scheduling. This paper presents LightningMoBA, which co-designs selection and execution through dynamic routing, KV-centric ownership, and runtime pipelining.
Full text
Accelerating Mixture of Block Attention via Dynamic Routing, KV-Centric Execution and Runtime Pipelining
Yingqi Shi Junru Lu Yun Liu Di Yin Xing Sun
Abstract
Mixture of Block Attention (MoBA) reduces long-context computation by selecting key-value (KV) blocks for each query. FlashMoBA implements MoBA efficiently but retains a fixed Top-kk budget and query-centric schedule. Fixed Top-kk applies one budget across different score distributions and couples routing to task granularity; query-range ownership fragments K/V reuse when several ranges select the same column. We present LightningMoBA, which co-designs selection and execution through dynamic routing, KV-centric ownership, and runtime pipelining. Dynamic routing thresholds MoBA’s per-query block scores and directly emits variable-cardinality sparse metadata. KV-centric ownership groups selected rows by (query head,KV block)(\text{query head},\text{KV block}) column, splits high-fanout columns into bounded tasks, and reuses each task’s loaded KV tile. On Hopper, pipelining overlaps regular K/V transfers and irregular query gathers with Tensor Core computation. Together, these mechanisms decouple retention from GPU scheduling granularity. At matched Top-1616 retention across five checkpoints, LightningMoBA accelerates end-to-end generation by 1.20 to 1.60×\times while raising mean RULER score from 74.42 to 75.28.
1 Introduction
Dense self-attention evaluates all pairwise query-key interactions in a sequence, whereas sparse attention reduces this work by routing each query to only a subset of keys. On GPUs, however, fewer interactions do not automatically yield proportionally lower latency: score selection, sparse-index construction, irregular data movement, and load imbalance can offset the savings in arithmetic. An efficient sparse attention method must therefore jointly optimize the routing decision and the GPU workload that follows from it.
Mixture of Block Attention (MoBA) [1] routes at block granularity: it scores each KV block against the query, retains a fixed number kk of blocks, and computes softmax attention over their tokens. FlashMoBA [2] implements this efficiently, maintaining an ordered Top-kk candidate set per query and assigning each CTA a logical query range whose selected rows are gathered into dense Tensor Core tiles.
Figure 1: Overview of LightningMoBA. Dynamic threshold routing adapts the retained block count per query and constructs sparse metadata directly; KV-centric execution organizes selected rows by (query head,KV block)(\text{query head},\text{KV block}) column and splits high-fanout columns into bounded work items; runtime pipelining keeps each work unit’s KV tile resident while its query gathers overlap Tensor Core work.
Nevertheless, two limitations remain. Fixed Top-kk routing assigns the same block budget to queries with different score distributions, retaining weakly related blocks for concentrated scores while discarding relevant blocks for diffuse ones. Maintaining the ordered candidate set also adds routing latency. FlashMoBA organizes metadata by KV block but assigns CTA ownership by logical query range, so several CTAs may claim one (query head,KV block)(\text{query head},\text{KV block}) column and leave cross-CTA reuse to the cache hierarchy. Coupling query-group size to kk also makes a routing parameter determine scheduling granularity.
We present LightningMoBA, which preserves MoBA’s parameter-free query-block scoring rule and computes standard softmax attention over the retained tokens, but reorganizes routing and execution along three axes (Fig. 1). Our contributions are as follows:
•
Dynamic routing. LightningMoBA replaces the fixed per-query Top-kk budget with a threshold-based variable retention rule and emits sparse metadata during score evaluation, without an ordered candidate set or a learned auxiliary indexer.
•
KV-centric execution and runtime pipelining. Work is organized by columns indexed by query head and KV block, so each unit reuses its loaded KV tile across the selected rows assigned to it. Fanout-aware partitioning bounds the rows per task, while an asymmetric Hopper pipeline overlaps TMA-based K/V transfers and irregular query gathers with Tensor Core computation.
•
Evaluation and analysis. At matched Top-1616 retention across five checkpoints, LightningMoBA accelerates end-to-end generation by 1.20 to 1.60×\times while increasing mean RULER score from 74.42 to 75.28. Controlled complete-device-path replays on two models reach 2.16 to 2.56×\times speedup at Top-3232, while routing and CSC construction reach 6.99 to 11.25×\times at Top-6464. Same-backend comparisons distinguish execution-backend gains, which dominate at Top-1616, from dynamic routing’s increasing benefit at larger budgets: routing alone yields 1.16 to 1.56×\times end-to-end speedups at Top-6464 on the evaluated two models (Sec. 4.3.2). Training integration using the adapted FlashMoBA backward implementation achieves a 1.602×\times speedup on Qwen3-4B at Top-6464 with closely aligned observed training losses (Sec. 4.2.2).
2 Related Work
The efficiency of sparse attention is determined jointly by its selection rule and by the execution path that consumes the resulting sparse pattern.
2.1 Scoring, Granularity, and Budgeting in Sparse Attention
Prescribed patterns [3, 4] fix connectivity by design; input-derived patterns [5, 6] cluster or hash tokens, and SSA [7] aligns sparse- with full-attention representations during training. Within content-dependent selection, granularity, scoring, and budget are separate choices: MoBA [1] scores blocks with mean-pooled keys under a fixed per-query budget; NSA [8] combines compressed, selected, and sliding-window branches; DSA [9] and MSA [10] use learned indexers, the latter per GQA group; HiLS [11], HySparse [12] and HySparse2 [13] learn summaries or derive routing from a preceding full-attention layer. Candidate discovery is amortized along different axes [14, 15, 16, 17], and adaptive budgets also appear [18, 19]. Thresholding avoids sorting: FlashPrefill [20] shares one block-level cutoff per query block, while LightningMoBA thresholds MoBA’s per-query block scores, so retention differs across individual queries. FlexPrefill [21] adapts each head’s sparse pattern to the input context, switching between query-specific and predefined patterns. Its cumulative-attention-based index selection retains query–key indices until a target attention mass is reached. In contrast, LightningMoBA preserves MoBA’s block scorer and uses per-query block-score thresholds with KV-centric execution. Retrofitting pretrained checkpoints through a learned selector is another route [22, 23]. LightningMoBA changes neither weights nor scorer; it replaces only the rule that maps scores to a retained set and co-designs the representation and execution of the resulting variable pattern.
2.2 Co-Design of Sparse Attention and Systems
The cost of a pattern depends on how its interactions are represented and scheduled: FlashMoBA [2] assigns CTA ownership by logical query range; FAST-Prefill [24] co-designs index generation and caching for an FPGA. KV-outer execution is established [1, 25, 10]: fixed Top-kk fixes the query-side row degree but leaves the KV-side column degree varying, so reverse indices and splitting popular columns are existing techniques.
LightningMoBA extends KV-outer execution to routing graphs with variable query-side degree: under per-query thresholding, both the row degree of each query and the total number of retained pairs are set at run time. It co-designs the routing representation and execution interface, decoupling the per-query budget from the kernel’s logical task size and dividing high-fanout columns into tasks with bounded query rows.
2.3 High-Performance Kernels on Modern GPUs
FlashAttention [26, 27] tiles attention with online softmax; FlashAttention-3 [28] and FlashAttention-4 [29] add warp-specialized producer-consumer pipelines on Hopper and Blackwell. PagedAttention [30] manages KV-cache storage at the serving level, orthogonal to the organization of data movement inside one attention call. These kernels show that tile shape, work ownership, and operand access determine what can overlap. LightningMoBA applies these principles to a KV-centric sparse workload with asymmetric operand access: contiguous K/V tiles move through TMA while indexed query rows move through a dedicated producer path, both overlapping Tensor Core computation, with tile shapes and warp roles independent of how many blocks each query retains.
3 Methodology
3.1 Preliminaries: MoBA and FlashMoBA
We consider causal prefill over NN tokens with head dimension dd. Batch and head indices are omitted unless needed, and gg query heads share each key/value head under GQA. We write 𝐪i\mathbf{q}_{i}, 𝐤t\mathbf{k}_{t} and 𝐯t\mathbf{v}_{t} for a token’s query, key and value vectors, reserving the unsubscripted kk for the number of blocks a query retains, with bold type keeping the two apart.
Mixture of Block Attention (MoBA) [1] partitions the sequence into n=⌈N/B⌉n=\lceil N/B\rceil blocks of at most BB tokens; the keys of block jj form 𝐊j∈ℝB×d\mathbf{K}_{j}\in\mathbb{R}^{B\times d}. For each query 𝐪i\mathbf{q}_{i}, MoBA scores each block against the query through the block’s mean-pooled key and keeps the best kk causally available blocks, including the query’s own block ci=⌈i/B⌉c_{i}=\lceil i/B\rceil:
𝐤~j=1|𝐊j|∑𝐤∈𝐊j𝐤,rij=𝐪i⊤𝐤~j,𝒮i={ci}∪TopKmin{k−1,ci−1}({rij}1≤j<ci).\tilde{\mathbf{k}}_{j}=\frac{1}{|\mathbf{K}_{j}|}\sum_{\mathbf{k}\in\mathbf{K}_{j}}\mathbf{k},\qquad r_{ij}=\mathbf{q}_{i}^{\top}\tilde{\mathbf{k}}_{j},\qquad\mathcal{S}_{i}=\{c_{i}\}\cup\operatorname*{TopK}_{\min\{k-1,\,c_{i}-1\}}\bigl(\{r_{ij}\}_{1\leq j<c_{i}}\bigr). (1)
Pooling is used only for routing; attention is computed over all tokens in the retained blocks. We write Z=∑i|𝒮i|Z=\sum_{i}|\mathcal{S}_{i}| for the number of selected query-block pairs, i.e., the sparse pattern executed by the attention stage. Under Eq. (1) each query retains 1+min{k−1,ci−1}≤k1+\min\{k-1,\,c_{i}-1\}\leq k blocks, so the budget bounds the pattern size as well: Z≤NkZ\leq Nk. Sec. 3.2 replaces the fixed budget by a score criterion, after which |𝒮i||\mathcal{S}_{i}| and ZZ are set by the scores rather than by kk. Sec. A.1 gives the corresponding worst-case bound.
FlashMoBA [2] provides an efficient GPU implementation of MoBA. It evaluates the fixed Top-kk routing rule, constructs the corresponding sparse metadata, and executes block-sparse attention over the selected query-block pairs. Its reference router maintains a running Top-kk candidate set for each query, while the attention kernel organizes execution around logical query ranges and gathers selected query rows into dense Tensor Core tiles.
We retain MoBA’s query-block score based on mean-pooled keys and its standard softmax attention over retained tokens, while redesigning block selection and the sparse workload that follows.
3.2 Dynamic Routing
MoBA assigns the same routing budget kk to every query, regardless of how its block scores are distributed. This fixed-cardinality constraint need not match the amount of relevant context available to each query: for a concentrated score distribution, selecting exactly kk blocks may retain weakly related blocks only to satisfy the budget, whereas for a more diffuse distribution the same budget may discard additional blocks with non-negligible scores. Top-kk therefore fixes how many blocks are retained rather than what score level is sufficient for retention.
Inspired by FlashPrefill [20], we instead apply a query-local threshold to MoBA’s original block scores. The scoring function is unchanged; only the rule that converts scores into a retained set is replaced. As illustrated in Fig. 2, this fixes a relative score criterion while allowing the retained cardinality to adapt independently for each query.
Figure 2: Fixed Top-kk and threshold-based selection applied to the same block scores of two queries. A fixed budget keeps the same number of blocks for both, retaining weakly scored blocks when the scores are concentrated and discarding a comparably scored one when they are diffuse. The threshold rule keeps every block within a fixed margin Δ=−logα/σ\Delta=-\log\alpha/\sigma of each query’s strongest preceding-block score, so the retained count follows the score distribution. Compulsory blocks are omitted so that the comparison isolates the selection rule.
3.2.1 Threshold-Based Block Selection
Query 𝐪i\mathbf{q}_{i} may attend to the causally admissible blocks 𝒞i={j:1≤j≤ci}\mathcal{C}_{i}=\{j:1\leq j\leq c_{i}\}. We retain unconditionally up to nsinkn_{\mathrm{sink}} initial sink blocks and a local window of w≥1w\geq 1 blocks ending at the query’s own block,
ℱi={1,…,min(nsink,ci)}∪{max(1,ci−w+1),…,ci}⊆𝒞i.\mathcal{F}_{i}=\{1,\ldots,\min(n_{\mathrm{sink}},c_{i})\}\cup\{\max(1,c_{i}-w+1),\ldots,c_{i}\}\subseteq\mathcal{C}_{i}. (2)
Let σ>0\sigma>0 denote the softmax scale and ri⋆=max1≤j<cirijr_{i}^{\star}=\max_{1\leq j<c_{i}}r_{ij} the largest score over complete blocks preceding the query’s own block. If ci=1c_{i}=1, set ri⋆=0r_{i}^{\star}=0; there are no optional blocks. We retain
𝒮i=ℱi∪{j∈{1,…,ci−1}∖ℱi:exp(σ(rij−ri⋆))≥α},α∈(0,1].\mathcal{S}_{i}=\mathcal{F}_{i}\cup\Bigl\{j\in\{1,\ldots,c_{i}-1\}\setminus\mathcal{F}_{i}:\exp\bigl(\sigma(r_{ij}-r_{i}^{\star})\bigr)\geq\alpha\Bigr\},\qquad\alpha\in(0,1]. (3)
Equivalently, taking logarithms on both sides of the inequality in Eq. (3) and dividing by σ\sigma, an optional block is retained when
ri⋆−rij≤−logασ=:Δ,r_{i}^{\star}-r_{ij}\leq-\frac{\log\alpha}{\sigma}=:\Delta, (4)
i.e., when its score lies within a fixed margin Δ\Delta of the strongest preceding block for that query. The decision therefore requires no softmax normalization. Unlike fixed-budget routing, which fixes cardinality and induces a query-dependent score cutoff, our rule fixes a query-relative score criterion and lets |𝒮i||\mathcal{S}_{i}| vary across queries and heads.
The current block is retained with token-level causal masking, but its pooled key is excluded from the reference maximum because it can contain future tokens. Preceding forced blocks still contribute to the maximum. The rule depends only on score differences relative to ri⋆r_{i}^{\star}, so it is invariant to an additive shift of the preceding-block scores for a query.
The router emits this representation with a two-pass streaming construction and a prefix-sum scatter into compact CSC metadata; we defer those details to Appendix E and only use their output here.
3.3 KV-Centric Execution
The sparse pattern produced by routing is content-dependent, whether or not its per-query cardinality is fixed. FlashMoBA assigns work by logical query ranges, so queries in different ranges that select the same KV block are handled by different CTAs, causing repeated KV loads and fragmented reuse across query-owned tasks. This limits the useful work each loaded KV tile can amortize.
We instead organize the sparse pattern by (query head,KV block)(\text{query head},\text{KV block}) columns and assign execution around these columns (Sec. 2.2). Queries selecting the same block under a query head are therefore grouped under the same KV-owned column, allowing each work unit to reuse its loaded KV tile across its assigned rows. Because a column’s task size is proportional to its fanout, assigning an entire column to one CTA can leave a long-running tail. We therefore separate two decisions: ownership, which determines which KV column a task belongs to, and task size, which partitions high-fanout columns into bounded work units. Alg. 1 illustrates the KV-centric ownership pattern before fanout-aware splitting: execution is organized by KV columns rather than query ranges. Sec. 3.3.2 then partitions high-fanout columns into bounded work items without changing this ownership rule.
3.3.1 KV-Centric Ownership
In FlashMoBA a CTA, scheduled as a unit onto an SM, owns a band of MM consecutive query rows and walks the blocks those rows selected, gathering only that band’s rows so that routing induces no masked lanes. Write
𝒬h,j={i:j∈𝒮h,i},λh,j(M)=|{b:𝒬h,j∩[bM,(b+1)M)≠∅}|,\mathcal{Q}_{h,j}=\{\,i:j\in\mathcal{S}_{h,i}\,\},\qquad\lambda_{h,j}(M)=\bigl|\{\,b:\mathcal{Q}_{h,j}\cap[bM,(b{+}1)M)\neq\emptyset\,\}\bigr|, (5)
for the queries that selected column (h,j)(h,j) and the number of bands claiming it, where a column is block ℬj\mathcal{B}_{j} as seen by query head hh. With CC selected columns, the mean over columns
Λ(M)=1C∑h,jλh,j(M)=|𝒱(M)|C,μ(M)=Z|𝒱(M)|=f¯Λ(M),f¯=ZC,\Lambda(M)=\frac{1}{C}\sum_{h,j}\lambda_{h,j}(M)\;=\;\frac{|\mathcal{V}(M)|}{C},\qquad\mu(M)=\frac{Z}{|\mathcal{V}(M)|}=\frac{\bar{f}}{\Lambda(M)},\qquad\bar{f}=\frac{Z}{C}, (6)
counts how many times a column is claimed, where 𝒱(M)\mathcal{V}(M) is the set of (band, column) pairs carrying work and f¯\bar{f} is the mean fanout. These two quantities determine the costs of band ownership. A column’s K/V is requested λh,j(M)\lambda_{h,j}(M) times rather than once. Since a physical KV block may appear in up to gg query-head columns, band fragmentation compounds with GQA sharing: Λ(M)\Lambda(M) measures the mean multiplicity at the column level, while the request count of a physical block is the sum of the multiplicities of its participating columns, and how many of those requests reach HBM depends on cache behaviour. Independently, each request exposes only μ(M)\mu(M) selected rows with which to fill an MMA tile that holds mMMAm_{\mathrm{MMA}} rows. The two are not independent: ZZ and CC are fixed by the routing pattern, and MM changes only |𝒱(M)||\mathcal{V}(M)|, so dividing numerator and denominator by CC gives
μ(M)=Z|𝒱(M)|=Z/C|𝒱(M)|/C=f¯Λ(M).\mu(M)=\frac{Z}{|\mathcal{V}(M)|}=\frac{Z/C}{|\mathcal{V}(M)|/C}=\frac{\bar{f}}{\Lambda(M)}.
At fixed f¯\bar{f}, raising Λ\Lambda multiplies the requests per column and divides the work each request carries by the same factor. The arithmetic is unchanged: fragmenting leaves the routing pattern untouched, so the same ZZ query-block pairs must still be evaluated, and CC, f¯\bar{f} and the 𝒪(ZBd)\mathcal{O}(ZBd) term of Sec. A.1 are unaffected. What changes is how often the K/V operand is fetched and how much of each MMA tile is filled with work that is actually useful.
Within fixed query-band ownership, increasing MM is the direct way to reduce this fragmentation (shared memory caps it, but the cap is not what makes MM the wrong knob): a band claims a column alone only if it contains every query that selected it. The resulting necessary condition is stated below, with sh,j=max𝒬h,j−min𝒬h,j+1s_{h,j}=\max\mathcal{Q}_{h,j}-\min\mathcal{Q}_{h,j}+1 denoting the query span of a column.
Lemma 3.1 (Span bound for unit multiplicity).
Unit multiplicity requires
Λ(M)=1⟹M≥maxh,jsh,j.\Lambda(M)=1\quad\Longrightarrow\quad M\ \geq\ \max_{h,j}s_{h,j}. (7)
The converse does not hold.
The requirement is not an artifact of how MM is chosen. Band ownership groups queries by position, whereas routing groups them by content, so the two partitions coincide only by accident; a column selected by both an early and a late query spans the sequence, and no band shorter than that span can own it alone. This is a property of content-based routing rather than of thresholding, so it also applies to a fixed budget. It explains why query-band ownership works for local or structured sparsity, where each block’s consumers are contiguous by construction, but fragments content-dependent patterns. Fig. 9 evaluates Eq. (6) on captured routing patterns; we use it to quantify how much fragmentation remains at practical values of MM.
Making the selected column (h,j)(h,j) the ownership domain removes band fragmentation: all selected rows of a column are grouped under the same KV owner before tasks are formed, and MM leaves the ownership decision entirely. This gives unit multiplicity at the column level before deliberate fanout splitting, λh,j=1\lambda_{h,j}=1, and makes every selected row available for tile formation. It does not fix how many CTAs serve the column: Sec. 3.3.2 partitions it into several work units that each load the tile for their assigned rows, emit a partial output and local log-sum-exp, and are combined by the online-softmax merge, which is exact in real arithmetic.
3.3.2 Fanout-Aware Workload Decomposition
KV-centric ownership removes band fragmentation by grouping all consumers of a column under one ownership domain. A column’s work grows with its fanout fh,j=|𝒬h,j|f_{h,j}=|\mathcal{Q}_{h,j}|, the number of queries selecting it. A task covering the column serially processes
wh,j=⌈fh,j/mMMA⌉w_{h,j}=\bigl\lceil f_{h,j}/m_{\mathrm{MMA}}\bigr\rceil (8)
MMA tiles, so fanout becomes task size and dispersion in fanout becomes dispersion in task duration. Fixed Top-kk bounds how many blocks a query keeps, not how many queries keep a block, so this dispersion is a property of the routing rather than of thresholding.
Dispersion in task size affects scheduling differently from total work. Lemma E.1 in Appendix E.2 specializes the classical list-scheduling bound to these work units: Eq. (11) separates an aggregate throughput term W/PW/P from a largest-work-item term maxh,jwh,j\max_{h,j}w_{h,j}. Fanout dispersion raises the latter. Additional hardware parallelism reduces the first term but cannot subdivide the largest item, and tensor parallelism sharpens this tail because a synchronized step waits for its slowest device.
Fig. 4 characterizes the source of that tail on captured routing patterns. It reports the mean and selected column-fanout quantiles for one high-tail prefill trace, normalized by its mean of 2,031 query rows. The median is only 0.064×0.064\times the mean, whereas P99 and the maximum reach 28.1×28.1\times and 64.5×64.5\times the mean. This dispersion raises the largest-work-item term in Eq. (11). The corresponding control trade-off is shown in Fig. 7 in Appendix F: decreasing the task-size cap TT reduces the largest work item while increasing load multiplicity. Both figures use the same actual all-pair columns, including compulsory sink and local selections; they characterize execution work rather than content-selected fanout alone. The optional-only tail is reported separately in Appendix B.1.4.
Fanout-aware splitting
To prevent a high-fanout column from defining the critical path, we decouple ownership from task size. Ownership remains KV-centric; only the granularity of individual work units changes. Capping the number of selected query rows a unit may claim, we partition each column into work units (h,j,ibegin,icount)(h,j,i_{\text{begin}},i_{\text{count}}) subject to icount≤Ti_{\text{count}}\leq T. The cap is on the count of selected rows in the column’s compact row list, not on the sequence positions those rows occupy, so it is unrelated to the span sh,js_{h,j} of Sec. 3.3.1. A count cap suffices because the metadata of Sec. E.1 lists a column’s selected rows contiguously even when those queries are scattered over the input sequence.
The exchange is explicit in both directions: maxh,jwh,j\max_{h,j}w_{h,j} is bounded by ⌈T/mMMA⌉\lceil T/m_{\mathrm{MMA}}\rceil, while a column of fanout fh,jf_{h,j} becomes ⌈fh,j/T⌉\lceil f_{h,j}/T\rceil units that each independently load the column’s K/V operand and produce the outputs for their assigned query rows, raising the multiplicity to
ΛT=1C∑h,j⌈fh,j/T⌉< 1+f¯/T.\Lambda_{T}=\frac{1}{C}\sum_{h,j}\bigl\lceil f_{h,j}/T\bigr\rceil\;<\;1+\bar{f}/T. (9)
Fanout-aware splitting therefore trades within-unit KV reuse for finer-grained parallelism and improved load distribution, rather than obtaining them alongside it, and both sides of the trade are computable from the routing metadata before the attention work is launched: lowering TT reduces the largest-task term in Eq. (11) while increasing ΛT\Lambda_{T}. Taking TT to be a multiple of mMMAm_{\mathrm{MMA}} preserves the total number of MMA row tiles for each column while redistributing those tiles among independently schedulable work units. Sec. 4.3.4 (Table 7) sweeps TT and reports the setting we use.
3.4 Runtime Pipelining
KV-centric execution creates asymmetric operand access: K/V tiles remain contiguous, whereas Q rows must be gathered from the routing metadata. We assign regular K/V transfers to the Tensor Memory Accelerator (TMA) and irregular Q gathers to a producer path, allowing both paths to be staged concurrently with Tensor Core computation. The implementation targets Hopper’s TMA, WGMMA and warp-specialized execution model; Sec. B.3 discusses portability.
4 Experiments
4.1 Experimental Setup
We evaluate end-to-end inference and training performance, controlled execution and routing costs, and long-context quality. Inference comparisons are training-free; training integration is evaluated separately under the settings in Sec. 4.2.2. Model configurations, benchmark settings, and timing scopes are specified below and in the corresponding result sections.
Models
The evaluation uses five checkpoints: Qwen3-4B-Instruct-2507 (Qwen3-4B), Qwen3-235B-A22B, Qwen3-32B-YaRN4, Llama-3.1-8B-Instruct (Llama-3.1-8B), and GQA-30B-A3B, an internal GQA model (30B total, 3B active) whose architecture follows Qwen3 and is trained from scratch. These span dense and mixture-of-experts architectures as well as different GQA grouping ratios; checkpoint sources are listed with the full setup in Appendix D. Each reported configuration identifies the checkpoint revision, tested context lengths, head configuration, and the layers to which sparse attention is applied.
We compare against an optimized implementation of the same block-attention method; its baseline definition, training-free protocol and model sources are in Appendix D.
4.2 End-to-End Performance
4.2.1 End-to-End Inference Performance
Table 1 reports the end-to-end comparison on RULER at the Top-1616 reference budget. The baseline is FlashMoBA with fixed Top-1616 routing, one sink block and a two-block local window; LightningMoBA uses dynamic threshold prefill with the same compulsory blocks and a matched average Top-1616 retention budget. Both variants use dense decoding with CUDA Graph enabled. Every entry aggregates the full 6,500-example RULER suite at tensor parallelism eight and batch size one, and times are generation-loop wall times from the first request to the completion of the last, excluding model loading, data preparation and result serialization. Top-1616 is the routing budget used by the deployed baseline. We further extend the end-to-end evaluation to the Top-3232 reference budget using the same five checkpoints and full 6,500-example RULER suite (Appendix C.1, Table 14). LightningMoBA achieves lower generation time in all 30 model–length configurations, with per-model average speedups of 1.23–1.49×\times over FlashMoBA. Controlled routing-cost measurements across different retention budgets are reported separately in Section 4.3.2.
Table 1: End-to-end RULER results at the Top-1616 reference budget over the full 6,500-example suite. Each cell reports Base / Ours; wall times are in minutes with speedup in parentheses. Averages are over the six context lengths for each model, and the better value is bolded.
Length Qwen3-4B GQA-30B-A3B Qwen3-235B-A22B Llama-3.1-8B Qwen3-32B-YaRN4
(a) RULER score: Base / Ours
4K 93.75 / 94.29 94.40 / 94.10 95.26 / 94.86 95.27 / 94.96 89.15 / 88.98
8K 85.98 / 91.87 88.20 / 89.15 94.27 / 94.65 91.82 / 93.54 85.90 / 87.80
16K 79.45 / 87.18 82.47 / 80.88 89.18 / 93.26 83.62 / 87.25 74.97 / 74.11
32K 69.98 / 79.93 74.09 / 70.82 79.72 / 84.03 70.73 / 77.76 67.44 / 69.96
64K 59.08 / 53.41 57.14 / 55.72 70.87 / 68.78 61.05 / 57.57 61.58 / 56.32
128K 46.35 / 46.51 34.92 / 35.40 63.78 / 62.90 41.45 / 41.75 50.85 / 50.54
Average 72.43 / 75.53 71.87 / 71.01 82.18 / 83.08 73.99 / 75.47 71.65 / 71.29
(b) Wall time: Base / Ours (speedup)
4K 145.9 / 60.5 (2.41×\times) 119.1 / 36.8 (3.24×\times) 325.6 / 123.9 (2.63×\times) 135.6 / 70.9 (1.91×\times) 270.6 / 171.6 (1.58×\times)
8K 153.1 / 69.5 (2.20×\times) 118.7 / 41.1 (2.89×\times) 368.6 / 150.8 (2.44×\times) 142.7 / 75.6 (1.89×\times) 290.3 / 195.0 (1.49×\times)
16K 165.0 / 87.7 (1.88×\times) 116.5 / 59.3 (1.96×\times) 458.2 / 206.0 (2.22×\times) 150.7 / 88.2 (1.71×\times) 317.8 / 240.0 (1.32×\times)
32K 174.8 / 114.3 (1.53×\times) 146.2 / 103.9 (1.41×\times) 579.3 / 341.2 (1.70×\times) 166.1 / 117.0 (1.42×\times) 394.1 / 327.7 (1.20×\times)
64K 206.7 / 153.1 (1.35×\times) 178.8 / 122.5 (1.46×\times) 785.0 / 537.1 (1.46×\times) 192.8 / 158.3 (1.22×\times) 504.1 / 446.8 (1.13×\times)
128K 284.2 / 251.6 (1.13×\times) 293.0 / 253.1 (1.16×\times) 1180.1 / 957.4 (1.23×\times) 277.9 / 241.3 (1.15×\times) 812.3 / 770.9 (1.05×\times)
Average 188.3 / 122.8 (1.53×\times) 162.1 / 102.8 (1.58×\times) 616.1 / 386.1 (1.60×\times) 177.6 / 125.2 (1.42×\times) 431.5 / 358.7 (1.20×\times)
Additional comparisons with MoBA and FlexPrefill are provided in Appendix C.3.
4.2.2 End-to-End Training Performance
Training integration.
We further extend LightningMoBA to end-to-end training and evaluate its performance on Qwen3-4B. The forward pass uses dynamic threshold routing and the optimized KV-centric attention implementation, while the backward pass adapts the original FlashMoBA CUDA kernels to consume dynamically generated CSC metadata and merged forward outputs, including the global log-sum-exp (LSE). The backward implementation also handles gradient accumulation and grouped key/value gradient reductions for GQA. Routing decisions are treated as non-differentiable, and gradients are computed over the retained sparse attention pattern. We compare dynamic threshold routing against fixed TopK under three matched average retention budgets: Top-16, Top-32, and Top-64. Both configurations share the optimized KV-centric forward and adapted backward implementation; only the routing strategy differs.
Training setup.
All six runs start from the same Qwen3-4B-Instruct-2507 weights and use identical PG19 windows at 128K for 100 training steps (seed 1234). We use tensor parallelism of 1, global and micro batch sizes of 1, BF16 precision, and AdamW with a learning rate of 10−510^{-5} and weight decay of 0.1. Block size is 128; one sink block and a two-block local window count toward the retention budget. Thresholds are calibrated on disjoint PG19 windows and fixed throughout each run. Reported averages include all 100 steps. TFLOPS/GPU is an effective throughput metric: analytically estimated sparse GEMM FLOPs, including model linear layers and retained attention, are divided by the measured end-to-end training step time and GPU count, aggregated over the run. The FLOP estimate excludes recomputation, routing, and non-GEMM operations, but their execution costs remain in the measured step time. This metric is not a hardware-counter measurement or model FLOPs utilization (MFU).
Table 2: End-to-end training performance of Qwen3-4B under Top-16, Top-32, and Top-64 reference budgets. We compare fixed TopK routing with LightningMoBA’s dynamic threshold routing at matched average retained-block budgets, using the same optimized KV-centric forward and adapted backward implementations. Avg. CE and Last-10 CE denote the average training cross-entropy over all 100 steps and the final 10 steps, respectively. Step (s) is the mean end-to-end training iteration time in seconds; TFLOPS/GPU reports effective sparse-computation throughput. Blocks denotes the average number of retained KV blocks per query. Speedup is the ratio of fixed-TopK to dynamic-threshold step times.
Budget Method Avg. CE ↓\downarrow Last-10 CE ↓\downarrow Step (s) ↓\downarrow TFLOPS/GPU ↑\uparrow Blocks Speedup
16 TopK 2.2147 2.2756 22.280 162.48 15.883 1.000×\times
16 LightningMoBA 2.2222 2.2780 22.361 161.96 15.933 0.996×\times
32 TopK 2.1909 2.2595 39.558 103.24 31.516 1.000×\times
32 LightningMoBA 2.1967 2.2615 32.284 126.41 31.405 1.225×\times
64 TopK 2.1802 2.2531 85.645 58.26 62.031 1.000×\times
64 LightningMoBA 2.1810 2.2507 53.470 94.58 62.295 1.602×\times
Training results.
Table 2 summarizes end-to-end training performance across three retention budgets. The reported average block counts differ by less than 0.5% between fixed TopK and dynamic threshold routing. At Top-16, the two configurations achieve nearly identical training speed (0.996×\times). With the same execution backend, dynamic routing achieves 1.225×\times and 1.602×\times higher end-to-end training throughput at Top-32 and Top-64, respectively. Both the average and final-10-step training CE remain close to the corresponding fixed-TopK results. These findings demonstrate the applicability of LightningMoBA to training and the increasing benefit of dynamic routing at larger retention budgets over the evaluated run.
4.3 System Efficiency and Ablation Studies
4.3.1 Cumulative Implementation Ablation
Adding one component at a time to FlashMoBA Top-kk gives the cumulative device-path latency of Table 3, each variant timed on the same captured BF16 Q/K/V with routing, CSC construction, attention and merge included. The budget exposes the scaling difference: the final path is 1.56 to 1.65×\times faster than FlashMoBA at Top-1616 and 2.16 to 2.56×\times at Top-3232, because ordered Top-kk maintenance grows with kk while the threshold stage applies one scalar test per recomputed score.
Table 3: Cumulative device-path latency (ms): FlashMoBA, specialized unsplit KV kernel, split-40964096 worklist, SM90 pipeline, and matched-density threshold routing. The first four variants share saved Top-kk CSC metadata. Speedup compares the final variant with FlashMoBA.
Model Length kk FlashMoBA +KV +Split +Pipeline +Threshold Speedup
Qwen3-4B 32K 16 25.529 22.091 20.564 16.520 15.632 1.63×\times
Qwen3-4B 128K 16 137.243 104.740 97.489 81.813 83.284 1.65×\times
GQA-30B-A3B 32K 16 24.543 21.574 19.961 15.866 15.786 1.56×\times
GQA-30B-A3B 128K 16 131.002 102.606 95.244 79.751 82.249 1.59×\times
Qwen3-4B 32K 32 56.083 48.102 47.797 37.821 24.291 2.31×\times
Qwen3-4B 128K 32 312.387 262.775 261.526 221.020 121.845 2.56×\times
GQA-30B-A3B 32K 32 52.660 45.178 45.107 35.154 24.380 2.16×\times
GQA-30B-A3B 128K 32 287.236 243.851 242.549 202.464 120.512 2.38×\times
Timing scopes and stage breakdown.
The fixed Top-kk variants share saved CSC metadata; the threshold variant uses a separately saved pattern with matched valid pair counts. Adjacent variants also change kernel families; single-factor tests follow below. The replay and validation protocol appears in Appendix B.1.1.
Fig. 3 complements the cumulative table with a stage breakdown against FlashMoBA and dense FA3 at 32K and 128K. Complete attention device-path totals include pooling, routing/CSC construction, reverse-index work, attention, and merge; isolated kernel timings exclude the surrounding stages. Full-model generation loops in Sec. 4.2.1 also include other model computation and dense decoding. Replay speedups are therefore not full-model inference speedups. At 32K, LightningMoBA’s complete attention path remains slower than dense FA3 (15.8 vs. 13.0 ms) due to sparse routing and metadata overheads, whereas at 128K, the reduction in attention computation outweighs these costs, yielding a 2.56×\times speedup over FA3 (83.2 vs. 213.2 ms).
Figure 3: Complete forward attention-path timing breakdown for Qwen3-4B at (a) 32K and (b) 128K on one SM90 GPU. Each panel compares dense FA3, FlashMoBA Top-1616, and the final LightningMoBA configuration. The panels use separate latency axes; sparse bars include pooling, routing/CSC construction, attention, reverse-index work, and merge.
4.3.2 Routing Efficiency and End-to-End Impact
Isolating routing cost shows why the budget matters. Against FlashMoBA’s ordered Top-kk candidate set at matched realized pair counts, threshold routing is faster by 3.15 to 4.13×\times at k=32k=32 and 6.99 to 11.25×\times at k=64k=64, and the gap widens with kk (Table 4).
Table 4: Complete routing sweep including CSC construction. Each group reports Top-kk and threshold latency (ms) with their ratio; each threshold setting matches the valid CSC pair count within 0.05%. Timings use CUDA Graph replay.
k=16k=16 k=32k=32 k=64k=64
Model Length Top-kk Thresh. Speedup Top-kk Thresh. Speedup Top-kk Thresh. Speedup
Qwen3-4B 16K 1.86 1.40 1.33×\times 6.32 1.91 3.31×\times 20.63 2.88 7.17×\times
Qwen3-4B 32K 4.06 3.39 1.19×\times 17.93 4.34 4.13×\times 64.01 6.02 10.64×\times
Qwen3-4B 64K 10.25 10.36 0.99×\times 47.47 12.65 3.75×\times 177.28 17.26 10.27×\times
Qwen3-4B 128K 30.70 32.28 0.95×\times 135.33 36.81 3.68×\times 514.42 45.74 11.25×\times
GQA-30B-A3B 16K 1.66 1.35 1.23×\times 5.58 1.77 3.15×\times 19.22 2.75 6.99×\times
GQA-30B-A3B 32K 3.52 3.26 1.08×\times 15.28 4.12 3.71×\times 54.65 6.22 8.79×\times
GQA-30B-A3B 64K 10.01 9.68 1.03×\times 44.82 11.62 3.86×\times 159.34 15.99 9.96×\times
GQA-30B-A3B 128K 28.57 31.85 0.90×\times 116.96 35.48 3.30×\times 416.01 44.56 9.34×\times
These CUDA Graph timings include scoring, selection, and CSC construction, but exclude mean pooling and attention. They therefore characterize the router, not the complete device path or full-model inference. Each model–length–budget configuration is calibrated outside timing to match attention-visible pair counts; Appendix B.1.2 details this protocol and the separate decision-stage measurement. At Top-1616, the complete-router advantage varies with context length and reverses at 128K in both models; larger budgets favor threshold selection in every measured case.
Controlled end-to-end impact.
Three configurations distinguish execution from routing: FlashMoBA uses fixed Top-kk with its original execution backend; optimized fixed Top-kk uses the LightningMoBA backend; and LightningMoBA uses dynamic threshold routing with that same optimized backend. Table 5 and Fig. 6 in Appendix F compare the latter two at matched average retention budgets of 16, 32, and 64 on Qwen3-4B and GQA-30B-A3B at 32K, 64K, and 128K. Every entry aggregates the full 6,500-example RULER suite at tensor parallelism eight and batch size one. Both use the same KV-centric execution, fanout-aware splitting, and runtime pipelining; times cover the complete generation loop, not isolated routing or attention kernels. These fixed Top-kk results are therefore distinct from the original FlashMoBA baseline in Table 1.
For corresponding measurements on the same model, length, and budget, let tFt_{\mathrm{F}}, tKt_{\mathrm{K}}, and tDt_{\mathrm{D}} denote the complete runtimes of original FlashMoBA, optimized fixed Top-kk, and dynamic routing. The total speedup factors as tF/tD=(tF/tK)(tK/tD)t_{\mathrm{F}}/t_{\mathrm{D}}=(t_{\mathrm{F}}/t_{\mathrm{K}})(t_{\mathrm{K}}/t_{\mathrm{D}}): an execution-backend ratio times a routing-policy ratio. These are ratios of measured complete runtimes, not independently additive kernel-time contributions; the identity does not apply across different timing protocols or unmatched configurations.
Table 5: Controlled end-to-end comparison of fixed Top-kk and dynamic threshold routing under the identical optimized LightningMoBA execution backend. Results cover Qwen3-4B and GQA-30B-A3B at 32K, 64K, and 128K, with matched average retention budgets of 16, 32, and 64. Both strategies use the same KV-centric execution, fanout-aware splitting, and runtime pipelining. RULER scores are reported in percentages, generation times in minutes, and speedups are calculated as Fixed / Dynamic.
Model Length Budget RULER score E2E (min) Speedup
Fixed Dynamic Fixed Dynamic
Qwen3-4B 32K 16 70.48 79.93 123.52 114.33 1.08×\times
32K 32 78.30 87.87 150.54 131.43 1.15×\times
32K 64 86.06 91.13