Over 107tok/s Qwen3.8-Flash-Next on M3 Ultra

2026-09-09 posted in AI, Performance

Ours: 107.16 tok/s · llama.cpp (MTP3): 55.74 tok/s · +92.2%.

Decode speed on the Rust sorted-slice merge task with a 4,096-token prompt and 512-token output: llama.cpp MTP3 55.74 tok/s; Ours 107.16 tok/s. Three-run medians.

Figure 1. Rust merge benchmark.

tldr

I optimized llama.cpp to keep cached attention keys after speculative decoding rejects tokens. The rejected tokens change only the end of the sequence, so rebuilding keys for the earlier history wastes work. Keeping those keys leaves a small update; padding it to eight rows gives the runtime a consistent shape to execute.

The implementation is public on GitHub. These runs use Huihui’s Qwen3.8-Flash-Next abliterated GGUF. The target weights are unchanged.

For the comparison above, I measured the Unsloth build of llama.cpp with MTP3 against our fastest saved result with MTP5. The task is to write a Rust function that merges two sorted integer slices. Both runs process the same 4,096-token prompt and generate 512 tokens; the reported speed covers generation only. MTP3 and MTP5 allow up to three and five draft tokens per step, respectively.

Speed as the prompt grows

The context sweep compares the same two configurations from 512 to 6,144 prompt tokens, with 128 generated tokens per request. Each point is a median of five runs on the Rust merge task. Ours reaches 87.13 tok/s at 4K here; the headline 107.16 tok/s comes from the separate run generating 512 tokens.

Decode speed in tok/s, llama.cpp MTP3 versus Ours, by prompt tokens. 512: 52.08 versus 104.53; 1K: 47.93 versus 97.67; 2K: 48.71 versus 91.02; 4K: 49.33 versus 87.13; 6K: 48.81 versus 84.95. Each request generates 128 tokens; points show five-run medians and whiskers show the run range.

Figure 2. llama.cpp and Ours across prompt lengths.

Why rollback was rebuilding history

Qwen Sparse Attention (QSA) groups earlier tokens into blocks and computes a summary vector, or pooled key, for each block. Each new query scores these keys to decide which parts of the history to read. Earlier work explores reuse across layers and speculative queries [1], [2], [3]. Here, we reuse keys across rollback.

Speculative decoding proposes several tokens, checks them together, and removes the rejected suffix. Earlier tokens survive. Their pooled keys also remain in memory, but rollback cleared the records that told the runtime those keys were valid. The next verification could then compute them again.

We changed rollback to retain the records for blocks whose inputs are still valid.

flowchart TB
  accTitle: Speculative decoding and rollback maintenance pipeline
  accDescr: Draft and verify candidates, commit the accepted prefix, then remove the rejected suffix. Validate historical QSA blocks and reuse valid keys while materializing changed blocks for the next round.
  subgraph ROUND["1 · SPECULATIVE ROUND"]
    direction LR
    Draft["Draft candidate<br/>tokens"] --> Verify["Verify candidates<br/>update QSA + save state"]
    Verify --> Accept["Commit accepted<br/>token prefix"]
  end
  subgraph ROLLBACK["2 · SUFFIX REMOVAL"]
    direction LR
    Recurrent["Restore recurrent<br/>state"] --> Remove["Remove rejected<br/>indexer + KV cells"]
    Remove --> Pending["Discard pending<br/>QSA layout"]
  end
  subgraph MAINTAIN["3 · RETAIN VALID HISTORICAL STATE"]
    direction TB
    Guard["Validate historical<br/>prefix descriptors"]
    Guard -->|validated prefix| Keep["REUSE<br/>Retain pooled key"]
    Guard -->|remaining descriptors| Delta["INVALIDATE<br/>Drop stale descriptors"]
    Keep --> Ready["Retained prefix metadata<br/>ready for the next verification"]
    Delta --> Ready
  end
  ROUND --> ROLLBACK
  ROLLBACK --> MAINTAIN
  MAINTAIN --> Next["Next speculative round"]
  class Keep reuse
  class Delta work
  class Draft,Verify,Accept,Recurrent,Remove,Pending,Guard,Ready,Next state
  classDef reuse fill:#e8f1fc,stroke:#356da8,color:#173b63,stroke-width:2px,stroke-dasharray:5 3
  classDef work fill:#fff2d9,stroke:#ad7620,color:#66430e,stroke-width:2px
  classDef state fill:#eef1f4,stroke:#64748b,color:#263545

Figure 3. Speculative decoding and rollback.

Keep the pooled keys; score them again

To reuse a pooled key, we need to know that its source keys and positions have not changed. A position alone is insufficient because cache storage can be recycled or shared. We check each complete block from oldest to newest, keeping it only while its cells still belong to the expected sequence and hold the expected positions. We stop at the first invalid block.

At the next verification, we compare each block’s source cells, positions, destination, and stream with its retained record. Only new or changed blocks need keys. The append path can supply this update list directly when its validity checks pass.

The current query still scores both retained and updated keys to choose which blocks to read. Cache scanning and query scoring continue to grow with history length.

flowchart TB
  accTitle: Incremental QSA pipeline with historical-key reuse
  accDescr: Retained keys bypass materialization. Changed or new blocks become a compact update list, pass through pooling and normalization, and write back into the cache. The current query scores both retained and updated keys.
  Layout["Current blocks + retained descriptors"] --> Compare["Compare descriptors<br/>identify the real delta"]
  Compare -->|unchanged prefix| Cache[("REUSE<br/>Historical pooled keys")]
  Compare -->|changed or new| Pack["Pack real updates + eligible fillers<br/>C update rows"]
  subgraph UPDATE["POOLED-KEY MATERIALIZATION"]
    direction TB
    Pool["Gather r source keys per row<br/>mean pool to one block key"] --> Transform["Cache-type round trip<br/>RMSNorm → RoPE"]
    Transform --> Write["SET_ROWS<br/>write C cache destinations"]
  end
  Pack --> UPDATE
  UPDATE --> Score["Current-query block scores<br/>causal mask → top-k selection"]
  Cache -. bypass materialization .-> Score
  Query["Current query<br/>projection + norm + RoPE"] --> Score
  Score --> Attention["Attention over selected blocks<br/>plus the current token tail"]
  class Cache reuse
  class Pack,Pool,Transform,Write work
  class Layout,Compare,Score,Query,Attention state
  classDef reuse fill:#e8f1fc,stroke:#356da8,color:#173b63,stroke-width:2px,stroke-dasharray:5 3
  classDef work fill:#fff2d9,stroke:#ad7620,color:#66430e,stroke-width:2px
  classDef state fill:#eef1f4,stroke:#64748b,color:#263545

Figure 4. Reusing cached keys.

Recurrent-state removal runs first because it can refuse the rollback. Once indexer and attention-cache removal succeed, we discard the pending update layout and trim the records of computed blocks.

This path handles one index-cache stream, sequence 0, and removal from a nonnegative position to the end. Other removals clear the records.

Here, $\operatorname{ValidBlock}(b,r,p_0,M)$ requires exactly $r$ source cells, cache stream 0, a nonnegative start, and $\mathrm{start}+r\le p_0$ . The key destination must be one of those cells. All four stored positional components equal the block start. Each source cell must be in bounds, live, exclusively owned by sequence 0, and at logical position $\mathrm{start}+i$ .

Algorithm 1. Retain a rollback-valid QSA prefix

Input: Cached block vectors $\mathcal{B}_{r}$ , surviving cell metadata $M$ , sequence $q$ , removal bounds $p_0,p_1$ (negative $p_1$ denotes an unbounded suffix).

Output: A retained descriptor prefix for each pooling ratio; surviving key values are unchanged.

Require recurrent removal accepted and attention removal succeeded
$\mathcal{P}\gets\varnothing$ ▷ discard the pending layout
if preservation is disabled, indexer absent, streams ≠ 1, $q\ne0$ , $p_0<0$ , or $p_1\ge0$ then
return $\varnothing$ ▷ invalidate all cached descriptors
for each pooling ratio $r$ do
if the entry does not contain exactly one stream vector then
$\mathcal{B}_{r}\gets\varnothing$ ; continue
$k\gets0$
for each block $b$ in $\mathcal{B}_{r}$ , in prefix order do
if $\neg\operatorname{ValidBlock}(b,r,p_0,M)$ then break
$k\gets k+1$
$\mathcal{B}_{r}\gets\mathcal{B}_{r}[0:k]$
return $\{\mathcal{B}_{r}\}_{r}$

Invariant: Every retained block is complete, exclusively owned, and strictly before the removed suffix.

Algorithm 2. Incrementally materialize pooled indexer keys

Input: Pooling ratio $r$ , current complete blocks $\mathcal{B}$ , retained descriptors $\widehat{\mathcal{B}}$ , source keys $X$ , pooled-key cache $K$ .

Output: Updated pooled-key cache and descriptor cache for the current layout.

$\Delta\gets[\,]$
for each stream $s$ and current block index $b$ do
if $\widehat{\mathcal{B}}_{s,b}$ is absent or differs from $\mathcal{B}_{s,b}$ then
Append $(s,b)$ to $\Delta$
$(U,C)\gets\operatorname{UpdateList}(\Delta,\mathcal{B})$ ▷ Algorithm 3
for each descriptor $d\in U$ do ▷ evaluated as a batched tensor graph
$x\gets\operatorname{Gather}(X,d.\mathrm{cells})$
$z\gets\frac{1}{r}\sum_{i=0}^{r-1}x_i$
$z\gets\operatorname{CacheTypeRoundTrip}(z)$ ▷ only when the types differ
$z\gets\operatorname{RMSNorm}(z)$
$z\gets\operatorname{RoPE}(z,d.\mathrm{pos})$
Write $z$ to $K[\operatorname{Destination}(d)]$ via SET_ROWS
Resize $\widehat{\mathcal{B}}$ to the current stream/block dimensions
for each $(s,b)\in\Delta$ do $\widehat{\mathcal{B}}_{s,b}\gets\mathcal{B}_{s,b}$
return $(K,\widehat{\mathcal{B}})$

Invariant: Blocks outside the real delta retain valid values; optional filler writes are idempotent.

After preservation, one step may need a single new key and the next may need three. That variation became the next thing to address.

Why doing eight rows can beat doing two

Changing the number of update rows changes the graph’s tensor shapes. We pad small updates to eight rows so the runtime can reuse the same update shape.

When two blocks change, their keys occupy the first two rows. Six unchanged blocks fill the rest. Each filler keeps its original inputs and writes back to its own location, reproducing the value already there. We choose distinct destinations so the extra rows cannot write to the same address in parallel.

flowchart TB
  accTitle: Pad8 with unique cache destinations
  accDescr: Two real update blocks are combined with six unchanged blocks whose write destinations are distinct. Eight rows pass through one update shape, producing two new keys and six idempotent writes.
  Real["REAL UPDATES<br/>B6, B7 → d6, d7"] --> Used["Reserve real destinations<br/>used = d6, d7"]
  Existing["EXISTING BLOCKS<br/>B0 … B7"] --> Filter["Scan in order<br/>skip destinations already used"]
  Used --> Filter
  Filter --> Fill["FILLER UPDATES<br/>B0 … B5 → d0 … d5"]
  Real --> Rows["Fixed update list · C = 8<br/>B6, B7 · real<br/>B0 … B5 · fillers"]
  Fill --> Rows
  Rows --> Compute["Materialize 8 rows<br/>gather → pool → norm → RoPE"]
  Compute --> Write["SET_ROWS<br/>8 distinct real destinations"]
  Write --> Changed["2 newly materialized keys<br/>d6, d7"]
  Write --> Same["6 idempotent writes<br/>d0 … d5 keep their values"]
  class Existing,Fill,Same reuse
  class Real,Changed,Compute work
  class Used,Filter,Rows,Write state
  classDef reuse fill:#e8f1fc,stroke:#356da8,color:#173b63,stroke-width:2px,stroke-dasharray:5 3
  classDef work fill:#fff2d9,stroke:#ad7620,color:#66430e,stroke-width:2px
  classDef state fill:#eef1f4,stroke:#64748b,color:#263545

Figure 5. Two real updates padded to eight rows.

We spend a little extra computation to avoid changing the update shape. This helps when recomputing the filler keys costs less than the overhead it removes.

The implementation supports targets of eight or sixteen rows; these results use eight. Padding requires one stream, at most 16 tokens in the microbatch, an update count that fits the target, and enough complete blocks to supply distinct destinations. Larger updates use their ordinary size. A linear scan is sufficient for such a small list.

Even an empty update has a minimum capacity of one row. It uses an existing complete block when possible, or the general path’s fallback descriptor otherwise. With at least eight complete blocks, an empty delta can use Pad8 and recompute eight existing keys.

Algorithm 3. Construct a fixed-capacity update list

Input: Real update list $\Delta$ , current complete blocks $\mathcal{B}$ , target $P\in\{8,16\}$ , microbatch size $m$ , stream count $S$ .

Output: Ordered update list $U$ with capacity $C$ . This procedure describes execution, not graph reservation.

$C\gets\max(1,|\Delta|)$ ; $U\gets\operatorname{Descriptors}(\Delta,\mathcal{B})$
if padding is enabled, $m\le16$ , $S=1$ , $C\le P$ , and $|\mathcal{B}_0|\ge P$ then
$C\gets P$
if $U=\varnothing$ then
if a complete block exists then append the first such block to $U$
else append the existing minimum-row fallback descriptor to $U$
if padding is enabled and $S=1$ then
for each $b\in\mathcal{B}_0$ , in order do
if $|U|=C$ then break
$d\gets b.\mathrm{stream}\cdot\mathrm{cacheSize}+b.\mathrm{keyCell}$
if $d\notin\operatorname{Destinations}(U)$ then append $b$ to $U$
assert $|U|=C$
else extend $U$ to $C$ with the legacy first-row fill, if required
return $(U,C)$

Invariant: Every real update is retained; every added filler targets an existing destination not already in $U$ .

Drafting five tokens at a time

MTP3 is a practical choice in existing setups: one published depth sweep chose it for regular use, although MTP4 was slightly faster in that benchmark [4]. Drafting further gives each verification more chances to accept tokens, but also adds work that may be discarded.

Our changes reduce the cost of maintaining state between those rounds. That makes deeper drafting more affordable, and our fastest saved configuration uses MTP5. The useful depth depends on the runtime’s costs as well as the model’s acceptance rate.

The Rust merge benchmark supplies reference code and asks for a function that merges two sorted i32 slices into a Vec<i32>, preserving duplicates. Its input is exactly 4,096 tokens, and each run generates 512 tokens. The table reports median decode speed over three runs on this task.

Inference configuration	Decode speed
llama.cpp (MTP3, Unsloth build)	55.74 tok/s
Ours	107.16 tok/s

Ours is 92.2% faster on this workload. That includes the runtime changes, MTP5, and a Q4 draft output head; llama.cpp uses MTP3 with the supplied Q8 draft. This comparison measures their combined effect.

Measurement details

The measurements were taken in separate sessions: new llama.cpp runs and saved runs of our fastest configuration. Both use the same target GGUF and exact prompt tokens, greedy decoding, one sequence, Q8 K/V caches, an 8K context capacity, a 32-token warmup, and disabled prompt caching. For the headline comparison, the raw server logs record 4,096 prompt tokens and 512 generated tokens for every scored run. No production overlap was sampled. Different draft settings can change acceptance rates as well as execution cost.

A later extension: load snapshot history once

Rollback needs saved convolution histories for different numbers of accepted tokens. These snapshots are overlapping windows into the same short history row. Making each copy separately reads some of those values again and launches another GPU operation.

A later optimization combines those copies. Each thread loads one row, then writes every required snapshot from that local copy. For an eight-element row and three snapshots of three elements each, this reduces logical source reads from 17 to eight, and GPU dispatches from four to one. The throughput results above do not include this later change.

flowchart LR
  accTitle: Separate convolution snapshot passes compared with one fused pass
  accDescr: The separate path writes a concatenation and rereads overlapping windows for each snapshot. The fused path loads each channel and sequence row into local storage once, then writes the concatenation and every safe snapshot from that load.
  subgraph SEPARATE["SEPARATE PASSES"]
    direction TB
    SourceA["State S + new activations T"] --> Concat["Concatenate X<br/>read s + n elements"]
    Concat --> Copies["k snapshot copies<br/>reread k × s elements"]
    Copies --> CostA["Source reads: s + n + k × s<br/>Dispatches: 1 + k"]
  end
  subgraph FUSED["ONE FUSED PASS"]
    direction TB
    SourceB["State S + new activations T"] --> Load["One thread per channel / sequence<br/>load s + n words into V"]
    Load --> Joined["Write concatenation X<br/>from V"]
    Load --> Windows["Write every snapshot window<br/>from the same V"]
    Joined --> CostB["Source reads: s + n<br/>Dispatches: 1"]
    Windows --> CostB
  end
  SEPARATE ~~~ FUSED
  class Load,Windows reuse
  class Copies work
  class SourceA,SourceB,Concat,Joined,CostA,CostB state
  classDef reuse fill:#e8f1fc,stroke:#356da8,color:#173b63,stroke-width:2px,stroke-dasharray:5 3
  classDef work fill:#fff2d9,stroke:#ad7620,color:#66430e,stroke-width:2px
  classDef state fill:#eef1f4,stroke:#64748b,color:#263545

Figure 6. Separate and fused snapshot execution.

Before combining the copies, the CPU checks that they write to separate regions and waits for earlier GPU work that uses those regions. If a group fails the checks, it uses the original copy operations.

The specialization accepts F32 rows with $1\le s,n\le8$ and $s+n\le16$ . It examines up to eight following copy nodes and requires at least two eligible snapshots. Shapes and strides must match the supported layout, and destinations must be in bounds and disjoint from the inputs, concatenation, and each other. The first unsupported candidate ends the scan. Every output is tracked by the scheduler; a dependency barrier is inserted when needed.

Algorithm 4. Recognize and dispatch a safe snapshot group

Input: A concatenation node $X=S\Vert T$ , following copy nodes, and the encoder’s pending memory accesses.

Output: One fused dispatch consuming $k+1$ nodes, or the ordinary execution path.

if fusion is disabled or $\neg\operatorname{SupportedConcat}(X)$ then return ordinary path
$J\gets[\,]$ ; $D\gets\varnothing$
for $i=1,\ldots,8$ , while a following node exists do
$c\gets\operatorname{NextNode}(i)$
if $c$ is not a supported F32 copy of a view of $X$ then break
if ownership, shape, stride, or window-offset checks fail then break
if $c$ ’s destination is out of bounds or overlaps $S,T,X,D$ then break
Append $(\mathrm{windowStart},\mathrm{cacheOffset},\mathrm{sequenceStride})$ to $J$
Add $c$ ’s destination range to $D$
$k\gets|J|$ ; if $k<2$ then return ordinary path
if a destination conflicts with prior encoded work then insert a dependency barrier
Dispatch Algorithm 5 over all channel/sequence rows
Record memory accesses for the concatenation and every accepted copy
return $k+1$ consumed nodes

Invariant: Accepted destinations are in bounds, pairwise disjoint, and do not alias $S$ , $T$ , or $X$ .

These are logical read counts; actual memory traffic depends on caches and the compiler. The kernel writes the same outputs.

Each thread handles one channel/sequence row, using a fixed uint values[16] array. Integer copies preserve F32 bit patterns. The dispatch uses groups of 128 threads; source and destination strides preserve the supported sequence layouts.

Algorithm 5. Materialize concatenation and snapshot windows

Input: State $S$ of width $s$ , new activations $T$ of width $n$ , $H$ channels, $Q$ sequences, and safe window descriptors $J$ .

Output: Concatenated rows $X$ and all snapshot windows $R_j$ .

in parallel for each GPU thread index $g$ do
if $g\ge HQ$ then return
$h\gets g\bmod H$ ; $q\gets\lfloor g/H\rfloor$
Allocate thread-local $V[16]$ of unsigned 32-bit words
for $i=0,\ldots,s-1$ do $V[i]\gets\operatorname{LoadBits}(S_{i,h,q})$
for $i=0,\ldots,n-1$ do $V[s+i]\gets\operatorname{LoadBits}(T_{i,h,q})$
for $i=0,\ldots,s+n-1$ do $\operatorname{StoreBits}(X_{i,h,q},V[i])$
for each snapshot descriptor $j\in J$ do
$a\gets j.\mathrm{windowStart}$
$d\gets\operatorname{Uint32View}(\mathrm{cache}+j.\mathrm{offset}+q\cdot j.\mathrm{sequenceStride})$
for $i=0,\ldots,s-1$ do $\operatorname{StoreBits}(d[h s+i],V[a+i])$

Invariant: Each valid GPU thread owns one row and reads each source element once; output bit patterns are preserved.

References

[1] Yushi Bai et al. IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse. arXiv:2603.12201, 2026.

[2] Harsh Shah. SpecAttn: Speculating Sparse Attention. arXiv:2510.27641, 2025.

[3] Zhibin Wang et al. SSV: Sparse Speculative Verification for Efficient LLM Inference. arXiv:2605.19893, 2026.

[4] On The Stack. Qwen3.8-Flash-Next on the EVO-X2: 125B Parameters in 96GB of Unified Memory. 2026.