![\"\"](\"https:\/\/www.newsbeep.com\/ie\/wp-content\/uploads\/2026\/06\/Screenshot-2026-06-24-at-12.00.06-AM-1.png\")
https:\/\/developer.nvidia.com\/blog\/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash-speculative-decoding\/<\/p>\nAutoregressive large language models generate text one token at a time. Each token waits for the one before it. This serial loop leaves modern GPUs underused and keeps inference slow. The cost grows worse with long Chain-of-Thought reasoning models. Their lengthy outputs make latency the dominant part of generation.<\/p>\n
Speculative decoding is the standard fix. A small draft model proposes future tokens. The large target model verifies those tokens in parallel. Accepted tokens are kept, so the output stays lossless. But most methods, including the state-of-the-art EAGLE-3, still draft autoregressively. That serial drafting caps real-world speedups near 2\u20133\u00d7.<\/p>\n
DFlash, introduced by research team from UC San Diego team (z-lab), takes a different route. It is a lightweight block diffusion model built for drafting. Instead of drafting tokens one at a time, it proposes a whole block in a single forward pass. The target model then verifies that block in parallel.<\/p>\n
The research team reports over 6\u00d7 lossless acceleration across a range of models and tasks. It reaches up to 2.5\u00d7 higher speedup than EAGLE-3. On NVIDIA Blackwell, NVIDIA engineering team reports up to 15\u00d7 higher throughput for gpt-oss-120b. That figure holds at the same user interactivity target.<\/p>\n
https:\/\/developer.nvidia.com\/blog\/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash-speculative-decoding\/<\/p>\n
What block diffusion drafting changes<\/p>\n
Block diffusion models denoise a block of masked tokens at once. They blend parallel generation with autoregressive block structure. DFlash applies this idea only to the drafting stage. Verification stays with the trusted autoregressive target model.<\/p>\n
This split matters for quality. Standalone diffusion LLMs often trail autoregressive models on accuracy. They also need many denoising steps, which slows their raw inference speed. DFlash sidesteps both problems. The draft only needs to be good enough to be accepted. The target\u2019s parallel verification guarantees the final output distribution.<\/p>\n
A second benefit is drafting cost. An autoregressive drafter\u2019s cost grows linearly with the number of speculative tokens. A diffusion drafter generates all tokens in one parallel pass. So drafting latency stays largely flat as the block grows. This frees DFlash to use deeper, more expressive draft models without adding latency.<\/p>\n
This separates DFlash from earlier diffusion-drafter work. Methods like DiffuSpec and SpecDiff-2 used massive 7B drafters, capping speedups near 3\u20134\u00d7. DFlash instead uses a small five-layer drafter (eight layers for Qwen3-Coder).<\/p>\n
The \u201ctarget knows best\u201d insight<\/p>\n
DFlash\u2019s core idea is simple: the target knows best. Large autoregressive models\u2019 hidden features encode information about multiple future tokens. DFlash extracts hidden states from several target layers. It fuses them into one compact target context feature. This feature then conditions the draft model.<\/p>\n
DFlash injects this feature differently than EAGLE-3. EAGLE-3 fuses target features into the draft\u2019s input embeddings only. As draft depth grows, that signal gets diluted. DFlash instead injects the feature into the Key and Value projections of every draft layer. The projected features sit in the draft\u2019s KV cache and persist across drafting iterations.<\/p>\n
This KV injection lets acceptance length scale with draft depth. A five-layer DFlash drafter generating 16 tokens beats EAGLE-3 generating 8 tokens. It is both lower-latency and higher-acceptance in the paper\u2019s tests. The draft model effectively becomes a diffusion adapter on top of the target.<\/p>\n
Two speedup numbers, measured differently<\/p>\n
The DFlash research\u2019s 6\u00d7 is single-stream lossless acceleration. On Qwen3-8B with greedy decoding (Transformers backend), DFlash averages a 4.86\u00d7 speedup. EAGLE-3 averages 1.76\u00d7 at tree size 16 and 2.02\u00d7 at tree size 60. DFlash peaks at 6.08\u00d7 on MATH-500 (\u03c4 = 7.87) and averages \u03c4 = 6.49 across tasks.<\/p>\n
NVIDIA\u2019s 15\u00d7 is throughput at a fixed interactivity target. It applies to gpt-oss-120b on eight NVIDIA Blackwell GPUs in a DGX B300 system, using TensorRT-LLM. At the 500\u2013600 tokens\/sec per-user range, DFlash serves more than 15\u00d7 the throughput of autoregressive decoding. That is about 1.5\u00d7 more than EAGLE-3 at the same point.<\/p>\n
The table below shows the paper\u2019s per-task speedups on Qwen3-8B at temperature 0 (Transformers backend).<\/p>\n
Task (Qwen3-8B, temp=0)BaselineEAGLE-3 (16)DFlash (16)DFlash \u03c4GSM8K1.00\u00d71.94\u00d75.15\u00d76.54MATH-5001.00\u00d71.81\u00d76.08\u00d77.87AIME251.00\u00d71.79\u00d75.62\u00d77.08HumanEval1.00\u00d71.89\u00d75.14\u00d76.50MBPP1.00\u00d71.69\u00d74.65\u00d75.95LiveCodeBench1.00\u00d71.57\u00d75.51\u00d77.27MT-Bench1.00\u00d71.63\u00d72.75\u00d74.24Average1.00\u00d71.76\u00d74.86\u00d76.49<\/p>\n
A separate NVIDIA Speed-Bench comparison measures interactivity speedups at matched concurrency. On gpt-oss-120b, DFlash averages 2.3\u00d7 versus EAGLE-3\u2019s 1.7\u00d7. On Llama 3.1 8B Instruct, DFlash averages 2.8\u00d7 versus EAGLE-3\u2019s 2.2\u00d7.<\/p>\n
Use cases with examples<\/p>\n
DFlash targets latency-sensitive serving where token-by-token generation hurts. Three patterns fit well:<\/p>\n
Coding agents: Code generation needs fast, interactive responses. On Gemma 4 31B with vLLM, NVIDIA reports up to 5.8\u00d7 on Math500 at concurrency 1. HumanEval reaches 5.6\u00d7. Faster drafts mean shorter wait times inside agent loops.<\/p>\n
Reasoning models: Long Chain-of-Thought traces dominate generation time. With thinking mode enabled, DFlash holds roughly 4.5\u00d7 under greedy decoding on Qwen3-4B and Qwen3-8B. Under sampling, it holds about 3.9\u00d7. This cuts the cost of long reasoning outputs.<\/p>\n
Serving and throughput: DFlash also raises serving throughput. On SGLang with a B200 GPU, it reaches up to 5.1\u00d7 on Qwen3-8B (Math500, concurrency 1). Gains taper as concurrency rises but stay positive, so serving cost still drops.<\/p>\n
Running DFlash<\/p>\n
DFlash ships with checkpoints and framework support, so adoption needs little code. On vLLM, you swap an EAGLE-3 config for a DFlash one. No application refactoring is required.<\/p>\n
vllm serve Qwen\/Qwen3.5-27B \\
\n –speculative-config ‘{“method”: “dflash”, “model”: “z-lab\/Qwen3.5-27B-DFlash”, “num_speculative_tokens”: 15}’ \\
\n –attention-backend flash_attn \\
\n –max-num-batched-tokens 32768<\/p>\n
The Transformers backend supports Qwen3 and LLaMA-3.1 models. It exposes a spec_generate call that pairs a draft model with a target model.<\/p>\n
from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer<\/p>\n
draft = AutoModel.from_pretrained(
\n “z-lab\/Qwen3-8B-DFlash-b16″, trust_remote_code=True,
\n dtype=”auto”, device_map=”cuda:0″).eval()
\ntarget = AutoModelForCausalLM.from_pretrained(
\n “Qwen\/Qwen3-8B”, dtype=”auto”, device_map=”cuda:0″).eval()
\ntokenizer = AutoTokenizer.from_pretrained(“Qwen\/Qwen3-8B”)<\/p>\n
messages = [{“role”: “user”, “content”: “How many positive whole-number divisors does 196 have?”}]
\ninput_ids = tokenizer.apply_chat_template(
\n messages, return_tensors=”pt”, add_generation_prompt=True,
\n enable_thinking=False).to(draft.device)<\/p>\n
output = draft.spec_generate(
\n input_ids=input_ids, max_new_tokens=2048, temperature=0.0,
\n target=target, stop_token_ids=[tokenizer.eos_token_id])
\nprint(tokenizer.decode(output[0], skip_special_tokens=False))<\/p>\n
Key Takeaways<\/p>\n
DFlash drafts an entire token block in one forward pass, not one token at a time.<\/p>\n
It injects target hidden features into every draft layer\u2019s KV cache, scaling acceptance length with depth.<\/p>\n
Research Paper\u2019s metrics: up to 6.08\u00d7 lossless speedup on Qwen3-8B; NVIDIA test: up to 15\u00d7 throughput on Blackwell at fixed interactivity.<\/p>\n
A lightweight five-layer drafter replaces the 7B drafters that capped earlier diffusion methods near 3\u20134\u00d7.<\/p>\n
Interactive Explainer<\/p>\n
\n=TOTAL) return;
\n arCells[i].classList.add(‘accept’);
\n i++;
\n arSteps.textContent=i; arTok.textContent=i; arTime.textContent=i;
\n timers.push(setTimeout(arStep, 70));
\n }
\n arStep();<\/p>\n
\/\/ —- animate DFlash: block by block —-
\n var pos=0, pass=0;
\n function dfBlock(){
\n if(pos>=TOTAL){ finish(); return; }
\n pass++;
\n var blockEnd=Math.min(pos+blockSize, TOTAL);
\n \/\/ 1) show drafted block (amber); clear any prior reject state at the frontier
\n for(var j=pos;j in ‘+dfPasses+
\n ‘ target passes vs ‘+arPasses+’ for autoregressive. Fewer sequential ‘+
\n ‘verifications \u2192 ‘+spd.toFixed(2)+’\u00d7 faster at \u03c4=’+tau+’, block size ‘+blockSize+’.’;
\n }
\n }
\n runBtn.addEventListener(‘click’, run);<\/p>\n
syncLabels();
\n})();<\/p>\n","protected":false},"excerpt":{"rendered":"Autoregressive large language models generate text one token at a time. Each token waits for the one before…\n","protected":false},"author":2,"featured_media":516181,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[61,60,80],"class_list":["post-516180","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technology","tag-ie","tag-ireland","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts\/516180","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/comments?post=516180"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/posts\/516180\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/media\/516181"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/media?parent=516180"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/categories?post=516180"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ie\/wp-json\/wp\/v2\/tags?post=516180"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}