dzimplei

Small language models that write in blocks

Pick a model and talk to it

Working report, 15 October 2026. Not peer reviewed; every number comes from a single training run.

Block-diffusion language models on a small budget: what DZ.6, DZ.9-K2 and a looped 128-token decoder taught us

Abstract

We converted two small autoregressive models, Qwen3-0.6B and K2-Horizon-0.9B, into block-diffusion models that write eight tokens at a time, and then tried to push the second one to 128-token blocks with a looped, adaptive-depth decoder that we re-implemented from its paper. The eight-token models reached 47.8% (DZ.6-Q3-8) and 52.0% (DZ.9-K2-8) on GSM8K for about 10 and 3.5 dollars of GPU credit: a tie with a published model in the first case, seven points under the autoregressive parent in the second. The 128-token looped model trained stably and its loss on unseen text fell by 14%, but its GSM8K accuracy dropped from 54.5% to 40.0%, its code score from 11.0% to 6.7%, and its long reasoning turned into repetition. Giving it more passes per step made it worse (49.0% with one pass, 36.5% with four). On a CPU, eight denoising steps per block was half as fast as autoregressive decoding; one step per block was 2.25 times faster. We report the failures and the cost alongside the results.

1 Introduction: how far can one person with rented GPUs get with diffusion language models?

An autoregressive model writes one token per forward pass. A block-diffusion model starts a block of tokens as blanks and fills them in over several denoising steps, so it can commit several tokens per pass [1]. Open recipes convert an autoregressive checkpoint with little compute [2], but the published conversions we compared against used about 64 A100 GPUs for about 10 hours.

We asked what a single person can do with free Kaggle time and about 200 Colab compute units. We built three models, put the first two on a public CPU server, wrote the third's training and decoding code in JAX, and measured everything we could. This report covers what worked, what did not, and what each thing cost.

  • Two eight-token-block models, DZ.6-Q3-8 and DZ.9-K2-8, that you can talk to above.
  • A clean-room JAX trainer and sampler for the adaptive-depth looped decoder ALoDLM [3], checked against reference implementations, and its first small-scale test.
  • Measurements of speed against autoregressive decoding, of cost, and of the failures.

2 Models: the name says the size, the base and the block

A name reads DZ.<size in tenths of a billion parameters>-<base>-<block size>. Q3 is Qwen3 and K2 is K2-Horizon. DZ.9-K2-128 is a 0.9B K2 model that writes 128 tokens per block.

ModelBaseBlockDecoderCost
DZ.6-Q3-8Qwen3-0.6B8Plainabout $10
DZ.9-K2-8K2-Horizon-0.9B8Plainabout $3.5
DZ.9-K2-128DZ.9-K2-8, continued128Looped, up to 4 passesabout $4.7

3 Method: a standard conversion, a clean-room looped decoder, and a trainer we had to make fast

3.1 Conversion. We followed the open A2D recipe for block diffusion [1][2]. Attention becomes causal across blocks and bidirectional inside a block. Training masks a random fraction t of a sequence's answer tokens and minimises the cross-entropy on them, weighted by 1/t, while a clean copy of the sequence supplies the earlier blocks. To decode, a block starts fully masked and each step commits the most confident tokens; with as many steps as tokens it writes one token per step.

3.2 Data. DZ.6 used MetaMathQA, Magicoder, OpenCoder and SmolTalk, plus 15% answers written by a larger Qwen model (27B). DZ.9-K2 used K2's own open training set, TxT360-3efforts [5]: its low-effort split for the eight-token model (39 million tokens seen) and a 70/15/15 mix of high, medium and low effort, with sequences up to 8,192 tokens, for the 128-token run.

3.3 Looped decoder. ALoDLM [3] applies the network's layers to the masked block repeatedly, up to K = 4 passes, with a gate that decides per token how many passes it needs. During training, tokens that exit are fed back as their true embedding. We implemented it from the paper's text in JAX, because the reference code carries a non-commercial licence. We left out the paper's autoregressive auxiliary loss and used a gate learning rate of 1e-4. When decoding, a token commits once its entropy drops below tau = 0.3 nats, and the most certain token always commits so every step makes progress.

3.4 Trainer. Parameters in fp32 sharded by GSPMD, bf16 compute. Our first attention, which wrote fp32 score matrices, trained at 212 tokens per second on an A100. Moving to cuDNN's fused kernel over triangular key ranges gave 1,194 tokens per second (5.6 times) with the loss and gradients unchanged to within bf16 noise (gradient cosine 0.9997). We checked the model against dLLM's PyTorch implementation at block sizes 8, 32 and 128 on small models (logit differences below 6e-8) and the sampler against an independent PyTorch engine (identical tokens).

4 Results: two eight-token models that work and a 128-token model that does not yet

4.1 Setup. Maths is GSM8K and code is HumanEval, run with lm-evaluation-harness [6], zero-shot and greedy. DZ.6 and DZ.9-K2-8 were scored with dLLM's sampler at eight steps per block, DZ.9-K2-128 with our JAX sampler (effort low, 512 new tokens, tau = 0.3). Standard errors are about 1.4 points on the full maths set, 2.5 on 400 problems, 3.5 on 200 and 3 to 4 on code. The settings differ between models, so compare within a table, not across.

4.2 DZ.6-Q3-8 ties the published model on maths and trails it on code. It scored 47.8 on maths against 46.2 for the published Tiny-A2D [4] (a tie within the noise) and 23.2 on code against 47.0.

Maths rose from 17 to 47.8 over five training stages

Figure 1. GSM8K by Colab training step (32 examples per step), DZ.6-Q3-8. The first point is the model trained on free Kaggle T4s.

4.3 DZ.9-K2-8 keeps most of K2's maths. Converted in five hours on one A100 with K2's low-effort data, it scored 52.0 on maths against 59.2 for K2 as released at brief thinking. K2's brief-thinking score is held down by the 512-token cap, so the true gap is probably wider. Code fell from 49.4 to 28.7.

MathsCode
K2 as released, brief thinking (512 tokens)59.249.4
K2 as released, longer thinking (2,048 tokens)76.049.4
DZ.9-K2-8, brief thinking52.028.7

The code prompt starts the answer without a thinking step, an unusual format for both models, so read the code column as a rough guide.

4.4 DZ.9-K2-128 lost accuracy at every stage we measured. Starting from DZ.9-K2-8 plus 14 warm-up steps at block 128, two rounds of looped training (713 and then 1,427 steps, 24 million real tokens) lowered GSM8K from 54.5 to 40.0 and HumanEval from 11.0 to 6.7. The trained models needed fewer passes per answer, but their answers were worse.

Maths (200)Passes per answerCode (164)
Start: DZ.9-K2-8 plus 14 steps at block 12854.533011.0
After round 1, 713 steps37.5170–
DZ.9-K2-128, round 2, 2,140 steps40.02246.7

Yet the model did learn its task. On 67 unseen training-style rows with fixed masks, the loss of the first pass fell from 3.56 to 3.25 to 3.06 nats per masked token. Figure 2 shows that deeper passes do not help when nothing is revealed between them.

Training cut the first-pass loss by 14%, but extra passes without revealed tokens still make it worse

Figure 2. Loss on masked tokens of 67 rows never used for training, after 1 to 4 passes with nothing committed in between (lower is better).

The loop did not help; one pass per step was best. A fixed-schedule decoder using only round 2's first pass scored 32.0 at one token per step and 15.5 at four (200 problems), far below the adaptive decoder's 40.0. But the adaptive entropy rule, not the loop, closes that gap. Running the same adaptive decoder with a single pass per step scored 49.0 on the first 96 problems, against 44.8 when the gate chose the depth and 39.6 when we forced all four passes in every step. More passes meant lower accuracy.

Decoder setting (first 96 problems)MathsPasses per answer
Start model, one pass per step56.2329
Round 2, one pass per step49.0183
Round 2, the gate chooses the depth44.8215
Round 2, all four passes every step39.6626
Round 2, four passes, one token per step36.52,035

Each score is uncertain by about 5 points, so only the trend is solid: it matches Figure 2, where deeper passes without revealed tokens raise the loss.

At high effort the model produced K2-style reasoning that stayed coherent for a few hundred tokens and then repeated one closing line until the budget ran out: 9 of 16 answers did not finish within 3,072 tokens and 4 of 16 were correct.

4.5 Against autoregressive decoding, eight steps per block is half as fast and one step is 2.25 times faster. We timed the same engine and weights on three CPU threads, generating 64 tokens, with the autoregressive baseline being the same network run one token per pass.

Speed depends on the steps per block: from half the autoregressive speed to 2.25 times

Figure 3. Tokens per second, DZ.6-Q3-8, three CPU threads, mean of three runs (spread under 7%). A block of 8 costs (steps + 1) passes over 8 tokens, and an 8-token pass costs about 1.8 times a one-token pass on this CPU.

The public playground runs on a shared four-core server with three threads. An eight-token block takes 2.5 seconds at eight steps and 0.5 seconds at one step for DZ.6-Q3-8, and 3.7 and 1.0 seconds for DZ.9-K2-8.

5 Discussion: what the 128-token result does and does not tell us

The first pass got better at the task it was trained on while the finished decoder got worse at answering questions. We do not know why. Three suspects stand out. The jump from blocks of 8 to blocks of 128 changes what the model must do at the start of every block, and 24 million real tokens is a small fraction of the paper's training run. The looped objective reveals true tokens to later passes, which decoding cannot do. And 70% of the data was high-effort reasoning while we scored low effort.

We did not train a plain block-diffusion control at block 128 with the same data and steps, and that is the experiment that would separate the first suspect from the second. Until it is run, we cannot say whether the drop comes from the loop or from the block size. What we can say is that at this training length the loop was a cost, not a benefit: a single pass per step beat the gate's choice, and four passes were worst.

6 Cost: the three models cost about 18 dollars of compute in total

ModelColab unitsAboutHardware
DZ.6-Q3-8105$10A100, after a free Kaggle start
DZ.9-K2-834.7$3.5A100, 5.1 hours of training
DZ.9-K2-12846.7$4.7A100, 6 hours of training

A Colab unit costs about 10 cents and an A100 hour 5.3 units. Serving is a shared CPU server we already ran. Not counted: our own time.

7 What did not work: five things we would warn the next person about

  • Kaggle TPUs. Our training job sat in the queue for over nine hours without starting, so we moved to Colab.
  • A five-hour evaluation with no result. It crashed on its last batch and kept no partial output. That was our mistake: no dry run and no cap.
  • 8-bit weights. End to end 1.7 times faster on the playground CPU, but 15 points worse on maths (47.5 to 32.5 on 40 problems), so the playground runs at full precision.
  • Guessing the bottleneck. We expected memory limits and found an attention implementation that wrote every score to memory. Profiling one layer found it in minutes.
  • Long reasoning. The 128-token model repeats itself at high effort. Published conversions are mostly scored with thinking off, which fits what we saw.

8 Limitations

  • Single training runs and small test sets: differences under about 7 points on 200 problems are within noise.
  • Different harnesses across models, so cross-table comparisons are rough.
  • No plain-diffusion control at block 128, no repeated seeds, no human evaluation, English only.
  • Small models make mistakes and can write text that is wrong or unsuitable.

9 Conclusion

Block-diffusion conversion of a small model is within reach of one person: two eight-token models for under 15 dollars, one tying a published model on maths. Pushing to 128-token blocks with a looped decoder trained cleanly but lost accuracy, and extra passes made it worse. A plain control run at block 128 is the next step before we scale to a 7B model.

Availability and licences

The playground and this report are public. The server stores no messages. Base models: Qwen3-0.6B and K2-Horizon-0.9B, both Apache-2.0. Data: MetaMathQA, Magicoder, OpenCoder, SmolTalk, and K2's TxT360-3efforts (CC BY 4.0).

References

  1. Arriola et al. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models. ICLR 2025.
  2. dLLM, an open library for diffusion language models (Apache-2.0), including the A2D conversion recipe.
  3. ALoDLM, adaptive looped diffusion language model, arXiv:2610.04198. We implemented it from the paper text only.
  4. Tiny-A2D, the published Qwen3-0.6B block-diffusion model.
  5. IFM. K2-Horizon-0.9B and TxT360-3efforts.
  6. EleutherAI. lm-evaluation-harness.