Writing 4 min
A diffusion language model as a brain-to-text decoder
Training the language model in a speech decoder by block diffusion, not left to right, cut word error rate by about ten points. The gain disappeared once the encoder had been trained on phonemes.
BIT is a brain-to-text decoder that reads attempted speech from intracortical recordings. A transformer encoder turns neural activity into a sequence of tokens. A language model then writes the sentence from those tokens, one word after another.The language model is Aero-1-Audio, an audio-tuned Qwen2.5-1.5B, fine-tuned with LoRA.
I wanted to know what happens if the language model writes the sentence by diffusion.
Why diffusion
The decoder sees the whole trial’s neural tokens before it writes anything. A diffusion model could fill in the confident words first and settle the uncertain ones from both sides. A left-to-right model has to commit to each word before it has written the next.
Two ways to write the same sentence. The autoregressive decoder commits one word at a time, left to right. The diffusion decoder starts with every word masked, scores all of them on each pass, and commits the ones whose confidence passes the 0.9 mark. An illustration, not a real trial.
The setup
The objective is block diffusion, taken from Fast-dLLM v2. The sentence’s tokens are masked at a random ratio and the model predicts the masked ones. Attention is bidirectional within a 32-token block and causal across blocks. The neural tokens and the prompt are a clean prefix that is never masked. At decoding time the model proposes every masked token and commits the ones it is confident in, eight positions at a time.
Everything else matches the left-to-right run: the same self-supervised encoder, projector, LoRA, prompt and number of epochs, with the checkpoint chosen on a validation split.
The data is Brain-to-Text ‘25, one participant (T15). The score is word error rate on 713 held-out trials, by free generation with greedy decoding: the model writes the sentence on its own.
BIT trains its encoder on phoneme labels before connecting it to the language model. The first set of runs skips that stage.
Without a phoneme stage, diffusion wins
| Language model | Objective | Test WER |
|---|---|---|
| Aero-1-Audio | block diffusion | 21.76% |
| Qwen2.5-1.5B-Instruct | block diffusion | 23.65% |
| Fast-dLLM v2 1.5B | block diffusion | 25.22% |
| Aero-1-Audio | autoregressive | 34.23% |
| Qwen2.5-1.5B base, no prompt | autoregressive | 43.75% |
| Qwen2.5-1.5B-Instruct | autoregressive | 55.66% |
Over three seeds with Aero, the autoregressive decoder averages 31.68% and the diffusion decoder 21.36%. The difference is 10.31 points, with a 95% interval of 9.10 to 11.59. Diffusion is also steadier: its seeds differ by 0.40 points (standard deviation), against 3.05 for autoregressive.With the neural input shuffled, replaced by noise, or removed, every run gives 99 to 142% WER. The output depends on the neural input.
The choice of language model matters less under diffusion. Aero’s audio post-training is worth about 21 points with autoregressive training and about 1.9 with diffusion.
It isn’t the decoding order
My reason for trying diffusion was the order in which words get written. That turned out not to be where the gain comes from.
In the run where I tested it, forcing the trained diffusion model to decode strictly left to right cost only 1.2 points. The gain comes from training, not from the order of generation.
It also isn’t a property of a purpose-built diffusion model. Plain Qwen2.5-1.5B-Instruct trained with the diffusion objective (23.65%) does slightly better than Fast-dLLM v2 (25.22%), which is that same model after about a billion tokens of conversion training.
The errors are different
- They sound like the target. Among wrong words, 25.9% of the diffusion decoder’s substitutions are phonemically similar to the right word, against 16.1% for the autoregressive decoder and about 2.7% by chance. My reading is that diffusion decoders lean more on the neural signal and less on the language model’s prior. The signal comes from speech-motor cortex, so their slips are sound-based.
- They derail less. 2.0% of the diffusion decoder’s sentences have a word error rate of 100% or more, against 9.3%.
- But each error keeps less meaning. Matched on the same trial and the same number of word errors, the autoregressive decoder’s wrong sentences stay slightly closer in meaning to the truth (BERTScore F1 difference 0.041, interval 0.018 to 0.065).
Autoregressive decoding is all-or-nothing. Diffusion fails more gracefully, with slips that sound right and mean less.
With the phoneme stage, the gain disappears
Then I ran both objectives after each of three stages: BIT’s phoneme training, nothing, and contrastive alignment.
| Stage before the language model | Autoregressive | Block diffusion |
|---|---|---|
| Phoneme training (BIT’s recipe) | 15.53% | 15.58% |
| None | 31.68% | 21.36% |
| Contrastive alignment | 29.16% | 22.66% |
The middle row is the mean of three seeds. The other two rows are one seed each.
With a phoneme-trained encoder the two objectives tie. The phoneme stage is worth 16.14 points under autoregressive training and 5.79 under diffusion. So diffusion makes up about ten of the sixteen points that skipping phonemes costs, and adds nothing on top of them.
Limits
- One participant and one dataset.
- Not the competition holdout. These are a split of the official validation set. BIT’s paper reports 11.06% on the holdout, with nucleus sampling and an encoder pretrained on 366.8 hours of human and monkey recordings. Mine uses greedy decoding and 60 hours of human pretraining. The numbers are not directly comparable.
- One seed for the phoneme and contrastive rows.
- Seen sentences. 11.4% of test trials repeat a training sentence. The repo reports seen and unseen separately.
Every run, its controls and the full audit are in the repo.