Writing 4 min
Aligning a neural encoder to a language model, without phonemes
Contrastive alignment taught a speech encoder to pick the right sentence out of 704 a third of the time, with no phoneme labels. It did not make the encoder a better starting point for decoding.
BIT decodes attempted speech from intracortical recordings with a neural encoder feeding a language model. Before the two are connected, the encoder is trained on phoneme labels, and the paper reports that skipping that stage fails.
The question here: can contrastive alignment to the language model’s own representations replace the phoneme stage?
Three arms
Every arm starts from the same self-supervised encoder and ends with the same sentence-level training of the language model. They differ only in the middle.
| Arm | Middle stage |
|---|---|
| A | Phoneme training (BIT’s recipe) |
| B | Nothing |
| C | Contrastive alignment |
In arm C, the encoder’s summary of a trial is pulled toward the language model’s representation of that trial’s sentence, and pushed away from the other sentences in the batch.The target is the mean of the language model’s middle hidden layer (14 of 28) over the sentence. The loss is symmetric InfoNCE with a learned temperature, on batches of 1,024 pairs. The targets are fixed. No language model runs during this stage and no phoneme label is used. It takes 51 minutes.
What alignment does. The red rings are sentence targets from the language model, and they never move. The dots are neural trials. Training reshapes the encoder’s space until each trial sits near its own sentence and away from the others. It doesn’t get all of them. An illustration in two dimensions, not data.
The data is Brain-to-Text ‘25, one participant (T15), with 713 held-out test trials.
The alignment works on its own
The test is retrieval: rank the trial’s true sentence among every distinct sentence in the split.
| Test trials | Top-1 | Top-5 | Candidates |
|---|---|---|---|
| All | 32.3% | 61.2% | 704 |
| Sentences never seen in training | 32.3% | 61.4% | 704 |
Chance for top-1 is 0.14%. Retrieval is no better on sentences seen in training, so the alignment generalizes to new sentences.
A leak in my first number
I first reported 35.8%. That figure was inflated.
The evaluation padded trials into batches, and the input smoothing then leaked the pad value into the last bins of every shorter trial. That carries the trial’s length relative to its batch, which is a cue to which sentence it is. Scored one trial at a time, the same checkpoint gives 32.3%. About 3.5 points had come from padding.
The alignment stage also trained with that leak, and its checkpoint was chosen on the inflated validation figure. The sentence-level stage has a padding fix and is unaffected.
It holds less phoneme information than its input
A linear probe reads the phoneme in each frame of the encoder’s output.
| Encoder | Phoneme probe accuracy |
|---|---|
| Always guess the most common | 41.2% |
| Arm C, after alignment | 49.2% |
| Self-supervised encoder | 54.7% |
| Raw input | 54.8% |
| Arm B | 63.2% |
| Arm A, phoneme-trained | 81.2% |
Alignment produces an encoder with less frame-level phoneme information than the input it started from. Sentence-level training then puts it back: after that stage the arm C encoders reach 61.5 to 63.1%, the same as arm B.
It does not help decoding
Word error rate on the test trials, by free generation. The diffusion post covers the two decoders.
| Arm | Autoregressive | Block diffusion |
|---|---|---|
| A, phonemes | 15.53% | 15.58% |
| B, nothing | 34.23 / 28.30 / 32.49% | 21.76 / 21.36 / 20.96% |
| C, contrastive | 29.16% | 22.66% |
Arm B has three seeds. Arms A and C have one.
Against arm B’s three-seed mean, arm C is 2.52 points better under autoregressive training and 1.30 points worse under diffusion. Arm B’s autoregressive seeds have a standard deviation of 3.05 points, and arm C sits inside their range. With one seed, arm C ends up no better than doing nothing.
Why not
To see what each encoder holds, I froze it, averaged its output over the trial, and fit a fresh linear map to retrieve the sentence. This is the readout the alignment was trained for, so it is the test most favourable to it.
| Encoder | Sentence retrieval, top-1 of 704 |
|---|---|
| Self-supervised | 19.4% |
| Arm C, after alignment | 44.5% |
| Arm C, after sentence training (diffusion) | 64.8% |
| Arm B, after sentence training (diffusion) | 64.8% |
| Arm B, after sentence training (autoregressive) | 70.5% |
| Arm C, after sentence training (autoregressive) | 73.1% |
| Arm A, after sentence training (autoregressive) | 92.0% |
- Alignment adds sentence identity, but less than word-level training does. The aligned encoder is 25 points above the self-supervised one, and 20 to 26 points below either arm B encoder.
- After sentence training, C and B hold the same amount. The differences are 2.5 points (interval −0.7 to 5.9) and 0.0 (−3.4 to 3.5).
- Phonemes and sentence identity go together. The phoneme-trained encoder is far ahead at 92.0%. A trial average of frames that each carry a phoneme is a bag of phonemes, and that picks out one sentence in 704 almost every time.
- Sentence training moves the encoder away from the alignment. Read through the alignment stage’s own head, retrieval falls from 32.3% to 17.3% and 13.5%. Sentence identity gets rebuilt, mostly in new directions.
So the head start from alignment is small, and sentence training overwrites much of it on the way to learning what it needs.
What I don’t know yet
Whether an aligned encoder can drive the decoder if sentence training isn’t allowed to rewrite it, and whether arm C’s small gain under autoregressive training is real. The next runs freeze the aligned encoder and add seeds. The rule for reading each one is written down before it runs.
Everything here is one participant, and arm C is one seed.