Flow Matching for Speech Generation

Introduction

Codec-token language models made zero-shot voice cloning work, but they decode token by token, and in Voicebox's comparison they lost on word error rate. Voicebox showed the alternative: generate a continuous mel spectrogram for the whole utterance in parallel, trained by masked infilling, and came out well ahead of VALL-E on word error rate [1]. Since then flow matching has become the workhorse acoustic generator, either on its own, as in F5-TTS, or behind an autoregressive token model, as in CosyVoice 2. A third pattern skips speech tokens altogether: the LLM's hidden state conditions a small diffusion head that generates each continuous latent frame, as in VibeVoice. The catch is that every solver step is a full forward pass of a large network, so the real game is keeping quality while cutting steps.

This post goes from diffusion to flow matching, then through how these models are trained, sampled and made fast enough to stream. It ends with the papers that matter, where they still break, and what to use.

From Diffusion to Flow Matching

The original diffusion sampler spends a thousand network calls on one sample. A real-time TTS system can afford tens. Several recent open and shipped TTS systems closed that gap with flow matching. It regresses a velocity along a straight path from noise to data. Voicebox, F5-TTS and CosyVoice 2 all use it and sample with classifier-free guidance.

Denoising diffusion

Denoising diffusion learns to undo a fixed noising process.

DDPM noises clean data $x_0$ over $T$ steps, and any step has a closed form. A network $\epsilon_\theta$ learns to recover the added noise:

Here $\epsilon \sim \mathcal{N}(0, I)$ is the noise. $\beta_s$ is the noise variance added at step $s$. $\alpha_t^2 = \bar\alpha_t = \prod_{s \le t}(1 - \beta_s)$ is the signal variance left. $\sigma_t^2 = 1 - \bar\alpha_t$ is the noise variance. The loss is plain regression, with $t$ uniform over the steps. It drops the weights of the variational bound, which gave better samples.

The standard setup uses $T = 1000$ steps [2]. Sampling applies the learned reverse step once per level: a thousand sequential calls. The noise predictor is also a score model. The score $\nabla \log p_t$ points uphill in the noisy data density. A short derivation gives $\epsilon_\theta \approx -\sigma_t\,\nabla \log p_t$.

The SDE view and the probability flow ODE

Take $T \to \infty$ and the noising chain becomes a stochastic differential equation (SDE). DDPM's chain is a discretisation of the variance-preserving (VP) SDE. Every such SDE also has a deterministic twin with the same marginals $p_t(x)$. It is the probability flow ODE:

Here $f$ is the drift and $g$ the noise scale of the forward SDE. Same marginals means you can use any ODE solver, with error set by the step count. My read is that the ODE removes randomness, not curvature. Diffusion trajectories bend, and the conditional path can overshoot the sample and backtrack. Every bend forces the solver into short steps.

Diffusion can also regress $x_0$ or $v = \alpha_t\epsilon - \sigma_t x_0$. Noise prediction ignores data errors at pure noise, so distillation uses v- or x-prediction. This $v$ is a derivative along an angle on a circle, not the flow-matching velocity.

Flow matching

Flow matching starts from the ODE. It picks a path from noise to data and regresses the velocity field that generates it. The field of the whole distribution is intractable, so training regresses per-sample fields. That gives the same gradient, with no ODE simulation. The straight path gives conditional flow matching. It is the objective most flow-matching TTS systems use:

Conditional flow matching on the optimal-transport (OT) path moves noise to data in a straight line. Draw noise $x_0 \sim \mathcal{N}(0, I)$, a data sample $x_1$ and a time $t \in [0, 1]$:

The original formulation samples $t$ uniformly. $\sigma_{\min}$ is a small constant, so the path ends in a narrow Gaussian around $x_1$. The target depends only on the endpoints, so each path runs at constant speed. With $\sigma_{\min} = 0$ the target is simply $x_1 - x_0$. Sampling integrates $\mathrm{d}x/\mathrm{d}t = v_\theta(x, t)$ from $t = 0$ to $t = 1$.

Voicebox trains with exactly this path and target [1]. Fewer than 10 steps already gives it high quality, though it defaults to 32. Time-step schedules covers where to spend the steps.

Classifier-free guidance

Every system above samples with guidance.

Classifier-free guidance trains one network both with and without the condition $c$. During training, $c$ is replaced by a null token $\varnothing$ at a small rate. At sampling time it sees the noisy input $x_t$ twice, with $c$ and without. The guidance scale $w$ extrapolates away from the unconditional prediction:

$w = 0$ is the plain conditional model, and the velocity form is the same rule. Two forward passes per step means guidance doubles the network calls.

Settings vary: Voicebox uses $w = 0.7$ [1], F5-TTS $w = 2$ at 16 or 32 steps [3]. The cost of guidance covers serving and removal.

Rectified flow and straight paths

Rectified flow reaches the same loss from another side. It is conditional flow matching with $\sigma_{\min} = 0$, regressing onto $x_1 - x_0$. Straight paths matter because one Euler step follows a straight, constant-speed flow exactly. The evidence is on ImageNet-32 images, against a 1000-step reference solution. The straight path reaches the same error with about 60% of the solver steps [4].

The regression has a closed-form minimiser, $v(x, t) = \mathbb{E}[x_1 - x_0 \mid x_t = x]$. That is the average direction of every training line through $x$. Random pairs cross, the field averages them where they do, and trajectories bend. Learned flows come out straighter than diffusion, not straight.

Reflow straightens them by re-pairing. Run the trained ODE, record each noise sample with its output, and fit a new rectified flow. Those pairs come from one deterministic ODE, so their lines cross less. Straightness is the mean squared gap between velocity and the endpoint direction. The guarantee covers the best of $K$ rounds: its straightness is bounded by a $1/K$ term. On images, one round gives nearly straight flows that work with one Euler step [5]. Tradeoff: each round fits the last model's outputs, so errors accumulate. The authors advise against many rounds. Few-step generation covers the alternatives.

How the two families relate

Stochastic interpolants make both families one framework. Each path comes with an ODE and a family of SDEs, all with the same marginals.

Tradeoff: the straight path buys fewer steps and a target with a stable scale. It gives up the stochastic sampler, which plain flow matching does not have. Stochastic interpolants restore one, but it needs the score as a second learned field. Several recent open and shipped TTS systems take the deterministic side of that trade. Guidance, solvers and distillation built for diffusion carry over to it.

Training a Flow Matching TTS Model

The objective in the previous section turns noise into samples. TTS needs specific words, in a specific voice, over a specific number of frames. The default recipe, which came from Voicebox, is to hide most of an utterance and train the flow to fill it back in, given the text. What separates shipped systems is what the flow generates, how the text reaches it, and who decides the length.

What to generate

Most shipped flow decoders still generate a log mel spectrogram and hand it to a separately trained vocoder; CosyVoice's decoder writes a mel for HiFi-GAN. Continuous latents from an audio codec show up when the model must also render music and sound effects, and in at least one large production stack.

The mel choice makes the vocoder a separate project with its own ceiling: Voicebox's alone took 7.5 days on 8 GPUs [1]. The price, as I see it, is that the flow can never repair what the vocoder smears.

Masked infilling and prompting

Each training example hides one contiguous span covering 70 to 100% of the target mel $x_1$. A binary frame mask $m$ zeroes those frames to give the context $x_{ctx} = (1-m) \odot x_1$. With $x_0$ as Gaussian noise, the path and target are the straight ones from the previous section, with $\sigma_{\min}$ a small constant:

The network predicts a velocity from the noisy mel, the context, the text $c$ and the flow step. In Voicebox the loss is graded only on masked frames:

The model sees every frame but is graded only on the frames it had to invent. It pays for itself: in a reduced Voicebox setup, masking the loss gave a clear gain in speaker similarity at almost no cost in intelligibility [1]. My read is that scoring visible frames rewards copying, which is not the skill you need at inference.

Prompting falls out for free, and that is a large part of why the recipe spread. At inference you lay down a reference clip, zeros for the speech to generate, and the text "reference transcript, target text". The voice is the visible context, so there is no speaker encoder to train, and the 70 to 100% mask matches that shape: a short prompt, then a long blank.

The catch is that the prompt transfers everything at once: voice, style and recording conditions come as a bundle. I would expect a reverberant prompt to come back as reverberant output. When the prompt is instead a separate input, the opposite failure appears: unless the loss excludes the prompt segment, the model learns to copy-paste it.

Text conditioning and alignment

The hard part of TTS is deciding which frames say which sounds. There are three places to put that decision, and each moves the cost somewhere else.

Large production stacks such as Seed-TTS and the CosyVoice family take the third row. An AR model writes semantic tokens, and a diffusion or flow decoder adds the acoustic detail.

The character route pays in convergence. In one comparison, the aligned model was already intelligible at 10% of training, while the character model needed significantly more training before it finished ahead [6]. A group reproducing a character model on small Mandarin models saw a stubborn share of test samples fail outright throughout training, and re-ranking could not fix them [3]. They attribute it to entangled semantic and acoustic features.

The token route has its own leak. CosyVoice 2 removed the speaker embedding from its language model because that vector carried language and paralinguistic information that hurt prosody naturalness and cross-lingual ability. My read: a conditioning vector is rarely only the thing you named it.

Duration

A non-autoregressive model must know how many frames to generate before it starts. The older answer is a separate per-phone duration model. Newer character-based recipes predict only the total length, often with an estimate from the character counts of the prompt and target text. In AR-plus-flow stacks the question disappears, because the tokens already fix the length.

Tradeoff: one total length doubles as a speech-rate control; one model scaled it from 0.7 to 1.3 with only a moderate WER increase [6]. The cost is that the network must place every phone itself, with no timing hint beyond the total.

Time sampling and guidance dropout

Speech flow models almost all sample $t \sim \mathcal{U}[0, 1]$ in training, not the logit-normal sampling image models use. The non-uniformity goes into the inference schedule, which can change without retraining.

Classifier-free guidance needs the same network to work without its condition, so training drops the conditions at random, about 20% of the time in most recipes. What you drop decides what you can guide. Dropping text and audio together gives one guidance direction for everything. My read is that adding a separate drop of only the audio context also teaches the model a text-only mode. With $v_c$ and $v_u$ the conditional and unconditional velocities, the two common ways of writing guidance, $(1+w)v_c - w v_u$ and $v_c + w(v_c - v_u)$, are algebraically the same, so strengths read on one scale. Guidance doubles the cost per step; see the inference section.

Architectures and scale

The open zero-shot recipes (Voicebox, E2 TTS, F5-TTS) are Transformers of about 330M parameters, trained on tens of thousands of hours. Newer ones use a DiT that injects the flow step through adaLN. Streaming decoders such as CosyVoice 2's train one network under several attention masks, from full to chunk-causal, so the same weights serve offline and streaming.

The pieces are coupled. In one set of ablations, a plain adaLN DiT without the convolutional text branch failed to learn alignment from padded characters [3]. An MMDiT "learned fast and collapsed fast" into repeated utterances with wild timbre and prosody.

Scale is less of a barrier than the big models suggest. The same group trained 158M models stably on LibriTTS (585 hours) and on single-speaker LJSpeech (24 hours), learning alignment from characters with no phonemizer [3].

Inference: Solving the ODE

To get speech you integrate $\mathrm{d}x_t/\mathrm{d}t = v_\theta(x_t, t, c)$ from Gaussian noise at $t = 0$ to a mel spectrogram at $t = 1$. Every solver step is a full network pass over the whole utterance, so the cost of inference is the number of function evaluations (NFE), multiplied by the guidance passes. A common recipe in recent open and published systems is Euler at roughly 10 to 32 steps, a warped time grid and batched guidance passes, with a vocoder that becomes the bottleneck once the flow model is cheap.

Solvers and NFE

The simplest solver is first-order Euler: one network evaluation per step, moving $x_t$ along the guided velocity $\hat v$ (defined in the next subsection):

If the path from noise to data were perfectly straight, a single Euler step would be exact. Flow-matching paths are straighter than diffusion paths but not straight (From Diffusion to Flow Matching), so Euler needs a handful of steps. Higher-order solvers such as midpoint and Heun-3 spend two and three evaluations per step to cancel the curvature. Most recent flow decoders (CosyVoice 2, F5-TTS, ZipVoice) run Euler anyway, at roughly 10 to 32 steps. The extra spend rarely pays, and the schedule matters more than the solver.

In one widely used zero-shot model with a warped schedule, doubling Euler from 16 to 32 steps bought only a small gain in intelligibility and doubled real-time factor, while Heun-3 at 16 steps did slightly better again at nearly three times Euler's cost [3]. The same step count hides very different compute across solvers.

Quality degrades slowly as steps drop, which is why teams keep pushing the count down. A small acoustic model loses little going from 10 steps to 2 [7]. The trap is that WER flatters low-step samples, which are easier for ASR to transcribe, and what they lose is diversity, which WER does not measure.

The cost of guidance

Most zero-shot flow decoders run classifier-free guidance at every step. Each step evaluates the network twice, once with the text and prompt $c$ and once with the null condition $\varnothing$, and extrapolates away from the unconditional prediction:

Here $w$ is the guidance strength: $w = 0$ is the plain conditional model, and a strength of 2 puts a weight of 3 on the conditional velocity. The derivation is in From Diffusion to Flow Matching.

The price is a doubled network cost per step. A common implementation batches the two passes into one call, as in the diagram. That saves a sequential launch but not the doubled FLOPs and activation memory, so it helps only when the GPU has headroom. Papers differ on whether NFE counts solver steps or network calls, but a guided sampler at 10 Euler steps runs 20 network passes however a paper counts.

Guidance and step count are not independent knobs. In one large zero-shot model, at 4 steps or fewer, weaker guidance ($w = 0$ or 0.3) gave higher speaker similarity, while from 8 steps up stronger guidance helped [1]. That makes an easy regression in speed-up work: cut the steps on a model tuned at 32 and forget to retune $w$. The usual fix is to remove the second pass by folding guidance into a distilled model, covered in Making It Fast and Streaming.

Time-step schedules

The solver decides how each step is taken; the schedule decides where the steps go. A uniform grid spends as many steps near the data as near the noise. Every schedule in common use moves steps toward $t = 0$, the noise end, and does it at inference only, with no retraining.

Sway Sampling warps a uniform grid $u \in [0, 1]$ with a single coefficient $s$:

With $s = 0$ the grid stays uniform, and negative $s$ packs steps at small $t$. The usual setting, $s = -1$, reduces to the cosine schedule $t = 1 - \cos(\tfrac{\pi}{2}u)$ that CosyVoice 2 ships. With the midpoint solver at 32 steps, it gave a clear intelligibility gain at zero extra cost [3]. Without the schedule, Euler degrades; the two ship together.

Pruning goes one step further. Empirically Pruned Step Sampling (EPSS) keeps the dense early steps and drops late ones. At 7 steps it was far more intelligible than an unscheduled 7-step run [8]. The floor is hard: below about 6 steps quality falls sharply, and only a model trained to take big steps goes further.

The vocoder

The ODE ends at a mel spectrogram, and a separately trained vocoder turns it into a waveform. It sits outside every NFE number above, yet it can move quality more than the solver does. In the same model, swapping the default vocoder for BigVGAN improved intelligibility more than doubling the steps did [3].

Streaming systems budget the vocoder as its own term in first-packet latency, alongside the language model and the flow decoder. Its right context costs time too: in one production system the vocoder's look-ahead alone adds 130 ms [9]. Once distillation cuts the flow model to 2 to 4 steps, my read is that the vocoder and its look-ahead become the larger cost, because the flow cost shrank and the vocoder cost did not. Making It Fast and Streaming picks up from there.

Making It Fast and Streaming

A better time-step schedule gets a flow-matching TTS model to about 7 NFE, and then quality "sharply declines below 6 NFE" [8]. Past that point the network itself has to change. Streaming is a separate problem. A non-causal decoder that attends over the whole utterance cannot start until the text is finished, however few steps it takes. Open streaming decoders such as CosyVoice 2 and StreamFlow run about 10 guided steps. The fastest pair a 2-step decoder with block-wise causal attention and reach a first packet in the low hundreds of milliseconds.

Few-step generation

The learned velocity field curves even though each training path was straight, so one big Euler step lands in the wrong place (From Diffusion to Flow Matching). Every few-step method is a way to get a model that can take that big step anyway. In practice that means distillation: you train a normal multi-step teacher, then a 1 to 4 step student from it.

A 4-step student runs about 4.5 times faster than a 16-step teacher [10], and about 14 times faster than a 128-step teacher [11]. In another case a 4-step student matches its 32-step teacher while running about 5 times faster [12].

I would expect a student trained only to match the teacher's velocity to blur at low NFE, because regressing onto an average gives an averaged mel. The students that hold quality add a signal at the level of distributions, metrics or a discriminator, which is why distribution matching with a GAN loss is a common choice. Treat a metric loss with care: a student trained with a CTC loss scored a lower WER than the ground-truth recordings [11]. Ground truth is not a perfect floor, but beating it is a warning sign that the student is partly optimising the evaluator.

Folding guidance into the model

Classifier-free guidance doubles every step, and step distillation alone does not remove it. My read is that a guided 2-NFE student still makes four model calls, half of them for the unconditional pass.

The fix is to fold guidance into the student, so it produces the guided output in one pass. The student learns to reproduce the guided teacher directly, so the unconditional branch disappears at inference. The best version conditions the student on the guidance strength as an input, so one network covers a range of strengths.

The cost is the knob. In Voicebox, the best guidance strength depends on NFE: at 4 steps or fewer, weaker guidance gives higher speaker similarity. Baking in one strength fixes the text-versus-speaker balance (Failure Modes and Evaluation) for every prompt. Keeping the strength as an input keeps both the single pass and the knob.

Streaming flow matching

Streaming needs a chunk-aware causal mask: the decoder sees past frames plus a bounded look-ahead of future frames, and renders one chunk at a time. The look-ahead gives each chunk some right context at its boundary, from a right-padded convolution or from the first few tokens of the next chunk. Chunks of about a second with a few tokens of look-ahead are typical, and some systems crossfade chunk edges.

You do not need two models for streaming and offline use. CosyVoice 2, a widely used AR-plus-flow streaming system, trains one decoder by sampling one of four masks per example: non-causal, full-causal, chunk-$M$ and chunk-$2M$. It writes the first-packet latency as $M$ tokens passing through each stage [13]:

Here $d_{\text{lm}}$, $d_{\text{fm}}$ and $d_{\text{voc}}$ are the per-token times of the LM, the flow decoder and the vocoder. Every term scales with $M$, so chunk size is the latency dial. The formula leaves out two waits: the look-ahead frames, and the text tokens the LM buffers before it emits its first speech token. Every frame of look-ahead is paid once, in first-packet time, which is why production systems keep it to a few tokens.

Serving under load

Distrust real-time factor (RTF) comparisons across papers. The same open model at 32 NFE is reported at RTF 0.31, 0.30 and 0.17 on three different GPUs [3, 10, 12]. Implementation matters as much as hardware: on the same GPU class, another paper measures that model at 0.123 against 0.31 [8]. Only ratios measured on one machine mean anything: in one, a 2-NFE streaming system against CosyVoice 2 at 10 NFE cut first packet from 843 ms to 377 ms on one RTX 4090 [14].

Concurrency changes the ranking. In one system that ships both decoders at 0.6B parameters, the DiT has the better RTF at one stream (0.234 against 0.288), but at six streams it reaches 0.709 against 0.434 for the causal decoder [9]. Its first packet grows from 138 ms to 481 ms, while the causal decoder's grows from 97 ms to 299 ms. A single-stream benchmark would pick the wrong decoder for a fleet, so benchmark at the concurrency you will serve.

The two decoders also differ in token rate, but my read is that part of the gap is structural. The AR stage reuses its KV cache from token to token. The flow stage cannot reuse activations across ODE steps, because its input $x_t$ changes at every step. A bounded context window is the nearest substitute, and my read is that it also caps memory per stream.

Flow-stage cost is NFE, times one plus the guidance pass, times model size and length, so step distillation and guidance folding multiply. My read is that once the flow model is down to 2 to 4 NFE, the vocoder becomes the next bottleneck, because its cost did not shrink. In the same system, the vocoder's right-context look-ahead alone adds 130 ms [9].

Pairing Flow Matching with an LLM

A language model is good at deciding what to say, in what order and with what emphasis. It is poor at writing 50 mel frames a second of spectral detail, and a flow decoder is the reverse. A common design splits the job: the LLM plans content, timing and prosody, and a flow or diffusion decoder renders voice and acoustics from noise. The real decision is what crosses the boundary: it sets the token rate, what the decoder must invent, and whether the halves train apart.

What crosses the interface

The most common answer is discrete semantic tokens. CosyVoice 2 puts a quantizer inside a speech recognition encoder and trains it on the recognition objective, so its 25 Hz tokens carry mostly what is said and when, and little of who says it [13]. The LLM predicts them like text, so sampling, KV caching and token-level RL carry over unchanged.

The second answer passes hidden states between two models before any audio token exists. In a Thinker-Talker design such as Qwen2.5-Omni, the Talker reads the Thinker's hidden representations, emits audio tokens, and a sliding-window DiT renders them.

The third drops quantization. The LLM's last hidden state conditions a small diffusion or flow head that turns one frame of noise into the next continuous latent, which is fed back as the next input. VibeVoice runs this at 7.5 latent frames a second, so a 10-second clip costs 75 LLM steps [15].

Tradeoff: a lower token rate is cheaper for the LLM, and the flow stage pays for it. Kimi-Audio runs 12.5 Hz tokens into a 50 Hz mel, a 4x upsample against CosyVoice 2's 2x, so its decoder fills in more timing detail per token [16]. Keeping voice out of the tokens is the same logic as dropping the speaker embedding from the LM in Training. The flip side is that a weak prompt or decoder shows up as voice drift, not as wrong words.

Conditioning and training the decoder

In the token route, the decoder takes three conditions. Token embeddings, upsampled to the mel frame rate, give content and timing. A speaker embedding from a separate speaker model gives identity, and the prompt's mel, laid down as unmasked context, gives the exact voice. This is the masked infilling recipe with tokens in place of padded characters, and the token count fixes the mel length, so there is no duration model.

The two halves train apart. The LLM learns cross-entropy on tokens from the frozen tokenizer; the decoder learns flow matching on real token and mel pairs. That decoupling is the main operational win: you can swap the LLM, retrain the decoder for a new domain, or run RL on the token stage without touching the other half.

The cost is a distribution gap. The decoder only ever saw tokens that the tokenizer produced from real speech, and at inference it reads tokens the LLM sampled, so errors in the token stream become audible. The decoder renders whatever sequence it gets and cannot repair one that no real utterance would produce.

The continuous route trains end to end. The head's diffusion loss flows back through the hidden state into the LLM, so the LLM learns to pack in what the head needs for the next frame, and only the audio tokenizer stays frozen. The price is exposure drift: training reads true latents, generation reads its own samples, and noise scale and guidance are the main controls.

Speech LLMs and the flow decoder

Several dialogue and omni models took the token route first, with a flow matching module and a vocoder rendering chunk by chunk. The look-ahead is the knob: Kimi-Audio peeks at next-chunk tokens, which delays only the first chunk [16], and Qwen2.5-Omni's DiT sees one block ahead and two behind [17]. The masks are in Streaming flow matching.

Full duplex is where the flow stage starts to hurt. First audio waits for the LM's first chunk, the look-ahead, the solver passes and the vocoder, and a duplex model also has a per-frame deadline. Qwen3-Omni dropped block-wise diffusion: its Talker predicts multi-codebook codec frames and a lightweight causal ConvNet decodes each frame as soon as it exists. For the whole pipeline it reports a theoretical cold-start first packet of 234 ms [18].

Tradeoff: the LM now has to predict every codebook of every frame, through a multi-token prediction module, so acoustic detail moves back into the LM. My read is that flow decoders stay the default where voice quality and cloning matter, and duplex assistants with a per-frame deadline keep moving to causal codec decoders.

Papers and Trade-offs at a Glance

A common shipped design is an autoregressive model that writes speech tokens and a flow matching or diffusion decoder that renders them for a vocoder. In Seed-TTS, deployed latency fell to 0.028 times the offline system's, with WER and similarity unchanged [19]. Flow matching became the common choice with Voicebox, and the table below groups the systems by family.

The table keeps the systems that set the designs in use today, with rows grouped by family: diffusion, non-autoregressive flow matching, AR tokens plus a renderer, speech LLM decoders, and systems with no ODE at all.

Five axes separate these systems, and the "what breaks" column is my read of the papers. Grad-TTS shows gains flattening after about 10 steps, and CFG doubles the forward passes behind every step.

Failure Modes and Evaluation

Most zero-shot TTS teams report the same two numbers, and at the top of the field neither separates a good system from a very good one. The failures that reach users (skipped words, odd pacing, flat delivery, degraded output from a noisy prompt) barely move a corpus-level average.

How these models are scored

The standard harness is cheap and automatic. WER comes from transcribing the generated audio with an off-the-shelf ASR model, Whisper for English and Paraformer for Mandarin. SIM is the cosine similarity between speaker-verification embeddings of the prompt and the output, usually from a WavLM-based encoder. Most labs run it on a LibriSpeech-PC test-clean subset and on the English and Mandarin sets in ByteDance's seed-tts-eval repository, then add a small listening test for naturalness.

The harness saturates first. On the seed-tts-eval English set, F5-TTS scores better on WER than the ground-truth recordings [3]. A model is not more intelligible than a person; my read is that it produces cleaner, more canonical speech that the ASR model finds easier to transcribe. At that level WER measures fit to the evaluator more than quality, and a small WER gap between two systems tells you little about which one listeners prefer.

The evaluator is also part of the result. If you rerank samples with one ASR model and score with another, the winner changes with the ASR family, and a same-family pair flatters itself. The same goes across papers: each lab picks its own test subset, ASR model and text normalisation, so WER numbers from two papers rarely measure the same thing. SIM scores from different speaker encoders do not compare, so a SIM number only means something next to a baseline scored with the same encoder. Predicted MOS models such as UTMOS and DNSMOS catch audible defects but stop agreeing with listeners once both clips are clean, which is exactly where a mature system lives. In practice that means holding the ASR model and speaker encoder fixed across every comparison, and keeping human listening tests for the decisions that matter.

Where they break

Most failures trace back to one root. The model discovers text alignment and total duration implicitly, must find where the prompt ends on its own, and inherits everything in the prompt.

Skipping and repetition shows up in training before it shows up in production, so watch per-sample WER, not just the mean. In F5-TTS's reproduction of E2 TTS, about 7% of Mandarin test samples came out badly garbled during training, and re-ranking could not rescue them [3]. The F5-TTS authors also found that an MMDiT variant collapsed into repeated utterances [3].

Post-training with rewards

Because WER and SIM are cheap to compute, a common recent practice is to turn them into rewards and post-train with RL. The catch is that a deterministic ODE sampler gives no likelihood for a policy gradient. One route turns the ODE into an SDE so GRPO has a likelihood to work with. The other leaves the flow decoder alone and puts the reward where likelihoods are natural: a differentiable reward on the speech-token language model, or RL on the duration predictor alone. The token-level route pays off: CosyVoice 3's reward post-training clearly cuts its Mandarin character error rate [22].

Reward hacking

The reward is an ASR model or a quality predictor, and the policy learns to satisfy it rather than the listener. CosyVoice 3 already shows the cost, with speaker similarity dropping slightly after RL. A single quality predictor as the reward makes human ratings worse, while an equal-weighted ensemble of predictors resists it.

Prosody is where this bites hardest. GRPO on error-rate and likelihood rewards lowered error rates but made speech flat, in a study on a non-flow TTS model. A few hundred human preference pairs per round of iterative DPO recovered the naturalness. There is no verifiable reward for prosody, so the policy trades it away for what is measured.

Safety

A model that clones a voice from a few seconds of audio is a spoofing tool by construction, and shipped systems use two defences. The first is an audio watermark trained jointly with a detector, which can localise the mark down to individual samples. Audiobox reports that its detector stays reliable across the audio edits it was tested against [21].

A watermark only protects outputs of a model that applies it, and anyone running released weights can skip that step. That is the case for the second defence, keeping weights in-house: ByteDance releases seed-tts-eval but withholds the Seed-TTS model "due to considerations for AI safety". Tradeoff: one of the most-reported benchmarks comes from a model outsiders cannot run.

Online RL for Flow-Matching TTS

Pretraining regresses a velocity field onto data, and nothing in that loss targets WER, speaker similarity or naturalness, which the evaluation harness already scores cheaply. GRPO closes that gap by scoring a group of samples per prompt and pushing toward the better ones, but it assumes a stochastic policy with a likelihood at every step, and an Euler ODE sampler has neither.

Why GRPO does not fit a flow model

For a prompt with $G$ rollouts and scalar rewards $R^i$, the advantage is:

A rollout counts as good only if it beats its siblings, so no learned critic is needed. If every rollout scores the same, the standard deviation is zero and the group carries no signal, which is why the recipe below drops such groups. The update multiplies the advantage by a clipped ratio of new to old step likelihoods, with a KL penalty to a frozen reference. That ratio is where a flow model breaks: once the initial noise is drawn every ODE step is deterministic, so a step has no density, and the group explores only through its starting noise.

Four ways around it

The four workarounds differ in what becomes the policy. The first is to turn the ODE into an SDE for rollouts. Adding noise with a chosen rollout schedule $\sigma_t$, and correcting the drift with a score term that for rectified flow is a function of $v_\theta$, keeps the marginals at every $t$:

Here $\mu_\theta$ is the velocity plus that correction and $z \sim \mathcal{N}(0, I)$ is a fresh draw per step, so each step is a Gaussian with a closed-form density and GRPO applies unchanged. Rollouts dominate the cost, so they run short: for Flow-GRPO on images, cutting rollout steps from 40 to 10 trained over 4x faster with no loss in final reward [24]. The FlowTTS-GRPO recipe also injects noise only in a short window of early steps, trains only those, and drops CFG from rollouts, which converged faster and still helped with CFG back on at inference [25]. The catch is that the injected noise is itself a defect: an SDE baseline on an AR-diffusion TTS model lost UTMOS on average [26].

The second retrains the network to output a Gaussian over the velocity, so every step has a density without changing the sampler. It works, but F5R-TTS needed a 1M-update pretraining run before its 1,100 GRPO updates [27].

The third skips the likelihood. Roll out with the ordinary ODE sampler, take each rollout's final latent $x_1^{i}$, and re-noise it along the straight path with $x_0 \sim \mathcal{N}(0, I)$ and $\sigma_{\min} = 0$:

The flow matching loss is then weighted by the group advantage $\hat A^{i}$, with a second term that anchors the velocity to the frozen reference velocity $v_{\text{ref}}$ at weight $\beta$:

The advantages have zero mean, so the loss pulls toward above-average rollouts and away from the rest. The sign matters: WER and SIM rewards cluster tightly within a group, so positive-only weights mostly teach the model to imitate its own samples.

The fourth leaves the flow model alone and runs RL where likelihoods already exist: the speech-token LM in two-stage systems, or a categorical duration predictor in F5-TTS-style models. Industry has used this route most: Seed-TTS ran REINFORCE on SIM and WER rewards [19], and CosyVoice 3 backpropagated a differentiable reward into its tokens [22]. In those systems the LM sets intelligibility; when CosyVoice 3's flow stage was trained with GRPO, its ASR reward stalled close to its ceiling while similarity kept climbing [25]. The reverse holds too: a flow decoder and vocoder trained on clean audio partly undo an LM-only MOS reward, and speaker attributes respond weakly because the speaker enters at the flow stage.

A working recipe

Start from a converged checkpoint, which doubles as the frozen reference, and build prompts as (prompt audio, prompt text, new target text) triples so RL sees the zero-shot task. Sample 8 to 16 rollouts per prompt: for Flow-GRPO on images, groups of 6 and 12 collapsed where 24 held [24], and a duration policy was unstable at 8 but not at 16 [12]. Score each rollout with a fixed ASR model (one minus WER or CER), a speaker-verification cosine to the prompt and optionally a MOS predictor. Normalise each reward by its batch standard deviation before summing so the noisiest one does not dominate; FlowTTS-GRPO weights similarity and ASR at 1.0 and DNSMOS at 0.4 [25]. Drop groups whose rewards barely differ. Train a LoRA adapter (rank 32 is under 3% of CosyVoice 3's flow model [25]) with a KL or velocity anchor, for hundreds to about ten thousand updates against hundreds of thousands to a million in pretraining.

Reward hacking

The policy optimises the reward models, not the listener. Seed-TTS lowered WER by speaking slowly and over-articulating, at a cost in naturalness [19]. Learned reward models get gamed by acoustic artifacts, much as a single MOS predictor does in the previous section. Training length is its own risk: the DMOSpeech 2 duration policy degraded sharply after about 1.5K to 2K steps [12].

I would use three defences. Sum several rewards and keep the KL anchor on; on images, KL and early stopping were not interchangeable. Evaluate with judges the policy never saw, a different ASR model and speaker encoder than the rewards use; in FlowTTS-GRPO, WavLM similarity, which was not one of the rewards, still rose clearly [25]. Stop on a held-out listening set that tracks speaking rate, not on training reward.

Techniques Simplified: What to Use?

Start from the constraint you face first; each row gives a common choice and what it costs.

Open Problems

• Rewards for the flow stage: A deterministic ODE sampler has no likelihood for a policy gradient, so current practice rewards the token model or duration predictor instead. One route turns the ODE into an SDE, which also changes the sampler.
• The few-step floor: Schedules are usable to about 7 steps and break below about 6. Students go further but lose diversity and can learn to game the evaluator.
• Evaluation that saturates: Some top systems already beat ground-truth recordings on WER, and predicted MOS stops agreeing with listeners once both clips are clean. Neither WER nor SIM measures diversity.

Key Takeaways

1. Straight training paths cut steps, not to one: the learned field averages crossing paths, so trajectories still bend.
2. Masked infilling gives prompting for free: scoring only the hidden frames turns the reference clip into the voice condition, with no speaker encoder.
3. Alignment and length are the hard part: phones need an aligner, padded characters converge slowly, and the large stacks hand both to an AR token model.
4. Cut cost in order: a warped or pruned schedule with Euler is usable to about 7 steps and breaks below about 6, and past that a distilled student with guidance folded in runs several-fold faster.
5. Retune guidance when you cut steps: the best strength shifts with step count, and weaker guidance wins at 4 steps or fewer.
6. Benchmark where you serve: RTF does not transfer across papers, and the decoder that wins at one stream can lose at six.
7. WER and SIM saturate, and rewards get hacked: hold evaluators fixed, keep listening tests, and use human preferences to protect prosody.
8. Watermarks are not enough: they protect only outputs of models that apply them, which is why at least one lab withholds its weights.

References

1. Le et al. (2023). Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale. arXiv:2306.15687.
2. Ho, Jain, and Abbeel (2020). Denoising Diffusion Probabilistic Models. arXiv:2006.11239.
3. Chen et al. (2024). F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. arXiv:2410.06885.
4. Lipman et al. (2022). Flow Matching for Generative Modeling. arXiv:2210.02747.
5. Liu, Gong, and Liu (2022). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. arXiv:2209.03003.
6. Eskimez et al. (2024). E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS. arXiv:2406.18009.
7. Mehta et al. (2023). Matcha-TTS: A Fast TTS Architecture with Conditional Flow Matching. arXiv:2309.03199.
8. Zheng et al. (2025). Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling. arXiv:2505.19931.
9. Hu et al. (2026). Qwen3-TTS Technical Report. arXiv:2601.15621.
10. Zhu et al. (2025). ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching. arXiv:2506.13053.
11. Li et al. (2024). DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis. arXiv:2410.11097.
12. Li et al. (2025). DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis. arXiv:2507.14988.
13. Du et al. (2024). CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. arXiv:2412.10117.
14. Xie et al. (2026). FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation. arXiv:2606.09141.
15. Peng et al. (2025). VibeVoice Technical Report. arXiv:2508.19205.
16. KimiTeam (2025). Kimi-Audio Technical Report. arXiv:2504.18425.
17. Xu et al. (2025). Qwen2.5-Omni Technical Report. arXiv:2503.20215.
18. Xu et al. (2025). Qwen3-Omni Technical Report. arXiv:2509.17765.
19. Anastassiou et al. (2024). Seed-TTS: A Family of High-Quality Versatile Speech Generation Models. arXiv:2406.02430.
20. Popov et al. (2021). Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. arXiv:2105.06337.
21. Vyas et al. (2023). Audiobox: Unified Audio Generation with Natural Language Prompts. arXiv:2312.15821.
22. Du et al. (2025). CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training. arXiv:2505.17589.
23. Wang et al. (2024). MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer. arXiv:2409.00750.
24. Liu et al. (2025). Flow-GRPO: Training Flow Matching Models via Online RL. arXiv:2505.05470.
25. Wang et al. (2026). FlowTTS-GRPO: Online Reinforcement Learning with Multi-Objective Reward Optimization for Flow-Matching Based Text-to-Speech. arXiv:2606.23190.
26. Yang et al. (2026). GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model. arXiv:2608.03215.
27. Sun et al. (2025). F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimization. arXiv:2504.02407.