Efficient Autoregressive TTS

WAND: Windowed Attention and Knowledge Distillation
for Efficient Autoregressive Text-to-Speech Models

Adapt pretrained AR-TTS models to constant compute and memory — up to 66.2% KV-cache reduction and near-constant per-step latency, with quality preserved.

Hanna Lee1, Tan Dat Nguyen1, Jaehoon Kang2, Kyuhong Shim2
1Korea Advanced Institute of Science and Technology (KAIST)  ยท  2Sungkyunkwan University (SKKU)
๐Ÿ“„ Paper (arXiv) </> Code (coming soon) ๐Ÿ”Š Audio Samples

Abstract

Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically with sequence length due to full self-attention. In this paper, we propose WAND, Windowed Attention and Knowledge Distillation, a framework that adapts pretrained AR-TTS models to operate with constant computational and memory complexity. WAND separates the attention mechanism into two: persistent global attention over conditioning tokens and local sliding-window attention over generated tokens. To stabilize fine-tuning, we employ a curriculum learning strategy that progressively tightens the attention window. We further utilize knowledge distillation from a full-attention teacher to recover high-fidelity synthesis quality with high data efficiency. Evaluated on three modern AR-TTS models, WAND preserves the original quality while achieving up to 66.2% KV cache memory reduction and length-invariant, near-constant per-step latency.

Method

WAND combines three ingredients to make autoregressive TTS decoding cheaper without retraining from scratch:

01

Windowed Attention

Persistent global attention over conditioning tokens, plus local sliding-window attention over generated tokens — turning a growing KV cache into a constant-memory, constant-cost operation.

02

Knowledge Distillation

The windowed student is distilled from the original full-attention teacher (skewed-KL on logits + CE on ground-truth tokens), recovering quality the hard window would otherwise lose.

03

Window Curriculum

A cosine schedule anneals the window from W=128 โ†’ 32 with a soft-to-hard mask temperature, easing the model into aggressive locality.

Overview of the WAND framework
Overview of the WAND framework. Conditioning tokens (system prompt, text, reference audio) retain global attention access, while generated acoustic tokens are restricted to a fixed-size sliding window of size W. A curriculum progressively tightens the window from the early to the late training stage under a cosine schedule.

Why a bounded window suffices. Analyzing the attention of the original full-attention models shows that the conditioning prefix and a small local window already capture 85–91% of all attention mass — so restricting the decode region to a fixed window discards only a small fraction of what the model actually uses.

Model Prompt (%) Generated (%) Local-W / Gen. (%) Coverage (%)
CosyVoice 2 (W=32)58.541.570.287.6
IndexTTS 1.5 (W=32)64.635.457.184.8
SparkTTS (W=64)47.952.182.891.0

Attention of vanilla full-attention models during decoding, averaged over 5 long utterances. Local-W/Gen. is the share within a window of size W among generated tokens; Coverage = Prompt + Generated ร— Local/Gen.

Results

On three architecturally diverse backbones, WAND turns a linearly-growing KV cache into a bounded one — up to 66.2% cache reduction and 1.89ร— speedup — while UTMOS, NMOS, SSIM, and WER stay on par with (or better than) the full-attention baseline.

ModelAttention KV Cache โ†“Reduction โ†‘ GFLOPs โ†“Speedup โ†‘ UTMOS โ†‘NMOS โ†‘SSIM โ†‘WER โ†“
Ground truth– –––– 3.533.791001.44
CosyVoice 2-0.5BFull (W=โˆž) 10.48 MB–11.55– 4.184.0194.61.94
WAND (W=32) 5.25 MB49.9%7.441.57ร— 4.214.0294.51.72โˆ’0.22
IndexTTS 1.5Full (W=โˆž) 38.44 MB–6.18– 3.974.1394.40.98
WAND (W=32) 13.01 MB66.2%3.281.89ร— 3.984.1394.40.91โˆ’0.07
SparkTTS-0.5BFull (W=โˆž) 18.09 MB–48.12– 3.933.9492.63.27
WAND (W=64) 7.15 MB60.5%31.741.51ร— 3.933.9792.83.11โˆ’0.16

Seed-TTS-eval test-en. Efficiency (KV cache, GFLOPs) accumulated for 10 s of audio; KV cache in fp32. All WAND models fine-tuned on 100 h of English data for 1 epoch.

Per-step decoding latency
Per-step decoding latency vs. number of generated tokens. Full attention grows linearly (๐’ช(n) per step), while sliding-window attention stays constant (๐’ช(1) per step) regardless of output length.

Audio Samples

Zero-shot voice cloning on the Seed-TTS test-en set. For each backbone we compare the original model against its WAND (windowed + distilled) variant. Ground truth is the target speaker reference.

# Ground Truth Baseline WAND (ours)

Showing 12 representative samples. Full sample sets are released with the code.

Code

Code coming soon. The implementation is being cleaned up and will be released here together with training scripts, distillation recipes, and the full evaluation suite.

In the meantime, the WAND recipe applies on top of public checkpoints of CosyVoice 2, IndexTTS, and Spark-TTS. Check back, or watch this page for the release.

BibTeX

@inproceedings{lee2026wand,
  title     = {WAND: Windowed Attention and Knowledge Distillation for
               Efficient Autoregressive Text-to-Speech Models},
  author    = {Lee, Hanna and Nguyen, Tan Dat and Kang, Jaehoon and Shim, Kyuhong},
  booktitle = {Proc. Interspeech 2026},
  year      = {2026}
}