SteerTTS 发布 23M 参数英文 TTS 模型，可用数值控制音高语速等参数

I was looking for small TTS model which is steerable but couldn't find any, specially something I can adjust contextually by providing controls from LLM during a conversation. So I built one.

What it is:

9.6M-param acoustic model (attention + bidirectional Mamba-3, FastPitch-style) + Vocos vocoder = 23M total
12 controls, each a z-score relative to the chosen voice’s own neutral speech: rate, loudness, median pitch, pitch range, final rise/fall, pauses, spectral tilt, breathiness-related measures.
Also: 51 style tags, free-text prompts (resolved into an inspectable JSON request), inline markup (*emphasis*, [pause 400ms], <slow>…</slow>), sentence-level streaming
2,825 built-in (anonymized) voices, plus experimental zero-shot cloning

Speed:

- RTFx ~60–80 on an RTX 5070
- ~4–8 on CPU without the optional flow refiner. First streamed audio in ~0.5 s on CPU.

Quality:

~1.8% average WER on the Open TTS Leaderboard English sets (run locally, not an official submission, have reached out for it).

What doesn’t work well (we tried to be upfront):

Loudness, pitch, and tilt steer reliably; pause, voicing, and CPPS controls are weak
Emotion tags behave as style switches; an independent emotion classifier mostly can’t tell them apart
Cloning similarity is low. Don’t expect a convincing likeness
Free-text prompts struggle with negation (“not fast” can come out fast). More data needed for prompt to controls module.
English only, no human evals yet

License: weights CC BY-NC 4.0 (non-commercial), code MIT.

Model + full eval tables: https://huggingface.co/WhissleAI/steer-tts

A new architecture implemented from scratch. Would love feedback, especially on which controls are actually useful, and where the steering feels off.

submitted by /u/Working_Hat5120
[link] [留言]