Semantic Speech Tokenization

NormToken Speaker- and Duration-Normalized Semantic Speech Tokens
via Iterative S2U–T2U Refinement

Hanlin Zhang1,*‡ Daxin Tan2,* Dehua Tao2,* Chengxi Deng3 Xiao Chen2,† Linqi Song1,†

1City University of Hong Kong 2AI Lab, Leibniz Research Center, Huawei 3Chinese University of Hong Kong

* Equal contribution. † Corresponding authors. ‡ Work done during an internship at Huawei.

Abstract

01

Method

Figure PDF
NormToken method: initialize a HuBERT-based semantic tokenizer, alternate S2U and T2U training, and synthesize speech with a token-conditioned de-tokenizer.
Iterative S2U–T2U refinement and token-conditioned speech synthesis.
02

Text-to-Speech

Seed-TTS

Text + reference speech

03

Voice Cloning

Seed-TTS

Source content + reference voice