Talking avatar and lip sync

A talking avatar is two engines in harness: one creates the face and facial expressions, the second synchronizes lips with sound (lip sync). The neural network breaks speech into phonemes - small sounds - and selects the shape of the mouth to match them. Hence the main rule: write the script as THEY SAY, not as they write. Short phrases, simple words, without long participial phrases - otherwise the lips will “chew” and the out-of-sync will be obvious. Mechanics of facial expressions: the model reads the emotion from the description and punctuation marks, so clearly indicate the mood (“friendly, with a slight smile”) and punctuate pauses. Insider: keep one emotion per line. A sharp change of “joy - anger - sadness” in three seconds breaks the face into an unnatural mask. The second technique is about 2 words per second: if you cram a lot of text into a short clip, the lip sync will get messed up. Calculate the length of the voiceover in advance. A typical mistake is to write robotic text with abbreviations and numbers (“ROI increased by 27.4%”): the model will pronounce it crookedly. Decipher it in words and test it by ear.