J. Edge, "Techniques for the Synthesis of Visual Speech", PhD, 2004. (Supervisor: Dr Steve Maddock). [pdf]
PhD Abstract: Face-to-face dialogue is the natural mode of communication between humans. We see changes in expression and hear changes in intonation, and the combination of these provides semantic information that communicates ideas, feelings, and concepts. This is exhibited not only in the changes in speech, which confers the majority of the meaning, and the properties of vocalisation (e.g. tone, tempo and loudness), but also in changes of facial expression. This thesis investigates techniques for the synthesis of visual speech movements from the initial data capture process through to the final animation of a talking head.
Synthesis can be split into three general processes: modelling, capture, and animation. Modelling requires techniques to represent and parameterise changes in facial expression during speech production. Capture is the retrieval of information about speech articulation from real speakers, which can be either static poses (visual-phonemes/visemes) or dynamic speech movements. Finally, animation techniques take captured information about speech articulation and use it to generate trajectories through the parametric space of a facial model.
This thesis presents novel methods in each of these categories, in the framework of several systems for text-to-visual speech synthesis. Modelling is performed using geometric free-form deformation techniques to manipulate two- (image) and three-dimensional (mesh) representations of faces. Statistical techniques are used to parameterise the manipulation of facial expression. A novel technique for the retargetting of captured motions to meshes, which vary in both shape and scale from the original actor, is introduced. Animation is performed using target-based models of coarticulation, and by concatenating captured motion fragments. A novel technique for the target-based modelling of coarticulation, based upon constrained-optimization techniques, is reported.