- JPB-UG-1: Active Shape Modelling for Lip Tracking (Tomos Lee)
- JPB-UG-2: Video Based Speech Detection
- JPB-UG-3: A Web Tool for Building Bibliographic Databases (Daneil Chu)
JPB-UG-1: Active Shape Modelling for Lip Tracking
Description
Lip tracking is the task of following the outline of a speaker’s lips through a sequence of video frames. This task is an important component of many audio-visual speech processing applications – including audio-visual speech recognition. The most successful lip tracking systems employ a technique know as active shape modelling (ASM). This technique employs a statistical model of the shape (and possibly, appearance) of the speaker’s lips that has been learnt from a small number of video frames in which the lip outlines have been traced by hand. The tracking system then examines the video and employs an iterative search to find a sequence of smoothly changing lip shapes that fit well to this model.
This project aims to build a Java-based demonstration of the ASM technique. For evaluation purposes the project will use part of a large corpus of audio-visual speech data that has recently been collected at Sheffield. Much of the necessary preprocessing has already been performed allowing the project to have a running start.
The references below provide an overview of Cootes’ ASM technique that will underpin the project (do not be put off by the maths - at heart the idea is quite simple and it works equally well whether you understand the maths or not!). For further background see Cootes’ web page.
Requirements
- good Java programming skills</BLOCKQUOTE>
Reading
- T.F.Cootes, C.J. Taylor, D.H.Cooper and J.Graham (1995) Active Shape Models - Their training and application, Computer Vision and Image Understanding 61(1) pp. 38-59
- T.F. Cootes and C.J. Taylor (2001), Statistical models of appearance for medical image analysis and computer vision. Proc. SPIE Medical Imaging
- I. Matthews, T.F. Cootes and J.A. Bangham (2002), Extraction of Visual Features for Lipreading. IEEE PAMI Vol.24, No.2, pp.198-213
- D.Cristinacce and T.F.Cootes (2004), A comparison of shape constrained facial feature detectors. Proc. Int.Conf on Face and Gesture Recognition
Data
[TOP]
JPB-UG-2: Video Based Speech Detection
Description
Automatic detection of whether or not someone is speaking has useful applications in speech recognition and telecommunications. There are many audio-based techniques for speech detection, but these techniques can be unreliable in the presence of background noise. Speech may also be detected using visual lip movement information. However, the problem is not as trivial as it may first seem. A trivial solution would simply detect lip motion and then assume that lips are moving if and only if the person is speaking. However, people often move their lips while not speaking. A better solution would need to discriminate between the type of lip movements that accompany speech, and lip movements that occur during non-speech periods.
For evaluation purposes the project will use part of a large corpus of audio-visual speech data that has recently been collected at Sheffield. The audio signal will be used to determine when the speaker starts and stops talking. The aim of the project will be to attempt to estimate these start and stop times using only the video data. Much of the necessary video and audio preprocessing has already been performed allowing the project to have a running start.
Requirements
- good Java programming skills</BLOCKQUOTE>
Reading
- Iyengar and Neti (2001) A Vision-based Microphone Switch for Speech Intent Detection
- Talking Heads Website - Useful visual speech resource
JPB-UG-3: A Web Tool for Building Bibliographic Databases
Description
Maintaining bibliographic databases is a chore that often involves retyping bibliographic details, or cutting and pasting them from the web. In an manner analogous to the way in which iTunes automatically retrieves CD details from the Gracenote online database, this project would retrieve bibliographic details from the citeseer ‘digital library’. The user will supply partial details of a reference and the program will use the Google Web API to locate the required reference in the citeseer `digitial library’. It will then automatically extract and store any bibliographic details that are available. The project could either be built in to an existing bibliographic database manager such as the open source project, Jabref. Or, it could be designed as a standalone Java applet or application.
If you are interested in this project I recommend that you come and see me so that I can demonstrate to you what the project is all about.
Requirements
- Java programming skills</BLOCKQUOTE>
Reading
- Jabref - open source bibliography manager
- Bibtex - a bibliography file format
- citeseer - scientific literature digital library
- Google Web API - the Google web search API