The Mug-Shot Search Problem

Abstract

Mug-shot search is the classic example of the general problem of searching a large facial image database when starting out with only a mental image of the sought-after face. We have implemented a prototype content-based image retrieval system that integrates composite face creation methods with a face-recognition technique (Eigenfaces) so that a user can both create faces and search for them automatically in a database. These two functions are fully integrated so that interim created composites may be used to search the data and interim search results may, likewise, be used to modify an evolving composite. Although the Eigenface method has been studied extensively for its ability to perform face identification tasks (in which the input to the system is an on-line facial image to identify), little research has been done to determine how effective it is as applied to the mug shot search problem (in which there is no on-line input image at the outset). With our prototype system, we have conducted a pilot user study that looks at the usefulness of eigenfaces as applied to this problem. The study shows that the eigenface method, though helpful, is an imperfect model of human perception of similarity between faces. Using a novel evaluation methodology, we have made progress at identifying specific search strategies that, given an imperfect correlation between the system and human similarity metrics, use whatever correlation does exist to the best advantage. We have also shown that the use of facial composites as query images is advantageous compared to restricting users to database images for their queries.