Psychophysical Evaluation of Deep Re-Identification Models

Abstract

Pedestrian re-identification (ReID) is the task of continuously recognising the same individual across time and camera views. Researchers of pedestrian ReID and their GPUs spend enormous energy producing novel algorithms, challenging datasets, and readily accessible tools to successfully improve results on standard metrics. Yet practitioners in biometrics, surveillance, and autonomous driving have not realized benefits that reflect these metrics. Different detections, slight occlusions, changes in perspective, and other banal perturbations render the best neural networks virtually useless. This work makes two contributions. First, we introduce the ReID community to a budding area of computer vision research in model evaluation. By adapting established principles of psychophysical evaluation from psychology, we can quantify the performance degradation and begin research that will improve the utility of pedestrian ReID models; not just their performance on test sets. Second, we introduce NuscenesReID, a challenging new ReID dataset designed to reflect the real world autonomous vehicle conditions in which ReID algorithms are used. We show that, despite performing well on existing ReID datasets, most models are not robust to synthetic augmentations or to the more realistic NuscenesReID data.