Leveraging Human Brain Activity to Improve Object Classification

Abstract

Today, most object detection algorithms differ drastically from how humans tackle visual problems. In this thesis, I present a new paradigm for improving machine vision algorithms by designing them to better mimic how humans approach these tasks. Specifically, I demonstrate how human brain activity from functional magnetic resonance imaging (fMRI) can be leveraged to improve object classification.

Inspired by the graduated manner in which humans learn, I present a novel algorithm that simulates learning in a similar fashion by more aggressively penalizing the misclassification of certain training datum. I propose a method to learn annotations that capture the difficulty of detecting an object in an image from auxilliary brain activity data. I then demonstrate how to leverage these annotations by using a modified definition of Support Vector Machines (SVMs) that uses these annotations to weight training data in an object classification task. An experimental comparison between my procedure and a parallel control shows that my techniques provide significant improvements in object classification. In particular, my protocol empirically halved the gap in classification accuracy between SVM classifiers that used state-of-the-art, yet computationally intensive convolutional neural net (CNN) features and those that used out-of-the box, efficient histogram of oriented gradients (HOG) descriptors. Further analysis demonstrates that my experimental results support findings in neuroimaging literature about the roles different cortical regions play in object recognition.