NandiNet: A Facial Movement-Activated Switch For Aphonic Communication

Abstract

My seventeen-year-old younger sister, Nandita, is severely physically disabled but is cognitively fine. As a result of her physical impairment, she communicates non-audibly by mouthing words. Familiar caregivers can understand her aphonic communication with a full vocabulary, but unfamiliar caregivers cannot and may even struggle to tell if she is trying to communication at all. In this thesis, I present NandiNet, a classifier that aims to classify short clips of Nandita by whether she is speaking in them or not. On a fifty-video survey, NandiNet successfully classified forty videos (80% accuracy). This exceeded the performance on the same survey by three unfamiliar individuals without speech pathology experience (average 48%), three speech pathologists who have not met Nandita (average 51%), and my two parents (66% and 76%). Although the survey does not allow for the use of conversational context which helps many familiar caregivers understand Nandita with ease, it serves as a reasonable proxy for interactions that Nandita may have with unfamiliar caregivers. I hope to test a continuous monitoring program that implements NandiNet repeatedly on 2.5-second intervals in the near future.