Novel applications of deep learning to computational ethology: behavior classification and full-body pose estimation

Abstract

The purpose of nervous systems is to generate adaptive behaviors that increase the animal’s fitness in its environment. Computational analysis of videos of animal behavior can help us both collect detailed measurements of animal movements and classify these observations into behaviors. Researchers often have prior knowledge of the behaviors-of-interest that are relevant for their study of neural function. However, manually labeling each behavior-of-interest is time-consuming, so researchers instead simply count bouts or limit the number of behaviors they study. To solve this problem, we created DeepEthogram: software to automatically label every frame of a video sequence with one or more human-defined behaviors. DeepEthogram uses convolutional neural networks to estimate motion, compute relevant features from motion and video frames, and to classify these features into behaviors. DeepEthogram is highly accurate, matching human-level performance on nine datasets across mice and flies. Neural activity across cortex in mice is infused with information about the animal’s ongoing movements. To better understand that activity, we developed a hardware recording setup and software analysis pipeline to fully reconstruct a mouse’s full 3D body posture during virtual navigation. We recorded videos of mice performing a virtual navigation task from multiple depth cameras. We use these videos and convolutional neural networks to estimate the 3D point cloud that makes up the mouse’s surface, 2D keypoints of specific anatomical features, and the silhouette of the mouse. We use these estimates to optimize the bone lengths and joint angles of a triangular mesh model. This procedure uses dense information from the environment and leverages prior knowledge about the mouse’s anatomy to estimate the mouse’s shape and posture with high fidelity. This rich behavioral representation can be used to improve our understanding of the relationship between ongoing behavior and cortical function. In the future, we will extend this pipeline to freely moving animals to study neural function and the effects of pharmacological therapies.