Geometric Deep Learning Enables 3D Kinematic Profiling Across Species and Environments

Abstract

Comprehensive descriptions of animal behavior require precise measurements of 3D whole-body movements. Although 2D approaches can track visible landmarks in restrictive environments, performance drops significantly in freely moving animals, where occlusions and appearance changes are ubiquitous. To enable robust 3D tracking, we designed DANNCE, a method using projective geometry to construct inputs to a convolutional neural network that leverages learned 3D geometric reasoning to track anatomical landmarks across species and behaviors. We trained and benchmarked DANNCE using a new 7-million frame dataset relating color videos and rodent 3D poses. In rats and mice, DANNCE robustly tracked dozens of landmarks on the head, trunk, and limbs of freely moving animals in naturalistic settings, achieving over an order of magnitude better accuracy than prior techniques. We extend DANNCE to rat pups, marmosets, and chickadees, and demonstrate a novel ability to quantitatively profile behavioral lineage over development. DANNCE offers unprecedented analytical access to animal behavior across species and environments.