Neural Symbolic Machine Reasoning in the Physical World

Abstract

The ability to understand physics lies at the core of human intelligence. It has been a long-standing research goal in artificial intelligence to build algorithms able to perceive, explain and predict the physical world. The recent success of deep neural networks has triggered a surge of advancements in models and datasets based on visual observations. However, these efforts mainly focused on recognizing patterns, while physics understanding and reasoning remain underexplored. In this dissertation, I will discuss a series of works on physical reasoning beyond pattern recognition, via a hybrid approach that combines neural networks and symbolic representations. I will first discuss the application of the neural symbolic framework to visual question answering on static images. Then I will present a video dataset that focuses on dynamic and causal reasoning on simple physics experiments that involve classic object collision. A neural symbolic model that tackles these challenging tasks is also introduced. Next I will study dynamics modeling and prediction on more complex physical systems such as rigid bodies and deformable objects. Finally, I will conclude the dissertation by briefly discussing future directions of applying machine intelligence to open scientific questions.