Finkbeiner, DouglasMahadevan, LakshminarayananYi, Kexin2021-07-1320212021-05-112021-05Yi, Kexin. 2021. Neural Symbolic Machine Reasoning in the Physical World. Doctoral dissertation, Harvard University Graduate School of Arts and Sciences.28498436https://nrs.harvard.edu/URN-3:HUL.INSTREPOS:37368350The ability to understand physics lies at the core of human intelligence. It has been a long-standing research goal in artificial intelligence to build algorithms able to perceive, explain and predict the physical world. The recent success of deep neural networks has triggered a surge of advancements in models and datasets based on visual observations. However, these efforts mainly focused on recognizing patterns, while physics understanding and reasoning remain underexplored. In this dissertation, I will discuss a series of works on physical reasoning beyond pattern recognition, via a hybrid approach that combines neural networks and symbolic representations. I will first discuss the application of the neural symbolic framework to visual question answering on static images. Then I will present a video dataset that focuses on dynamic and causal reasoning on simple physics experiments that involve classic object collision. A neural symbolic model that tackles these challenging tasks is also introduced. Next I will study dynamics modeling and prediction on more complex physical systems such as rigid bodies and deformable objects. Finally, I will conclude the dissertation by briefly discussing future directions of applying machine intelligence to open scientific questions.application/pdfenPhysicsThesis or Dissertation2021-07-130000-0002-6007-2407