Structure Modeling for Language Models

Abstract

Neural language models are probabilistic models of text parameterized by neural networks. They are widely applicable to applications with outputs consisting of discrete sequences, such as document summarization, question answering, and image captioning. The minimal assumptions about data enable advancements in language modeling to drive improvements across a diverse array of applications.

In natural language, structures are both pervasive and essential. For example, a book is organized into chapters, with a logical flow connecting them; without this structure, the book would lose its coherence. Therefore, effectively understanding and modeling textual sequences requires comprehending and representing the inherent structures. This thesis focuses on structure modeling for language models.

The thesis is organized into two primary sections: structure analysis for language models and structure modeling techniques. The first section investigates the modeling of various structural aspects in language model generations, including section transition structures, coreference structures, and topic correlation structures. Emphasizing the need for a comprehensive understanding of these components, the thesis assesses language model performance at the structural level. Adapting a statistical framework to evaluate high-level coherence in machine-generated texts, the research uncovers that even large language models display limitations in capturing discourse coherence and coreference. Furthermore, it demonstrates that improvements in surface-level modeling do not necessarily guarantee better structure modeling.

The second section of the thesis presents a variety of structure modeling techniques aimed at improving or customizing language models. These techniques are organized into three categories: factorized structure modeling, hierarchical structure modeling, and global structure modeling. They can improve the structural coherence, transparency, computational efficiency, and data efficiency of language models.

In conclusion, this research conducts an in-depth exploration of structure analysis and modeling techniques for language models. Through the development of various structure analysis methods and modeling approaches, the thesis endeavors to deepen the understanding and improvement of language models concerning their capacity to represent structures. The proposed techniques hold the potential to boost language model performance across a broad array of applications, ultimately advancing not only natural language generation but also other domains where the output space comprises discrete sequences, such as computer vision, robotics, and genomics.