Pointing Isn't Always Rude: Using Pointer Networks to Improve Word Prediction in a Language Model

Abstract

One of the problems in natural language generation is that of rare and unknown words. Pointer networks presents a method to mitigate this problem by allowing models to reference words in the source text and directly copy them. In this paper I propose the application of a pointer mechanism to the log bilinear language model, and analyze the effects on it compared to the original model. The results show that while pointer networks improve the log bilinear model's performance on a smaller dataset, it does not produce improved results on a larger dataset. I therefore present theories for these results and suggest further work that can be done to rectify shortcomings.