Subword tokenization for low resource, morphologically rich languages: A morphologically constrained tokenizer for Lithuanian

Abstract

Many Slavic and Baltic languages remain underrepresented in training data, resulting in disparities in language model performance and natural language processing (NLP) resources. I address these low-resource conditions by adapting subword tokenizer architecture to leverage the morphologically rich structure of Lithuanian, a particularly low-resource Baltic language. Language models learn the relative meaning of tokens formed by merging characters or subwords based on the frequency with which they appear adjacent in training data. I present the MorphRich tokenizer, which is constrained by Lithuanian morphology to produce tokens that preserve the boundaries of morphemes that densely encode grammatical information. By training a small decoder model, I compare MorphRich-equipped model performance against a baseline model with a frequency-only BPE tokenizer. Although the MorphRich tokenizer produces more tokens-per-sequence, morphologically constrained tokens improve the model’s bits-per-character performance and the model learns more language information per token. The MorphRich tokenizer produces an embedding matrix in which vectors more densely encode grammatical and semantic information. To examine the downstream impact of this, I demonstrate that the model more effectively learns case agreement from surrounding morphological context rather than relying on word position, which is particularly significant for Lithuanian as a free word order language. In building the MorphRich tokenizer, I also contribute NLP resources currently underdeveloped for Lithuanian, including a morphological analyzer, parts-of-speech tagger, and morpheme boundary detector. This thesis suggests that language-specific tokenizers may improve model performance for low-resource, morphologically rich languages.