Automated Agile User Story Quality Assessment with Natural Language Processing

Abstract

In this thesis, we explored different methods for assessing the quality of Agile User Stories, by using Natural Language Processing (NLP) and Machine Learning to measure the User Stories' quality. The study's primary question was whether the state-of-the-art Bidirectional Encoder Representations from Transformers (BERT) model can evaluate User Stories' quality. We initiated the operationalization of AQUSA and StoryLine, subsequently training BERT utilizing the data generated by both, followed by a comparative analysis of the results. BERT when trained with StoryLine shows a commendable detection rate of 88.50%, which is not as high as StoryLine itself but substantially higher than AQUSA. BERT when trained with AQUSA has a detection rate slightly better than AQUSA itself at 71.02%, but it's still much lower than BERT trained with StoryLine or StoryLine on its own. We concluded if one is looking for consistency in defect detection, BERT trained with AQUSA is recommended as it aligns almost perfectly with AQUSA's decisions. The significant difference between AQUSA and StoryLine suggests that combining insights from both models could provide a more holistic view of defect detection, potentially catching more nuances than using either model alone.