Unifying annotated discourse hierarchies to create a gold standard
View/ Open
Published Version
http://www.sigdial.org/workshops/workshop5/proceedings/pdf/carbone.pdfhttp://www.aclweb.org/anthology-new/W/W04/W04-2323.pdf
Metadata
Show full item recordCitation
Marco Carbone, Kobi Gal, Stuart M. Shieber, and Barbara Grosz. Unifying annotated discourse hierarchies to create a gold standard. In Proceedings of the Fifth SIGdial Workshop on Discourse and Dialogue, Boston, MA, April 30-May 1 2004.Abstract
Human annotation of discourse corpora typically results in segmentation hierarchies that vary in their degree of agreement. This paper presents several techniques for unifying multiple discourse annotations into a single hierarchy, deemed a “gold standard ” — the segmentation that best captures the underlying linguistic structure of the discourse. It proposes and analyzes methods that consider the level of embeddedness of a segmentation as well as methods that do not. A corpus containing annotated hierarchical discourses, the Boston Directions Corpus, was used to evaluate the “goodness” of each technique, by comparing the similarity of the segmentation it derives to the original annotations in the corpus. Several metrics of similarity between hierarchical segmentations are computed: precision/recall of matching utterances, pairwise inter-reliability scores ( ¡), and non-crossing-brackets. A novel method for unification that minimizes conflicts among annotators outperforms methods that require consensus among a majority for the ¡ and recall metrics, while capturing much of the structure of the discourse. When higher recall is preferred, methods requiring a majority are preferable to those that demand full consensus among annotators.Terms of Use
This article is made available under the terms and conditions applicable to Other Posted Material, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#LAACitable link to this page
http://nrs.harvard.edu/urn-3:HUL.InstRepos:2252611
Collections
- FAS Scholarly Articles [18292]
Contact administrator regarding this item (to report mistakes or request changes)