Unifying annotated discourse hierarchies to create a gold standard

DSpace/Manakin Repository

Unifying annotated discourse hierarchies to create a gold standard

Citable link to this page


Title: Unifying annotated discourse hierarchies to create a gold standard
Author: Carbone, Marco; Shieber, Stuart ORCID  0000-0002-7733-8195 ; Gal, Ya'akov Kobi

Note: Order does not necessarily reflect citation order of authors.

Citation: Marco Carbone, Kobi Gal, Stuart M. Shieber, and Barbara Grosz. Unifying annotated discourse hierarchies to create a gold standard. In Proceedings of the Fifth SIGdial Workshop on Discourse and Dialogue, Boston, MA, April 30-May 1 2004.
Full Text & Related Files:
Abstract: Human annotation of discourse corpora typically results in segmentation hierarchies that vary in their degree of agreement. This paper presents several techniques for unifying multiple discourse annotations into a single hierarchy, deemed a “gold standard ” — the segmentation that best captures the underlying linguistic structure of the discourse. It proposes and analyzes methods that consider the level of embeddedness of a segmentation as well as methods that do not. A corpus containing annotated hierarchical discourses, the Boston Directions Corpus, was used to evaluate the “goodness” of each technique, by comparing the similarity of the segmentation it derives to the original annotations in the corpus. Several metrics of similarity between hierarchical segmentations are computed: precision/recall of matching utterances, pairwise inter-reliability scores ( ¡), and non-crossing-brackets. A novel method for unification that minimizes conflicts among annotators outperforms methods that require consensus among a majority for the ¡ and recall metrics, while capturing much of the structure of the discourse. When higher recall is preferred, methods requiring a majority are preferable to those that demand full consensus among annotators.
Published Version: http://www.sigdial.org/workshops/workshop5/proceedings/pdf/carbone.pdf
Terms of Use: This article is made available under the terms and conditions applicable to Other Posted Material, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#LAA
Citable link to this page: http://nrs.harvard.edu/urn-3:HUL.InstRepos:2252611
Downloads of this work:

Show full Dublin Core record

This item appears in the following Collection(s)


Search DASH

Advanced Search