Unifying annotated discourse hierarchies to create a gold standard
| Title: | Unifying annotated discourse hierarchies to create a gold standard |
| Author: |
Carbone, Marco; Shieber, Stuart; Gal, Ya'akov Kobi
Note: Order does not necessarily reflect citation order of authors. |
| Citation: | Marco Carbone, Kobi Gal, Stuart M. Shieber, and Barbara Grosz. Unifying annotated discourse hierarchies to create a gold standard. In Proceedings of the Fifth SIGdial Workshop on Discourse and Dialogue, Boston, MA, April 30-May 1 2004. |
| Full Text & Related Files: |
Shieber_UnifyingAnnotated.pdf (110.8Kb; PDF) 
|
| Abstract: | Human annotation of discourse corpora typically results in segmentation hierarchies that vary in their degree of agreement. This paper presents several techniques for unifying multiple discourse annotations into a single hierarchy, deemed a “gold standard ” — the segmentation that best captures the underlying linguistic structure of the discourse. It proposes and analyzes methods that consider the level of embeddedness of a segmentation as well as methods that do not. A corpus containing annotated hierarchical discourses, the Boston Directions Corpus, was used to evaluate the “goodness” of each technique, by comparing the similarity of the segmentation it derives to the original annotations in the corpus. Several metrics of similarity between hierarchical segmentations are computed: precision/recall of matching utterances, pairwise inter-reliability scores ( ¡), and non-crossing-brackets. A novel method for unification that minimizes conflicts among annotators outperforms methods that require consensus among a majority for the ¡ and recall metrics, while capturing much of the structure of the discourse. When higher recall is preferred, methods requiring a majority are preferable to those that demand full consensus among annotators. |
| Published Version: | http://www.sigdial.org/workshops/workshop5/proceedings/pdf/carbone.pdf |
| Terms of Use: | This article is made available under the terms and conditions applicable to Other Posted Material, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#LAA |
| Citable link to this page: | http://nrs.harvard.edu/urn-3:HUL.InstRepos:2252611 |
This item appears in the following Collection(s)
-
FAS Scholarly Articles [5137]
Peer reviewed scholarly articles from the Faculty of Arts and Sciences of Harvard University
Contact administrator regarding this item (to report mistakes or request changes)
