Concise comparative summaries (CCS) of large text corpora with a human experiment

DSpace/Manakin Repository

Concise comparative summaries (CCS) of large text corpora with a human experiment

Citable link to this page


Title: Concise comparative summaries (CCS) of large text corpora with a human experiment
Author: Jia, Jinzhu; Miratrix, Luke Weisman; Yu, Bin; Gawalt, Brian; El Ghaoui, Laurent; Barnesmoore, Luke; Clavier, Sophie

Note: Order does not necessarily reflect citation order of authors.

Citation: Jia, Jinzhu, Luke Miratrix, Bin Yu, Brian Gawalt, Laurent El Ghaoui, Luke Barnesmoore, and Sophie Clavier. 2014. “Concise Comparative Summaries (CCS) of Large Text Corpora with a Human Experiment.” Ann. Appl. Stat. 8 (1) (March): 499–529. doi:10.1214/13-aoas698.
Full Text & Related Files:
Abstract: In this paper we propose a general framework for topic-specific summarization of large text corpora and illustrate how it can be used for the analysis of news databases. Our framework, concise comparative summarization (CCS), is built on sparse classification methods. CCS is a lightweight and flexible tool that offers a compromise between simple word frequency based methods currently in wide use and more heavyweight, model-intensive methods such as latent Dirichlet allocation (LDA). We argue that sparse methods have much to offer for text analysis and hope CCS opens the door for a new branch of research in this important field.

For a particular topic of interest (e.g., China or energy), CSS automatically labels documents as being either on- or off-topic (usually via keyword search), and then uses sparse classification methods to predict these labels with the high-dimensional counts of all the other words and phrases in the documents. The resulting small set of phrases found as predictive are then harvested as the summary.

To validate our tool, we, using news articles from the New York Times international section, designed and conducted a human survey to compare the different summarizers with human understanding. We demonstrate our approach with two case studies, a media analysis of the framing of “Egypt” in the New York Times throughout the Arab Spring and an informal comparison of the New York Times’ and Wall Street Journal’s coverage of “energy.” Overall, we find that the Lasso with L2 normalization can be effectively and usefully used to summarize large corpora, regardless of document size.
Published Version: doi:10.1214/13-aoas698
Other Sources:
Terms of Use: This article is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at
Citable link to this page:
Downloads of this work:

Show full Dublin Core record

This item appears in the following Collection(s)


Search DASH

Advanced Search