Pradhan, SameerElhadad, NoémieSouth, Brett RMartinez, DavidChristensen, LeeVogel, AmySuominen, HannaChapman, Wendy WSavova, Guergana2016-02-012014Pradhan, Sameer, Noémie Elhadad, Brett R South, David Martinez, Lee Christensen, Amy Vogel, Hanna Suominen, Wendy W Chapman, and Guergana Savova. 2014. “Evaluating the state of the art in disorder recognition and normalization of the clinical narrative.” Journal of the American Medical Informatics Association : JAMIA 22 (1): 143-154. doi:10.1136/amiajnl-2013-002544. http://dx.doi.org/10.1136/amiajnl-2013-002544.1067-5027http://nrs.harvard.edu/urn-3:HUL.InstRepos:24983942Objective: The ShARe/CLEF eHealth 2013 Evaluation Lab Task 1 was organized to evaluate the state of the art on the clinical text in (i) disorder mention identification/recognition based on Unified Medical Language System (UMLS) definition (Task 1a) and (ii) disorder mention normalization to an ontology (Task 1b). Such a community evaluation has not been previously executed. Task 1a included a total of 22 system submissions, and Task 1b included 17. Most of the systems employed a combination of rules and machine learners. Materials and methods We used a subset of the Shared Annotated Resources (ShARe) corpus of annotated clinical text—199 clinical notes for training and 99 for testing (roughly 180 K words in total). We provided the community with the annotated gold standard training documents to build systems to identify and normalize disorder mentions. The systems were tested on a held-out gold standard test set to measure their performance. Results: For Task 1a, the best-performing system achieved an F1 score of 0.75 (0.80 precision; 0.71 recall). For Task 1b, another system performed best with an accuracy of 0.59. Discussion Most of the participating systems used a hybrid approach by supplementing machine-learning algorithms with features generated by rules and gazetteers created from the training data and from external resources. Conclusions: The task of disorder normalization is more challenging than that of identification. The ShARe corpus is available to the community as a reference standard for future studies.en-USNatural Language ProcessingDisorder IdentifciationNamed Entity RecognitionInformation ExtractionWord Sense DisambiguationClinical NotesJournal Article2016-02-0110.1136/amiajnl-2013-002544