OUP user menu

Automated identification of patients with a diagnosis of binge eating disorder from narrative electronic health records

Brandon K Bellows, Joanne LaFleur, Aaron W C Kamauu, Thomas Ginter, Tyler B Forbush, Stephen Agbor, Dylan Supina, Paul Hodgkins, Scott L DuVall
DOI: http://dx.doi.org/10.1136/amiajnl-2013-001859 e163-e168 First published online: 1 February 2014

Abstract

Binge eating disorder (BED) does not have an International Classification of Diseases, 9th or 10th edition code, but is included under ‘eating disorder not otherwise specified’ (EDNOS). This historical cohort study identified patients with clinician-diagnosed BED from electronic health records (EHR) in the Department of Veterans Affairs between 2000 and 2011 using natural language processing (NLP) and compared their characteristics to patients identified by EDNOS diagnosis codes. NLP identified 1487 BED patients with classification accuracy of 91.8% and sensitivity of 96.2% compared to human review. After applying study inclusion criteria, 525 patients had NLP-identified BED only, 1354 had EDNOS only, and 68 had both BED and EDNOS. Patient characteristics were similar between the groups. This is the first study to use NLP as a method to identify BED patients from EHR data and will allow further epidemiological study of patients with BED in systems with adequate clinical notes.

Keywords
  • binge eating disorder
  • natural language processing
  • electronic health record
  • eating disorder

Background and significance

Binge eating disorder (BED) is characterized by regular, excessive consumption of food (occurring, on average, once per week for at least 3 months), without an associated inappropriate compensatory behavior such as purging, fasting, or engaging in compulsive exercise.1 The lifetime prevalence of BED in adults in the USA ranges from 2.0% to 3.5%,2 ,3 but may be underestimated because individuals with BED tend to conceal their illness.4

The advancement of electronic health records (EHR) to gather and store patient data58 has enabled researchers greater access to patient information.9 ,10 However, BED was not officially classified as an eating disorder until the release of the Diagnostic and Statistical Manual of Mental Disorders (DSM), fifth edition in 2013.1 Before this, BED was included in the DSM, fourth edition (IV) as an area for further research under eating disorder not otherwise specified (EDNOS).11 In addition, DSM diagnoses are not captured in structured data and the International Classification of Diseases, 9th and 10th edition (ICD-9 and ICD-10) billing codes only include disorders from the DSM-IV. Thus, BED is currently captured using the EDNOS ICD-9 code, which includes other eating disorders.12 ,13 Therefore, despite an increased ability of clinicians to screen for BED,1417 it remains difficult to study in large populations. Epidemiological research in BED has relied on time and resource-intensive prospective questionnaires and patient registries,3 ,4 ,1821 which do not allow the identification of patients with BED in other datasets.

Natural language processing (NLP) includes a set of methods developed to recognize distinctive word and phrase patterns from written text, in a manner similar to human chart review.10 ,2225 NLP has been used to identify patients with diseases that do not have diagnosis codes, or for which the diagnosis codes are poorly documented, and for test results absent in structured EHR data.24 ,26 ,27 The objective of this study was to identify patients with clinician-diagnosed BED from narrative clinic notes in the Department of Veterans Affairs (VA) EHR using NLP; then, compare characteristics (ie, demographics, comorbidities, and medication use) of patients identified to have only BED, only EDNOS, and both BED and EDNOS.

Materials and methods

Data acquisition and management

This historical cohort study used EHR data from the national population of US veterans who received care in the VA. Administrative and clinical information recorded in the EHR during patient–provider interactions within the VA was obtained, including patient demographics, diagnostic codes, vital signs, medications, and narrative clinical notes. Examining both diagnostic codes and clinical notes allowed for a comparison of how a BED diagnosis appears in the EHR (figure 1). This study was reviewed and approved by the University of Utah Institutional Review Board and the VA Salt Lake City Health Care System Research and Development Service.

Figure 1

Current method of diagnosing EDNOS from structured data compared to NLP method for identifying patients diagnosed with BED. BED, binge eating disorder; EDNOS, eating disorder not otherwise specified; NLP, natural language processing.

NLP development for BED patient identification

Initial term identification

To identify patients with clinician-diagnosed BED, narrative clinic notes from inpatient and outpatient encounters were selected from 1 January 2000 to 31 December 2011. A random set of 1000 of these notes was sampled from adult patients (≥18 years) with at least one diagnosis of EDNOS (ICD-9 307.50). Clinicians and informaticists reviewed this sample of notes for terms and phrases used by clinicians to describe a diagnosis of BED.

The initial set of terms and phrases found included different tenses of the words ‘eat’ or ‘eater’; preceded by descriptors such as ‘addictive’, ‘binge’, or ‘over’; or followed by ‘disorder’ or ‘episode’. ‘EDNOS’, ‘NOS’, and ‘NEC’—‘not elsewhere classified’ were also commonly found.

NLP development

The goal of the NLP algorithm was not to infer BED through descriptions of behavior, but to identify explicit physician-documented diagnoses (eg, medical record stated ‘the patient has BED’). A rule-based approach was used and was iteratively developed starting with the initial terms and phrases found in the EDNOS patient notes. The VA EHR includes many templates (eg, questionnaires) and informative documents (eg, patient handouts) that may be used by providers, but do not necessarily indicate a diagnosis. Therefore a customized algorithm, similar to ConText,28 was used to identify instances when a term or phrase was found in a clinical note indicating a diagnosis of BED was negated, hypothetical, historical, given to someone other than the patient, or included in informative text. This approach excluded references to family history, key terms used in other contexts (eg, alcoholic binge), and descriptions in informative text (eg, ‘symptoms of BED include …’). Windows of 10 words were created around each instance to help distinguish whether a clinician was affirming (‘yes’), ruling out (‘no’), or considering a BED diagnosis as part of a differential diagnosis (‘possible’). Each instance in a narrative clinical note was classified into one of these three diagnostic categories (figure 2). The NLP system did not use grammatical parsing and syntactic and semantic normalization were performed during rule development.

Figure 2

Diagram of the natural language processing algorithm.

NLP validation

The NLP system went through five iterations of rule development, in which results were manually validated. Patterns of the most commonly missed or misclassified concepts were identified, and the rules were adjusted accordingly. For each of the five iterations, clinical annotators manually reviewed 1000 randomly selected (using an automated randomization sequence in Microsoft SQL server) instances to compare their classification to the NLP system output. Validation centered on two criteria: first, that the term found was actually relevant to BED, and second, that the inference made by the tool with regard to the ‘yes,’ ‘no,’ or ‘possible’ classification was correct. Accuracy results were then tabulated and reported. Error analyses were performed with each iteration to determine the utility of each term in the full set of notes. Frequency analysis helped find commonalities occurring in the surrounding text in both correct and incorrect NLP system classifications. This process identified related terms along with misspellings, abbreviations, and synonyms of the concepts of interest. Terms with the least entropy, that is, terms with a large number of instances but a small percentage of which were relevant to BED diagnosis, were identified and removed. Through the validation process, the initial set of phrases was narrowed down until it only included variations of ‘BED’, including acronyms, abbreviations, and differences in word order (more information available in the supplementary data, available online only). After final validation, the BED NLP tool was then run on all notes associated with inpatient and outpatient encounters in the VA system from 1 January 2000 to 31 December 2011. In order to assess the sensitivity of the NLP tool, a random sample of 200 patient records containing any of the initial terms and phrases and 530 patient records of patients with an EDNOS diagnosis that were not identified and classified by NLP as having BED was manually reviewed.

Patient selection

Patients with at least one diagnosis of BED identified by NLP in a clinical note were classified as having BED (first diagnosis defined index date). EDNOS patients were identified by ICD-9 code (first diagnosis defined index date). Patients were included in the final cohort if they were aged 18 years or over, had 1 year or more of activity (eg, an office visit, prescription fill, hospital visit) before and after the index date, had a body mass index (BMI) value within ±60 days of the index date, and no diagnosis for other eating disorders (ie, bulimia nervosa (ICD-9 307.51), anorexia nervosa (307.1), pica (307.52), rumination disorder (307.53), and psychogenic vomiting (307.54)). Three groups were created for comparison of patient characteristics based on the method of identification: patients with only NLP-identified BED (BED only), patients with only EDNOS (EDNOS only), and patients with both BED and EDNOS (BED+EDNOS).

Statistical analysis

Descriptive statistics (frequencies and percentages for categorical variables; means and SD for continuous variables) were used to characterize patients in terms of demographics (age, sex, race, BMI), comorbidities (identified in 1-year post-index period by ICD-9 code), and medication use (identified in 1-year post-index period by prescription fill records). Pair-wise t tests and χ2 tests were used to compare characteristics between groups with a p value less than 0.05 considered significant.

Results

Approximately 10.9 million patients (comprising >1.7 billion clinical notes) had a medical encounter in the VA during the study period. Over 193 000 clinical notes contained instances of the initial terms and phrases. Manual review of 1000 instances from the NLP output revealed the NLP tool was able to classify BED diagnoses correctly in clinical notes in 82.4% of instances in the first iteration, which improved to 91.8% in the final iteration. In addition, in the final iteration, the NLP system correctly identified patients as having BED in 90.7% of instances, as not having BED in 96.6% of instances, and as having a possible BED diagnosis, including differential diagnosis, in 91.3% of instances (table 1). Review of patient records found the NLP tool to be 96.2% sensitive in identifying and correctly classifying patients with BED, 99.5% sensitive for patient records containing any of the initial terms and phrases and 94.7% sensitive for patients with EDNOS diagnosis codes (table 1). The NLP tool found 1487 unique patients with a confirmed BED diagnosis. After applying study eligibility criteria, there were 525 BED-only, 1354 EDNOS-only, and 68 BED+EDNOS patients (figure 3).

View this table:
Table 1

Measurement properties of the final iteration of the NLP tool

Accuracy*
Instances reviewedInstances annotator agreed with NLP classificationBED diagnosis
‘Yes’‘No’‘Possible’†
NLP-identifiedAnnotator agreedNLP-identifiedAnnotator agreedNLP-identifiedAnnotator agreed
NN%NN%NN%NN%
100091891.873166390.717717196.6928491.3
Sensitivity‡
 Patient records reviewedMissed BED diagnosesSensitivity
Sample selection categoryNN%
   Initial keyword list§200199.5
   EDNOS¶5302794.7
   Total7302896.2
  • *Used 1000 randomly selected instances of variants of the phrase ‘BED’.

  • †Includes differential diagnoses.

  • ‡Used 200 patient records containing any of the initial terms and phrases and 530 patient records of patients with an EDNOS diagnosis that were not identified and classified by NLP as having BED.

  • §Patient records were selected that contained one of the initial keywords or phrases (‘addictive eating’, ‘BED’, ‘binge eat’, ‘binge eater’, ‘binge eating’, ‘eating disorder’, ‘EDNOS’, ‘eating episode’, ‘over eat’, ‘over eater’, ‘over eating’).

  • ¶Patient records were selected that contained a diagnosis code for EDNOS (ICD-9 307.50).

  • BED, binge eating disorder; EDNOS, eating disorder not otherwise specified; ICD-9, International Classification of Diseases, 9th edition; NLP, natural language processing.

Figure 3

Attrition summary. AN, anorexia nervosa; BED, binge eating disorder; BN, bulimia nervosa; ED, eating disorder; EDNOS, eating disorder not otherwise specified; CD-9, International Classification of Diseases, 9th edition; NLP, natural language processing; VA, Department of Veterans Affairs.

Patient characteristics were similar between the groups (table 2). There were no significant differences between the BED+EDNOS group and the other groups with regard to demographics and only a few with regard to comorbidities and medication use. However, the BED-only and EDNOS-only groups were significantly different in terms of demographics as well as several comorbidities and medication classes.

View this table:
Table 2

Characteristics of patients with only NLP-identified binge eating disorder (BED only), those with only a diagnosis of eating disorder not otherwise specified (EDNOS only) and those with both BED and EDNOS (BED+EDNOS)

VariableBED+EDNOS (N=68)BED only (N=525)EDNOS only (N=1354)
N/mean%/SDN/meanN/meanp Value*N/mean%/SDp Value*p Value†
Age at index date (mean, SD), years48.510.948.710.30.8949.812.50.340.05
   18–2434.4101.90.16332.40.25<0.001
   25–34710.3407.616512.2
   35–441014.711722.321115.6
   45–542232.419837.742731.5
   55–642536.813826.339729.3
   65+11.5224.21218.9
Sex
   Male4566.238373.00.2485062.80.57<0.001
Race
   White5580.936769.90.11102475.60.650.01
   Black710.38816.817112.6
   Hispanic22.961.1292.1
   Other/unknown45.96412.21309.6
BMI at index date (mean, SD)38.711.140.39.70.2637.011.20.22<0.001
   <18.50010.20.60342.50.43<0.001
   18.5–24.957.4244.619414.3
   25.0–29.9913.2499.315711.6
   30.0–34.9913.28015.219614.5
   35.0–39.91826.511221.324117.8
   40.0–44.91014.711622.122116.3
   45.0–49.9811.85710.914310.6
   50.0–54.934.4499.3836.1
   ≥55.068.8377.0856.3
Post-index comorbidities
   Depression1725.010620.20.3643632.20.21<0.001
   Anxiety2029.46412.2<0.00116912.5<0.0010.86
   Hyperlipidemia3652.926550.50.7064047.30.360.21
   Hypertension4160.331660.20.9970952.40.200.002
   Diabetes2841.218835.80.3949236.30.420.83
   Heart disease811.88015.20.4519614.50.530.68
   Overweight11.5132.50.61322.40.630.89
   Obesity4464.733664.00.9179959.00.350.05
   Morbid obesity2029.416932.20.6436426.90.650.02
   Asthma913.2377.00.07997.30.070.84
   Sleep apnea1927.915329.10.8431623.30.380.009
   Osteoarthritis1826.512223.20.5633524.70.750.50
   Gallbladder disease11.5132.50.61131.00.680.01
   Cerebrovascular disease00.000.000.0
   Back pain/disorders1927.910920.80.1833024.40.510.10
   Non-alcoholic steatohepatitis00.081.50.31161.20.370.55
Post-index medication use
   Antidepressants5175.028854.90.00292668.40.25<0.001
   Antipsychotics1217.67514.30.4626519.60.700.008
   Anxiolytics2232.412123.00.0943332.00.95<0.001
   Antihypertensives4058.834665.90.2580259.20.950.008
   Anti-diabetics2435.317633.50.7745133.30.740.93
   Antihyperlipidemics3145.625147.80.7359243.70.760.11
   Weight loss agents68.8234.40.11664.90.150.65
   Opioids3247.119837.70.1454840.50.280.27
  • p Values in bold indicate significance.

  • *p Value for comparison to BED+EDNOS group.

  • †p Value for comparison to BED-only group.

  • BED, binge eating disorder; BMI, body mass index; EDNOS, eating disorder not otherwise specified; NLP, natural language processing.

Discussion

This study demonstrated the ability to apply NLP to narrative clinical notes within the EHR of the VA to identify patients with a diagnosis of BED with greater than 90% classification accuracy and sensitivity. In addition, this study showed that BED is not commonly coded using the EDNOS ICD-9 code in the VA population. Given the differences in the characteristics between the BED-only and EDNOS-only populations and the lack of overlap between the populations, this study highlights the need for a specific identifier of BED in structured data, such as an ICD-9 code. The lack of a specific BED diagnosis code limits epidemiological study and hinders retrospective analyses of BED. The use of NLP may help to provide useful insights into the BED population, when suitable physician notes are available, and may be used to identify other conditions that do not have a specific ICD-9 code.

The study was conducted exclusively within the VA system and, therefore, represents veterans with BED and EDNOS, but may not necessarily represent the general population of patients with these diseases. Other institutions may possibly use different language to denote a BED diagnosis that was not considered in this study. However, VA physicians often work in healthcare centers outside of the VA system, thus institutional bias pertaining to diagnostic diction may be minimal. In addition, clinician diagnosis of BED was identified using only variations of a single concept: ‘BED’. This limited lexicon and the set of terms and phrases that filtered the initial documents for processing could have potentially reduced the number of patients identified and may not have the same results in other institutions. However, the NLP tool benefitted from the frequent use of semistructured data and was able to identify and classify patients with BED with a high sensitivity using a limited lexicon. Despite these limitations, these methods have provided the only existing automated means by which to capture information on this population and enable a platform for future work.

Conclusion

As there is no ICD-9 diagnosis code for BED, before this study there was no applicable method of automating a specific search for patients diagnosed with BED. In this study patients with BED were identified in the VA system using NLP with greater than 90% accuracy and sensitivity. After inclusion criteria, only 68 patients identified by NLP also had a diagnosis of EDNOS. While there were few differences between the BED+EDNOS group and the other two groups, BED-only patients were younger, more obese, and more were men than the EDNOS-only patients. These differences, combined with the lack of overlap between BED and EDNOS, highlight the need for a specific identifier of BED in structured data. However, NLP provides a new method for better identifying patients diagnosed with BED from unstructured EHR data.

Contributors

BKB was responsible for the conception and design of the study, interpretation of data, drafting and revising the manuscript, and approving the final manuscript. SLD was responsible for the conception and design of the study, analysis and interpretation of data, drafting and revising the manuscript, and approving the final manuscript. JL and AWCK were responsible for the conception and design of the study, interpretation of data, revising the manuscript, and approving the final manuscript. TG and TBF were responsible for the analysis and interpretation of data, revising the manuscript, and approving the final manuscript. SA, DS, and PH were responsible for the conception of the study, revising the manuscript, and approving the final manuscript.

Funding

This work was supported by a grant from Shire Development Inc., of which DS and PH are employees. The work was also funded by Anolinx, of which AWCK is the owner and SA is an employee.

Competing interests

Outside the submitted work, JL has received funding for research from Amgen Inc., Merck & Co., Inc., Genentech LLC, and Anolinx LLC. SLD has received grant funding for research from Genentech LLC, F. Hoffmann-La Roche LTD, Merck & Co, Inc, Mylan Specialty LP, Anolinx LLC, and from Amgen Inc. The remaining authors have no other disclosures.

Ethics approval

This study was reviewed and approved by the University of Utah Institutional Review Board and the VA Salt Lake City Health Care System Research and Development Service.

Provenance and peer review

Not commissioned; externally peer reviewed.

Acknowledgements

The authors wish to thank Kristin Knippenberg for writing assistance; Jacob Crook for assistance with the statistical analysis; and Beniel Kevin Malohi, James Potter, Corinne Halls, Kristi Gregory, and Kelli Henricksen for coordination, annotation, and chart review.

References

View Abstract