OUP user menu

★ Research Paper ★

The Relationship between Electronic Health Record Use and Quality of Care over Time

Li Zhou , Christine S. Soran , Chelsea A. Jenter , Lynn A. Volk , E. John Orav , David W. Bates , Steven R. Simon
DOI: http://dx.doi.org/10.1197/jamia.M3128 457-464 First published online: 1 July 2009


Objective: Electronic health records (EHRs) have the potential to advance the quality of care, but studies have shown mixed results. The authors sought to examine the extent of EHR usage and how the quality of care delivered in ambulatory care practices varied according to duration of EHR availability.

Methods: The study linked two data sources: a statewide survey of physicians' adoption and use of EHR and claims data reflecting quality of care as indicated by physicians' performance on widely used quality measures. Using four years of measurement, we combined 18 quality measures into 6 clinical condition categories. While the survey of physicians was cross-sectional, respondents indicated the year in which they adopted EHR. In an analysis accounting for duration of EHR use, we examined the relationship between EHR adoption and quality of care.

Results: The percent of physicians reporting adoption of EHR and availability of EHR core functions more than doubled between 2000 and 2005. Among EHR users in 2005, the average duration of EHR use was 4.8 years. For all 6 clinical conditions, there was no difference in performance between EHR users and non-users. In addition, for these 6 clinical conditions, there was no consistent pattern between length of time using an EHR and physicians performance on quality measures in both bivariate and multivariate analyses.

Conclusions: In this cross-sectional study, we found no association between duration of using an EHR and performance with respect to quality of care, although power was limited. Intensifying the use of key EHR features, such as clinical decision support, may be needed to realize quality improvement from EHRs. Future studies should examine the relationship between the extent to which physicians use key EHR functions and their performance on quality measures over time.


Electronic health records (EHRs) have the potential to advance the quality of health care by providing timely access to patients' health information, tracking patients over time to ensure that they receive guideline-recommended care, and offering decision-support mechanisms to reduce medical errors.16 However, cross-sectional studies have failed to show a direct correlation between having an EHR and high levels of quality of care,7,8 suggesting that simply having an EHR may not be sufficient to improve quality and safety of health care. Nonetheless, randomized controlled trials demonstrate clearly that quality improvement occurs when specific decision support is in place.9,10 Other factors, such as presence of order entry and better training and implementation of EHR systems, are likely also needed to achieve higher levels of quality and safety.11

In addition, it is conceivable that quality and safety benefits of EHR adoption and use may be time-dependent, possibly taking some years after implementation to occur, as users become more facile with the applications.12,13 As with other new technologies,14 there may be considerable lag in comprehensive usage and consequent delay in realizing the benefits attributable to EHR adoption. Therefore, we undertook the present study to examine how the quality of care delivered in ambulatory care practices varied according to duration of EHR adoption and usage.


The study design involved two data sources: (1) a statewide survey of physicians' adoption and use of EHR and (2) statewide data on physicians' quality of care as indicated by their performance on widely used quality measures. These two sets of data were integrated and linked for each included physician. We examined the relationship between EHR adoption and quality of care in two ways. We first examined the cross-sectional relationship between having an EHR and concurrent indicators of quality of care. Then, we carried out a longitudinal analysis to study the trend of EHR adoption and usage as well as to examine the association between the duration of EHR usage and quality of care.

Statewide Survey of Physicians' Use of Electronic Health Records

In 2005, we surveyed a representative sample of physicians in Massachusetts regarding adoption and use of EHRs. The methods for sampling, questionnaire development, survey administration and data collection have been described elsewhere.11 The following presents a brief summary of the methods. The survey and research protocol were approved by the Human Studies Committee of Partners Healthcare.


Of 6,174 active medical and surgical practices in Massachusetts, we drew a stratified random sample of 1977 practices and then randomly selected one physician per practice. After excluding ineligible practices and physicians, the remaining sample included 1884 physicians, of whom 1,345 (71%) responded to the survey. We further excluded respondents who indicated that they do not see any outpatients or who did not complete the principal questions on EHR prevalence, resulting in 1,181 respondents for subsequent analysis. Demographic characteristics of respondents and non-respondents were similar.11

EHR Adoption and Use

An eight-page questionnaire collected demographic and practice information, as well as measures of health information technology (HIT) adoption and usage. For this analysis, we used the reported date of EHR adoption, use of EHR features, demographics and practice information. Respondents indicated how long they had been associated with their main practice and if their main practice had an EHR. If a practice was currently using an EHR, respondents indicated when their practice first began using it and noted which EHR features were available and, if available, the extent to which they used each feature (most or all of the time; some of the time; do not use). The EHR features are listed in Table 1. Because we hypothesized that certain essential EHR functions would be more likely than others to be associated with quality of care, we took the following approach to define these key features. Starting with the Institute of Medicine's eight core EHR functions,15 a government advisory panel identified four functions essential to a functioning EHR16: “health information and data”, “result management”, “order entry management” and “decision support”; we added “electronic communication and connectivity” and mapped use of the ten features included in our survey to these five core EHR functions, as shown in Table 1.

View this table:
Table 1

Core EHR Functions and Associated Features that were Studied in the Survey

Core EHR Functions*Features in Statewide Survey
Health information and data*Electronic visit notes
   (access to key information, such as patients' diagnoses, allergies, and medications)Medication list
Problem list
Result management*Laboratory test results viewing
   (ability to manage results of all types electronically)Radiology test result viewing
Order entry and management*Laboratory order entry
   (ability to enter and store orders for prescriptions, tests, and other services into a computer)Radiology order entry
Electronic transmission or faxing of prescriptions
Decision Support*Reminders for care activities (e.g., overdue health maintenance)
   (computer reminders, alerts, prompts and computerized decision support systems to improve prevention, diagnosis and management of patient disease)
Electronic communication and connectivity*E-referrals or clinical messages
   (efficient, secure, and readily accessible electronic communication between providers and patients)(e.g., via e-mail) between providers
  • EHR = electronic health record.

  • * Health information and data, result management, order entry and management, and decision support were defined as core EHR functions by an Institute of Medicine Panel.16 The authors added electronic communication and connectivity as an additional core function.

In addition, to assess financial considerations, respondents were asked to indicate whether their practice's income or their personal earnings were eligible for incentive payments for quality of care, patient satisfaction, adoption of HIT, or actual use of HIT.

Statewide Data on Physicians' Quality of Care

Data Source and Measures

The quality of care measures in this study were based on the National Committee for Quality Assurance (NCQA)—Healthcare Effectiveness Data and Information Set (HEDIS®). The Massachusetts Health Quality Partners (MHQP) has aggregated HEDIS® data for commercially insured members from five major health plans in Massachusetts, accounting for approximately 75% of the state's population, and provided annual physician-level data for survey respondents for 2001–2004. After removing specialists and other survey respondents who did not have any HEDIS® data, the resulting analytic sample included 506 physicians (42.8% of the survey respondents) with at least 1 year of HEDIS® data, of whom 445 (87.9%) had 4 years of data. From a set of 18 available quality measures, we aggregated measures based on six previously defined clinical categories of quality,17 as shown in Table 2.

View this table:
Table 2

HEDIS® Quality Measures Aggregated into Six Clinical Categories*

Clinical CategoryMeasures
Asthma careAppropriate asthma medications for children
Appropriate asthma medications for adults ages 18–56 yrs
Behavioral and mental healthEffective acute phase treatment
Effective continuation phase treatment
Optimal practitioner contacts during acute phase
Cancer screeningBreast cancer screening
Cervical cancer screening
Diabetes careLDL-C screening
LDL-C controlled (less than 130 mg/dL)
Eye examinations
HbA1c testing
Poor HbA1c control
Monitoring diabetic nephropathy
Well child and adolescent visitWell-child visits first 15 mo of life
Well-child visits ages 3–6
Well care visits for adolescents
Women's healthChlamydia screening in women ages 16–20
Chlamydia screening in women ages 21–25
  • HEDIS = Healthcare Effectiveness Data and Information Set.

  • * Not all physicians were eligible for each clinical category, as inclusion depended on having at least five eligible patients for each HEDIS® measure group. The clinical categories were adopted from prior study.17

  • Only applied to physicians who practiced in pediatrics and family medicine.

Definitions of Performance on Quality Measures

Each physician's performance on individual HEDIS® measures was defined as the proportion of patients receiving the required care among the eligible patients for that measure. We created an annual score for each physician for each of the six clinical categories for which they were eligible by summing the numerators for each of the component HEDIS® measures and dividing by the sum of the denominators. For example, considering the cancer screening category, if a physician had 15 of 20 eligible patients completing mammography and 22 of 25 patients completing cervical cancer screening, the score for that category would be (15 + 22)/(20 + 25) = 0.82, or 82%. For each category we excluded physicians who had less than 5 eligible patients, i.e., the sum of the denominators was less than 5, because estimates based on such small samples may not be statistically valid.

Data Analysis

All analyses were performed with SAS statistical software (version 9.1; SAS Institute, Inc, Cary, NC, United States). All p values were 2-tailed, and p < 0.05 was considered statistically significant. We determined the proportion of physicians whose main practices had EHR available and the characteristics of individual physicians and their practices. We further compared these characteristics between physicians with and without EHRs using two-tailed χ2 tests and t tests.

If a physician indicated in the 2005 survey that a feature was available in his or her EHR system, we assumed that the feature had been available since the time when the practice first began using an EHR. The same assumption was also applied to the extent of usage of the EHR feature. Based on these assumptions, we estimated EHR adoption and availability and use of EHR core functions by year.

We compared the HEDIS® quality measures of respondents to the length of time using an EHR. We calculated each physician's duration of EHR use based on responses to how long the physician's practice had been using an EHR and how long the physician had been working in that practice. We placed physicians into the following categories: No EHR use; < 1 year; 1 year; 2 years; 3 years; 4 years; 5 years; 6–9 years; and > = 10 years of EHR use. Generalized estimating equations (GEE) regression models were used to produce adjusted mean HEDIS® rates for each of these duration categories, along with 95% confidence intervals. These analyses were adjusted for available characteristics of physicians and their practices, financial considerations regarding EHR usage and quality of care, and the year in which the HEDIS® measures were collected. The adjusted means are interpreted as the mean HEDIS® rate within each duration category, for a physician with average values for the covariates in the GEE models. In addition, to account for the variability of eligible patients per physician, we created a weight for each GEE model defined as the total number of eligible patients for each physician divided by the average number of eligible patients for all physicians.

GEE regression models were also used to assess the association between the duration of EHR usage with physicians' quality measures, and whether there were differences in quality measures between EHR users and non-users. These analyses are adjusted for the same covariates and weights as above.


Characteristics of Survey Respondents

We compared characteristics of survey respondents who matched to HEDIS data and were thus included in the analyses with those who did not match and were thus excluded from the analysis. Physicians who had HEDIS® data were younger (47.2 versus 50.0 yrs; p < 0.05), graduated from medical school more recently (19.6 versus 22.6 yrs; p < 0.05), practiced in groups with fewer physicians (3.7 v. 4.8 physicians per practice; p < 0.01), and had more patient visits in a typical week (91.1 v. 73.6 patients; p < 0.01). In addition, physicians with HEDIS® data were more likely to be female (38.3 versus 26.5%; p < 0.01) and to be in non-hospital-based practice settings (85.5 versus 67.7%; p < 0.01), compared with physicians who did not have HEDIS® data and were thus excluded.

The physicians having HEDIS® data were stratified by EHR adoption status (Table 3). Among these 506 physicians, 164 (32.4%) physicians reported having an EHR in their main practice. Physicians using an EHR tended to be younger (45.4 versus 49.3 yrs; p < 0.001) and graduated from medical school more recently (17.6 versus 21.8 yrs; p < 0.001). Physicians practicing in an urban setting and in groups with more physicians were more likely to have an EHR.

View this table:
Table 3

Characteristics of Respondents Who Had and Did Not Have EHR in Their Practice*

CharacteristicAll Respondents (N = 506)Has EHR in Practice?
Yes (N = 164)No (N = 342)p Value
Physician characteristics
   Age, years< 0.001
      ≤ 4026.535.422.2
      ≥ 6110.14.912.6
   Race or ethnic group0.16
   Time since graduation from medical school, years< 0.001
      < 1010.511.010.2
      ≥ 3018.88.523.7
      Internal medicine42.145.740.4
      Family practice25.325.025.4
Practice characteristics
   Practice type0.52
      Primary care69.667.770.5
   Clinical setting0.12
   Practice size (# of Physicians)< 0.001
      Small (1–2)
      Medium (3–5)32.632.932.5
      Large (≥ 6)19.436.013.7
   Practice ownership< 0.001
   Residents or students present within the past year< 0.001
   Number of outpatient visits/week0.66
      > 2004.73.75.3
  • EHR = electronic health record.

  • * Comparison of the characteristics between respondents who had EHR in their practices and who did not have an EHR was done using of two-tailed chi-squares tests and t tests. Percentages may not total 100 because of missing data.

EHR Adoption and Use of EHR Functions

EHR Adoption

Among the 164 EHR adopters, 137 physicians provided the year in which their practice first began using an EHR. By 2005, the average duration of using EHR in this study population was 4.8 years. To estimate the cumulative percentage of practices that had adopted EHR by year, we assumed that the 27 EHR users who indicated they had an EHR, but who did not report an adoption year, were distributed by year proportionate to those who did report the adoption year. Actual reported and estimated years of adoption and cumulative adoption by year are shown in Table 4. Only physicians who reported an actual adoption year were included in analyses examining the relationship between duration of EHR use and quality of care.

View this table:
Table 4

Estimated EHR Use by Year

Year of Adoption(A) Number of Respondents Reporting an Adoption Year(B) Estimated Number of Respondents Not Reporting an Adoption Year*(C) Estimated Accumulated Total Number of EHR Adopters(D) Estimated Accumulated Percentage of EHR Adoption (%)
before 1990s4151.0
  • EHR = electronic health record.

  • * Numbers in column B were calculated using equation “27*A / 137” where 27 is the total number of EHR users who did not report an adoption year, “A” represents the number of respondents reporting an adoption year, and 137 is the total number of physicians who reported an adoption year.

  • Numbers in column D were calculated using numbers in column C divided by 506, the total number of included respondents in this study.

Availability and Use of EHR Core Functions

Availability and use of IOM-defined core EHR functions increased over time from 2000 to 2005 (Fig 1). For each function assessed, the percent of physicians reporting that the function was available more than doubled between 2000 and 2005. Appendix 1 (available as an online data supplement at www.jamia.org) shows use and availability of ten EHR features.

Figure 1

Among Physicians with EHRs, Availability and Usage of EHR Core Functions over Time. The length of each bar indicates the percentage of physicians who have each function available in their system. The component segments of each bar reflect the percentage of physicians using each function most or all of the time, some of the time, or not at all.

Quality Performance and EHR Adoption

EHR Users vs. Non-users

We compared quality performance between EHR users and non-users regardless of when their EHRs were implemented. For all 6 clinical conditions, there was no difference in performance between EHR users and non-users. There was a non-significant trend suggesting better performance on cancer screening among EHR users (coefficient = 0.011; 95% CI: −0.001, 0.024; p = 0.08).

Longitudinal Analysis

The trend of physicians' performance on each HEDIS® measure group is presented in Fig 2A–F. For all six clinical conditions, there was no consistent pattern relating EHR adoption to physicians' performance over time. The adjusted average performance and trend for cancer screening and diabetes care showed slight variability in relation to duration of EHR use. Performance on asthma only visibly increased for EHR duration of 10 years or more. For the well-child and adolescent visit measure, only physicians who practiced in pediatrics and family medicine were included, none of whom had used the EHR for 10 years or more. Performance on behavioral and mental health and women's health appeared to fluctuate over time, with no consistent pattern observed across measure categories.

Figure 2

A–F. display the adjusted averages of HEDIS® measure rates by the duration of EHR use for each of the six clinical conditions. The X axis indicates the years of using an EHR. “0” indicates the year of beginning implementing an EHR. Multivariate regression models shows there was no association between duration of using an EHR and physicians' performance for each group of HEDIS® quality measures.

Figure 2A. Cancer Screening.

Figure 2B. Diabetes Care.

Figure 2C. Asthma Care.

Figure 2D. Well Child and Adolescent Visit (Only included respondents who practice in pediatrics and family medicine).

Figure 2E. Behavioral and Mental Health.

Figure 2F. Women's Health.

Multivariate GEE regression models showed that there was no association between duration of using an EHR and physicians' performance for each group of HEDIS® quality measures.

Financial Considerations Regarding EHR Usage and Quality of Care

Physicians who indicated that the types of electronic systems they have and the amount they use the electronic information systems contributes to their practice's income were more likely to have an EHR (have: 37.7 versus 28.6%; p < 0.05 and use: 40.2 versus 28.7%; p < 0.05). Having an EHR was not related to physicians' reported financial incentives for patient satisfaction or clinical quality.


Studies have emphasized the adoption of EHRs,6,1820 and a few studies have examined the relationship between EHR adoption and quality of care.7,8 However, these studies have mostly been cross-sectional, providing snapshots of frequency and characteristics of the correlation at a particular point in time. In contrast, this study provides a longitudinal view of the relationship between EHRs and health care quality, taking into account the changes in correlates over time.

Gartner14 suggests in the “Hype Cycle” that the impact of new technologies can be characterized by a peak in benefits from the new technology due to hype or excitement followed by a period of decreased benefits and disappointment and an eventual plateau in outcomes. Such a pattern was not shown for EHRs' impact on quality of care using available datasets in our current study. There is no apparent or consistent pattern for physicians' performance over the time of using an EHR. We found no evidence that quality of care improved with increasing duration of EHR usage. Our findings are consistent with a previous national study by Linder, et al7 in that, as implemented, EHR use was not associated with better quality ambulatory care. Taken together, these results suggest that simply implementing EHRs is unlikely to result in improved quality. Other strategies, such as paying more for higher quality care and ensuring that physicians are using EHRs to their full capacity through education and workflow transformation, may be essential.

Several studies have demonstrated that decision support delivered through electronic records can improve quality,2129 especially for specific domains such as preventive care and care of some chronic conditions like diabetes.9,10,30 However, the level of improvement is often modest, and large gaps often persist. Furthermore, some studies have failed to show an effect of computerized clinical decision support on quality of care, especially for complex conditions.3134 Our study showed that the usage of decision support among EHR users was quite low, only 23.5% in 2005, compared to its availability, which was 65.0% among EHR adopters. To improve quality of care, it is becoming increasingly clear that implementing an electronic record is not sufficient and this tool needs to be coupled with other system supports such as registry functions, and care delivery transformations such as team-oriented approaches. In the UK, with the combination of near-universal EHR implementation, decision support, and substantial pay-for-performance, very high levels of quality performance have been achieved, although it is debatable which of these factors has been most important.3537

In the United States, a great deal of emphasis has focused on the adoption of EHRs, with the implicit assumption that improved quality will result. According to our estimates, the EHR adoption rate in Massachusetts has increased relatively rapidly in the last two decades. It was less than 1% before 1990s, around 12% in 2000, and then doubled in the succeeding five years, from 16% in 2001 to 32% in 2005. To achieve the national goal of universal EHR adoption by 2014,38 the 2005 EHR adoption rate will need to triple in less than a decade, which represents a challenge for health care policy.

For the use of key EHR functions, our study shows that the usage of health information and results management had grown at the same pace as their availability. In contrast, the usage of decision support and communication were relatively low, as compared with the fast growth of their availability. Among physicians who had EHRs in their main practice, only about one-third used decision support and around half used communication features most of the time. Taken together, these data suggest that stronger incentives or more extensive programs to support physician office transformation may be needed.

Beyond decision support, other factors may be important with respect to improving quality. In particular, the presence of some specific features and functionalities in an EHR may have an important impact on quality. For example, the long-term users might have relatively mature EHRs and well-designed clinical decision support and order entry management, and they might be more willing to use these features. In contrast, short-term EHR users might only have rudimentary EHRs with limited functionalities. They may not use these functions even if they are available within their EHR. Our survey did not record the length of time using specific EHR features, which prevented us from being able to determine the associations between the duration of using a specific EHR feature and quality of care. This analysis should be considered in future studies.

In addition, the sample sizes for long-time users were relatively small. For example, the EHR adoption rate was low in 1995, around 2.8% according to our study; therefore, in 2004 we only had 9 physicians who used an EHR greater than or equal to 10 years. Although we applied 4 years of repeated HEDIS® measures, we were unable to interpret and report the trend for quality measures, which had wide variations, thereby limiting our ability to detect differences which might become apparent with a larger sample.

Several additional limitations of this study should be considered. First, as in any observational study, unmeasured confounding factors may have obscured true associations; our models included all available covariates to control for confounding to the greatest extent possible. Secondly, we used HEDIS® rates to measure physicians' quality performance. Most HEDIS® measures examine the provision of preventive care and care for specific diseases. Although HEDIS® measures have been widely used by researchers and other healthcare related entities, they are derived from claims data. Actual clinical data may provide a more accurate representation of the quality of physician care. Moreover, HEDIS® indicators may not be adequately sensitive to detect improvements in quality of care attributable to the adoption and use of EHR. We also note that only physicians with adequate numbers of patients for each HEDIS® category were included in this study; excluding physicians with smaller-volume practices may have introduced some potential bias. It is conceivable that for these physicians, EHRs could be particularly useful in improving their adherence to quality measures through certain EHR features, such as reminders; this is a hypothesis that should be tested. Controlling for practice size, clinical volume, and whether the practice was hospital-based may have ameliorated some of this potential bias.

A third potential limitation is that EHR adoption and usage were self-reported by physicians, and social desirability bias may have led physicians to overestimate actual usage. Two lines of reasoning support the validity of self-reported data in this context. A nationally representative survey6 using similar self-reported measures found similar levels of EHR usage as in our statewide survey.39 Furthermore, our follow-up statewide survey40 demonstrated stability of usage rates of some EHR functions and considerably increased usage rates of others, suggesting that self-reported measures may adequately detect changes of usage over time. Nonetheless, future studies should consider actual measurement of EHR usage. Finally, this survey was conducted in a single state, so the generalization of our findings to the rest of the United States may be limited.


In summary, our results show that there was no association between duration of using an EHR and quality of ambulatory care, and in general, EHR use was not associated with improved quality of care. Health care policies should consider strategies to increase the efficient use of clinical decision support and other possible tools to improve quality of care. Future studies may be needed to measure the relationships between the quality of care and EHR use over time.


View Abstract