OUP user menu

★ Research Paper ★

Take Note(s): Differential EHR Satisfaction with Two Implementations under One Roof

Ryan T. O'Connell MD, Christine Cho MD, MPH, Nidhi Shah MD, Karen Brown MD, Richard N. Shiffman MD, MCIS
DOI: http://dx.doi.org/10.1197/jamia.M1409 43-49 First published online: 1 January 2004

Abstract

Objective: The aim of this study was to rigorously evaluate perceived differences in satisfaction with an electronic health record (EHR) between residents of two medical specialties who share the same health record, practice location, administration, and information technology support.

Design: A cross-sectional survey was used comparing user satisfaction between pediatrics residents and internal medicine residents in an academic practice.

Measurements: The survey was designed to measure baseline demographic characteristics, attitudes toward computers, general satisfaction with an EHR, and perceived practicality of use, variation from familiar practice, organizational support, and impact on delivery of care.

Results: Medicine subjects were similar to pediatrics subjects in baseline demographic characteristics. Satisfaction with the EHR implementation was very high for both sets of subjects, but internal medicine residents were significantly less likely to be satisfied with the EHR implementation (relative risk [RR] = 0.84, 95% confidence interval [CI] = 0.73–0.98) and considerably less likely to believe that their colleagues were satisfied with it (RR = 0.56, 95% CI = 0.41–0.77). The only surveyed characteristic independently predicting satisfaction was medical specialty (p = 0.04). Medicine subjects were less likely to believe template-based documentation improved their efficiency (RR = 0.64, 95% CI = 0.46–0.88). They were significantly more likely to believe the system had been designed to improve billing (RR = 1.50, 95% CI = 1.05–2.04) and not to improve patient care (RR = 0.61, 95% CI = 0.44–0.85).

Conclusion: The authors found a difference in satisfaction between internal medicine and pediatrics users of an EHR. Although many potential factors that influence satisfaction were similar between subjects in the two specialties, differences in previous experience may have influenced the results. Medicine residents had more previous experience with a different EHR implementation, which they may have perceived as superior to the one involved in this study. Pediatric residents had more previous experience with structured data entry prior to EHR implementation and more preventive care patient encounters for which structured data entry may be well suited. Since successful implementations generally require satisfied users, understanding what factors affect satisfaction can improve chances of a system's success.

The history of electronic health record (EHR) implementations has been marked by enduring successes and grand failures. Whereas implementations such as the Regenstrief Medical Record System have been used effectively for many years,1 other implementations have been expensive failures.2,3 Along with cost, barriers such as resistance by physicians, concerns about security, and risk of investing in a product that may not be supported in the long term4 all contribute to the fact that the majority of ambulatory practices in the United States have not implemented EHRs.57

In spite of electronic records' presently low penetration into the ambulatory care environment, their potential benefits to patients in terms of quality and safety cannot be ignored. It has been proposed that integrating electronic records into ambulatory settings will become the “standard of care.”4,8 As ambulatory practices across the country implement new systems, participants will be eager to optimize their chances of successful implementation.

Physician satisfaction is an important component of the success of an electronic implementation.9,10 Mazzoleni et al.11 describe user satisfaction as “essential to the survival” of a system. Implementations that have failed2 or have been plagued with difficulty3,12 have often been those with which physician–users are dissatisfied.

Satisfaction with electronic health record systems is multifactorial. A number of studies have identified factors contributing to or predicting user satisfaction with new electronic implementations.3,1316 Among factors that may affect user satisfaction with computer systems are gender,1719 age,11,17,20,21 and computer sophistication or familiarity with technology.11,13,14,22 For each of these factors, controversies exist about the degree and direction of its influence on satisfaction.

Other factors related to satisfaction pertain to the users' perceptions of the implementation. Satisfaction with a system is related to how useful it is perceived to be,11 and systems that are hard to use often yield dissatisfied users.2 A strong correlation was observed in one implementation between the ability to perform work efficiently and user satisfaction.10 Sittig et al.23 found that physician satisfaction correlated best with users' “ability to use the system to carry out their assigned tasks.”

Commitment of clinical leadership is another critical ingredient in a successful electronic implementation.24 Lack of support from clinical or administrative leadership has led to dissatisfaction with implementations of computer systems,12,2527 while support by implementation staff15,25,26 and adequate training22,26 are factors associated with successful implementations.

Newly deployed EHRs can affect delivery of care in both positive and negative ways. Although EHRs may be expected to provide improved access to medical information, process improvement,28 and improved quality of care, some electronic implementations have been shown to negatively affect workflow.15,27 In addition, some physicians worry about the way in which computers will affect their role as the care provider29 and have expressed concerns that using a computer system negatively affects the interaction between doctor and patient.14,22,30

A natural experiment to explore user satisfaction was created when Yale–New Haven Hospital's Primary Care Center (PCC) implemented the same EHR in clinics of two different medical specialties. Anecdotal reports suggested that pediatrics residents as a group were generally quite satisfied with the implementation and that medicine residents as a group were generally dissatisfied. This differential satisfaction occurred in the practices located literally at two ends of a hallway, using the same installed hardware, the same administrative support, and the same implementation staff.

The purpose of this work was (1) to determine if a difference in satisfaction truly existed between groups defined by specialty, (2) to determine which constitutive factors of the EHR users were predictive of satisfaction, and (3) to quantify the relative importance of each of several perceptions that could affect user satisfaction.

Methods

Setting

Yale–New Haven Hospital's Primary Care Center (PCC) is an ambulatory clinic that primarily serves an inner-city population. Within the PCC, the adult medicine clinic is responsible for 16,500 visits annually, and the pediatrics clinic accommodates 20,000 visits annually. All internal medicine and pediatrics residents provide longitudinal care for a panel of patients on an ongoing basis in the PCC under the supervision of faculty members.

Subjects

All 41 internal medicine residents and all 45 pediatrics residents who have weekly, half-day continuity clinics in the PCC were invited to participate in a questionnaire-based survey. Surveys were distributed to and completed by all participants within the month of November 2002. Surveys were completed and returned anonymously. A textbook lottery provided an incentive to return surveys. Yale University's Human Investigation Committee approved the research plan.

EHR Implementation

The Logician ambulatory care information system (GE Medical Systems Information Technology, Hillsboro, OR) was deployed in the pediatrics clinic in December 2000 and in the adult medicine clinic in September 2001. Its user interface can be customized using a screen editor and proprietary expression language.

For the purposes of this investigation, we have chosen to use the Institute of Medicine (IOM) definition of an Electronic Health Record.31 The IOM identifies eight core functions of an EHR System, which include health information and data, results management, order entry/management, decision support, electronic communication and connectivity, patient support, administrative processes, and reporting and population health management.

As implemented in the PCC, the electronic record is used principally for documentation, and all clinical documentation currently is electronic. All notes are typed; there is no dictation. The system also serves to electronically generate orders such as radiology and laboratory orders as well as creation of prescriptions. Finally, the system serves to report results, such as laboratory and radiologic study data. The system under investigation did not generate any billing data or coding data.

All care providers use the EHR, and resident physicians training in pediatrics or internal medicine use it in a similar way. Attending physicians also use the system, but their usage is variable. Nearly all attending physicians in pediatrics use the system principally to make brief addendums to resident notes, since they do not have their own panel of patients. In the internal medicine clinic, many attending physicians supervise residents, but they also have their own patients and, therefore, must document full visits as well as review results and enter orders. Because of the consistent usage patterns by residents of both specialties and inconsistent usage patterns by attending physicians, we surveyed only residents.

For three years prior to the implementation of the EHR, pediatric residents documented health maintenance encounters using structured, machine-scannable paper forms. Urgent visits were recorded using free text. While the internal medicine residents had access to structured paper-based forms for management of common diseases, such as diabetes, such forms were supplemental to the progress note and were not used consistently.

Prior to the system deployment, internists and pediatricians met independently with the same team of IT implementers to design structured forms (templates) for a variety of visit types. Based on user feedback that was favorable toward structured, paper-based forms in the pediatrics clinic,32 the design of the Logician-based system used a similarly structured data-entry format.

In the internal medicine implementation, three main types of encounter templates were developed: new patient, return patient, and urgent visit. Optional supplemental forms to document diabetes, asthma, and hypertension and to provide prompts regarding appropriate care were also created. In the pediatric implementation, more than 15 different templates were developed for health maintenance encounters at various ages. Other forms were devised for generic acute care visits and for chronic asthma management. In both the adult medicine and pediatrics implementations, clinicians had the option of supplementing structured notes with free text.

Survey Development and Deployment

Based on a literature review, we constructed a model to clarify the types of factors, or dimensions, that affect satisfaction. We identified several potential constitutive predictors such as age, gender, and computer sophistication, included in the dimension “user characteristics.” Additional dimensions that affect satisfaction we identified were “user expectations,” and the perceptions of each of the following: “practicality of use,” “impact on delivery of care,” “organizational support,” and “variation from familiar practice.” Each of these dimensions would serve as a subheading for survey items.

To gain a deeper understanding of factors affecting satisfaction in the particular implementation of this electronic record, structured interviews were conducted with several clinician-users of the EHR system and the information systems project manager for the implementation.

Using results derived from the interviews and literature search, we developed survey items to collect data on user characteristics (demographic information and optimism that computers can benefit medical care), levels of satisfaction, expectations, and perceptions of the EHR system. An item from a validated survey instrument was included to assess computer sophistication.33 The survey contained 30 statements about EHRs in general or the specific implementation of the EHR in the PCC. Agreement or disagreement with these statements was assessed using a 4-point Likert scale, with the options “strongly disagree,” “disagree,” “agree,” or “strongly agree.” Neutrality was intentionally omitted. The complete survey, included in Appendix A, is available as an online data supplement at <www.jamia.org>.

We postulated that differences in satisfaction between the two groups of subjects (internal medicine and pediatrics residents) could be explained by differences in the factors that affect satisfaction with the EHR. Thus, to analyze components contributing to satisfaction or dissatisfaction, the primary unit of analysis was the dimension, or subheading on the survey, which encompassed two or more survey items.

Statistical Analysis

To compare baseline characteristics, differences in continuous variables were compared by t-tests. Fisher's exact tests were used to test for differences in categorical variables. Logistic regression was performed to assess the relationship between user characteristics and satisfaction.

To compare survey results between specialties, calculations were performed to determine relative risk of agreement between medicine and pediatrics subjects for each survey item ranked on the Likert scale. Since relative risk calculation requires a binary comparison, we conflated responses of “agree” and “strongly agree” into agreement and “disagree” and “strongly disagree” into disagreement. Ninety-five percent confidence intervals (CIs) were also calculated. In the case in which agreement was 100% with a survey item, a correction factor was equally applied to both sides of the calculation to determine relative risk.

Results

All 45 eligible pediatrics residents completed surveys (100%), while 40 of 41 (98%) eligible medicine residents completed surveys. The one eligible internal medicine resident not included was in Nepal at the time of the survey and could not be reached.

Characteristics of pediatrics and internal medicine subjects were generally similar (Table 1). No differences were observed in mean age and relative proportions of PGY-year. A higher proportion of pediatric residents were women (p = 0.04). The mean self-reported number of hours per week using a computer was similar. Pediatrics subjects' mean length of use of the EHR implementation was 2.8 months longer than that of medicine subjects.

View this table:
Table 1

Baseline Characteristics of Subjects

CharacteristicMedicine SubjectsPediatrics Subjectsp-value
Female (%)43670.04
Mean age (yr)28.428.40.99
PGY-I (%)35331.00
PGY-II (%)38361.00
PGY-III (%)27310.90
Time per week using a computer (hours)17.915.60.43
Mean length of time using Logician (months)11.714.50.06
  • P-values represent the probability of a difference between medicine and pediatrics subjects for each characteristic occurring by chance.

Medicine subjects were more likely to describe themselves as “very sophisticated” or “sophisticated” computer users than were pediatrics subjects (relative risk = 1.88, 95 % CI = 1.16–3.02). There was no significant difference in computer optimism. Fifty-eight percent of pediatrics subjects and 68% of medicine subjects responded that they were “very optimistic” that computers could benefit medical practice; the remainder of subjects responded that they were “optimistic.”

Medical specialty was the only independent predictor of satisfaction (p = 0.04). Other predictor variables, including gender, PGY-year, computer sophistication, and computer optimism did not predict satisfaction or dissatisfaction. There was no correlation between gender and computer sophistication.

Table 2 summarizes survey items that, when analyzed by survey-defined category, differentiated between medicine and pediatrics subjects. Medicine users were less likely to be satisfied with the EHR implementation. They were also less likely to believe their colleagues were satisfied. Ninety-one percent of pediatrics residents compared with 51% of medicine residents believed their colleagues were satisfied.

View this table:
Table 2

Survey Items Categorized under “General,” “User Expectations,” and “Practicality of Use”

Survey ItemMedicine Subjects' Agreement (%)Pediatrics Subjects' Agreement (%)Relative Risk, Medicine vs. Pediatrics95% Confidence Interval
General
   Generally satisfied.83980.84*0.73–0.98
   Colleagues have a favorable opinion.51910.56*0.41–0.77
   Wish I could just write on paper.8140.550.15–2.10
   Benefits of EHR outweigh inconveniences.981000.97*0.95–0.99
   System easier to use as I have become familiar.901000.90*0.87–0.93
User expectations
   Experienced with the VA Hospital's CPRS.95146.65*3.16–14.00
   Use of an EHR can improve care delivery.981000.97*0.95–0.99
   Screens were designed to promote improved billing.77511.50*1.05–2.14
   Screens were designed to optimize patient care.51840.61*0.44–0.85
   Screens were designed as a teaching tool.5600.09*0.02–0.34
Practicality of use
   Structured forms facilitate visit documentation.641000.64*0.60–0.69
   Templates teach about recommended practices.56910.62*0.46–0.83
   Templates remind about anticipatory guidance.64960.67*0.53–0.86
   Templates make the care I provide more efficient.54840.64*0.46–0.88
   Templates are well suited to my specialty.661000.66*0.61–0.71
   For urgent visits, I would prefer using free text.76741.020.79–1.31
   For continuity visits, I would prefer using free text.46410.38*2.57–41.97
   Order entry is straightforward.28470.590.33–1.06
   Writing prescriptions is straightforward.72870.830.66–1.04
   I rarely have questions that cannot be answered.60790.750.55–1.01
   I generally finish my notes before I leave clinic.74910.81*0.66–0.99
  • Values in boldface and marked by an asterisk(*) represent relative risks for which the 95% confidence intervals do not include 1.0.

Despite the difference in satisfaction between the specialty groups, the majority (91%) of all subjects reported general satisfaction with the EHR implementation. Ninety-eight percent of pediatrics subjects and 83% of medicine subjects reported they were “generally satisfied.” Almost all agreed that the benefits of health records outweighed the disadvantages. Both medicine and pediatrics users preferred to use electronic notes instead of paper-based recording to document visits.

Items addressing expectations about the EHR yielded different responses from each specialty group. While medicine users were more likely to believe the design of the EHR's user interface was to improve billing, pediatrics users were more likely to believe the design was intended to improve patient care or be used as a teaching tool for residents.

Pediatrics users found Logician more practical to use. Although medicine and pediatrics subjects felt similarly about ease of prescription writing and order entry, medicine subjects were less likely to finish notes before leaving the clinic. Of five survey items pertaining to structured documentation, all indicated a more favorable perception by pediatrics subjects. Furthermore, medicine residents were much more likely to prefer documentation of scheduled health maintenance visits using free text (no templates) as compared with pediatrics residents.

Table 3 displays survey items within dimensions that did not differentiate between medicine and pediatrics subjects. Subjects of both specialties agreed almost unanimously with statements pertaining to the positive impact of EHRs on care delivery.

View this table:
Table 3

Survey Items Categorized under “Impact on Delivery of Care,” “Organizational Support,” and “Variation from Familiar Practice”

Survey ItemMedicine Subjects' Agreement (%)Pediatrics Subjects' Agreement (%)Relative Risk, Medicine vs. Pediatrics95% Confidence Interval
Impact on delivery of care
   System has a positive impact on patient care.95931.020.92–1.13
   System promotes comprehensiveness of care.95911.050.93–1.18
Organizational support
   I was adequately trained in the use.45530.840.54–1.31
   Attending physicians view the system favorably.88890.980.84–1.16
   Technical support for the system is available.50650.770.52–1.13
Variation from familiar practice
   Interferes with the doctor–patient relationship.23240.970.44–2.13
   Limits my ability to include important content.18 91.970.62–6.24
   Makes my encounter documentation more efficient.68750.910.70–1.20
   Fits in with the way I'm accustomed to writing notes.49760.65*0.45–0.93
  • Values in boldface and marked by an asterisk(*) represent relative risks for which the 95% confidence intervals do not include 1.0.

Overall, responses to items pertaining to “variation from familiar practice” were consistent between medicine and pediatrics users. There was a high level of agreement in both specialties that use of the system did not interfere with the doctor–patient relationship. However, pediatrics residents were more likely to agree that they were accustomed to writing notes in the way the new system required.

Neither pediatrics nor medicine users perceived using the system as a disruption in their usual practice patterns. Three of four survey items within the dimension “variation from familiar practice” yielded similar responses both by medicine and pediatrics subjects.

Perceptions of organizational support between groups of users were also similar. Pediatrics and medicine subjects reported similar feelings about adequacy of training and technical support for the EHR. Most residents of both specialties agreed that their attending physicians had a favorable opinion of the new EHR.

Discussion

While survey results confirmed the anecdotal observation that pediatrics residents were more satisfied than internal medicine residents, most users in the PCC in both specialties were quite satisfied with the EHR. The high survey response rate suggests the results reflected the opinions of the whole population. The disparity between the survey results and initial anecdotal reports that internal medicine residents were generally dissatisfied may be explained by the “silent majority” phenomenon. Specifically, those unhappy with Logician, almost all of whom were medicine users, were likely to be more vocal in their dislike of the system than were satisfied users in their endorsement of the system. This explanation is supported by the fact that while most (83%) medicine users were satisfied with Logician, only about half (51%) believed their colleagues were satisfied.

Many factors contribute to satisfaction. Constitutive factors of the subjects aside from medical specialty appeared to play a small role in determining satisfaction. In findings consistent with those of Brown et al.18 who studied computer anxiety in interns, we found no correlation between gender and computer sophistication or satisfaction, nor between age and satisfaction. The finding that increased sophistication with computers does not correlate with greater EHR satisfaction was consistent with the findings of Penrod and Gadd13 who compared satisfaction of community physicians with that of academic physicians.

Having observed no difference in the dimension “impact on delivery of care,” we cannot ascribe differences in satisfaction to a dichotomy in perceptions of the EHR's impact on care delivery. Organizational support can be an important factor in satisfaction as evidenced by the findings of Travers and Downs15 who found that degree of clinical leadership and system support differentiated between a satisfied practice and dissatisfied practice in North Carolina. However, organizational support did not appear to play an important role in differentiating satisfaction in the PCC implementation we studied.

Neither pediatrics nor medicine users perceived using the system as a disruption in their usual practice patterns. Three survey items within the dimension “variation from familiar practice” yielded similar responses both by medicine and pediatrics subjects, but the last item within this dimension yielded different responses. This item, which addresses style of documentation, may have been better categorized under the dimension “practicality of use,” which assesses other aspects of electronic documentation.

The difference found in user expectations may be partially explained by prior experience of users. Almost all medicine users—but relatively few pediatrics users—had extensive experience with another ambulatory electronic health record. The Veterans Affairs Hospital's Computerized Patient Record System (CPRS) is used by all medical residents for several months during their training, whereas pediatrics residents have no consistent exposure to alternate EHR systems in the course of their training. The experience medicine users have with another EHR likely colors their point of view and expectations of an electronic record. Murff and Kannry10 compared a commercially available physician order entry (POE) system with CPRS, and found that for order entry purposes, residents were more satisfied with CPRS.

The notion is widespread (though false) among internal medicine residents that the Logician system was implemented to promote billing. The expectation that the interface was developed for billing purposes was summarized by one user we interviewed, who suggested that since the VA Hospital does not bill patients for services, the user interface design of CPRS does not take into account billing. By contrast, a commercial EHR, expected to be used in medical practices that do bill for services, would need to have the ability to bill.

Issues relating to structured as opposed to free text data entry may have important ramifications regarding satisfaction. Alhough points of view on many aspects of the impact of the EHR were the same across users from both specialties, perceptions of documentation templates consistently differed between pediatrics and internal medicine residents.

It is probable that several factors led to differential acceptance of documentation templates. Structured documentation has been shown to speed access to patient data in health records34 and to promote quality assurance.35 However, the time required to enter data in a structured format may be greater than that required to enter data in free text format.28 Rodriguez et al.36 found that electronic documentation took more time than paper-based documentation, but users were able to locate specific patient data faster. The finding that internal medicine subjects were less likely to finish notes before leaving clinic suggests the possibility that available documentation templates took more time for them to complete than for pediatrics subjects.

Pediatrics templates may have been better suited to users' needs than templates used by internal medicine residents. In contrast to the adult medicine clinic, the pediatrics clinic had iteratively refined its paper-based, structured documentation for three years prior to the deployment of the EHR.32 Folz-Murphy found that one reason physicians quit using electronic documentation was that “documentation templates did not fit with their practice requirements.”14 It appears that the structured format enhanced the efficiency of documentation for pediatrics residents by providing prompts and reminders, whereas the structured format appeared to afford medicine residents fewer benefits overall.

An alternate explanation for the disparity is the legacy of structured documentation prior to computerization in the pediatrics clinic that did not exist in adult medicine. In the pediatrics clinic, physicians were accustomed to structured scannable paper forms. Internal medicine subjects' responses indicate that they were less likely to be accustomed to taking notes using the system. Perhaps using computerized structured forms was more of a paradigm shift for internal medicine.

Since the PCC implementation is a dynamic process, the nine months of extra experience in the pediatrics clinic may have led to a more refined user interface for pediatricians. Gamm et al.16 write that “there appears to be strong support for the possibility that length of use may contribute to higher levels of acceptance.”

Specialty-specific differences in satisfaction with the EHR and documentation style emerged from our findings. It is notable that differences in responses of medicine and pediatrics subjects are reflective not only of medical specialty, but of differences in two similar yet separate implementations. Prior experience with the VA's CPRS, prior experience with structured documentation, and differential time of initial implementation certainly affected user satisfaction and acceptance of structured documentation as well. It would be valuable to attempt to replicate results in other institutions in which pediatricians and internists each use an implementation of the same EHR. Further research would also involve studying implementations across other specialties that use similar electronic records.

Limitations

We emphasize that our findings apply to a population of 85 resident physicians and may not be generalizable to internists and pediatricians in all practice settings. The study population was young (mean age, 28). If a typical subject in our study was born in 1974, she has been exposed to personal computers for most of her life. Subjects we surveyed were optimistic toward computers in medicine. They were accustomed to learning how to use new computer applications in a way that community physicians with many years' practice experience may not be.

Patient age is obviously different in adult and pediatrics clinics, but the makeup of visit types in each clinic is also different. For many reasons, including differences in insurance coverage between inner-city New Haven's child and adult populations, adults in the PCC are much less likely to be seen by physicians for preventive care visits, a visit type that appears well suited to template-based documentation. We can imagine that satisfaction with structured documentation might be equal between internists and pediatricians if investigators were to survey internists who care for an adult population who regularly seek preventive care. To generalize our findings to other practice settings would require validation in academic and community practices among fully trained physicians.

Conclusion

We found a difference in EHR satisfaction between internal medicine and pediatrics users in the implementations studied. Although many potential factors that influence satisfaction were similar between subjects in the two specialties, differences in previous experience may have influenced the results. Medicine residents had more previous experience with a different EHR implementation, which they may have perceived as superior to the one involved in this study. Pediatric residents had more previous experience with structured data entry prior to EHR implementation and more preventive care patient encounters for which structured data entry may be well suited. Since successful implementations generally require satisfied users, understanding what factors affect satisfaction can improve chances of a system's success.

Footnotes

  • The authors thank Drs. Prakash Nadkarni, Madeline Wilson, and Sharon Inouye for their suggestions and advice in writing the manuscript. Support for this research was provided by the National Library of Medicine Training Grant, Number #T15-LM07065, RO1 LM07199.

References

View Abstract