OUP user menu

★ Case Report ★

Use of Clinical Decision Support to Increase Influenza Vaccination: Multi-year Evolution of the System

Mary N. Gerard , William E. Trick , Krishna Das , Marjorie Charles-Damte , Gregory A. Murphy , Irene M. Benson
DOI: http://dx.doi.org/10.1197/jamia.M2698 776-779 First published online: 1 November 2008


Despite recognition that clinical decision support (CDS) can improve patient care, there has been poor penetration of this technology into healthcare settings. We used CDS to increase inpatient influenza vaccination during implementation of an electronic medical record, in which pharmacy and nursing transactions increasingly became electronic. Over three influenza seasons we evaluated standing orders, provider reminders, and pre-selected physician orders. A pre-intervention cross-sectional survey showed that most patients (95%) met criteria for vaccination. During our intervention, physicians were increasingly likely to accept pre-selected vaccination orders, Year 1 (47%), Year 2 (77%), Year 3 (83%); however vaccine administration by nurses was suboptimal. As electronic medical record functionality improved, patient receipt of vaccine increased dramatically, Year 1 [0/36; 0%], Year 2 [8/66; 12%], Year 3 [286/805; 36%]. Successful use of clinical decision support to increase inpatient influenza vaccination only occurred after initiation of CPOE for all medications and integration of an electronic medication administration record. Also, since most patients met criteria for influenza vaccination, complicated logic to identify high-risk patients was unnecessary.


Many improvements in patient-care derived from implementation of electronic medical records result from use of clinical decision support (CDS) systems triggered by computerized physician order entry (CPOE).14 Despite the benefits realized from use of electronic medical records, CPOE, and CDS, penetration of these systems in U.S. healthcare settings remains low.46 Increased adoption of CDS likely will follow demonstrations of successful implementations in institutions that have vendor-provided solutions.

Although influenza vaccination effectively prevents disease and reduces the risk of hospitalization, many high-risk people are not immunized.79 As such, the Joint Commission on Accreditation of Healthcare Organizations measures influenza vaccination of eligible pneumonia inpatients as an indicator of good quality healthcare. For these reasons, and because the population served by Stroger Hospital (formerly Cook County Hospital) historically has been undervaccinated,10 we designed a CDS-based intervention to increase influenza vaccination of hospital patients. Since components of our electronic medical record were introduced incrementally over three influenza seasons; we evaluated use of CDS during the maturation of our system. We report our challenges and successes during the implementation of our system.


Setting and Project Description

Stroger Hospital is a 464-bed public hospital. In 2001, we began installation of a new information system (Cerner Inc., Kansas City, MO.). Our information system gradually evolved as components were transferred from paper-based to electronic systems. We used the CDS system to improve influenza vaccination during the 20032004, 20052006, and 20062007 influenza seasons, i.e., Year 1, Year 2, and Year 3, respectively. Due to a nation-wide influenza vaccine shortage, there were no interventions during 20042005. We focused our intervention on internal medicine ward patients, who were admitted to one of three separate teams; admissions were consecutively assigned to a team, resulting in similar patient characteristics between teams. There were no education sessions for physicians during the three influenza seasons. We obtained approval from the IRB.

Year 1: 2003–2004 Influenza Season

During Year 1, we performed a cross-sectional survey to determine how often patients met high-priority category criteria for receipt of influenza vaccination;11 we performed bedside interviews.

Computerized provider order entry (CPOE) was available for laboratory tests, radiographic studies, diets, electrocardiograms, and admission/discharge orders; there was no medication CPOE. Nursing documentation was paper-based, including medication administration. Electronic orders were printed at the nursing station. No standing orders policy existed for influenza vaccination (Table 1).

View this table:
Table 1

Interventions Used to Increase Vaccination of Hospital Patients and Information System Maturation over Three Influenza Seasons

Influenza Season
   NursingNo interventionEducational sessions*No intervention
   InstitutionNo interventionStanding orders policy# initiatedStanding orders policy continued
Clinical decision support
   Team 1Electronic reminder to physicians to order vaccineElectronic nursing reminder to follow standing orders protocolPre-selected order triggered by “Admission to Bed” order
   Team 2No CDS interventionPre-selected order triggered by “Discharge Patient” orderPre-selected order triggered by “Discharge Patient” order
   Team 3Pre-selected order triggered by “Discharge Patient” orderNo CDS interventionBoth of the above
Information system capacityLaboratory, diet, radiology, EKG, and discharge orders electronic; these orders printed at the nursing station.
  • Nurses: kardex and patient activity list available electronically.

  • Physicians: increased electronic charting, electronic consult orders.

  • Nurses: medication orders populated e-MAR.

  • Physicians: CPOE available for medications (including vaccines).

  • CDS = clinical decision support; EKG = electrocardiogram; e-MAR = electronic medication administration record; CPOE = computerized physician order entry.

  • * Mandatory small-group sessions; we educated nurses about the influenza vaccination and the new standing orders policy.

  • An automated electronic reminder populated the nurses' kardex and patient activity list at the time of admission.

  • The clinical decision support system was not fully operable for the “Admission to Bed” order; therefore, these results are not presented.

  • #The standing orders policy enabled nurses to administer influenza vaccine to patients without an individual physician's order.

The cross-sectional survey showed that >95% of internal medicine patients met a high-priority criterion for vaccination; therefore, our CDS rule targeted all internal medicine inpatients. We tested separate strategies for each internal medicine team, as follows: 1) A pre-selected order “Administer Influenza Vaccine 0.5 ml IM” was presented to physicians in real time as a pop-up, triggered by their “Discharge Patient” order; 2) a pop-up reminder to order vaccine was presented to physicians upon entry of the “Discharge Patient” order; 3) no intervention, i.e., usual care.

Year 2: 2005–2006 Influenza Season

During Year 2, we augmented CDS rules with a written universally-applied standing-orders policy that enabled nurses to vaccinate inpatients even if there wasn't a patient-specific physician's order. We educated nurses about the policy and importance of influenza vaccination. Nurses viewed electronic orders through a patient-centered task list. Compared to Year 1, physicians entered more orders electronically, and increasingly documented patient evaluations in the electronic medical record. However, medication order entry still was paper-based (Table 1).

We compared the following strategies concurrently (i.e., one strategy per team): 1) A pre-selected order “Administer Influenza Vaccine 0.5 ml IM” was presented to physicians, triggered by the “Discharge Patient” order; 2) an electronic reminder to follow the standing orders policy populated nurses' task lists upon patient admission; 3) usual care.

Year 3: 2006–2007 Influenza Season

During Year 3, medication CPOE was active and orders were routed electronically to the electronic medication administration record (Table 1). Since despite physician orders, vaccine administration was uncommon during Year 2, in Year 3 we intended to test triggering the order at the time of patient admission. Unfortunately, we were unable to implement CDS using the “Admission to Bed” order; therefore, for Year 3, all results were for the “Discharge Patient” order—the same intervention as during Year 2.

Influenza Vaccination Assessment

During all three seasons, the pre-selected influenza vaccine order was enacted unless the physician de-selected the order. During Years 1 and 2, to determine patient vaccination receipt, we reviewed paper medication administration records and nursing notes on a random sample of patients. During Year 3, since vaccine administration was only documented electronically, we retrieved these data from the hospital information system for all patients; patient sampling was unnecessary.

To demonstrate changes in vaccination rates over time—during the transition from a hybrid paper-electronic system to an electronic system—we present the results from the intervention strategy that remained constant (i.e., pre-selected order triggered by the “Discharge Patient” order). We tested the trend in vaccination receipt by calculating the chi-square test for trend. All analyses were performed using Stata version 9.2 (StataCorp, College Station, TX).


Year 1

From the cross-sectional survey, approximately one-third of patients were vaccinated before hospitalization; reactions to eggs or previous influenza vaccination were rare and most patients met criteria that placed them in a high-priority category for vaccination (Table 2).11 The most common criterion met was age >49 years followed by diabetes, cardiac, pulmonary, or renal disease. For seven patients (7%) the sole criterion met could only be discerned through bedside interview; for example, living with a high-risk person or child aged less than two years.

View this table:
Table 2

Characteristics of Patients Evaluated during the Cross-sectional Survey, 2003–2004

CharacteristicNumber, N = 103%
   Black or African American6462
Influenza vaccine history
   Usually receives influenza vaccination3938
   Received influenza vaccination this year3130
   Egg allergy00
   Reaction to influenza vaccination11
Met criteria for vaccination(95% CI)
   Internal medicine or family practice, n = 666395 (87–99)
   Surgery, n = 373081 (65–92)
   Total9390 (83–95)
  • Median age = 52 years, inter-quartile range [43–62].

During Year 1, of 114 patients sampled for chart review, none were vaccinated. Of 36 patients admitted to the pre-selected order team, 17 (47%) times the physician accepted the order; however, the vaccine never was administered, Figure 1.

Figure 1

Influenza vaccination rates of hospital patients, by intervention and year.

Year 2

Of 204 patients sampled, 66 were admitted to the pre-selected order medical team. Compared to Year 1, physicians were much more likely to accept the pre-selected order (77% vs. 47%; p = 0.002) and patients were more likely to receive vaccine (8/66 [12%] vs. [0/36] 0%, p = 0.03), Figure 1. Despite having a standing orders policy, few patients admitted to the usual care or nursing reminder teams received influenza vaccine (1% and 6%, respectively).

Year 3

During Year 3—after implementation of the electronic medication administration record—among 805 patients discharged by the pre-selected order team, most physicians (n = 665; 83%) accepted the pre-selected order, and there was a dramatic increase in patient vaccination (n = 286; 36%), Figure 1. Of 665 patients who had a physician's order for vaccination, 43% were vaccinated. Since approximately 30% of patients were vaccinated before hospitalization, we estimate that 61% of eligible patients were vaccinated. Over the three influenza seasons, there was a significant increase in patient vaccination (Year 1 [0/36], Year 2 [8/66], Year 3 [286/805]; p < 0.001), Figure 1.


Over time, physicians were increasingly likely to accept pre-selected vaccination orders; by Year 3 83% of orders were accepted. A bigger challenge was improving vaccine administration after physician order, which improved dramatically after the medication administration record was integrated into the electronic medical record. Also, since most internal medicine patients met criteria for vaccination, sophisticated rule-building was unnecessary. In fact, since not all patients who meet high-priority criteria for vaccination could be identified using clinical data, building rules to selectively trigger the CDS system would have resulted in missed opportunities to vaccinate high-risk patients.

Our findings illustrate the tenet that to successfully implement CDS it is essential to address workflow integration, healthcare worker-system interaction, local culture, and transition of most processes to the electronic system.1,3,1215 During Year 1, we attempted CDS in a predominantly paper-based system; for example, electronic orders were printed at the nurses station, which delayed notification of nurses about the order during a time critical process. Since our CDS rule was triggered by the “Discharge Patient” order, nurses were required to vaccinate patients in the relatively short time between the discharge order and the patient's departure. Although we considered other CDS triggers, each potential solution had challenges. For example, during Year 2 we tested automated electronic reminders to nurses, triggered by patient admission. Unfortunately, nurses rarely followed the policy. We considered using temporal triggers, e.g., orders presented by hospital day, but this was not an option with our CDS system. Finally, we attempted to use the admission order as a trigger for the pre-selected order, but could not resolve technical difficulties.

During Year 1, physicians who were exposed to a reminder, rather than the pre-selected order, did not place the order. Likely because in part, there was no medication CPOE, and physicians either had to write the order in the paper chart or search for the electronic order at a time when there was no CPOE for medications. During Year 2, increased vaccine administration by nurses likely resulted from increased functionality of the electronic medical record; for example, availability of an electronic task list for nurses. In Year 2, despite improved physician acceptance of the order and increased vaccine administration, coverage levels remained low.

Since nursing and physician reminders were unsuccessful during Years 1 and 2, in Year 3 we focused on using pre-selected physician orders. To facilitate nursing administration of vaccine during the discharge process, we intended to test presenting the pre-selected order to physicians upon both patient admission and discharge. However, despite successful use of the rule in the test environment, we were unable to reliably use the order triggered by patient admission. Specifically, influenza orders triggered by patient admission usually were not accepted by the system; therefore, the pharmacist could not verify the order electronically. Since Year 3 vaccination rates were dramatically improved, we believe that the low vaccine administration rates during Years 1 and 2 were primarily due to incomplete maturation of the electronic system, rather than logistical challenges posed by triggering the pre-selected order on patient discharge.

Our experiences illustrate how CDS implementations require attention to local workflow, in particular, transition of workflow from paper-based to electronic systems. This may partly explain the relatively low penetration of CDS in healthcare settings, especially since many institutions do not have the expertise to tailor less flexible vendor-provided systems to their needs.4,16 Despite these challenges, we achieved significant and meaningful increases in vaccination coverage using CDS. By Year 3, over 50% of patients not vaccinated before hospitalization, were vaccinated during their hospital stay.

At our hospital, a written universally-applied standing orders policy was ineffective, even after augmenting the policy with electronic reminders to nurses. In addition to nurses' concerns about acting without an individual physician's order, during educational sessions many nurses expressed concerns about the influenza vaccine. After recognizing these substantial local barriers, during Year 3 we abandoned electronic nursing reminders—despite proven success at another institution.17 For the following reasons, we presented the pre-selected order to all patients: incomplete electronic problem lists compromise the sensitivity of electronic inferences of chronic medical conditions, some criteria are not electronically captured (e.g., living with a high-risk person), we were hesitant to increase nurses' workload by creating an electronic form to screen patients, and influenza vaccination is safe and effective for all patients.

Our findings are limited in that we evaluated patients in a single, large, urban public hospital, nurses at other hospitals may be more likely to respond to electronic reminders and standing orders policies.17 Also, we did not determine previous vaccination history through patient interview; however, a bedside interview may have influenced patients' desire for vaccination and biased our results.

We encountered several challenges during implementation of a CDS rule to increase influenza vaccination. These challenges included local cultural issues—nurses were reluctant to carry out standing orders—and technical issues, such as incomplete integration of our electronic medical record and lack of functionality of a vendor-provided system. Despite these challenges, we observed early and nearly complete physician acceptance of the CDS-generated order. After integration of the electronic medication administration record, there was a dramatic increase in nurses' administration of vaccine. Use of CDS can dramatically improve patient care, but success may be realized only after understanding the local workflow and culture, and near-complete transition from paper to electronic processes.


View Abstract