OUP user menu

★ Research Paper ★

Comparative Impact of Guidelines, Clinical Data, and Decision Support on Prescribing Decisions: An Interactive Web Experiment with Simulated Cases

Vitali Sintchenko , Enrico Coiera , Jonathan R. Iredell , Gwendolyn L. Gilbert
DOI: http://dx.doi.org/10.1197/jamia.M1166 71-77 First published online: 1 January 2004

Abstract

Objective: The aim of this study was to compare the clinical impact of computerized decision support with and without electronic access to clinical guidelines and laboratory data on antibiotic prescribing decisions.

Design: A crossover trial was conducted of four levels of computerized decision support—no support, antibiotic guidelines, laboratory reports, and laboratory reports plus a decision support system (DSS), randomly allocated to eight simulated clinical cases accessed by the Web.

Measurements: Rate of intervention adoption was measured by frequency of accessing information support, cost of use was measured by time taken to complete each case, and effectiveness of decision was measured by correctness of and self-reported confidence in individual prescribing decisions. Clinical impact score was measured by adoption rate and decision effectiveness.

Results: Thirty-one intensive care and infectious disease specialist physicians (ICPs and IDPs) participated in the study. Ventilator-associated pneumonia treatment guidelines were used in 24 (39%) of the 62 case scenarios for which they were available, microbiology reports in 36 (58%), and the DSS in 37 (60%). The use of all forms of information support did not affect clinicians' confidence in their decisions. Their use of the DSS plus microbiology report improved the agreement of decisions with those of an expert panel from 65% to 97% (p = 0.0002), or to 67% (p = 0.002) when antibiotic guidelines only were accessed. Significantly fewer IDPs than ICPs accessed information support in making treatment decisions. On average, it took 245 seconds to make a decision using the DSS compared with 113 seconds for unaided prescribing (p < 0.001). The DSS plus microbiology reports had the highest clinical impact score (0.58), greater than that of electronic guidelines (0.26) and electronic laboratory reports (0.45).

Conclusion: When used, computer-based decision support significantly improved decision quality. In measuring the impact of decision support systems, both their effectiveness in improving decisions and their likely rate of adoption in the clinical environment need to be considered. Clinicians chose to use antibiotic guidelines for one third and microbiology reports or the DSS for about two thirds of cases when they were available to assist their prescribing decisions.

There is a growing interest in the impact of guidelines and clinical decision support systems (DSSs) on decision making, especially in reducing practice variation and error.1 However, empirical evidence shows that adherence to guidelines and their impact on decision outcomes remain inconsistent and vary across clinical specialties and levels of expertise.24 Similarly, despite their promise to improve clinical practice and patient outcomes,3,5,6 computer-based guidelines and DSSs have not been widely adopted.5 The success rates of DSSs that target prescribing decisions vary from 1% to 99%, but little is known about the factors that determine whether clinicians use these systems.7

The paradox of a proliferation of evidence such as clinical practice guidelines, but its low uptake, has suggested that clinicians are already struggling with information oversupply and the competition for their limited reserves of attention.4,8 This has led to the suggestion that the notion of the “best” evidence should be replaced with a more complex notion of the “most effective” method of evidence delivery, which takes into account both the native potential of evidence to improve clinical decisions as well as the likelihood that the mode of delivery will be adopted.8

Studies of evidence-based interventions look at the effectiveness of those interventions in improving decisions under controlled conditions. However, no studies have yet looked at the impact such interventions have when free-willed adoption rates also are measured. Our objective was to determine which mode of decision support is most likely to be used if available, and, if used, most likely to be effective. To achieve these objectives we designed an experiment using the interactive features of the Internet. Ventilator-associated pneumonia (VAP) was chosen as it represents a complex prescribing decision, and there currently is much variability in the approaches to its management.9 This choice also allowed us to exclude the recognized impact of patient demand on antibiotic prescribing decisions.7

Participants and Methods

Participants

Using lists provided by the Royal Australasian College of Physicians and the Royal College of Pathologists of Australasia, we identified specialist intensive care and infectious disease practitioners (ICPs and IDPs) working full time in tertiary referral hospitals in Australia and invited them to participate in the study. Forty volunteered to participate, and 31 completed the experiment. No monetary incentive was offered. All respondents were accredited practitioners, representing all States of Australia, and 16.6% and 11.7% of the national samples of IDPs and ICPs, respectively.

Information Support

We developed eight hypothetical cases designed to cover a range of causes of pulmonary infiltrates, based on the types of referrals received by the Westmead Hospital intensive care unit. (A description of these cases is available as an online data supplement at <www.jamia.org>.) An expert panel, made up of one ICP (JRI), one IDP (GLG), and a clinical microbiologist (VS) reviewed the cases and agreed on the optimum prescribing decision for each based on best available evidence.

Two decision support tools were developed. First, we designed and validated antibiotic guidelines for VAP, which were tailored to recommendations of local professional bodies. The antibiotic guidelines described an algorithm for VAP management in critical care.10 Secondly, a DSS was developed to provide advice about appropriate management of VAP using the Clinical Pulmonary Infection Score (CPIS).10 The DSS included a CPIS calculator to provide patient-specific prior probabilities of VAP and a previously published clinical pathway for its application (Fig. 1).10 The CPIS is a VAP severity measure based on a patient's chest radiography, body temperature, duration of ventilation, oxygenation, and laboratory values (blood leukocyte count, microbial culture, and presence of tracheal secretions). This score identifies patients at risk of VAP for whom short-term antibiotic could be beneficial. Finally, the laboratory reports providing case-specific results of bronchoalveolar lavage or endotracheal aspirate bacterial cultures with antibiotic susceptibility information were designed for the simulated cases.

Figure 1

Case scenario and clinical pulmonary score input forms.

Web Experiment

A crossover design with intrasubject comparison was chosen to minimize the possible effects of variability between subjects. The participants completed a brief questionnaire asking them to rate their information technology skills. The experiment was described briefly to participants, and the availability of information support features to aid their prescribing decisions was explained. The participants then were presented with eight case scenarios requiring prescribing decisions. Four levels of support were offered: (1) no extra information (control level), (2) antibiotic guidelines, (3) microbiology laboratory report, and (4) DSS, with the same microbiology report as in (3).

The case scenarios were allocated randomly to the different levels of decision support (Fig. 2). However, the two initial cases presented to subjects were always control cases, the third and fourth had an option of antibiotic guideline use, the fifth and sixth offered laboratory reports and no guidelines, and the last two cases provided access to laboratory reports and DSS. The order of decision support availability was designed to avoid any learning effect between cases.

Figure 2

Online experiment design.

Participants were free to elect to use the support offered or ignore it but were not able to request any other information in addition to or instead of the information provided. Support had to be selected by clicking a button on a screen, and was not visible on screen until selected. Participants were asked to make a prescribing decision (start, stop, or modify antibiotic therapy); to indicate on a five-point Likert scale their levels of confidence in the diagnosis of VAP; their prescribing decision for each case; and, after completing all cases, the likelihood that they would use unit-specific antibiotic guidelines and DSSs in their practice. Pilot testing was done with five clinicians to determine the acceptability and clarity of the cases and questionnaire. Participants did not receive any specific training in using the decision support modalities prior to the experiment. Participants in the pilot were excluded from the final study. The experiment was implemented as a series of Web pages describing simulated case scenarios, and subjects could participate in the experiment at a location of their choice. Access to the experiment was protected by a single-use password. The case scenarios and decision support elements (Fig. 1) can be accessed at:<http://www.eng.unsw.edu.au/biomed/websurvey/surveyIntro.cfm?guest=user6guest6>.

Outcome Measures

Using a log-file, we monitored whether, and for how long, participants accessed each decision support feature. Time to complete a prescribing task and the prescribing decision for each case were documented. Participants' decisions to start/modify/stop antibiotic therapy were compared with expert-defined optimal decisions. Confidence in both the diagnosis of VAP and prescribing decision was determined. A “clinical impact” score to measure the effectiveness of decision support was calculated as the product of the adoption rate and efficacy of each option (percentage of decisions corresponding to expert panel recommendation). The clinical impact score is designed to reflect both evidence uptake and the effectiveness of evidence provision within the clinical environment in the same way that a citation index measures the impact of a scientific paper within a research community.8

Sample Size and Statistical Analysis

We calculated that 53 cases were required in each arm to detect a 25% increase in decision accuracy from 60% in a control arm with 90% power and two-sided α = 0.05. Enrollment of 31 clinicians in the experiment with every clinician completing eight cases (total 248), each of which was allocated randomly to the four levels of support (62 in each arm), ensured a sufficient number of cases. Descriptive statistics were generated using SPSS for Windows (version 10.0, Chicago, IL). One-way analysis of variance (ANOVA) and post-hoc Duncan test were applied to test significant differences between groups. Differences in scenario answers and responses to questions were also examined using χ2, independent t-test, and Spearman's rank correlation coefficient.

Results

Characteristics of Subjects

All doctors who participated in the study worked as consultants in tertiary referral hospitals, and 29 of 31 (93.5%) held a professional college fellowship. Seventy-three percent of ICPs rated their computer skills as “good” or “excellent” compared with only 38% of IDPs (Table 1).

View this table:
Table 1

Sample Characteristics of Participants

Total, n (%)IDPs, n (%)ICPs, n (%)
Number of clinicians311615
Professional college fellowship3014 (93)15 (100)
Self-assessed computer experience
   Excellent31 (6)2 (13)
   Good145 (31)9 (60)
   Satisfactory1410 (62)4 (27)
Likelihood of future use*
   VAP antibiotic guideline15/17 (88)4/5 (80)11/12 (92)
   DSS17/22 (77)6/8 (75)11/14 (77)
  • Abbreviations: IDPs, infectious disease practitioners; ICPs, intensive care practitioners.

  • * Includes only the respondents who actually accessed these support modalities.

Decision Support Use

The antibiotic guidelines, microbiology reports, and the DSS with microbiology reports were selected in 24 (39%), 36 (58%), and 37 (60%) cases, respectively (Table 3). Only 17 (55%) clinicians accessed antibiotic guidelines, and 22 (71%) accessed either microbiology reports or the DSS when provided (Table 2). Seven (44%) IDPs and one (7%) ICP did not use any of the information support available at the time of their decision making (χ2 = 5.56; p = 0.018). There was no correlation between the use of information support and self-rated level of computer skills.

View this table:
Table 2

Use of Information Support Features by Clinicians

TotalIDPsICPsChi-square*p-value*
Number of clinicians311615
Use of information support, n (%, 95% CI)
   Antibiotic guidelines17 (55, 38–72)5 (31, 8–53)12 (80, 60–100)7.430.006
   Laboratory report22 (71, 55–87)9 (56, 32–80)13 (87, 70–100)3.480.06 
   DSS22 (71, 55–87)8 (50, 26–74)14 (93, 80–100)7.060.008
  • Abbreviations: IDPs, infectious disease practitioners; ICPs, intensive care practitioners.

  • * Difference between IDPs and ICPs.

Most participants who accessed decision support features during the study indicated that they would use VAP treatment guidelines (88%) or the DSS (77%) in the future (Table 2); six of 14 (43%) participants who did not access guidelines during the experiment, indicated that they would do so in the future.

Effects of Decision Support

The use of a decision support option did not affect participants' confidence in their prescribing decisions but did significantly improve their agreement with the expert panel's decisions from 65% when no decision support was available to 97% when it was used (Table 3). Microbiology reports were accessed in 32 of 37 (87%) cases in which the DSS was also used. Access to VAP treatment guidelines was not associated with improvement in prescribing decision quality.

View this table:
Table 3

Impact of Guidelines, Clinical Text, and Decision Support on Prescribing Decisions

Information Support CategoriesOne-way ANOVA
None (Control)GuidelinesLaboratory ReportDSS + Laboratory ReportdfFp
Case scenarios/information support combinations presented62626262
Information support features used, n (%)62 + 89 = 151*24 (39)36 (58)37 (60)
Agreement with the expert panel, n98162836NA
% (95% CI)65 (57–73)67 (48–87)78 (65–91)97 (92–100)
Confident or highly confident of prescribing decision, n1031828273,2470.090.966
% (95% CI)68 (61–75)75 (58–92)78 (65–91)73 (59–87)
Time taken (seconds per case)
   Mean1132021232453,24716.03<0.001
   Median83117115202
   SD8411652170
  • * Cases from other categories in which no support options were accessed are included.

  • p < 0.01 (chi-square, compared with control group—no support); NA, not applicable for categorical data.

Clinical impact scores were 0.26, 0.45, and 0.58 for guidelines, laboratory report, and the DSS-supported decisions, respectively. Thus, DSS plus reports had a 1.28 times greater impact than reports alone and two times greater impact than access to guidelines. The improvement in prescribing decisions associated with use of DSS was offset by a significantly longer time required to complete the task. On average, it took 245 seconds to make a decision using the DSS features compared with 113 seconds required for unaided prescribing intent (t = −4.59; df = 40; p < 0.001).

Discussion

Variation in Clinical Decision Support Use

In this study, we have shown a significant variation between members of two internal medicine clinical subspecialties in their use of decision support for an antibiotic prescribing task. This may reflect well-documented variations in clinical training and decision making1 as well as the current limited understanding of appropriate prescribing in the complex area of VAP management. Physician practice styles persist over time11 and can be attributed to differences in an individual's practice style, risk handling, and intensity of care.12 For example, risk-averse physicians are more likely to seek additional information before deciding to treat or to do nothing, increasing their use of information resources.13

Our results confirmed previous observations that the use of a DSS significantly improves prescribing decisions and compliance with treatment protocols.2,3,14 Despite a low level of adoption (about one-third of decisions), information support significantly reduced the variability of decisions and improved their agreement with evidence-based recommendations. In our experiment, the improvement was apparently dependent on the use of laboratory data to estimate the probability of VAP. The magnitude of improvement in this study was comparable with that reported for a commercial DSS, which improved the diagnostic accuracy of clinicians confronted with difficult case scenarios by 11%.15

Reasons for Staff Not Using Information Support

Despite their potential benefits, decision aids were not utilized by clinicians in up to two thirds of prescribing decisions in this study. These findings suggest that independent of the native effectiveness of a decision support intervention, the rate of adoption by clinicians will have a significant bearing on its ultimate clinical impact. It has been postulated that the perceived usefulness of medical information is a function of its relevance, validity, and the effort involved in searching for it.16 However, these parameters were held constant in our study. The fact that participants in our study were unfamiliar with the DSS provided, particularly with the CPIS calculator, could contribute to the low adoption rate of information support in our study. Another possible explanation for this finding is the schema-driven decision making thought to be associated with experts. Experts seem to rely on their own experience and pattern recognition skills to solve problems, are confident about their decisions, and not easily influenced by the use of decision aids.17 Doctors often choose not to use available evidence at the time of decision making but rely on what they know and choose the strategy requiring least effort.18

The potential for time to be consumed by information seeking is another reason for not using decision support. In our study, it took on average 202 and 245 seconds per case to make decisions using guidelines and DSSs, respectively, in comparison with 113 for the unsupported group and 123 seconds on average per case using the more familiar format of laboratory reports (Table 3). One possible reason for this striking difference is that participants were not familiar with guidelines or DSS features. Importantly, for at least half of the decisions, participants could not ascertain the effort required to gain the information prior to clicking on the relevant button. We believe that doctors may overestimate cognitive and time cost required for information seeking and underestimate the potential gain in decision quality.

Previous research showed a lack of correlation between clinicians' confidence in their decisions and their correctness.1922 In one study,20 all 68 participating family doctors and internists were confident in their diagnoses despite accuracies of 46% and 67%, respectively. Similarly, we found no correlation between participants' confidence in their decisions and their quality, which may reflect the participants' extensive experience in VAP management.

Clinical Impact of Decision Support Use

Interestingly, computerized decision support, including pathology results, had a “clinical impact” more than two times that of antibiotic guidelines. The clinical impact score combines “availability” and “cost to access” into a single measure of adoption rate.8 Surprisingly, according to the impact score, the antibiotic guidelines were less effective than pathology report viewing or decision support.

Study Limitations

First, our study was small, and we tested the prescribing decision making of experts in a limited clinical domain using simulated cases. However, the use of simulated case scenarios has been shown to predict clinical performance.23 Second, this experiment relied on physicians' self-reported prescribing intentions and did not attempt to correlate intentions with actual decisions in the clinical workplace. The experiment did not test clinician preferences for decision support in future cases, when past experience might alter subsequent decisions. For example, the relatively large time cost of using the DSS may or may not influence future decisions to adopt it. However, because the information was presented in a standard way (e.g., laboratory reports) and in a clinically meaningful context, it is plausible that their responses reflect actual clinical decision making better than would responses to direct questions about treatment strategies.24 No training in the DSS use was provided for participants that could affect the usability of the system. To ensure that observed differences in decision making were due to the different information sources available and not to other factors, we used a repeated measures experiment with simulated cases rather than randomization of participants. Third, the experiment participants were limited to selecting only one answer per case, but more than one option may be used in practice. Expertise and experience influence what information is sought and how the patient is assessed more than they influence the interpretation of information provided.17 Finally, we cannot exclude the possibility of sampling bias. Study participants represented subsets of IDPs and ICPs with experience in Internet use, and our selection process could have attracted practitioners with an interest in clinical decision support. However, our participants came from two different specialties, were well trained, and were affiliated with academic medical centers. There is no reason to suspect that they were less likely to use clinical evidence than other doctors. Furthermore, external validity of our study is confirmed by the observed frequency of “correct” choices (65%) in the control set of unaided decisions, which is consistent with other studies of clinical decision making.25

Implications

The implicit assumption of a direct relationship between the accuracy of information provision and its clinical impact must be replaced with a more complex model. Clinicians' attitudes and the environment in which decisions are made influence the acceptance and adoption of decision support tools. Our results support the suggestion that simply providing clinicians with more accurate evidence may not be enough to improve their decisions or reduce practice variation.19 The dissemination of evidence-based decision support should take into account not only the potential efficacy of the interventions but also the likelihood of adoption by users, especially if the cost of using the intervention is high. The “clinical impact” score used here is a potential marker for the actual effect of an intervention in the workplace.

Unexpectedly, the impact of computerized decision support was much greater than that of guidelines. Our observations indirectly support a suggestion26 that prescribing decisions depend more on the doctor's decision-making style than on the actual clinical situation. Participants in our study who accessed treatment guidelines may not have followed them because of disagreement, outcome expectancy, or inertia of previous practice.4 The low adoption rates provide additional insight into the limitations of informatics interventions, and particularly treatment guidelines, in improving clinical practice.

We conclude that in measuring the impact of decision support systems both their effectiveness in improving decisions as well as their likely rate of adoption in the clinical environment need to be considered. Designing decision support systems that only emphasize decision accuracy, and not features that enhance clinical adoption, is likely to result in a significant reduction in the impact of the tools in the working clinical environment.

Acknowledgments

The authors acknowledge the cooperation of the volunteering participants without whom their experiment could not have been carried out. The authors are indebted to H. Garsden and A. Polyanovski for their input in the experiment's software development. The authors also thank Professor J. Westbrook and Dr. S. Gosling for their comments on the experimental design and earlier versions of this paper.

References

View Abstract