OUP user menu

A clinical data warehouse-based process for refining medication orders alerts

Abdelali Boussadi , Thibaut Caruba , Eric Zapletal , Brigitte Sabatier , Pierre Durieux , Patrice Degoulet
DOI: http://dx.doi.org/10.1136/amiajnl-2012-000850 782-785 First published online: 1 September 2012


The objective of this case report is to evaluate the use of a clinical data warehouse coupled with a clinical information system to test and refine alerts for medication orders control before they were fully implemented. A clinical decision rule refinement process was used to assess alerts. The criteria assessed were the frequencies of alerts for initial prescriptions of 10 medications whose dosage levels depend on renal function thresholds. In the first iteration of the process, the frequency of the ‘exceeds maximum daily dose’ alerts was 7.10% (617/8692), while that of the ‘under dose’ alerts was 3.14% (273/8692). Indicators were presented to the experts. During the different iterations of the process, 45 (16.07%) decision rules were removed, 105 (37.5%) were changed and 136 new rules were introduced. Extensive retrospective analysis of physicians' medication orders stored in a clinical data warehouse facilitates alert optimization toward the goal of maximizing the safety of the patient and minimizing overridden alerts.

  • Computer-assisted decision-making
  • decision support systems
  • clinical organization and administration
  • drug therapy
  • computer-assisted organization and administration
  • clinical data warehouse
  • clinical decision rule


Implementing a clinical decision rule involves three main steps: creating the rule, testing it, and assessing its impact on clinician behavior.1 In most rule-based decision support systems studies, assessing the impact of the clinical decision rules development step is over-emphasized to the detriment of the rule testing development step.2 ,3

Testing clinical decision rules after they have been fully implemented can be resource intensive and time consuming.4 First, operational databases relying upon computerized provider order entry (CPOE) and electronic medical record components are difficult to query for testing and validating the implemented clinical decision rules.5 The multiplicity of relational tables, which are optimized to facilitate patient management, impedes the efficiency of the statistical queries necessary for validating the rules. The CPOE and electronic medical record components architecture do not clearly separate the business logic from the technical logic of implementation. As a result, it is difficult to modify the rules based on the results of testing. Second, testing clinical decision rules in a production system is computationally intensive and requires specific expertise during alert development as described by Oppenheim et al.4

An alternative is to use a large ‘retrospective’ chart review.3 The present study used a clinical data warehouse (CDW) to test and refine (new) clinical decision rules for medication order alerts regarding drug dosages for inpatients with renal insufficiency before the rules were fully implemented and/or reintroduced.


Study site and settings

Hôpital Européen Georges Pompidou (HEGP) is a teaching hospital with 24 clinical departments and 795 beds. The HEGP clinical information system integrates an electronic health record with a CPOE (DxCare, MEDASYS).6 A CDW based on the i2b2 framework was integrated into the HEGP clinical information system in 2009.7

Data with the electronic health record database and the CPOE database are automatically reflected into the CDW database. The clinical data used in this study to assess decision rules were derived from medication orders and biological investigations ordered between January 2005 and March 2011 at the HEGP. The greatest benefit of using the i2b2 in relation to our objectives was the simplicity of the i2b2 star schema which includes only five tables.7

Medication list selection

After having analyzed existing guidelines,8 ,9 a local medical expert panel defined the dosage adjustments according to the estimated glomerular filtration rate (eGFR) of patients for the medications considered (table 1). The eGFR values were determined using the revised 4-components of an equation from the Modification of Diet in Renal Disease (MDRD) study,10 with the exception of the ethnic component, which has not been validated in a European context. Required dosage adjustments were based on the level of kidney function impairment. For each medication and for each eGFR category, the expert panel determined:

View this table:
Table 1

Comparison between alert prevalences before and after the first iteration of the clinical decision rules refinement process

First iterationSecond iteration
MedicationsPatientsMax daily dosage alerts frequencyUnder-dosage alerts frequencyTotal alertsMax daily dosage alerts frequencyUnder-dosage alerts frequencyTotal alerts
Amoxicillin and clavulanic acid28013253307 (9.44%)90 (2.77%)397 (12.20%)
Amoxicillin and clavulanic acid–oral19652273316 (13.90%)78 (3.43%)394 (17.33%)
Amoxicillin and clavulanic acid–IV–2000/20035420 (0%)3 (7.14%)3 (7.14%)
Amoxicillin and clavulanic acid–IV–1000/20080193811 (1.17%)8 (0.85%)19 (2.03%)
Cefotaxim36342817 (3.97%)20 (4.67%)37 (8.64%)8 (1.87%)13 (3.04%)21 (4.91%)
Ethambutol6710913 (11.93%)11 (10.09%)24 (22.02%)18 (16.51%)1 (0.92%)19 (17.43%)
Isoniazid871735 (2.89%)92 (53.18%)97 (56.07%)12 (6.94%)11 (6.36%)23 (13.29%)
Metronidazole5317302 (0.27%)7 (0.96%)9 (1.23%)32 (4.38%)12 (1.64%)44 (6.03%)
Acyclovir931344 (2.99%)4 (2.99%)8 (5.97%)
Amoxicillin1284202754 (2.66%)8 (0.39%)62 (3.06%)
Metformin9131131194 (17.15%)33 (2.92%)229 (20.25%)
Tramadol32235121 (5.98%)6 (1.71%)27 (7.69%)
Valacyclovir2663560 (0%)2 (0.56%)2 (0.56%)
  • The ‘minimum dosage’ per 24 h as the lowest medication dosage irrespective of indication;

  • The ‘maximum dosage’ per 24 h as the highest medication dosage irrespective of indication.

Clinical decision rule refinement

We developed a first set of clinical decision rules to monitor prescriptions for the medications listed in table 1. Clinical decision rules were represented as logical ‘If <Conditions>…Then <Actions>…Else’ statements to be followed to reach a conclusion.2 ,11 A rule is acted upon, that is, the <action> part is performed, when the <condition> part is true. A patient's eGFR value is an example of a decision rule condition. One or more eGFR values can be estimated for a patient during his hospital stay. The clinical decision rules have been designed to use the eGFR value which has the closest entered date/time attribute to the date/time attribute of the corresponding medication order.

Figure 1 describes the different steps of the clinical decision rule test and refinement process. We submitted these rules to the refinement process to determine the nature of the resulting changes which could impact the decision rules and the alert incidence.

Figure 1

The different steps of the clinical decision rule refinement process. CDW, clinical data warehouse; CPOE, computerized provider order entry; EMR, electronic medical record; HEGP, Hôpital Européen Georges Pompidou.


We implemented 280 clinical decision rules based on a business rule management system (BRMS) and connected them to the CDW database to retrospectively control orders of 10 medications (table 1). The details of the alert system architecture are described in online appendix B and were reported previously.2

Clinical decision rule refinement

Changes to the clinical decision rule condition value

The alert system analyzed 8692 drug prescriptions ordered between 2008 and 2011 for 6727 different patients in 17 different medical departments. The alert system triggered 892 alerts (10.26%), 617 (7.10%) exceeds max daily dose alerts and 273 (3.14%) under-dosage alerts (table 1). The indicators summarized in table 1 were presented to the HEGP expert panel (step 4 in figure 1). Based on the results of the first iteration of the refinement process, the panel decided to modify the medication dosage rates for four medications: cefotaxim, ethambutol, isoniazid, and metronidazole (see online appendix A for the overall modifications). In this case, steps 1−5 were followed in the refinement process.

Antibiotic under-dosage presents a serious risk of exposing the patient to resistant bacteria and thus worsening the infection.12 ,13 Pharmacists were notified of the under-dosage alert rates associated with the isoniazid and ethambutol prescriptions checked by the alert system (table 1). The pharmacists then discussed these rates with the HEGP physicians in charge of tuberculosis treatment. The physicians informed the pharmacists that the average weight of the patients associated with these prescription alerts was between 50 and 60 kg (online appendix A). The clinical decision rules of the first iteration test case had been configured to check the isoniazid and ethambutol prescriptions of patients weighing at least >70 kg (online appendix A). In this first test case iteration, a possible explanation for the under-dosage alert rate for ethambutol prescriptions (10.09%) and isoniazid prescriptions (53.18%) (see table 1) was that the pharmacists had initially overestimated the weights of the patients associated with these prescriptions. This hypothesis was confirmed in the second test case iteration, in which the under-dosage alert rates decreased for the checked prescriptions of isoniazid and ethambutol to 6.36% and 0.92%, respectively (table 1).

Similarly, the cefotaxim high under-dosage alert rate observed in the first test case (table 1) emphasized the relevance of the decision rules related to cefotaxim prescription. Cefotaxim dosages (online appendix A) configured by the pharmacists in the first clinical decision rules were derived from Drug Prescribing in Renal Failure.9 The medication dosages recommended by this resource are higher than those listed in the Guidelines for Prescription in Renal Disease (GPR) handbook.8 Indeed, according to Drug Prescribing in Renal Failure, the recommended daily cefotaxim dosages are between 2000 and 4000 mg for patients with eGFRs between 20 and 50 ml/min/1.73 m2 and between 1000 and 2000 mg for patients with eGFRs of <10 ml/min. In the GPR handbook, the listed minimum daily dosages are 1500 mg for patients with eGFRs between 15 and 30 ml/min/1.73 m2 and 750 mg for those with an eGFR of <15 ml/min/1.73 m2. French physicians often prefer to consult the GPR handbook for antibiotic dosage information. For this reason, the pharmacists reviewed and adjusted the initial clinical decision rules according to the GPR handbook recommendations.8

Alerts for high and low doses of metronidazole increased after the second iteration of the refinement process (from 0.27% to 4.38%, and from 0.96% to 1.64%, respectively; see table 1). Metronidazol dosages (online appendix A) configured by the pharmacists in the first iteration of the refinement process were also derived from Drug Prescribing in Renal Failure.9 Similarly to the case of cefotaxim, isoniazid, and ethambutol, and according to the daily practice of French physicians, pharmacists reconfigured the clinical decision rules with the dosage recommendations of the GPR handbook.8 These two guidelines differ only in the maximum daily dose recommended when the eGFR is <10 ml/min/1.73 m2: the GPR handbook recommends 750 mg and Drug Prescribing in Renal Failure recommends 1500 mg (online appendix A). Thus, the lower threshold for the maximum daily dose of metronidazole resulted in a substantially higher number of ‘exceeds maximum dosage’ alerts for the second configuration.

Adding new clinical information

In the first iteration of the refinement process, pharmacists informed the alert system administrator that the clinical alert recommendations for amoxicillin and clavulanic acid did not distinguish the branded names prescribed for oral administration from the branded names prescribed for intravenous administration (online appendix A).

A second iteration of the refinement process was performed and steps ‘a’ and ‘b’ (figure 1) were followed to include the medication administration route in the clinical decision rules as a new condition part.

New indicators were presented to the HEGP expert panel (table 1). Pharmacists decided in the second iteration to distinguish between the different amoxicillin and clavulanic acid forms for intravenous administration (amoxicillin and clavulanic acid–IV–2000/200 and amoxicillin and clavulanic acid–IV–2000/100). Table 1 compares the prevalence of alerts before and after adding these new criteria to the clinical decision rules for amoxicillin and clavulanic acid prescriptions.

Number of decision rules affected during the experiments

Table 2 reports the number of clinical decision rules removed, modified, or added in response to the policy changes made by the pharmacists during the execution of the refinement process.

View this table:
Table 2

Clinical decision rules affected by the refinement processes

Initial number of rulesRules removedRules changedRules added
First experiment
Second experiment
   Amoxicillin and clavulanic acid45450136
   Total28045 (16.07%)105 (37.5%)136


During the different iterations of the decision rule refinement process (figure 1), 45 (16.07%) decision rules were removed, 105 (37.5%) were changed, and 136 new rules were introduced (table 2). Executing a test development phase before proceeding with a prospective study allowed a priori control of the performance indicators of the alerts, which can lower alert fatigue syndrome. Some of the problems related to alert refinements that arise in prospective studies can be addressed and eliminated during a test development phase. This allows alerts to be improved before prospective assessment and full integration into the hospital information system, leaving study investigators with more time to focus on other problems related to clinical alert monitoring.

Both interruptive and non-interruptive alerts can be useful in daily clinical practice,14 although there is some difficulty in determining the threshold between a non-interruptive alert and an interruptive alert. Our approach could facilitate the definition of these thresholds by pre-analyzing the impact of clinical decision rules. Hence, experts can determine whether certain recommendations are less critical than others and maintain these in a non-interruptive mode. Our approach could be used to test and refine other decision rules templates for medication orders control,15 for example, decision rules for hepatic function medication orders control,16 drug–drug interactions,17 etc. In addition, this method is currently used to design and test decision rules for clinical trial patient recruitment,18 ,19 thus allowing our method to be tested in a different setting.

Our results show that using a CDW as an experimental platform to test clinical decision rules before they are fully implemented allows medication dose thresholds to be easily adjusted according to experts' opinions. This experimental platform also allows assessment of the impact of the modifications on alert frequencies and possibly the false-positive rate of the alerts. Such testing can result in radical modifications to clinical decision rules, such as the introduction or suppression of new facts in the condition part of the clinical decision rules.

In the first iterations of the clinical decision rule design and refinement process, decision rules were designed to control several generic drug names.15 After several iterations of the process, the number of conditions in the condition part of the rules increased and so the number of decision rules also increased. The case of amoxicillin and clavulanic acid is a good example (tables 1 and 2). In the first iteration of the process, the decision rules designed to control the amoxicillin and clavulanic acid medication orders did not consider the administration route as a condition. After the administration route was introduced as a condition, the number of rules increased from 45 to 136 (table 2).

Regardless of the clinical decision rule software design process and the degree of end-user involvement in this process, CDW-based retrospective decision rule testing enables the identification and correction of design errors and omissions in the clinical data used by the rules, supporting an ‘agile’ process.2 Nonetheless, careful testing before implementation does not eliminate the need for regular monitoring of the performance of clinical decision rules after implementation.

Limitations and perspectives

The decision rules we tested were only for adjusting the dosages of medications whose dose depends on the level of renal function. To date, we have not yet implemented and tested this refinement process for developing and maintaining more advanced decision rules, such as those involving complex pharmaco-dynamic models or sequential drug administration. However, our approach could be extended to such situations in principle.

Retrospective testing of clinical decision rules establishes a ‘partial’ external validity of these rules.20 However, the rules must be further validated by prospective testing.1 An evaluation study assessing the diagnostic performance of a clinical decision rule-based alert system developed using the proposed framework, will allow complete validation of clinical decision rules. The results of such a study will allow us to determine how our rate of false-positive alerts compares to that of other systems that did not use a clinical decision rules refinement step during the development phase of their alert system.


The most common reason for overriding alerts, which can impair patient safety, is alert fatigue.21 One way to reduce alert fatigue is to reduce the frequency of false-positive alerts or alerts not accepted by clinicians.22 This recommendation can be difficult to implement because it requires definition of the correct balance among the requirements of the clinicians, the hospital, and the safety of the patients.23 Testing decision rules before they are fully implemented may help determine the right balance. This can be accomplished by performing several test iterations on data downloaded from the CPOE or from the pharmacy systems of the clinical site for which the alert system is intended.24 The method described in this study can facilitate clinical alert optimization to maximize the safety of the patient and to minimize overridden clinical alerts.


AB made a substantial contribution to the conception and design of this study, acquisition, analysis and interpretation of data, and drafting of the article. TC, EZ, BS, and PDX made substantial contributions to the acquisition of data, analysis and interpretation of data, and critically revising the article. PD as PhD mentor of AB, made substantial contributions to framing the research objectives, analysis and interpretation of data, and critically revising the article.


This study and AB were supported by a grant from the Programme de recherche en qualité hospitalière (PREQHOS—PHRQ 1034 SADPM), The French Ministry of Health and the Electronic Health Record For Clinical Research (EHR4CR) project, Grant Agreement no. 115189.

Correction notice

This article has been corrected since it was published Online First. A mistake in the second paragraph of the Methods; study site and settings section has been corrected and the last sentence of the contributors section has been amended.

Competing interests


Provenance and peer review

Not commissioned; externally peer reviewed.


We thank IBM-ILOG for providing us with an evaluation version of IBM WebSphere ILOG JRules.


View Abstract