OUP user menu

★ Research Paper ★

Developing High-specificity Anti-hypertensive Alerts by Therapeutic State Analysis of Electronic Prescribing Records

Svetla Gadzhanova, Ivan I. Iankov, James R. Warren, Jan Stanek, Gary M. Misan, Zak Baig, Lorenzo Ponte
DOI: http://dx.doi.org/10.1197/jamia.M2169 100-109 First published online: 1 January 2007

Abstract

Objective: This paper presents a model for analysis of chronic disease prescribing action over time in terms of transitions in status of therapy as indicated in electronic prescribing records. The quality of alerts derived from these therapeutic state transitions is assessed in the context of antihypertensive prescribing.

Design: A set of alert criteria is developed based on analysis of state-transition in past antihypertensive prescribing of a rural Australian General Practice. Thirty active patients coded as hypertensive with alerts on six months of previously un-reviewed prescribing, and 30 hypertensive patients without alerts, are randomly sampled and independently reviewed by the practice's two main general practice physicians (GPs), each GP reviewing 20 alert and 20 non-alert cases (providing 10 alert and 10 non-alert cases for agreement assessment).

Measurements: GPs provide blind assessment of quality of hypertension management and retrospective assessment of alert relevance.

Results: Alerts were found on 66 of 611 cases with coded hypertension with 37 alerts on the 30 sampled alert cases. GPs assessed alerting sensitivity as 74% (CI 52% - 89%) and specificity as 61% (CI 45% - 74%) for the sample, which is estimated as 26% sensitivity and 93% specificity for the antihypertensive population. Agreement between the GPs on assessment of alert relevance was fair (kappa = 0.37).

Conclusions: Data-driven development of alerts from electronic prescribing records using analysis of therapeutic state transition shows promise for derivation of high-specificity alerts to improve the quality of chronic disease management activities.

Introduction

The World Health Organization (WHO) Global Strategy on Diet, Physical Activity and Health indicates that chronic diseases are now the major cause of death and disability worldwide, with non-communicable conditions—notably, cardiovascular diseases (CVD), diabetes, obesity, cancer and respiratory diseases—accounting for 59% of deaths annually and 46% of the global burden of disease.1 In the United States, chronic diseases account for 70% of all deaths, and the medical care costs of people with chronic diseases account for more than 75% of medical care costs.2 In New Zealand more than 80% of CVD, the number one killer, can be attributed to the major risks: high cholesterol, high blood pressure, low fruit and vegetable intake, inactive lifestyle and tobacco.3 The general practitioner (GP, family physician), has a key role in management of diabetes and significant comorbidities such as hypertension.45 Given the magnitude of the problem, even incremental improvements in chronic disease management can be expected to have meaningful impact on population wellness as well as on overall health care cost.

Prescribing alerts are generally aimed at addressing interactions created at the point in time of the alert: duplicate drug classes, drug-drug interaction, drug-problem interaction (including pregnancy), and drug-lab interaction.6 This model is noted for its tendency to create large numbers of inappropriate alerts that are frequently overridden by physicians.79 This model is poorly suited for capturing key aspects of prescribing for chronic disease management. Events such as allowing effective long-term therapy to run out, or initiating therapy with drugs other than the recommended first-line therapy so long as the drug is not contraindicated are not handled as “interactions” per se. These conditions currently fall outside of the traditional “safety” alerting framework, but are nonetheless important when trying to detect sub-optimal treatment over time.

In Australia, high rates of electronic prescribing have resulted in most practices having a local accumulation of prescription records that has the potential to provide the basis for retrospective analysis of prescribing-related practice patterns.10 Intuitively, the time-stamped, patient-specific and GP-specific record of prescriptions from electronic prescribing should provide a convenient and rich resource for analysis of this aspect of GP activity. A systematic literature review of electronic patient record data in primary care found that prescribing data is generally of higher quality than diagnostic or lifestyle data,11 which is an unsurprising result in that the prescription is directly linked to the execution of clinical workflow. On this basis we have formulated a model of therapeutic state-transition12,13 for audit of therapeutic practice which is based firstly on the actions of the GP over time (notably, prescribing), and exploiting other data (such as problems and observations) when available, but not requiring the latter as a basis for analysis. This contrasts with a model based firstly on the status of the patient, which while causally appealing and appropriate, we find to be poorly supported by the data actually available in current community based systems.

A particular concern in approaching electronic decision support for chronic disease management is to create decision support outputs that are perceived as helpful by the end users (the GPs), and in particular are not seen as overly conservative or not clinically significant. As Ash et al.14 point out, IT in health care can generate its own new set of problems, with a major category being “decision support overload” wherein the system is deemed unusable and is in fact resented. Shah et al6 demonstrate that careful expert formulation of alert criteria and a focus on the most critical alerts can bring specificity (i.e., rate at which alerts are accepted) to robust levels (67% in their case). Our approach focuses on examination of the actual prescribing data in a process of computerized iterative discovery and review (what can be considered a data mining approach) to establish the foundation for alert criteria. In our model changes in therapy, which we take as manifestations of the GP's decisions, are the basis for analysis.

In this paper, we present a model for analysis of chronic disease prescribing action over time in terms of transitions in status of therapy as indicated in electronic prescribing records. The quality of alerts derived from analysis of these therapeutic state transitions has been assessed in the context of antihypertensive prescribing in an Australian rural general practice wherein the local GPs participate directly in the process of analyzing their past prescribing to set evidence-based alert criteria for future prescribing. Quality is assessed in terms of sensitivity (an alert being generated when appropriate) and specificity (an alert not generated when not appropriate) when the GPs review cases where the alert criteria are applied to their subsequent antihypertensive prescribing practices. The key research question is whether therapeutic state transition analysis can lead to the development of locally-acceptable alerts to support improved chronic disease management.

Method

We desired a model that could provide GPs with an introspective view on the management of selected groups of patients. Our basic concept is to abstract information from electronic health records (EHRs) and transform it into a series of states and transitions in order to identify the extent of concordance of actual therapy with states and transitions described by a selected evidence-based clinical practice guideline, while still allowing clinicians to draw their own conclusions about appropriateness of actual practice.

Based on the chosen guideline, a number of Boolean variables are defined to represent major treatment options. In the present investigation a therapeutic state variable will be synonymous with a specific drug group (such as diuretics or beta-blocking agents), where drug groups are identified from the guideline. From EHR prescribing data we compute the state of a patient on each given day in an investigated period as the combination of state variables (i.e., state representing which of the drug groups are in use for that patient on that day). Thus, each patient history will be a sequence of states in the period of investigation and state transition will exist where the state value changes. A state-transition diagram is used to provide a visual representation of the identified prescribing histories of the group of patients of interest over time. State-transition paths that appear to differ from the “usual” practice of the GPs are brought to their attention for further consideration and form the foundation for candidate alert criteria.

Setting

We illustrate application of our model by investigating data from a rural medical practice in South Australia with two full-time GPs (co-authors ZB & LP), where six years of EHR data were available at the commencement of analysis, providing approximately 70,000 prescriptions on 4,000 patients. The practice is the only one in its town and serves a population with significant agricultural and retiree segments. The practice has an interest in diabetes and cardiovascular disease. Accordingly, EHRs for patients with diabetes, cardiovascular disease and diabetes are relatively comprehensive. Diabetic coding was validated against the presence of antiglycemic medications, glucose test strips, and HbA1C. Thoroughness of coding of hypertension in the EHR was validated by correlation with prescribing of one or more antihypertensive drugs, disambiguating other uses of those drugs via patient notes. Coding was present in greater than 98% and 95% of indicated cases, respectively. Laboratory tests were ordered electronically and results returned to the EHR. Some coded observations, notably blood pressures (BPs), were also available.

State-Transition Model

As introduced in Warren et al.,12 a model of therapeutic state-transition can be formalized as follows.

The coverage period of a prescription is defined by the estimated duration of the prescription (time in days that the prescription should last the patient in light of dose, instructions, pack size and number of repeats) expressed as a date range.

Let us define the following:

  • n is the number of observed drug groups (i.e., number of therapeutic state variables);

  • m is the number of observed patients; and

  • p is the number of days in the analysis period.

Let tk denote the kth day of the analysis period, 1 ≤ kp.

Define gijtk as:

gijtk = 0, if the jth patient is not covered by a prescription from the ithdrug group at tk; gijtk = 1, otherwise where 1 ≤ in and 1 ≤ jm.

Let the value of the therapeutic state of patient j at tk, denoted by Sjtk , be the Boolean vector Sjtk=[g1jtkg2jtkgnjtk] One can see that there are 2n mathematically possible values of a patient's therapeutic state. It is useful to refer to patient j being in the “zero state” at time tk if gijtk=0 , i=1,2,,n .

Where the value of SjtkSjtk+1 , we say patient j has made a transition at time k+1. The sequence of state values held by a patient defines that patient's path.

While the EHR data is likely to provide a highly accurate record of GP prescribing decisions for the prescribers at the practice under consideration, there are a number of sources of uncertainty external to the database for interpretation of the data as real-world events:

  • Non-adherence. It is not certain that the patient has filled the prescription, or, if so, takes the medication as directed and for the full duration specified;

  • Other prescribers and drug supply sources. The patient may see another GP, a specialist, or get a prescription from a hospital. These possibilities are somewhat diminished in an isolated rural practice where there are not alternative GPs in the town. Moreover, the patient may receive drug supply as sample packets (sometimes GPs give drug company samples to patients rather than have them go to waste) or by using someone else's medication (e.g., his or her spouse's);

  • Implicit stop/change events. At any time, the GP may decide to stop a medication, which will not have a record in the EHR. Moreover, the EHR does not generally indicate if a new medication prescription is in addition to an ongoing one or represents a replacement; and

  • Scheduling vagaries. A patient may arrive some amount of time early or late to get a new prescription for ongoing therapy with no specific relationship to a therapeutic decision.

As such, the prescribing data does not provide an entirely crisp indication of therapeutic decisions or the patient's actual therapy. Several heuristics are helpful to make the raw state-transition data more interpretable, notably merging relatively short states into their neighboring states on the assumption that they are artifacts of the analysis (see details in Warren et al.13).

In addition to the heuristics, there is a question of how many patient transitions of a given type constitute the minimum to highlight for analysis (in particular, in a graphical state transition display). One school of thought is to include every single transition, but a therapeutic path that is not repeated is unlikely to represent GP intent or a common mis-operation of the system, and as such has minimal value to highlight for quality improvement. We find that displaying only transitions that have been repeated about 5 to 10 times (for our size of data set) is most useful. This sort of thresholding is analogous to a minimum level of support in association rule data mining.

Procedure

Six meetings of the analysis team and GPs (the author set) were convened over about ten months, with each meeting approximately two hours in duration. Initial focus was on agreement of scope and the identification of acceptable guidelines for the management of hypertension, customization thereof, and identification of state variables (in this case, largely affirming author experience from another general practice setting). Subsequent meetings centered on review of state-transition graphical displays. EHRs of patients from de-identified state-transition cohorts of interest were looked-up by the GPs and their clinical characteristics discussed by the group. Over the course of meetings the cohorts and therapeutic events constituting the best candidates for alerts were identified. Alert criteria were formulated with particular attention to screening false-negatives from the cohorts of interest, often entailing use of non-state-variable data (e.g., diagnoses and lab data). At the penultimate meeting of the group a set of alert criteria and appropriately phrased alert templates were agreed on for subsequent evaluation.

Six months of EHR data not previously reviewed by the group, (November 1, 2004, to April 30, 2005, the most recent data at the time of the analysis) was utilized to generate alerts by the agreed criteria on active patients with coded hypertension (with or without comorbidities). Patients without regular practice case ID numbers (i.e., visitors, such as tourists) were excluded. Thirty hypertensive patients with alerts and 30 hypertensive patients without alerts were randomly sampled and independently reviewed by the practice's two main GPs, each GP reviewing 20 alert and 20 non-alert cases (providing 10 alert and 10 non-alert cases for agreement assessment).

Each GP independently reviewed their 40 cases (presented in five packs of eight) in the presence of their practice management software (the interface used in consultations, providing access to the EHR) and with an agreed local (Australian Heart Foundation) guideline in hand.15 For each pack the GP completed an assessment of the quality of hypertension management of all eight patients (blind to whether the alert algorithm had identified any alerts for that case); then the GP completed an assessment of the relevance of the alerts found (see Appendix). The assessment of quality of hypertension management (“Form A” in the appendix) provides three progressively more restrictive criteria for assessment of the patient's management. The assessment of each alert found (“Form B”) gives the GP the choice to Agree with the alert (in which case they specify which action to take), to assess the alert as Not Useful, or to take a middle ground that the alert Has Merit but provide details of why they disagree with alerting for the case at hand.

Alerts were assessed in terms of sensitivity and specificity vis-à-vis the GP assessment. Inter-GP agreement is assessed on the jointly-viewed cases via the kappa statistic. A final debriefing meeting of the analysis team and GPs reviewed assessments made.

Results

Applying State Transitions

For the purposes of our analysis the team (the author set, which includes three physicians [JS, ZB, LP] and a pharmacist [GM]) considered a synthesis of Australian and international guidelines.1520 We abstracted six relevant groups (therapeutic state variables) of therapeutic agents for management of hypertension, defined in terms of Anatomical Therapeutic Chemical (ATC) classification.21 The importance of rigorous hypertension management in diabetes was of particular concern to the analysts and GPs. For this reason the drug groups are labelled to reflect their general order and preference for antihypertensive therapy in diabetes.

  • A: ACEi (Angiotensin Converting Enzyme inhibitors) and ARBs (Angiotensin Receptor Blockers) (ATC: C02EA)

  • B1: Beta-blockers (ATC: C07, especially discerning selective beta-blockers: C07AB)

  • B2: Diuretics (ATC: C03AA—thiazides and C03C—loop diuretics)

  • B3: Non-dihydropyridine calcium channel blockers (ATC:C08CX, C08DA01, C08DB01)

  • C: Dihydropyridine Calcium Channel Blockers (DCCBs; ATC: C08CA)

  • D: Alpha blockers, hydralazines and clonidine (ATC: C02DB, C02CA, C02AC)

We also include in the analysis combination products that represent more than one of the groups—ATC: C07BA, C07BB, C03AB, C08GA01, C09BA, C09BB, and C09DA.

The analysis team presented the GPs with a series of state-transition diagrams (graphics produced with GraphViz—see http://www.graphviz.org/) for various cohorts as defined on ranges of dates and coded diagnoses. Of particular interest was the prescribing data from March 2003 (when the current practice management software was introduced, earlier data having been converted from a previous system) for patients coded with hypertension. This was generally reviewed in two distinct sub-cohorts of those with a diabetes diagnosis (excluding gestational diabetes) and all other hypertensive patients.

Figure 1 shows state transitions on the most recent 20 months of prescribing at the time of review (March 2003 to November 2004) for the cohort of patients with hypertension but not diabetes. Init_Out is defined as a state where the patient has had no prescription for any of the six drug groups of interest in the EHR history, while still having encounters with the practice for at least 100 days (they are “initially out” of therapy with respect to hypertension); conversely, Init_in patients show antihypertensive prescriptions from their first 100 days of contact with the practice or within the first 100 days of the start of the six-year EHR history. Zero state represents a patient that had had at least one antihypertensive prescription in the EHR history and has subsequently exceeded the duration of that therapy by a significant period; if the patient shows no further antihypertensive prescribing in the EHR history they are transitioned to PracEnd (i.e., a state where we have no evidence of the patient returning to hypertensive management with the practice). Heuristics (as discussed in the State-Transition Model section of the Method) were applied, with various values discussed with the GPs for face validity. Heuristics applied for Figure 1 are such that states of less than 30 days' duration are absorbed into the prior state, except for Zero state which is absorbed if less than 90 days. Figure 1 shows only transitions that occurred at least 5 times (with one patient often repeating the same transition more than once); states are not shown unless they received or were the source of at least five transitions.

Figure 1

Therapeutic state-transition graph for antihypertensive prescribing in a 20-month period.

Deriving Alerts

State-transition diagrams were used in review with the GPs to identify areas for more in-depth investigation. For instance, from Figure 1 we see 12 distinct patients make the transition from ACEi-Diuretic combination therapy (or possibly ARB-Diuretic combination therapy) to the Zero state (no active antihypertensive drug prescriptions) without going through an intermediate monotherapy state (A, B2 → Zero transition). Since this is unlikely to represent intended best-practice, cases where this transition was present were reviewed in detail by the GPs with the analysis group to understand the causes. A number of explanations emerge, including:

  • Patient not compliant with the doctors' advice;

  • Using another source (or suspecting another source of) for medication supply—e.g., another practice, existing stockpile, providing manufacturer sample packs to the patient;

  • Personal intolerance to a specific drug group (e.g., ACEi intolerance) causing a prior combination to be simplified (this can be applicable for A, B2 → B2, but is less likely for A, B2 → Zero);

  • Technical misinterpretations (either an error in the analysis logic or the EHR not reflecting the full story).

Over a series of meetings, a spectrum of alert criteria for antihypertensive prescribing were agreed upon based on discussion driven by analysis of patient subgroups identified chiefly through review of state transitions. A number of drug-problem interactions independent of the state-transition paradigm were also identified as the basis for alerts as it was deemed difficult to discuss quality antihypertensive prescribing without consideration of major relevant comorbidities, such as asthma, where for example, beta-blockers are generally contraindicated. Drug-drug interaction arose as a desirable alert criteria area only insofar as it corresponded with therapeutic states (notably the B1, B3 state), or as a qualifier or surrogate in definition of drug-problem alert. No alerts based on dosing levels were developed. State-transition based alerts, once identified, were often further qualified to provide higher specificity by extending the criteria based on comorbidities and/or laboratory test results. In all cases, alert criteria were identified on a basis that there may be diverse reasons why a case fitting a criterion was, or was not, actually in need of a change in management, but that on balance the situation warranted an alert. That is to say, the alerts were not expected to have 100% specificity, but were expected to be both explicable and, overall, a good use of clinician time. Our working measure was the GP agreeing, “I'd be happy to have an alert under those circumstances.” Figure 2 illustrates the taxonomy of alerts identified.

Figure 2

Taxonomy of antihypertensive prescribing alerts used in study.

For each type of alert identified, one or more alert templates were developed and reviewed with the GPs to arrive at appropriate phrasing, including the specific supporting facts to include in the alert description. In many cases this produces several sub-types of the alert; for example, for a drug-problem interaction the drug may be prescribed when the problem has previously been already diagnosed, the diagnosis may be made while there is a current prescription of the contraindicated drug, or the diagnosis and prescription may be recorded on the same visit—yielding three distinct phrasings for the alert. A specific alert template—for a patient who is in the “C” state and has diabetes, on the sub-case that the patient entered the C state before the commencement of the investigation period—is as follows:

  • Alert date: 1 Nov 2004 (start of investigated period)

  • Check therapy (DCCB mono-therapy)

  • —Mono-therapy with DCCB commenced on Date1

  • —DCBB continues as mono-therapy into investigated period

  • —Diabetic patient

Notable in the above alert phrasing is that it is a statement of the key facts for the GP to assess the situation; therapeutic states are not referred to per se.

Assessment

As described in the Procedure section of the Method, the agreed alert criteria were applied for a six-month period subsequent to that previously reviewed by the GPs and analysis group. Equal-size samples of hypertension patients with alerts and without alerts were reviewed by the GPs via forms as per the Appendix.

The sampled population consisted of 611 cases in the practice EHR that showed as Active patients at the start of the six-month analysis period and had a coded diagnosis of hypertension. One or more alert criteria were found applicable for an alert date during the analysis period on 66 of the 611 cases. Thirty of the 66 cases were randomly sampled for GPs' review with a total of 37 alerts on those 30 cases. Thirty “control” patients were randomly sampled from the remaining 545 non-alert cases.

Table 1 shows the interaction of GP's assessment of the applicability of alerts versus whether an alert was actually generated. For cases where an alert was found, the GP's assessment is simply his or her assessment of the alert(s) (from his or her Form B). Fractions are generated for the cases with alerts because: (a) some cases are assessed by both GPs and the GPs disagree; and (b) some cases have multiple alerts. In each situation the case is assigned fractionally to the outcomes in the table. For control cases, the GP's assessment is taken as “Agree to alerting” if they answered in the negative to any question about the overall quality of hypertension management of the patient (from Form A), or as “Disagree” otherwise. One case was removed from the control group because the patient was identified by the assessing GP as not active with the practice despite having shown as such in the EHR.

View this table:
Table 1

GP Assessment Versus Alert Status of Patient

Alert
YesNo
GP assessment for alertingAgree14.55*
Has Merit10.250 (N/A)
Not Useful 5.2524*
  • * Agree/No cell represents a GP assessment of No to at least one of the quality of hypertension management questions (Form A of Appendix) and no alerts; Not Useful/No cell represents all quality questions answered in the affirmative and no alerts.

The Table 1 data presents the main story of GP alert assessment. Table 2 shows the breakdown of GP assessment per alert (which is not implicit in Table 1 because several patients have multiple alerts) and subdivided per assessing GP. Figure 3 shows the breakdown of GP alert assessment by alert type. Also, it is worth noting that in twelve cases for one GP and two for the other, the GP assessed the quality of hypertension management as good (affirmative to all three Form A questions) but assessed “Agree” (on Form B) to at least one of the alerts found.

View this table:
Table 2

GP Assessment Per Alert

GP 1GP 2Total
GP assessment for alertingAgree15 (60%)9 (37%)24 (49%)
Has Merit 9 (36%)8 (33%)17 (35%)
Not Useful 1 (4%)7 (29%) 8 (16%)
Figure 3

GP alert assessment distribution on alert categories.

The figures from Table 1 are used to compute sensitivity and specificity of the alert. Table 3 shows four different interpretations based on two factors. First, sensitivity and specificity can be computed with the GP's middle-ground assessment of the alert, “Has Merit,” grouped either with “Agree” or “Disagree” (acknowledging that the latter, more critical, assessment is most conventional). Second, the results are reported for the sample and also estimated for the population. For the population figures, sensitivity and specificity estimates for the sample are algebraically adjusted to account for the 8.26:1 prevalence (545 vs. 66) of non-alert cases. Confidence intervals are computed on the sample using the Wilson's binomial method (yielding asymmetric intervals about the observation where proportion estimates approach zero or one).

View this table:
Table 3

Sensitivity and Specificity Measures for Alerts by GP Assessment

Grouping “Has Merit” with “Agree”Grouping “Has Merit” with “Not Useful”
ObservedPopulation Estimate*ObservedPopulation Estimate*
Sensitivity83.19% (CI 66.19%–92.60%)37.48% (CI 19.17%–60.23%)74.36% (CI 52.17%–88.50%)25.99% (CI 11.67%–48.24%)
Specificity82.05% (CI 64.75%–91.92%)97.42% (CI 93.82%–98.95%)60.76% (CI 45.25%–74.38%)92.75% (CI 87.22%–96.00%)
  • * Based on alert cases 8.26 times more prevalent than non-alert cases.

Based on the 20 cases rated by both GPs, agreement on Form A question 1 (free of significant contraindications) was substantial (Kappa = 0.70) and agreement on question 2 (optimized or acceptable process) was moderate (Kappa = 0.50). Question 3 (guideline compliance) was always answered in the affirmative when it was answered. Agreement between the GPs on the assessment of alerts, based on the three-point Form B questionnaire, was only fair (Kappa = 0.37) with the doctors agreeing in seven out of 12 cases and one doctor rating more negatively than the other in the remaining five cases.

Debriefing

Assessment results were reviewed with the GPs, including cases with discrepancies in Form A to Form B assessment; sources of lost sensitivity; and perceptions of the technology and specific areas for enhancement.

The majority of the cases where the GPs assessed overall quality as good (using Form A) and subsequently agreed with at least one alert on the same case were accounted for by patients moving out of an effective combination and by drug-problem interactions. In the former case, the practice management system display, based on a chronological list of all prescriptions for a patient, is poorly suited to highlighting when one of many drugs prescribed to a chronic/complex patient has run out; so this class of suboptimal therapeutic state transition is easily missed even when the GP is explicitly examining the quality of antihypertensive prescribing. With respect to drug-problem interactions, misses usually occurred when the diagnosis was made some years in the past. Some of these latter situations should trigger an alert in the current practice management system, but these were possibly lost in the large number of low specificity alerts offered by typical commercial prescribing alert logic.

Sensitivity loss (failure to alert in situations assessed as suboptimal) was found to be due to four sources: (a) alerting only on thiazide diuretic/gout interaction when other diuretics were also assessed by the GPs as contraindicated in gout; (b) failure to alert on COX-2 inhibitor use; (c) failure to alert on persistently high blood pressure; and (d) assessor error (i.e., on further review in debriefing the therapy did not have the deficit identified in assessment).

The GPs are interested to have the alert logic taken into production use, feeling the alerts are on the whole a good use of clinician time. The alerts should first be extended to address the sensitivity deficits as described above. All of the types of alerts used in the evaluation are deemed desirable to keep.

Discussion

An increasingly computer-based operating environment in health care provides us with an increasingly rich pool of data for analysis, with the potential to use this data to improve subsequent practice. For community based systems, some of the most comprehensive available data is around the common action of prescribing. These data can provide us with a basis for representing the therapy applied to patients over time. The temporal patterns in the prescribing actions alone, therapeutic state transitions, provide a useful (although incomplete) model of practice behavior.

We have illustrated the use of therapeutic state transitions to empirically identify areas of practice that are good candidates for deeper analysis and further improvement. In particular, through an iterative process of review of state transitions with practitioners, we have identified a set of chronic disease management alerts in the area of antihypertensive prescribing for patients in the community. Assessment of the alerts by the practitioners, when applied to a period of six months of previously un-reviewed prescribing, shows specificity of the alerts to be high (61% specificity observed on an alert-weighted sample, estimated at 93% specificity for the diagnosed hypertensive population). The alerts demonstrate strong face validity, with only a 16% rate of individual alerts attracting a GP assessment of “Not Useful” (Table 2) and 97% specificity with respect to avoiding a Not Useful designation on a per-patient basis (Table 3). These characteristics indicate that the alerts have good potential to avoid being systematically ignored.

Comparison of the level of alert acceptance to the results found by Shah et al.6 is interesting. Shah et al. achieved 67% agreement with interruptive, high-severity alerts (life threatening or “potential for serious injury”) with most of the accepted alerts being for duplicate class contraindications. We observe a somewhat lower agreement rate of 49% for alerts derived through the state-transition based analysis process. Moreover, that rate is in a controlled setting, wherein time pressure to override the alert is minimal. On the other hand, the state-transition alerts are generally of a much lower severity—in fact, in all cases in the present study the patients with alerts had these alerts merely as a result of sub-optimal treatment (no severe adverse events were observed in these patients, all of whom are “regulars” of the practice). This suggests that the present study is uncovering a largely distinct “low-severity” alert category that achieves reasonable acceptance. Shah et al. did not investigate negative (non-alert) cases, so sensitivity and specificity cannot be compared. Although based on a relatively small sample, the sensitivity and specificity estimates constitute a particular contribution of the present study.

The most significant limitation of the present study is its small scale in terms of number of patients, doctors, and setting. Although the patient sample is small, the confidence intervals are sufficiently tight to support discussion of the quality of the alerts. Specificity (i.e., no alert when an alert is not warranted) is sufficiently high so that triggered alerts are not seen as “time wasters.” The identified sources of loss of sensitivity (an alert being generated when warranted) are instructive and highlight the types of limitations that are likely to emerge on replication of the analysis technique in other settings. First is the challenge of knowledge engineering in this domain. Several false-negative cases concerned the issue of diuretic/gout interaction. The precise scope for this class of alert criterion had been explicitly considered in the analysis sessions leading up to alert formalization; however, the assessment made by GPs in the evaluation context was different from the analysis sessions (i.e., that in retrospect, an alert was warranted). A second source of false-negatives was the COX-2/hypertension interaction where in retrospect the issue of the effect of COX-2 inhibition on renal prostoglandins was underrepresented and did in fact warrant an alert. This highlights the need for ongoing review and maintenance of knowledge bases as therapies and accepted medical knowledge evolve—with change outpacing the methodology in the present study (wherein we selected reference guidelines early in the procedure). False-negatives were also identified in relation to persistently high blood pressure readings. This demonstrates a fundamental limitation of pure therapeutic state transition modeling: although the therapeutic pathway appeared valid, it was not deemed appropriate when correlated with the BP observations.

The use of a single general practice context, with only two main practitioners, must be acknowledged. Resource limitations necessitated restricting the context to a single practice in order to ensure comprehensive local contextual assessment of the state transitions. With possible advances in the therapy for hypertension in the future it is not possible to know whether the current alert criteria will still be applicable, say, in several years in this same practice or whether the contextual evaluation of alert criteria in this GP practice is applicable to other practices. However, the authors believe that the general principle of alert criteria based on local adaptation of evidence based guidelines (which must also be based on evidence) is valid. The focus must be on the alert development process, which in practice must be ongoing, rather than a focus on the specific alerts. It is clear that the two GPs exhibited substantial individual differences in assessment of alerts (e.g., from Table 2 and the inter-rater agreement statistics); however, a reasonable aggregate fit was achieved. A larger sample of prescribing records, practices, and GPs would assist in determining the extent to which the set of alerts defined for this practice is applicable in other practice settings.

In terms of model, our research is most closely related to other methods of temporal abstraction of medical data.2224 However, our method is based on events (which drugs have been prescribed, and when should prescribed supply have run out). This stands in contrast to quantitative measurements that are readily characterized as parameters' values (high or low), gradient of change of parameters (increasing or decreasing) and rate of change of parameters (fast or slow) as in Shahar's KBTA (Knowledge-Based Temporal Abstraction).22

While the present study has focused on use of therapeutic state transition based analysis in the context of potentially interruptive alerts, the fact is that these alerts are largely aimed at long-term benefit, as compared to urgent safety issues. An alternative model for use may be in terms of an internal audit or review-and-recall process.

An ideal outcome of chronic disease management in the community is minimization of morbidity and mortality with maximized quality of life, and in particular minimized risk of adverse outcomes such as CVD events. There appear to be large gaps in cardiovascular disease management prescribing in the community, especially for the elderly.2527 Putnam et al.25 use the somewhat optimistic phrasing that “the use of evidence-based cardiovascular medications is rising and perhaps approaching reasonable levels for some drug classes.” Rafter et al.26 found that only 28% of patients with documented cardiovascular disease were receiving the combination of lipid lowering and blood pressure lowering medications. It seems clear that there is room for clinical decision support to tune this treatment process to great societal benefit if alerts can be formulated with sufficient specificity to not be ignored. A broader objective would be also to promote non-pharmacological chronic disease management actions in the community, such as those related to diet, exercise, and smoking cessation.

Conclusion

We provide a model and method for application of therapeutic state transition based analysis to identify locally justified areas for practice improvement, which we have evaluated in the context of management of hypertension in a general medical practice setting. The prescribing actions recorded in the databases of community based practice management systems (as are common in Australia, New Zealand, the UK, and elsewhere) show promise for supporting formulation of high-specificity alerts for suboptimal chronic disease management prescribing. In light of the significant burden of chronic illness, there are substantial economies to be gained from methods that optimize treatment by real-time decision support (as interruptive alerts, or for other means of follow-up) that are of sufficiently high specificity and face validity as to not be ignored.

Appendix

Embedded Image

Embedded Image

Embedded Image

Footnotes

  • This work was supported by Australian Research Council Discovery Project Grant DP0346054.

  • Special thanks to practice manager Cherie Cleland for her assistance with study logistics.

  • The study protocol was approved by the Human Research Ethics Committee of the University of South Australia (protocol P005-04) and undertaken with a Memorandum of Understanding with Lubims Pty. Ltd.

References

View Abstract