Design of an Electronic Medical Record (EMR)-Based Clinical Decision Support System to Alert Clinicians to the Onset of Severe Sepsis

Stephanie Fountain, MD

James Perry III, MD

Brenda Stoffer

Robert Raschke, MD

Banner University Medical Center Phoenix

Phoenix, AZ, USA

Abstract

Background: The aim of our study was to design an electronic medical record ­based alert system to detect the onset of severe sepsis with sensitivity and positive predictive value (PPV) above 50%.

Methods: The PPV for each of seven potential criteria for suspected infection (white blood cell count (WBCC) >12 or <4 x 10⁹ /L, immature granulocyte count >0.1 K/uL or immature granulocyte % >1%, temperature >38 C. or <36 C. or the initiation of broad­spectrum antibiotics) was determined by chart review of 160 consecutive patients who had evidence of organ system failure (as defined by standard criteria)plus at least one of the proposed criteria. Then, using only criteria with calculated PPV >50%, the charts of sixty consecutive patients who met CMS criteria for severe sepsis were reviewed to calculate the sensitivity of organ dysfunction plus any one of the suspected infection criteria.

Results: Four proposed criteria for suspected infection had PPV >50%: WBCC >12 x 10 9 /L (69%; 95%CI:53­84%), Temperature >38C. (84%; 95%CI:68­100%), Temperature <36C. (57% 95%CI:36­78%), and initiation of antibiotics (70% 95%CI:56­84%). These four criteria were present in 53/60 of the patients with severe sepsis by CMS criteria, yielding a sensitivity of 88.3% (95%CI: 80.2­96.4%). Alert criteria were satisfied before the onset of severe sepsis in 25/53 cases, and within 90 minutes afterwards in 28/53 cases.

Conclusions: Our criteria for suspected infection plus organ dysfunction yields reasonable sensitivity and PPV for the detection of severe sepsis in real­time.

Editor's Note: For accompanying editorial click here

Introduction

The American College of Chest Physicians and the Society for Critical Care Medicine define sepsis as a systemic inflammatory syndrome in response to infection and defined sepsis as “severe” when associated with acute organ dysfunction (1,2). The incidence of severe sepsis varies depending on the method of data abstraction from 300 to >1,000 per 100,000 person-years with an in-hospital mortality of 14.7% to 29.9% (3). Severe sepsis was estimated to cost U.S. healthcare system more than $24 billion in 2007 (4). The incidence and mortality of severe sepsis is expected to continue to rise (3-6).

Early recognition of severe sepsis and rapid implementation of standardized treatment bundles is associated with improved patient outcomes (7-12), but compliance rates with standardized time-sensitive treatment bundles for severe sepsis are generally in the 30% range (13). One reason may be that clinicians do not always recognize the onset of severe sepsis and therefore don’t have the opportunity to initiate all the elements required for bundle compliance in time. Therefore, a system that could alert providers to the onset of severe sepsis could help them achieve bundle compliance.

Clinical Decision Support Systems (CDSSs) use innovative software incorporated into electronic medical records (EMRs) to augment the awareness and expert knowledge of clinicians by providing pertinent and timely information at the point of care. CDSSs are adept at performing surveillance of electronic data to identify patients with vital signs and laboratory findings consistent with clinical deterioration. Several researchers have previously attempted to identify patients with severe sepsis in real-time with EMR-based CDSSs, but these systems suffered poor positive predictive value (PPV) and uncertain sensitivity (14,15). The PPV of a CDSS surveillance alert is important because it is inversely related to the proportion of false alerts. False alerts lead to clinician alert fatigue and subsequent disregard of alert recommendations (16,17).  High sensitivity is another important operating characteristic, but sensitivity is typically only achievable at the cost of reducing PPV. 

The goal of this pilot study was to develop criteria that could be used in a CDSS to identify patients at the onset of severe sepsis in real-time in order to alert clinicians. We chose to operationalize severe sepsis as organ system dysfunction due to infection, without requiring systemic inflammatory response syndrome, since a recent study that showed that the requirement of SIRS in the definition of severe sepsis excludes 1-in-8 patients suffering organ system dysfunction due to infection (18).  Organ dysfunction already has a standard definition based on laboratory results and vital signs (2) that are  discrete and easily extracted from the EMR by CDSS logic, but suspected infection does not.   Thus, a specific aim of this study is to determine optimal EMR-based criteria to define suspected infection in relation to the diagnosis of severe sepsis.  Our hypothesis was that we could identify a set of criteria for suspected infection which would have acceptable sensitivity and PPV for severe sepsis when combined with standard organ system dysfunction criteria.      

Methods

We chose seven potential criteria to identify suspected infection: the presence of a white blood cell count (WBCC) >12 x 10⁹/L or <4 x 10⁹/L, immature granulocyte count >0.1 K/uL or immature granulocyte % >1%, temperature >38 C. or <36 C. or the initiation of broad-spectrum antibiotics (piperacillin/tazobactam, third or fourth-generation cephalosporin, aminoglycoside, carbapenem, or vancomycin).  Organ system dysfunction was identified in the EMR as previously described and delineated in table 1.  

Table 1. Suspected infection and organ dysfunction criteria.

Our study occurred in two phases.  In the first, we tested individual criteria related to suspected infection in order to determine which had PPV >50% and were therefore incorporated into the second phase of the study.  In the second phase, we combined those accepted criteria for suspected infection with organ system dysfunction criteria and calculated the sensitivity for the diagnosis of severe sepsis as defined by Centers for Medicare and Medicaid (CMS).

Phase 1.  We used Cerner Discern^® to access clinical data in our Cerner Millennium^® EMR (Cerner Corporation, North Kansas City MO, USA) in order to identify a retrospective cohort of 160  Banner Health inpatients who satisfied any one of the seven potential suspected infection criteria plus one organ system dysfunction criteria (Table 1) within an eight-hour window.

The cohort consisted of four groups of forty patients each based on the type of suspected infection criteria present: abnormal WBCC, abnormal temperature, elevated immature granulocytes and initiation of antibiotics.  Patients were also selected so that half met criteria in the emergency department and half on the hospital wards.  Patient selection was otherwise consecutive. Chart reviews were performed by physician research staff to determine whether each patient was suffering the onset of severe sepsis at the time suspected infection and organ dysfunction criteria were satisfied. Such patients were considered to be true positive for the purposes of calculating PPVs. We decided a-priori that individual criteria that did not achieve at least 50% PPV would not be used in our final list of accepted criteria for suspected infection to be used in phase 2 of our study. We also compared PPV for each criteria between emergency department patients and inpatients.

Phase 2. The charts of sixty consecutive patients who met CMS criteria for severe sepsis in Banner Health were reviewed to calculate sensitivity of the combination of any one of the suspected infection criteria accepted in phase 1, plus one organ system dysfunction criteria occurring together within a six-hour window. The gold standard for the diagnosis of severe sepsis, and the time of onset of severe sepsis, were determined using CMS criteria by trained Banner Health data extraction staff for the primary purpose of regulatory reporting to CMS. The chart of each patient identified with severe sepsis by CMS methodology was reviewed to determine how many exhibited criteria for suspected infection and organ system dysfunction within 8 hours before, or 90 minutes after the onset of severe sepsis determined by CMS methodology.  [The rationale for this time window was that a hypothetical alert triggered by these criteria would only be valuable if it identified patients before, or shortly after the onset of severe sepsis].  We considered these to be true positive for the purposes of calculating sensitivity.

Results

Phase 1: PPVs with 95% confidence intervals for each of the potential criteria for suspected infection are listed in Table 2 below.

Table 2. PPV and 95% CI for individual suspected infection criteria (when found in temporal association with organ system dysfunction) for the clinical diagnosis of severe sepsis.

Only WBCC had a significantly different PPV when used in the emergency department vs the inpatient wards:  84% vs 50% (p=0.03).   

Immature granulocytes and WBCC <4 x 10⁹/L had PPV <50% and could be excluded from the set of accepted criteria with no loss of sensitivity. The set of accepted criteria include: WBCC >12 x 10⁹/L. temperature >38 or <36 and initiation of antibiotics. Finding any one of these accepted criteria in association with organ system dysfunction yielded a PPV of 70% (95%CI: 61-78%) for the diagnosis of severe sepsis.  

In 35/115 cases in which patients with one of these accepted criteria for suspected infection were not suffering an infection (false positive) the actual diagnoses included: cardiovascular diseases (s/p coronary artery bypass, myocardial infarction, cardiogenic shock), post-operative state, endocrinological disorders (hypothyroidism, diabetic ketoacidosis, adrenal failure), central nervous system pathology (intracranial hemorrhage, subarachnoid hemorrhage, seizure), obstetrical complications (placenta previa, spontaneous hemorrhage), and gastrointestinal hemorrhage.

Conclusions

Our data suggests that the best criteria set for suspected infection are likely to be: WBCC >12 x 10⁹/L, temperature >38 or <36 C. or initiation of broad spectrum antibiotics. The PPV of this set of criteria is likely to be >60%.  Leukopenia, and elevated immature granulocyte counts each had poor PPV and their exclusion would not significantly diminish the sensitivity of the set of criteria. 

Compared to other alert systems, this logic is novel for its abandonment of the use of SIRS criteria and the inclusion of antibiotic initiation. It could be argued that initiation of antibiotics should not be used to identify suspected infection because the clinician starting antibiotics is obviously already aware of infection.  However, unpublished analysis of 323 Banner health patients who qualified for severe sepsis by CMS criteria showed that 76% of those who failed bundle compliance received appropriate and timely antibiotics, but failed other important aspects of care, such as getting blood cultures before starting antibiotics and assessing lactate concentration. This suggests that a severe sepsis alert, triggering when a clinician enters an order for antibiotics could potentially assist the clinician in ordering other bundle elements. Exclusion of antibiotic initiation from our accepted criteria would have reduced the sensitivity of our alert logic to 75%.

The operating characteristics of our CDSS compares favorably to four previously published severe sepsis surveillance CDSSs which utilized SIRS criteria (see table 3 below). 

Table 3. Operating characteristics of CDSSs designed to provide surveillance for severe sepsis.

One of the strengths of this alert logic is that is it widely generalizable. It only includes data that is collected on most, if not all, hospitalized patients. It does not require additional tests or measurements that may limit its utility to a smaller patient population. It does not require physicians or ancillary staff to perform additional tasks or deviate from their standard workflow. Another strength of this logic is that it was created within the software program Cerner Discern^® in our Cerner Millennium^® EMR, one of the most widely used EMRs across the country. This would potentially allow seamless integration into any hospital system using this software, improving patient care and fulfilling “meaningful use” mandate of the Affordable Care Act.  However, our study is only a small pilot study. These results will need further validation using a larger data set. Further studies are needed to show whether a CDSS using these criteria can improve clinical outcomes of patients with severe sepsis.

Refenences

1. Geroulanos S, Douka ET. Historical perspective of the word "sepsis". Intensive Care Med. 2006 Dec;32(12):2077. [CrossRef] [PubMed]
2. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, Schein RM, Sibbald WJ. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992 Jun;101(6):1644-55. [CrossRef] [PubMed]
3. Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med. 2013 May;41(5):1167-74. [CrossRef] [PubMed]
4. Lagu T, Rothberg MB, Shieh MS, Pekow PS, Steingrub JS, Lindenauer PK. Hospitalizations, costs, and outcomes of severe sepsis in the United States 2003 to 2007. Crit Care Med. 2012 Mar;40(3):754-61. [CrossRef] [PubMed]
5. Dombrovskiy VY, Martin AA, Sunderram J, Paz HL. Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med. 2007 May;35(5):1244-50. [Pubmed]
6. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001 Jul;29(7):1303-10. [CrossRef] [PubMed]
7. Miller RR 3rd, Dong L, Nelson NC, Brown SM, Kuttler KG, Probst DR, Allen TL, Clemmer TP; Intermountain Healthcare Intensive Medicine Clinical Program. Multicenter implementation of a severe sepsis and septic shock treatment bundle. Am J Respir Crit Care Med. 2013 Jul 1;188(1):77-82. [CrossRef] [PubMed]
8. Levy MM, Rhodes A, Phillips GS, Townsend SR, Schorr CA, Beale R, Osborn T, Lemeshow S, Chiche JD, Artigas A, Dellinger RP. Surviving Sepsis Campaign: association between performance metrics and outcomes in a 7.5-year study. Intensive Care Med. 2014 Nov;40(11):1623-33. [CrossRef] [PubMed]
9. Levy MM, Dellinger RP, Townsend SR, Linde-Zwirble WT, Marshall JC, Bion J, Schorr C, Artigas A, Ramsay G, Beale R, Parker MM, Gerlach H, Reinhart K, Silva E, Harvey M, Regan S, Angus DC. The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med. 2010 Feb;36(2):222-31. [CrossRef] [PubMed]
10. Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med. 2013 Feb;39(2):165-228. [CrossRef] [PubMed]
11. Levy MM, Pronovost PJ, Dellinger RP, Townsend S, Resar RK, Clemmer TP, Ramsay G. Sepsis change bundles: converting guidelines into meaningful change in behavior and clinical outcome. Crit Care Med. 2004 Nov;32(11 Suppl):S595-7. [CrossRef] [PubMed]
12. ProCESS Investigators, Yealy DM, Kellum JA, Huang DT, Barnato AE, Weissfeld LA, Pike F, Terndrup T, Wang HE, Hou PC, LoVecchio F, Filbin MR, Shapiro NI, Angus DC. A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014 May 1;370(18):1683-93. [CrossRef] [PubMed]
13. Gao F, Melody T, Daniels DF, Giles S, Fox S. The impact of compliance with 6-hour and 24-hour sepsis bundles on hospital mortality in patients with severe sepsis: a prospective observational study. Crit Care. 2005;9(6):R764-70. [CrossRef] [PubMed]
14. Tafelski S, Nachtigall I, Deja M, Tamarkin A, Trefzer T, Halle E, Wernecke KD, Spies C. Computer-assisted decision support for changing practice in severe sepsis and septic shock. J Int Med Res. 2010 Sep-Oct;38(5):1605-16. [CrossRef] [PubMed]
15. Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, French B, Fuchs BD. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med. 2015 Jan;10(1):26-31. [CrossRef] [PubMed]
16. Guidi JL, Clark K, Upton MT, et al. Clinician Perception of the Effectiveness of an Automated Early Warning and Response System for Sepsis in an Academic Medical Center. Ann Am Thorac Soc. 2015 Oct;12(10):1514-9. [CrossRef] [PubMed]
17. Shapiro NI, Howell MD, Talmor D, et al. Implementation and outcomes of the Multiple Urgent Sepsis Therapies (MUST) protocol. Crit Care Med. 2006 Apr;34(4):1025-32. [CrossRef] [PubMed]
18. Kaukonen KM, Bailey M, Pilcher D, Cooper DJ, Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med. 2015 Apr 23;372(17):1629-38. [CrossRef] [PubMed]
19. Alsolamy S, Al Salamah M, Al Thagafi M, et al. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak. 2014 Dec 5;14:105. [CrossRef] [PubMed]
20. Amland RC, Lyons JJ, Greene TL, Haley JM. A two-stage clinical decision support system for early recognition and stratification of patients with sepsis: an observational cohort study. JRSM Open. 2015 Oct 8;6(10):2054270415609004. [CrossRef] [PubMed]
21. Semler MW, Weavind L, Hooper MH, et al. An Electronic Tool for the Evaluation and Treatment of Sepsis in the ICU: A Randomized Controlled Trial. Crit Care Med. 2015 Aug;43(8):1595-602. [CrossRef] [PubMed]
22. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011 May;57(5):500-4. [CrossRef] [PubMed]

Cite as: Fountain S, Perry J III, Stoffer B, Raschke R. Design of an electronic medical record (EMR)-based clinical decision support system to alert clinicians to the onset of severe sepsis. Southwest J Pulm Crit Care. 2016 Apr;12(4):153-60. doi: http://dx.doi.org/10.13175/swjpcc021-16 PDF

Comments

[Rick Robbins, M.D.]

The Washington Post reports that the Centers for Disease Control and Prevention (CDC) has declared sepsis a medical emergency, reporting Tuesday that about 72 percent of patients with this fast-moving and deadly illness have recently been seen by doctors and nurses, representing missed opportunities to catch it early or prevent it. However, there was no mention in the Washington Post article on how the CDC proposed to identify sepsis. The above article by Fountain et al. represents an opportunity to identify sepsis early in its course.