Clinical Performance Of An Automated Systemic Inflammatory Response Syndrome (SIRS) / Organ Dysfunction Alert: A System-Based Patient Safety Project

Robert A Raschke, MD, MS

Huw Owen-Reece, MBBS

Hargobind Khurana, MD 

Robert H Groves Jr, MD

Steven C Curry, MD

Mary Martin, PharmD

Brenda Stoffer

Suresh Uppalapu, MD 

Heemesh Seth, DO

Nithya Menon, MD 

Banner Good Samaritan Medical Center, Phoenix Arizona

Abstract

Objective: We have employed our electronic medical record (EMR) in an effort to identify patients at the onset of severe sepsis through an automated analysis that identifies simultaneous occurrence of systemic inflammatory response syndrome (SIRS) and organ dysfunction. The purpose of this study was to determine the positive predictive value of this alert for severe sepsis and other important outcomes in hospitalized adults.

Design: Prospective cohort.

Setting: Banner Good Samaritan Medical Center, Phoenix AZ

Patients: Forty adult inpatients who triggered alert logic within our EMR indicating simultaneous occurrence of SIRS and organ dysfunction.

Interventions: Interview of bedside nurse and chart review within six hours of alert firing to determine the clinical event that triggered each alert.

Results: Eleven of 40 patients (28%) had a major clinical event (immediately life-threatening illness) associated with the alert firing. Severe sepsis or septic shock accounted for four of these – yielding a positive predictive value of 0.10 (95%CI: 0.04-0.23) of the alert for detection of severe sepsis. The positive predictive value of the alert for detection of major clinical events was 0.28 (95%CI: 0.16-0.43), and for detecting either a major or minor clinical event was 0.45 (95%CI: 0.31-0.60). Twenty-two of 40 patients (55%) experienced a false alert.

Conclusions: Our first-generation SIRS/organ dysfunction alert has a low positive predictive value for severe sepsis, and generates many false alerts, but shows promise for the detection of acute clinical events that require immediate attention. We are currently investigating refinements of our automated alert system which we believe have potential to enhance patient safety.

Introduction

Severe sepsis is defined as systemic inflammatory response syndrome (SIRS) of infectious etiology with secondary organ dysfunction. It is estimated that 750,000 patients suffer severe sepsis annually in the United States - 3 cases per 1000 population (1). Mortality has fallen over the past several decades, but ranges from 20-30% in recent studies (1,3). Results of recent treatment trials for severe sepsis are consistent with the hypothesis that early diagnosis and treatment are important (2,3), but reliable systems for early recognition of severe sepsis in hospitalized patients are not widely available.  

We have sought to improve patient safety at our institution by using our integrated electronic medical record (EMR) to identify patients at the onset of severe sepsis through a logic algorithm that analyzes vital signs and laboratory data. This logic function identifies patients with simultaneous systemic inflammatory response syndrome (SIRS) and organ dysfunction, but cannot distinguish whether an acute infection is the cause of these findings. The purpose of this study was to determine what clinical events – infectious or non-infectious - actually cause the vital sign and laboratory changes that trigger this alert, and what the positive predictive valve of the alert is for detecting the onset of severe sepsis in hospitalized adult patients.

Methods

This was a prospective cohort study carried out at Banner Good Samaritan Medical Center – a 700-bed University-affiliated teaching hospital in Phoenix AZ. It was part of an ongoing quality improvement project and was thereby exempted from IRB approval. The SIRS / organ dysfunction alert logic was developed at Banner Health using Cerner Discern Expert^®, Cerner Corporation, North Kansas City MO, USA. The logic function monitored the EMR for standard SIRS criteria and laboratory evidence of organ dysfunction with thresholds consistent with standard definition of severe sepsis (Table 1) (4,5).

Table 1. Specific criteria for the logic function of our SIRS/organ failure alert. 

When any single criterion for SIRS was met, the program searched the prior 6 hours for the most recent vital signs, and the prior 30 hours for the most recent white blood cell count. If a second SIRS criterion was met, the program identified the patient as exhibiting SIRS, but did not trigger an alert. When any single laboratory criterion for organ dysfunction was met (table 1), the program identified the patient as suffering organ dysfunction. If criteria for SIRS and organ dysfunction overlap in any 8 hour window, the alert fired, triggering a real-time notification in the Cerner Millenium^® EMR alerting clinicians to the possibility of severe sepsis or septic shock. The alert has been in clinical application since 2010. 

We sampled 40 non-consecutive inpatients in the first three months of 2014 by a nonrandom method blinded to the patient’s clinical condition. On days of data collection, all alerts that had fired within the prior 6 hours were reviewed, regardless of patient location or diagnosis. The patient bedside was visited by a physician researcher during the six-hour window after alert firing and the nurse interviewed in order to determine the circumstances that caused the alert to fire. The patient might be briefly examined if necessary to confirm the nursing impression. Chart review was also performed to assist in this determination. Demographics, admission diagnosis, vital signs and laboratory data that triggered the alert logic, and any treatment the associated clinical event required were also recorded. Chart review was repeated 48 hours later to review microbiological test results and physician progress notes that might shed further light on the clinical event that triggered the alert.

The “clinical event” associated with each alert was defined as the most likely acute explanation for the vital sign and laboratory abnormalities that triggered the alert. A clinical event might be an acute illness, such as pneumonia with septic shock, or a non-illness event, such as initiation of dialysis. Clinical events could include the illness that necessitated admission if the alert fired within 24 hours of admission, or secondary illnesses - for instance, a catheter-associated blood stream infection.

The severity of clinical events related to alert firings were classified into three tiers. 

1. Major clinical events were acute life-threatening illnesses that required emergent resuscitation with any one or more of the following: >1 L intravenous fluid resuscitation, vasopressor infusion, >2 units of packed red blood cell transfusion, endotracheal intubation, advanced cardiac life support, or emergent surgical intervention.
2. Minor clinical events were acute non-life-threatening illnesses that required urgent treatments not included in the definition of major clinical events above.
3. False alerts were said to have occurred when no acute illness was recognized in temporal relationship to the alert firing. 

The positive predictive value of the alert for detecting severe sepsis, major clinical events, and major or minor clinical events were calculated, with 95% confidence intervals.

Results

Nineteen women and 21 men, with ages ranging from 22 to 103 years were included. Twenty-two of forty (55%) were in the ICU at the time the alert fired, and 18 on the floors. Vital signs and laboratory values that triggered the alert logic are listed in Table 2. 

Table 2. SIRS / organ dysfunction alert trigger criteria in forty patients. 

Eleven of 40 patients (28%) had a major clinical event associated with the alert firing – two of these occurred outside the ICU. Severe sepsis or septic shock accounted for four of these major clinical events – yielding a positive predictive value of 0.10 (95%CI: 0.04-0.23) of the alert for detection of severe sepsis or septic shock. The seven remaining patients with major events suffered acute pulmonary edema, pulmonary embolism, ischemic bowel, pancreatitis, acute cardiogenic shock, acute right heart failure secondary to pulmonary hypertension, and an incarcerated enteric hernia. The positive predictive value of the alert for detection of major clinical events was 0.28 (95%CI: 0.16-0.43).

Major clinical events were clearly recognized before the alert fired in nine of 11 cases, as evidenced by the patient having been admitted or transferred to the intensive care unit specifically for the event of interest, and/or having received treatment such as intubation or initiation of intravenous vasopressors before the alert fired. In two cases, the alert fired at about the same time that treatment of the acute clinical event commenced, and it was unclear what role it played in clinical recognition of the event.

Seven of 40 patients (17%) had a minor clinical event associated with the alert firing. These included two patients with anemia, and one each with hypotension from an antihypertensive medication, dialysis disequilibrium, post-operative pain, dehydration, and paroxysmal atrial fibrillation. The positive predictive value of the alert for detecting either a major or minor clinical event was 0.45 (95%CI: 0.31-0.60).

Twenty-two of 40 patients (55%) were not experiencing any identifiable acute illness that could explain the alert firing - these were considered false alerts. Aberrant vital signs triggered false alerts during dialysis (2), turning or sitting-up post-operative patients (2), an endoscopy procedure, and a family argument. Other false alerts were attributable to the pharmacological effect of calcium channel blocker, oximeter malfunction, error in vital sign entry, and widely discrepant blood pressures between right and left arms. The remaining false alerts were triggered by slightly abnormal vital signs with no identifiable cause.

Four patients (10%) did not survive to discharge – two had major clinical events, one a minor clinical event and one a false alert – in the later two cases, the cause of death was unrelated to the clinical event that triggered the alert.  

We examined alert triggering criteria to better understand how the discriminant ability of the alert might be improved. We noted that 15 of 40 (37%) alerts triggered with respiratory rates of 21 or 22 bpm, however these included six alerts associated with major clinical events. Twelve of 40 (30%) alerts triggered with heart rates in 91-95 bpm range, including two alerts associated with major clinical events. Laboratory results contributed to 31 of 40 alert firings – but in 12 cases they were stable or improving at the time they triggered the alert. In no case was a stable or improving laboratory value associated with a major clinical event.

Discussion

It’s important to study the effects of any quality improvement project in order to determine whether it is having the desired results. Our small pilot study suggests that our first-generation SIRS/organ dysfunction alert has a low positive predictive value for severe sepsis, and generates many false alerts. This is partially a reflection of the low specificity of SIRS criteria for sepsis (6). The high number of false positive alerts has led to alert-fatigue among physicians and nurses providing bedside patient care – a phenomenon which is not unique to our institution (7).  

Our alert demonstrated greater potential utility to detect acute clinical deterioration than to detect sepsis. Buck and colleagues (7) used an EMR-based logic system to activate a sepsis alert similar to ours, and observed similar results in that only 17% of alert patients had a sepsis-related discharge diagnosis, but 40% had a major illness which required urgent intervention. We have used the results of our study to re-task future iterations of our alert to detect acute clinical deterioration rather than sepsis.

Other researchers provide guidance in this regard. Vital sign and laboratory result criteria similar to the ones used in our study have been previously shown to predict in-hospital cardiac arrest (8), predict 30-day mortality (9), generate early warning scores to detect acute clinical deterioration (9), and activate medical emergency teams (8,10). A recent large study by Churpek and colleagues (11) validated a risk stratification tool that utilized vital signs, laboratory findings and demographics to predict the combined outcome of cardiac arrest, ICU transfer or death on the wards. The model yielded notable discriminant accuracy with an area under the receiver operating curve (AUROC) of 0.77.

We are currently investigating revisions in our alert logic to improve detection of acute clinical deterioration. The alert logic now trends laboratory values associated with organ dysfunction. We are studying whether adding a reflex serum lactate to the automatic alert response might help identify patients who are acutely deteriorating (12).   

Our study has many apparent weaknesses, but it should be noted that it was carried out originally only to provide data to help guide local efforts to improve patient safety. In this regard, it succeeded in guiding our (and perhaps other’s) future efforts in what will more likely be a useful direction.

We failed to clearly determine what role our automated alert played in bedside decision-making. In most cases, clinicians were already evaluating or treating the clinical event that triggered the alert before the alert fired. However, we feel that a safety net is a wise precaution even in a high-reliability system. It should also be noted that our institution has medicine and surgery residency teaching programs, a critical care fellowship, 24/7 in-house intensivist coverage, and remote video ICU coverage. The benefit of EMR-based automated alerts is likely to be amplified in less well-staffed institutions. Refined versions of EMR-based automated alerts, such as the ones we are currently investigating, have potential to enhance patient safety. 

References

1. Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303-10. [CrossRef] [PubMed]
2. Rivers E, Nguyen B, Havstad S, Ressler J, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345:1368-77. [CrossRef] [PubMed]
3. The ProCESS investigators. A randomized controlled trial of protocol-based care for early septic shock. N Engl J Med. 2014;370(18):1683-93. [CrossRef] [PubMed]
4. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, Cohen J, Opal SM, Vincent JL, Ramsay G. International Sepsis Definitions Conference. Crit Care Med. 2003;31(4):1250-6. [CrossRef] [PubMed]
5. Dellinger RP, Levy MM, Rhodes A, et al, Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580. [CrossRef] [PubMed]
6. Pittet D, Range-Frausto S, Tarara LN, et al. Systemic inflammatory response syndrome, sepsis, severe sepsis and septic shock: incidence, morbidities and outcomes in surgical ICU patients. Intensive Care Med. 1995;21:302-9. [CrossRef] [PubMed]
7. Buck KM. Developing an early sepsis alert program. J Nurs Care Qual. 2014;29(2):124-32. [CrossRef] [PubMed]
8. Hodgetts TJ, Kenward G, Ioannis G, et al. The identification of risk factors for cardiac arrest and formulation of activation criteria to alert a medical emergency team. Resuscitation. 2002;54:125-31. [CrossRef] [PubMed]
9. Goldhill DR, McNarry AF. Physiological abnormalities in early warning scores are related to mortality in adult inpatients. Br J Anaesth. 2004;92:882-4. [CrossRef] [PubMed]
10. Kenward G, Castle N, Hodgetts T, Shaikh L. Evaluation of a medical emergency team one year after implementation. Resuscitation. 2004;61:257-63. [CrossRef] [PubMed]
11. Churpek MM, Yuen TC, Winslow C, et al. Multicenter development of validation of a risk stratification tool for ward patients. Am J Respir Crit Care Med. 2014;190:649-55. [CrossRef] [PubMed]
12. Bakker J, Nijsten MWN, Jansen TC. Clinical use of the lactate monitoring in critically-ill patients. Ann Intensive Care. 2013;3:12-20. [CrossRef] [PubMed] 

Reference as: Raschke RA, Owen-Reece H, Khurana H, Groves RH Jr, Curry SC, Martin M, Stoffer B, Uppalapu S, Seth H, Menon N. Clinical performance of an automated systemic inflammatory response syndrome (sirs) / organ dysfunction alert: a system-based patient safety project. Southwest J Pulm Crit Care. 2014;9(4):223-9. doi: http://dx.doi.org/10.13175/swjpcc121-14 PDF