Journal of Intensive and Critical Care Open Access

Keywords

Sepsis; Documentation; Clinical decision support systems; Patient monitoring; Heathcare quality indicators; Outcome and process assessment

Introduction

Sepsis Syndromes (sepsis, severe sepsis, cryptic septic shock, and septic shock) are major causes of morbidity and mortality for hospitalized patients. Approximately, 750,000 cases of severe sepsis occur each year with a mortality rate estimated at 30 to 50% [1,2]. Multiple evidence-based interventions have been developed to help improve sepsis outcomes. To facilitate their implementation, evidence-based practice guidelines from the Severe Sepsis Campaign recommend several interventions in the first 6 h, the sepsis resuscitation bundle, and other interventions within 24 h, the sepsis management bundle [3]. The sepsis resuscitation bundle includes serum lactate measurement to detect shock and risk-stratify patients [4-7], drawing blood cultures followed by administering appropriate antibiotics [5,8-10] and administering early and adequate intravenous fluids for signs of end-organ hypoperfusion [3,4,6,11-13]. These interventions are time-sensitive and must be instituted early to achieve better outcomes [3,9,11,12].

Early recognition of sepsis syndromes in hospitalized patients is challenging. The diagnosis relies on the detection of the systemic inflammatory response syndrome (SIRS) in patients with a suspected or proven infection [14]. In addition, severe sepsis requires recognition of new organ dysfunction. Regrettably, SIRS is overly sensitive, and not specific in the patient populations at greatest risk for developing sepsis. In addition, many hospitalized patients have baseline end-organ dysfunction, wherein subtle trends in laboratory abnormalities may escape detection. Last, detection may be compromised in current clinical care settings with increasing reliance on shift work [15,16]. Routine screening of all hospitalized patients for sepsis represents one strategy for trying to capture sepsis cases early. However, this is time consuming and requires extra effort to screen all patients to benefit only a few. In contrast, electronic detection and alert systems offer a more focused and efficient methodology. This strategy has been deployed successfully in other aspects of critical care, e.g. in the early recognition of acute lung injury (ALI) [17].

Given the concerns about the difficulty in recognizing sepsis patients and the time-sensitive nature of early sepsis care, we developed an electronic recognition and alert system to identify floor patients with sepsis syndromes. The goal of this study was to evaluate the positive predictive value of the new automated system in identifying sepsis cases. We also sought to determine the potential efficacy of the system by examining clinician compliance, when blinded to the alerts, with the evidence-based bundle for early sepsis management at the time the system identified the patient.

Methods

Setting

This study was performed at the Hospital of the University of Pennsylvania. This study protocol was reviewed and approved by the Institutional Review Board of the University of Pennsylvania with a waiver for the requirement of written informed consent from the participating subjects or their legally authorized representatives.

Sepsis detection and alert algorithm

An automated computer algorithm was designed to detect hospitalized patients with sepsis and provide additional information about associated end-organ dysfunction to identify when sepsis is severe. Since vital signs were not available electronically at the time of this study, to identify SIRS we included the WBC as one SIRS criterion. A blood culture order was included as a surrogate criterion for both fever (or hypothermia) and the suspicion for infection. In effect, the combination of the blood culture order and the WBC criteria was used to define sepsis in accordance with the established definition; i.e. suspected infection in the presence of two or more SIRS criteria [14,18].

Specifically, the system first scans the hospitals information system (Medview) for the presence of a blood culture order. It then looks for either leukocytosis (WBC>12,000/mm³) or leukopenia (WBC<4) in the 24 h before or after the time of the blood culture order. Then, to classify the severity of sepsis, the system screens for organ dysfunction by looking for abnormalities in lactic acid, partial pressure of arterial oxygen (PO₂ ), glucose, bilirubin, INR, creatinine, platelets, and PTT using established values for organ dysfunction criteria [14,18]. Of note, severe sepsis is defined by the onset of new organ dysfunction. Since baseline comorbidities are relatively common in hospitalized patients, the system compares the current values to the most recent ones, looking for a predefined change to indicate that the organ dysfunction is new. The software algorithm then automatically e-mails an alert to the study coordinator for patients who meet these criteria. Each of these data elements is included in the e-mailed sepsisalert. The algorithm is setup to create an alert only on patients who are not in the intensive care unit. It generates an alert at the first evidence of sepsis during a hospitalization, but not more than once. Clinicians were blinded to the alert. As a result, their management decisions were unaffected by whether the sepsis alert had been triggered.

Study design and data collection

This was a retrospective cohort study of 97 consecutive patients identified by the sepsis detection and alert computer algorithm from October 14 through October 27, 2011. For each patient for whom an alert was generated, reviewers manually evaluated each patient’s chart and recorded the outcome and exposure variable along with demographic information. Two independent data collectors determined if a sepsis syndrome was present for each patient. Study data were collected and managed using REDCap electronic data capture tools hosted at the University of Pennsylvania [19].

The primary objective of the study was to determine the positive predictive value of the computer generated sepsis alert. The secondary objective was to evaluate the potential value of the alert by first assessing, at the time of the alert, clinician adherence to the early sepsis treatment bundle that included drawing of a lactic acid, early administration of new broadspectrum antibiotics, and aggressive (≥ 500 cc IVF bolus) fluid administration when end-organ hypoperfusion was present. Second, we assessed for the occurrence of a serious adverse outcome that occurred following the alert, to assess whether earlier sepsis recognition could potentially improve patient care.

Determination of Positive Predictive Value (Ppv) for Sepsis

Data and definitions

Baseline characteristics were recorded from each patient and included race, hospital service, presence of SIRS criteria, and comorbidities. Sepsis was defined as suspected or proven infection in the presence of two or more SIRS criteria. Suspected infection was defined by the notation in the patient’s chart that either the provider had suspected infection or had ordered an intervention for the management of an infection (e.g. antibiotics) in the absence of an alternative diagnosis. Severe sepsis was defined as sepsis associated with organ dysfunction, hypoperfusion or hypotension [14,18]. Hypoperfusion was primarily defined as a serum lactate of ≥ 2 mmol/L. Definitions of hypotension and end organ dysfunction was based on the 2001 International Sepsis Definition Conference criteria [18] with the exception of hepatic dysfunction which was defined as bilirubin >2.0 mg/dL and an elevation above the patients baseline [4]. Septic shock was defined as hypotension (systolic blood pressure <90 mmHg) despite adequate fluid resuscitation (≥ 1500 mL) or the use of vasoactive agents [18]. To determine whether sepsis was recognized, we reviewed clinician documentation. Appropriate documentation of sepsis was defined by the presence of a term defining the proper syndrome (sepsis, severe sepsis, cryptic septic shock, or septic shock) documented in a clinician’s progress note at the time of or in the next progress note that followed the sepsis alert. Severe sepsis was also considered documented appropriately if sepsis was documented in the progress notes with the additional documentation of a new organ dysfunction within the same time frame [20].

Assessment of Potential Value of the Alert

In order for the sepsis alert system to have clinical value, it must fire prior to the initiation of evidenced-based sepsis interventions and occur prior to a serious adverse outcome. An intervention was considered performed if it occurred within the previous 24 h before (unless otherwise stated), and up to 6 h after, the time of the sepsis alert [3,21]. The interventions included drawing a serum lactate, administering a new broad-spectrum antibiotic (up to 48 h before, and 6 h after the alert, and administering a fluid bolus of ≥ 500 mL of fluid over a 30 min time period. For all patients with a sepsis syndrome, serum lactic was considered indicated based on the sepsis campaign practice guidelines [18]. For all patients with severe sepsis or septic shock, new broadspectrum antibiotics were considered indicated. For all patients with circulatory dysfunction, as evidenced by lactic acidosis, acute kidney injury, or hypotension, a fluid challenge (>500 cc IVF bolus) was considered indicated.

We assessed for serious adverse outcomes that occurred within the subsequent 48 h after the alert, including any rapid responses, codes, ICU transfers or deaths. In addition, we included in our list of important outcomes, mortality occurring any time following the alert.

Statistical Analysis

Concordance between the two independent reviewers was evaluated using the kappa statistic. Positive predictive value was calculated as the proportion of actual cases of sepsis within those patients for whom an alert was generated. No other values of alert accuracy could be determined due the manner in which the cohort of patients was derived. A complete assessment of accuracy would have required chart review of all hospital discharges to identify all cases of sepsis, which was beyond the scope of this investigation. Chi squared statistic or Fisher’s exact test were used to compare categorical variables. Student’s t test was performed for continuous variables. All tests were two-tailed and a p value of <0.05 was considered statistically significant. Statistical analysis was performed using Stata 11.2 software package.

Results

Patients

In ninety-seven patients a sepsis alert was generated by the computer algorithm during the study period. Of these 97 patients, 48 had sepsis only, 24 had severe sepsis (including 1 patient with septic shock) and 25 failed to meet criteria for any sepsis syndrome. The positive predictive value of the computer algorithm for determining the presence of a sepsis syndrome was 74% (95% CI=64% to 83%). Of all the patients with a sepsis syndrome, 21% (25% with sepsis and 13% with severe sepsis) were documented appropriately. Of the 25 patients that did not meet criteria for a sepsis syndrome (false +), 52% had less than two SIRS criteria and in 68% there was evidence of an alternative clinical diagnosis.

Baseline characteristics

The baseline characteristics of the cohort demonstrated a mean age of 53, with 63% of patients being white and 24% African American (Table 1). The majority (76%) of the patients were on the medicine service with 9/20 (45%) of those patients being oncology patients. The sources of infection included pneumonia (19%), urogenital (17%), skin/soft tissue/wound (2%), intraabdominal (9%), catheter/device-related (8%), bacteremia (10%), and unknown (25%). The baseline characteristics of the patients identified with a confirmed sepsis syndrome, including both sepsis and severe sepsis, are shown in Table 1.

Outcomes

The proportion of cases where there were missed clinical opportunities identified at the time of the sepsis alert are shown in Table 2, categorized by whether or not they were documented as sepsis appropriately in the chart. In 56% of cases with a sepsis syndrome a serum lactate was not measured and some aspect of the care bundle was not initiated in 65% of cases. The indicated interventions were performed significantly more likely in sepsis cases that were documented appropriately (67% vs. 28% for lactate; p=0.004, and 78% vs. 28% for the whole bundle, p<0.001). The data presented in Tables 3 and 4 suggest that adverse outcomes occurred more commonly in septic patients that were not documented appropriately.

Discussion

This study demonstrated that an automated sepsis detection and alert system can identify sepsis in hospitalized, non-ICU patients with a positive predictive value of 74%. We found that this system identified the presence of a sepsis syndrome often before treating clinicians documented it. Since we found a signal that adherence with the sepsis bundle at the time of the alert was low, our data suggests that the alert may have value to improve sepsis care. In support of this hypothesis is the finding that serious adverse outcomes occurring after the alert were more likely in patients where sepsis was not documented.

Table 1 Baseline demographics.

All Patients with Sepsis (including severe sepsis)

Table 2: Missed clinical opportunities identified at the time of the sepsis alert stratified by whether sepsis was documented appropriately.

Table 3 Adverse outcomes of patients identified by the sepsis alert system, stratified by whether sepsis was documented appropriately

There have been several prior studies that attempted to use electronic surveillance to diagnosis sepsis in a variety of venues. These studies have reported a positive predictive value that ranged from 19.5% to 54%. Nelson et al. used a real-time electronic surveillance algorithm in the emergency room that screened for SIRS and hypotension and found a 54% positive predictive value. The authors noted that their system usually failed to identify the patient before the clinician [22]. Meurer et al. reported on an electronic sepsis screen of geriatric patients in the emergency room that screened for SIRS and sent an automatic page to clinicians. It had a sensitivity of 36% and a specificity of 78%. In this study, they did not report a positive predictive value, however, based on the data presented it would be approximately 48% [23]. The positive predictive value of the screening systems in both of these studies was lower than ours. The screening systems in these studies differed from our algorithm in two major respects. First, both screened SIRS criteria, whereas our system was created to function without the use of vital signs. The most likely explanation for the higher PPV in our study is that the requirement for a blood culture order provides a more specific context than vital signs. Other than sepsis, few conditions are associated with a blood culture order, whereas many conditions cause vital sign abnormalities. In addition, in the study by Nelson et al they found that clinicians often identified sepsis before the alert system. This difference is most likely due to the ED setting where every patient in this study was undergoing rapid initial evaluation, while the ward patients in our study were not.

Computerized sepsis screening algorithms have also been utilized in the intensive care unit setting. Herasevich et al developed a computerized septic shock sniffer for the ICU, which employed a complex algorithm screening for SIRS criteria, microbiology data, and hypoperfusion. The positive predictive value reported was 34%, much lower than our study [24]. Although the specificity would be expected to be higher using microbiological data, the shock screening criteria likely negated any favorable effect on specificity, given the many diagnoses-other than sepsis-that cause hypoperfusion in an ICU population. Furthermore, this ICU sepsis screening system was designed as a research study screening method, not as a clinical tool to improve the care process or patient outcomes.

Kollef et al. was the first to report on the development of an electronic sepsis detection system for floor patients [25,26]. Our algorithm performs well compared to theirs. Their sepsis detection algorithm was designed using a statistical model that used laboratory values taken retrospectively from a cohort of known sepsis patients. When their system was tested in a validation cohort the positive predictive value was between 19.5 and 21.4% depending on which year they used to validate the cohort, which is much lower than ours [27]. Another advantage of our system is that it distinguishes sepsis severity by evaluating trends in laboratory values to identify acute organ dysfunction. Furthermore, our system may be more generalizable to other health care institutions since the algorithm was designed based on standardized sepsis criteria, in contrast to their unique criteria specific to the ward population of a single institution.

Table 4: Adverse outcomes of the severe sepsis patients identified by the sepsis alert system stratified by whether sepsis was documented appropriately.

We found that in the majority (80%) of the sepsis cases detected, sepsis was not documented by providers at the time the alert fired. Clearly, documentation is not a reliable perfect substitute for physician recognition; however, trends in the data-though not statistically significant-suggest that the lack of documentation was associated with non-compliance with the sepsis care bundle. This does provide support for the notion that a sepsis alert system that can improve patient recognition will improve sepsis management. By that standard, a sepsis alert may improve patient management.

We have also shown that the sepsis detection system provides alerts before early sepsis management occurs. Sepsis management is time-sensitive. Most evidence-based recommendations require rapid implementation to influence most significantly patients’ outcomes [3,9,12]. Management delays due to lack of recognition can be detrimental. We have shown that in 56% of the sepsis patients identified by the system, serum lactate had not been obtained to help stratify sepsis severity [4]. In addition, serum lactate was not measured in a significantly higher percentage of patients who did not have their sepsis syndrome documented appropriately.

Although these results suggest that the sepsis detection and alert system has the potential to provide for earlier, appropriate recognition and treatment of sepsis, for the system to truly make a difference it must ultimately improve patient outcomes. This study was not designed to evaluate whether or not the system could improve outcomes; however, it provided insight into adverse events in septic patients and how documentation and recognition might influence outcomes. Many would assume that those recognized would be more severely ill as they would be easier to identify. Our findings suggest that recognition and intervention might be more important than perceived acuity. This relationship needs to be studied in a larger cohort to see if this association becomes significant. The trend toward significance, however, supports the notion that this detection system could ultimately improve patient outcomes.

This study has multiple limitations. First, the algorithm is limited because of its lack of vital signs and its reliance on a blood culture order as a surrogate for fever. If informed by better electronic record systems including vital signs, the effectiveness of the algorithm is likely to improve. Of note, however, the inclusion of the blood culture order makes the SIRS designation more specific to sepsis, a feature absent from other early warning systems, which use vital signs to detect clinical deterioration. In addition, this more simplified system may be used now by institutions that do not yet have vital signs available electronically. Second, the study took place in a single center. While lack of recognition is likely a general problem, the detection system needs to be evaluated in different clinical environments. In addition, while documentation is not a true surrogate for clinician recognition, it is the best proxy we have, short of prospective survey, to assess physician’s thinking. Third, the study design did not allow us to evaluate what cases of sepsis were missed by the algorithm. Nevertheless, this study did suggest that the algorithm could increase detection. Fourth, we did not test the outcomes of the intervention in clinical practice-an evaluation that might reveal the incremental benefits and risks of deploying the system in usual settings.

In conclusion, despite several limitations, this study demonstrated that an automated sepsis detection and alert system was easily implemented and reasonably accurate in the real-time identification of septic patients. It provided these alerts before interventions of the sepsis bundle could be implemented leading to adverse outcomes. This system should improve patient care by identifying under-recognized septic patients and by alerting clinicians to provide aspects of sepsis management that are timesensitive and may otherwise not be instituted.

References

1. Levinson AT, Casserly BP, Levy MM (2011) Reducing mortality in severe sepsis and septic shock. Seminars in respiratory and critical care medicine. 32: 195-205.
2. MacRedmond R, Hollohan K, Stenstrom R, Nebre R, Jaswal D, et al. (2010) Introduction of a comprehensive management protocol for severe sepsis is associated with sustained improvements in timeliness of care and survival. Quality and Safety in Health Care 19: 1-7.
3. Dellinger RP, Levy MM, Carlet JM, Bion J, Parker MM, et al. (2008) Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2008. Critical care medicine 36: 296-327.
4. Mikkelsen ME, Miltiades AN, Gaieski DF, Goyal M, Fuchs BD, et al. (2009) Serum lactate is associated with mortality in severe sepsis independent of organ failure and shock. Critical care medicine 37: 1670-1677.
5. Jones AE,Puskarich MA (2009)Is lactate the "Holy Grail" of biomarkers for sepsis prognosis? Critical care medicine 37: 1812-1813.
6. Jansen TC, van Bommel J, Bakker J (2009) Blood lactate monitoring in critically ill patients: a systematic health technology assessment. Critical care medicine 37: 2827-2839.
7. Howell MD (2009) Lactate: finally ready for prime time? Critical care medicine 37: 2858-2859.
8. Levy MM, Pronovost PJ, Dellinger RP, Townsend S, Resar RK, et al. (2004) Sepsis change bundles: converting guidelines into meaningful change in behavior and clinical outcome. Critical care medicine 32: S595-S597.
9. Daniels R (2011) Surviving the first hours in sepsis: getting the basics right (an intensivist's perspective). Journal of Antimicrobial Chemotherapy 66: ii11-ii23.
10. Chamberlain DJ, Willis EM, Bersten AB (2011)The severe sepsis bundles as processes of care: A meta-analysis. Australian Critical Care 24: 229-243.
11. Nguyen HB, Jaehne AK, Jayaprakash N, Semler MW, Hegab S, et al. (2001) Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 345: 1368-1377.
12. Funk D, Sebat F, Kumar A (2009)A systems approach to the early recognition and rapid administration of best practice therapy in sepsis and septic shock. Current opinion in critical care15: 301-307.
13. Bozza FA, Carnevale R, Japiassú AM, Castro-Faria-Neto HC, Angus DC, et al. (2010) Early fluid resuscitation in sepsis: evidence and perspectives. Shock 1: 40-43.
14. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, et al. (1992) Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest 101: 1644-1655.
15. Jones GR (1998) Assessment criteria in identifying the sick sepsis patient. Journal of Infection 1: 24-29.
16. Chang VY, Arora VM, Lev-Ari S, D'Arcy M, Keysar B (2010) Interns Overestimate the Effectiveness of Their Hand-off Communication. Pediatrics 125: 491-496.
17. Azzam HC, Khalsa SS, Urbani R, Shah CV, Christie JD, et al. (2009) Validation study of an automated electronic acute lung injury screening tool. J Am Med Inform Assoc 16: 503-508.
18. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, et al. (2003) 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Critical care medicine 31: 1250-1256.
19. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, et al. (2009) Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 42: 377-381.
20. Association AH (2008) Coding ClinicFourth Quarter 2008. Coding Clinic Alphabetical Index-AHA Central.
21. Nguyen HB, Kuan WS, Batech M, Shrikhande P, Mahadevan M, et al. (2011) Outcome effectiveness of the severe sepsis resuscitation bundle with addition of lactate clearance as a bundle item: a multi-national evaluation. Critical Care 15: R229.
22. Nelson JL, Smith BL, Jared JD, Younger JG (2011) Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 57: 500-504.
23. Meurer WJ, Smith BL, Losman ED, Sherman D, Yaksich JD, et al. (2009) Real-Time Identification of Serious Infection in Geriatric Patients Using Clinical Information System Surveillance. Journal of the American Geriatrics Society 57: 40-45.
24. Herasevich V, Pieper MS, Pulido J, Gajic O (2011) Enrollment into a time sensitive clinical study in the critical care setting: results from computerized septic shock sniffer implementation. J Am Med Inform Assoc 18: 639-644.
25. Micek ST, Roubinian N, Heuring T, Bode M, Williams J, et al. (2006) Before-after study of a standardized hospital order set for the management of septic shock. Critical care medicine 34: 2707-2713.
26. Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, et al. (2010) Early prediction of septic shock in hospitalized patients. Journal of hospital medicine: an official publication of the Society of Hospital Medicine 5: 19-25.
27. Mato AR, Luger SM, Heitjan DF, Mikkelsen ME, Olson E, et al. (2010) Elevation in serum lactate at the time of Febrile Neutropenia (FN) in hemodynamically-stable patients with Hematologic Malignancies (HM) is associated with the development of septic shock within 48 h. Cancer Biology & Therapy 9: 585-589.