Analysis of Overall Level of Evidence Behind The Institute of Healthcare Improvement Ventilator-Associated Pneumonia Guidelines

Reference as: Padrnos L, Bui T, Pattee JJ, Whitmore EJ, Iqbal M, Lee S, Singarajah CU, Robbins RA. Analysis of overall level of evidence behind the Institute of Healthcare Improvement ventilator-associated pneumonia guidelines. Southwest J Pulm Crit Care 2011;3:40-8. (Click here for PDF version of manuscript)

Leslie Padrnos^1,4(lpadrnos@email.arizona.edu)

Tony Bui^1,4 (tony.bui@cox.net)

Justin J. Pattee^2,4 (backageyard@gmail.com)

Elsa J. Whitmore^2,4 (elsa_whitmore@hotmail.com)

 Maaz Iqbal^1,4 (maaziqbal@gmail.com)

Steven Lee^3,4 (timmah2k@gmail.com)

Clement U. Singarajah^2,4 (clement.singarajah@va.gov)

 Richard A. Robbins^1,4,5 (rickrobbins@cox.net)

¹University of Arizona College of Medicine

²Midwestern University-Arizona College of Osteopathic Medicine

³Kirksville College of Osteopathic Medicine

⁴Phoenix VA Medical Center

⁵Phoenix Pulmonary and Critical Care Research and Education Foundation

None of the authors report any significant conflicts of interest.

Abstract

Background 

Clinical practice guidelines are developed to assist in patient care but the evidence basis for many guidelines has recently been called into question.

Methods 

We conducted a literature review using PubMed and analyzed the overall quality of evidence and made strength of recommendation behind 6 Institute of Health Care (IHI) guidelines for prevention of ventilator associated pneumonia (VAP). Quality of evidence was assessed by the American Thoracic Society levels of evidence (levels I through III) with addition of level IV when evidence existed that the guideline increased VAP. We also examined our own intensive care units (ICUs) for evidence of a correlation between guideline compliance and the development of VAP.

Results 

None of the guidelines could be given more than a moderate recommendation. Only one of the guidelines (head of bed elevation) was graded at level II and could be given a moderate recommendation. One was graded at level IV (stress ulcer disease prophylaxis). The remainder were graded level III and given weak recommendations. In our ICUs compliance with the guidelines did not correlate with a reduction in VAP (p<0.05).

Conclusions 

Most of the IHI guidelines are based on level III evidence. Data from our ICUs did not support guideline compliance as a method of reducing VAP. Until more data from well-designed controlled clinical trials become available, physicians should remain cautious when using current IHI VAP guidelines to direct patient care decisions or as an assessment of the quality of care.

Introduction

The growth of guideline publications addressing nearly every aspect of patient care has been remarkable. Over the past 30 years numerous medical regulatory organizations have been founded to improve the quality of care. Many of these organizations have developed medical regulatory guidelines with 6870 listed in the National Guideline Clearinghouse (1). Many of these guidelines were rapidly adopted by healthcare organizations as a method to improve care.

Interest has grown in critically appraising not only individual clinical practice guidelines but also entire guideline sets of different medical (sub)specialties based on their rapid proliferation and in many instances an overall lack of efficacy in improving care (2,3). We assessed the quality of evidence underlying recommendations from one medical regulatory organization, the Institute for Healthcare Improvement (IHI), regarding one set of guidelines, the ventilator associated pneumonia (VAP) guidelines or VAP bundles (4). This was done by senior medical students during a month long rotation in the Phoenix Veterans Administration ICU. 

Methods

The study was approved by the Western Institutional Review Board.

Literature Search

In each instance PubMed was searched using VAP which was cross referenced with each component of the VAP bundle (as modified by the Veterans Administration) using the following MESH terms: 1. Elevation of the head of the bed; 2. Daily sedation vacation; 3. Daily readiness to wean or extubate; 4. Daily spontaneous breathing trial; 5. Peptic ulcer disease prophylaxis; and 6. Deep venous thrombosis prophylaxis. In addition, each individual component of the term was cross referenced with VAP. We also reviewed “Related citations” as listed on PubMed. Additional studies were identified using the “Related citations” in Pubmed from studies listed as supporting evidence on the IHI website and from the references of these studies.

Each study was assessed for appropriateness to the guideline. Studies were required to be prospective and controlled in design. Only studies demonstrating a reduction in VAP were considered, i.e., surrogate outcomes such as reduction in duration of mechanical ventilation were not considered. 

The American Thoracic Society grading system was used to assess the underlying quality of evidence for the IHI VAP guidelines (5) (Table 1). Only evidence supporting a reduction in VAP was considered. We added category IV when there was literature evidence of potentially increasing VAP with the use of the recommendation. A consensus was reached in each case. 

Table I. Levels of Evidence

Level of Evidence	Definition
Level I (high)	Evidence from well-conducted, randomized controlled trials.
Level II (moderate)	Evidence from well-designed, controlled trials without randomization (including cohort, patient series, and case-control Studies). Level II studies also include any large case series in which systematic analysis of disease patterns was conducted, as well as reports of data on new therapies that were not collected in a randomized fashion.
Level III (low)	Evidence from case studies and expert opinion. In some instances, therapy recommendations come from antibiotic susceptibility data without clinical observations.
Level IV	No evidence of improvement with some evidence of an increase in a negative outcome.

Guideline Compliance and VAP Incidence

We also assessed our ICUs for additional evidence of the effectiveness of the VAP bundle. Data was collected for a period of 50 months from January, 2007 through February, 2011. This was after the Veterans Administration requirements for VAP reporting and IHI compliance was instituted. Diagnosis and compliance were assessed by a single quality assurance nurse using a standardized protocol (6). Statistical analysis was done using a Pearson correlation coefficient with a two-tailed test. Significance was defined as p<0.05.

Results

Literature Review

Numbers of articles identified by PubMed search and used for grading the level of evidence and strength of recommendation are given in Table 2. Also included are the level of evidence and the strength of the recommendation.

Table 2.

Guideline

Total Articles

No. of Articles Used (references)

Level of Evidence

Strength of Recommendation

Elevation of the head of the bed	31	8 (7-14)	II	Moderate
Daily sedation vacation	66	4 (15-18)	III	Weak
Daily readiness to wean or extubate	47	3 (19-21)	III	Weak
Daily spontaneous breathing trial	29	1 (22)	III	Weak
Peptic ulcer disease prophylaxis	52	9 (23-29)	IV	Weak
Deep venous thrombosis prophylaxis.	14	2 (30-31)	III	Weak

Head of Bed Elevation

A literature search identified 31 articles of which 8 were used in evaluating this guideline (7-14). However, only 2 specifically studied head of bed elevation with one supporting and another not supporting the intervention (7,8). Consequently it was graded as level II and the strength of recommendation was graded as moderate.

Daily Spontaneous Breathing Trial, Daily Readiness to Wean, and Daily Sedation Vacation

From 1-4 studies were identified for each of these interventions, however, none demonstrated a reduction in VAP. Consequently, it was judged that the evidence basis was level III and the strength of recommendation was graded as weak.

Stress Ulcer Disease Prophylaxis

We found no evidence that stress ulcer disease prophylaxis decreased VAP (23-29). There was some evidence that acid suppressive therapy increased pneumonia and VAP. Consequently, it was judged to be a level IV (possibly increasing VAP). 

Deep Venous Thrombosis Prophylaxis

We could find no evidence that deep venous thrombosis prophylaxis decreased VAP (30,31).

Guideline Compliance and VAP Incidence

Beginning in the first quarter of fiscal year 2007 there was a significant decrease in the incidence of VAP in our hospital (33). This coincided with the requirement for the monitoring of VAP, compliance with the VAP bundles and our adoption of endotracheal aspiration with nonquantitative culture of the aspirate as opposed to bronchoalveolar lavage which had been out standard practice. We changed practices because bronchoalveolar lavage with quantitative cultures appeared to offer no improvement in clinical outcomes to endotracheal aspiration (34). In our medical and surgical ICUs, 5097 audits representing 5800 ventilator-days were assessed. Nineteen cases of VAP were identified with an average of 2.1 VAP infections/1000 ventilator-days.  We assessed our surgical and medical ICUs, combined and separately, for a correlation between total bundle compliance and each component of the VAP bundle with VAP incidence (Appendices 1-3). There was no significant correlation between compliance with the bundles and VAP (p<0.05).

Discussion

This manuscript questions the validity of the VAP bundles as proposed by the IHI. We found that a systematic review of the literature revealed predominately weak evidence to support these guidelines. Only one guideline (head of bed elevation) was supported by a randomized trial (7), but an additional, larger trial showed no decrease in VAP (8). Furthermore, data from our own ICUs showed no evidence of IHI VAP guideline compliance with a reduction in VAP.

Head of bed elevation is a relatively simple and easy to perform intervention which may reduce VAP. Studies examining aspiration have shown a reduction in critical care patients with the head of bed elevation but it is unclear whether this translates into a reduction in VAP (36,37). Drakulovic et al. (7) reported a randomized controlled trial in 86 mechanically ventilated patients assigned to semi-recumbent or supine body position.  The trial demonstrated that suspected cases of ventilator-associated pneumonia had an incidence of 34 percent while in the semi-recumbent position suspected cases had an incidence of 8 percent (p=0.003).  However, another study in 221 subjects demonstrated that the target head elevation of 45 degrees was not achieved for 85% of the study time, and these patients more frequently changed position than supine-positioned patients (8). The achieved difference in treatment position (28 degrees vs. 10 degrees) did not prevent the development of ventilator-associated pneumonia. The other 5 articles identified either did not identify head of bed elevation directly or as part of a bundle. Most were a before and after design and not randomized. Therefore, it is difficult to draw any meaningful conclusions.

The IHI groups daily "sedation vacations" and assessing the patient’s “readiness to extubate.” The logic is that more rapid extubation leads to a reduction in VAP. Kress et al. (15) conducted a randomized controlled trial in 128 adult patients on mechanical ventilation, randomized to either daily interruption of sedation irrespective of clinical state or interruption at the clinician’s discretion. Daily interruption resulted in a reduction in the duration of mechanical ventilation from 7.3 days to 4.9 days (p=0.004). However, in a retrospective review of the data, the authors were unable to show a significant reduction in VAP (16).

Stress ulcer prophylaxis and deep venous thrombosis prophylaxis are routine in most ICUs. However, stress ulcer prophylaxis with enteral feeding is probably as effective as acid suppressive therapy and acid suppressive therapy may increase the incidence of VAP (38). Deep venous thrombosis prophylaxis has been shown to decrease the incidence of pulmonary emboli but not improve mortality (32). Although we use these interventions in our ICU, we would suggest that these would be more appropriate for recommendations rather than guidelines.

The diagnosis of VAP is difficult, requiring clinical judgment even in the presence of objective clinical criteria (6). The difficulty in diagnosis, along with the negative consequences for failure to follow the IHI guidelines, makes before and after comparisons of the incidence of VAP unreliable. Therefore, we sought evidence for the effectiveness of VAP prevention guidelines reasoning that the better the compliance with the guidelines, the lower the incidence of VAP. We were unable to show that improved VAP guideline compliance correlated with a reduced incidence of VAP.

The IHI guidelines would not meet the criteria outlined earlier in an editorial in the Southwest Journal of Pulmonary and Critical Care for a good guideline:

Our study has several limitations. No literature review is totally comprehensive. It is possible that studies relevant to the IHI VAP guidelines, especially those written in a foreign language, were not identified. Second, the Phoenix VA data may be underpowered to show a small beneficial effect despite having over 5000 patient audits. Third, as with other healthcare facilities, the VAP guidelines at our institution were mandated and monitored. The threat of negative consequences may have compromised the objective assessment of the data, likely invalidating a before and after comparison. Fourth, correlation between guideline compliance and VAP incidence is not a substitute for a randomized trial. Unfortunately, the later is not possible given that guideline compliance is mandated.

It is unclear why the IHI guidelines have received such wide acceptance given their weak evidence basis. Agencies involved in guideline writing should show restraint in guideline formulation based on opinion or weak or conflicting evidence. Only those interventions based on strong evidence which can make a real difference to patients should be designated as guidelines.

Acknowledgements

The authors acknowledge Janice Allen, MSN, RN who collected the VAP data reported from the Phoenix VA.

References

1. http://www.guideline.gov/ Accessed 3-16-2011.
2. Lee DH, Vielemeyer O. Analysis of overall level of evidence behind infectious diseases society of America practice guidelines. Arch Intern Med 2011;171:18-22.
3. Kett DH, Cano E, Quartin AA, Mangino JE, Zervos MJ, Peyrani P, Cely CM, For KD, Scerpella EG, Ramirez JA. Implementation of guidelines for management of possible multidrug-resistant pneumonia in intensive care: an observational, multicentre cohort study.  Lancet Infect Dis 2011 Jan 19. [Epub ahead of print].
4. http://www.ihi.org/IHI/Topics/CriticalCare/IntensiveCare/Changes/ImplementtheVentilatorBundle.htm. Accessed 3-16-2011.
5. Schünemann H, Jaeschke R, Cook DJ, et al. An Official ATS Statement: Grading the Quality of Evidence and Strength of Recommendations in ATS Guidelines and Recommendations. Am J Resp Crit Care Med 2006;174:605-14.
6. Horan TC, Andrus M, Dudeck MA. CDC/NHSN surveillance definition of health care-associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 2008;36:309-32.
7. Drakulovic MB, Torres A, Bauer TT, Nicolas JM, Nogue S, Ferrer M. Supine body position as a risk factor for nosocomial pneumonia in mechanically ventilated patients: A randomised trial. Lancet 1999;354:1851-8.
8. van Nieuwenhoven CA, Vandenbroucke-Grauls C, van Tiel FH, Joore HC, van Schijndel RJ, van der Tweel I, Ramsay G, Bonten MJ. Feasibility and effects of the semirecumbent position to prevent ventilator-associated pneumonia: a randomized study. Crit Care Med 2006;34:396-402.
9. Baxter AD, Allan J, Bedard J, Malone-Tucker S, Slivar S, Langill M, Perreault M, Jansen O. Adherence to simple and effective measures reduces the incidence of ventilator-associated pneumonia. Can J Anaesth 2005;52:535-41.
10. Muscedere J, Dodek P, Keenan S, Fowler R, Cook D, Heyland D; VAP Guidelines Committee and the Canadian Critical Care Trials Group. Comprehensive evidence-based clinical practice guidelines for ventilator-associated pneumonia: prevention. J Crit Care 2008;23:126-37.
11. Wip C, Napolitano L. Bundles to prevent ventilator-associated pneumonia: how valuable are they? Curr Opin Infect Dis. 2009;22:159-66.
12. Laux L, Dysert K, Kiely S, Weimerskirch J. Trauma VAP SWAT team: a rapid response to infection prevention. Crit Care Nurs Q 2010;33:126-31.
13. Bird D, Zambuto A, O'Donnell C, Silva J, Korn C, Burke R, Burke P, Agarwal S. Adherence to ventilator-associated pneumonia bundle and incidence of ventilator-associated pneumonia in the surgical intensive care unit.  Arch Surg 2010;145:465-70.
14. Torres A, Serra-Batlles J, Ros E, Piera C, Puig de la Bellacasa J, Cobos A, Lomeña F, Rodríguez-Roisin R. Pulmonary aspiration of gastric contents in patients receiving mechanical ventilation: the effect of body position. Ann Intern Med 1992;116:540-3.
15. Kress JP, Pohlman, AS, O'Connor, MF, Hall,JB. Daily interruption of sedative infusions in critically ill patients undergoing mechanical ventilation. N Engl J Med 2000; 342:1471-1477.
16. Schweickert WD, Gehlbach BK, Pohlman AS, Hall JB,  Kress JP. Daily interruption of sedative infusions and complications of critical illness in mechanically ventilated patients. Crit Care Med 2004; 32:1272–6.
17. Mehta, S. A randomized trial of daily awakening in critically ill patients managed with a sedation protocol: a pilot trial. Critical Care Medicine 2008; 36:2092-9.
18. Girard TD, Kress JP, Fuchs BD, et al. Efficacy and safety of a paired sedation and ventilator weaning protocol for mechanically ventilated patients in intensive care (Awakening and Breathing Controlled Trial): A randomized contolled trial.  Lancet 2008;371:126-34.
19. Marelich GP, Murin S, Battistella F, Inciardi J, Vierra T, Roby M. Protocol weaning of mechanical ventilation in medical and surgical patients by respiratory care practitioners and nurses: Effect on weaning time and incidence of ventilator-associated pneumonia. Chest 2000;118:459-67.
20. Jain M, Miller L, Belt D, King D, Berwick DM.  Decline in ICU adverse events, nosocomial infections and cost through a quality improvement initiative focusing on teamwork and culture change.  Qual Saf Health Care 2006; 15: 235–239.
21. Resar RK. Making noncatastrophic health care processes reliable: learning to walk before running in creating high-reliability organizations. Health Serv Res 2006; 41: 1677–89.
22. Liang JF, Tian R, Feng L. Clinical experience of spontaneous breathing trial in weaning mechanical ventilation.  Zhongguo Wei Zhong Bing Ji Jiu Yi Xue 2009;21:617-20.
23. Yildizdas D, Yapicioglu H, Yilmaz HL. Occurrence of ventilator-associated pneumonia in mechanically ventilated pediatric intensive care patients during stress ulcer prophylaxis with sucralfate, ranitidine, and omeprazole. J Crit Care 2002;17:240-5.
24. Lopriore E, Markhorst DG,  Gemke RJ. Ventilator-associated pneumonia and upper airway colonization with Gram negative bacilli: the role of stress ulcer prophylaxis in : the role of stress ulcer prophylaxis in children. Intensive Care Med 2002;28:763–767.
25. Dellinger RP, Carlet JM, Masur H, et al. Surviving Sepsis Campaign guidelines for management of severe sepsis and septic shock. Crit Care Med 2004;32:858-873.
26. Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia. Am J Respir Crit Care Med 2005;171:388-416.
27. Laheij RJ, Sturkenboom MC, Hassing RJ, Dieleman J, Stricker BH, Jansen JB. Risk of community-acquired pneumonia and use of gastric acid-suppressive drugs. JAMA 2004;292:1955-60.
28. Cook DJ, Laine LA, Guyatt GH, Raffin TA. Nosocomial pneumonia and the role of gastric pH: A meta-analysis. Chest 1991;100:7-13.
29. Chua LC, Mehta M, MD, Bhutani S, Schorr C, Milcarek B, Gerber D.  Early ventilator-associated pneumonia in patients on outpatient acid-suppressive therapy.  Chest  2010;138:730A [Abstract].
30. Wahl WL, Talsma A, Dawson C, Dickinson S, Pennington K, Wilson D, Arbabi S, Taheri PA. Use of computerized ICU documentation to capture ICU core measures Surgery 2006; 140:684-9.
31. Pronovost PJ, Berenholtz SM, Ngo K, McDowell M, Holzmueller C, Haraden C, Resar R, Rainey T, Nolan T, Dorman T. Developing & pilot testing quality indicators in the intensive care unit. Journal of Critical Care 2003;18:145-55.
32. Dentali F, Douketis JD, Gianni M, Lim W, Crowther MA. Meta-analysis: anticoagulant prophylaxis to prevent symptomatic venous thromboembolism in hospitalized medical patients. Ann Intern Med 2007;146:278-88.
33. Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Safe Health Care 2003 ;12:458-64.
34. Canadian Critical Care Trials Group. A randomized trial of diagnostic techniques for ventilator-associated pneumonia. N Engl J Med 2006;355:2619-30.
35. Robbins RA, Thomas AT Raschke RA. Guidelines, recommendations and improvement in healthcare. Southwest J Pulm Crit Care 2011;2;34-37.
36. Torres A, Serra-Batlles J, Ros E, Piera C, Puig de la Bellacasa J, Cobos A, Lomeña F, Rodríguez-Roisin R. Pulmonary aspiration of gastric contents in patients receiving mechanical ventilation: the effect of body position. Ann Intern Med 1992;116:540-3.
37. Orozco-Levi M, Torres A, Ferrer M, Piera C, el-Ebiary M, de la Bellacasa JP, Rodriguez-Roisin R. Semirecumbent position protects from pulmonary aspiration but not completely from gastroesophageal reflux in mechanically ventilated patients. Am J Respir Crit Care Med 1995;152:1387-90.
38. Marik PE, Vasu T, Hirani A, Pachinburavan M. Stress ulcer prophylaxis in the new millennium: A systematic review and meta-analysis. Critical Care Medicine 2010:38;2222-8.

Appendices

Click here for Appendix 1. VAP rate in all ICUs

Click here for Appendix 2. VAP in medical ICU

Click here for Appendix 3. VAP in surgical ICU