Analysis of the impact of AHRQ exclusions on the variation in Patient Safety Indicator (PSI) values by Vladimir Stevanovic Patient Safety Subgroup meeting Paris, 26 May 2011
Background HCQI Expert Group meeting in June 2009 Concerns were raised that PSI data may reflect rather coding and registration practices than actual differences in patient safety Several countries expressed reservations about the publication of PSIs in Health at a Glance 2009 due to perceived risk of misinterpretation The Secretariat proposed to undertake further analysis with the aim of validating the methodological approach
Background PSI Subgroup meeting in October 2009 The Secretariat presented a preliminary analysis on the impact of AHRQ exclusions by using NZ data The findings implied that the exclusions have varying impact on the country results and may distort indicator values apart from the obstetric ones The Secretariat proposed further analysis to be undertaken through a voluntary subsample of countries by replicating the same methods and using UPIs
Background HCQI Expert Group meeting in June 2010 Ten countries participated in the replication analysis: CAN, DNK, FIN, ISR, NOR, SNG, SPA, SWE, SWI, USA The results confirmed the previous findings that differences in coding practice, admission type definition and long LOS exclusion are likely to be the most important driving factor behind variations The decision was made to collect additional data by the means of regular HCQI 2010/11 data collection in order to better understand the impact of exclusions
Objectives To improve international comparability of PSIs by: Assessing the impact of admission type (ADMT) and and length of stay (LOS) exclusions on the rates for - PSI 07 Catheter-related bloodstream infection, - PSI 12 Pulmonary embolism or deep vein thrombosis, - PSI 13 Postoperative sepsis Exploring whether these exclusions account for any undesired or increased variation across countries The scope does NOT include exclusions that are inherent to the concept of an indicator by their nature
Methods All hospital discharges of one year Age =15 y or >15 y? yes PDx is immunocompromised state (list C-1 / W-1), or cancer (list C-2/W-2)? no SDx is immunocompromised state (list C-1 / W-1), or cancer (list C-2/W-2)? no PDx is identical to the numerator definition? Catheter-Related Bloodstream Infection Perform pre-exclusion calculations, see seperate flowchart Case is assigned to MDC 14 no no Exclude or the PDx is listed in table M3? yes yes Exclude Catheter-related bloodstram infections pre-exclusions Count and report number of All hospital discharges of one year, cases / admissions, ALOS, and yes Exclude age =15 years or > 15 years and mean number of secondary LOS < 24 hours or one day diagnoses (1) Count and report number of PDx is patients (countries with UPI identical to the yes only) / discharges (2) numerator definition? yes Exclude no Pre-exclusion and Post-exclusion stats (all discharges in a year AND age>=15 years AND LOS >= 24h) - the total number of discharges - the average length of stay - the average number of SDX no End Catheter-related bloodstram infections impact of length of stay (LOS) exclusions Count and report denominator cases (3) SDx is identical to the numerator definition? yes Count and report numerator cases (3) LOS > or = 48 hours or 2 days? yes no LOS is < 48 hours or < 2 days? yes LOS is < 24 hours or < 1 day? Add case to denominator population Calculate mean number of secondary diagnoses and ALOS of denominator population and report (6) Count and report denominator cases (7) yes Count denominator cases and report (4) SDx is identical to the numerator definition? yes yes Count denominator cases and report (5) SDx is identical to the numerator definition? yes Length of stay and Admission type exclusions Count numerator cases and report (4) Count numerator cases and report (5) SDx is identical to the numerator definition? yes Add case to numerator population - the numerator data - the denominator data Count numerator cases and report (8)
Additional data
Countries Australia Belgium Canada Denmark Finland France Iceland Israel Italy New Zealand Singapore Spain Sweden Switzerland United States
Results PSI 07 Catheter-related bloodstream infection (LOS excl)
Results PSI 07 Catheter-related bloodstream infection (LOS excl) Spearman-rank test (no LOS excl vs <1 day, <2 days) = 0.89, 0.84 (both p<0.01)
Results PSI 12 Postoperative PE or DVT (LOS excl)
Results PSI 12 Postoperative PE or DTV (LOS excl) Spearman-rank test (no LOS excl vs <1 day, <2 days) = 0.78,(p<0.01), 0.59 (p=0.07)
Results PSI 13 Postoperative sepsis (LOS excl)
Results PSI 12 Postoperative sepsis (LOS excl) Spearman-rank test (no LOS excl vs <1-3 days, <4 days) = 0.93-97, 0.78
Results PSI 12 Postoperative sepsis (LOS excl)
Results PSI 13 Postoperative sepsis (LOS & ADMT excl)
Results PSI 12 Postoperative sepsis (LOS & ADMT excl) Spearman-rank test (no ADMT vs with ADMT) = -0.19 (p=0.60)
Results PSI 12 Postoperative sepsis (ADMT distribution)
Results PSI 12 Postoperative sepsis (modified excl criteria)
Results PSI 12 Postoperative sepsis (modified excl criteria)
Publication PSI 07 Catheter-related bloodstream infection Min/Max ratio = 100 fold Variation in coding practices? Inflammatory conditions?
Recommendations 1. The exclusion criteria for PSI 07 Catheter-related bloodstream infections and PSI 12 Postoperative pulmonary embolism and deep vein thrombosis are appropriate, hence there is no need to change the existing LOS<2 days exclusion. 2. For PSI 13 Postoperative sepsis, exclusions of discharges with LOS<4 days and non-elective (acute) type admissions account for an increased variation between countries and cause bias in the resulting postoperative sepsis rates. It is recommended to drop the admission type exclusion criterion from the algorithm and to use the modified LOS<3 days exclusion instead.
Recommendations - cont. 3. For PSI 07 Catheter-related bloodstream infection, remaining ambiguities in the definition make this indicator not fit for reporting at this moment. 4. The following indicators seem to be robust enough and are therefore recommended for publication in Health at a Glance 2011: - PSI 05 Foreign body left in during procedure - PSI 12 Post-operative pulm. embolism or deep vein thrombosis - PSI 13 Post-operative sepsis - PSI 15 Accidental puncture or laceration - PSI 18 Obstetric trauma - vaginal delivery with instrument - PSI 19 Obstetric trauma - vaginal delivery without instrument
Optimal system requirements Unique patient identifier (UPI) Near miss and adverse events register Present on admission (POA) flag Standardised registration/coding practice Mapping between classification systems Calculation method/algorithm adjusted for international comparison - exclusions inherent to the concept of PSI - adjustments for the effect of confounders
Members of the Patient Safety Subgroup are invited to Comment on findings from the analysis, Decide on whether to change the algorithm used for the calculation of PSI13 Postoperative sepsis as recommended by this report, Decide on whether to adjust patient safety indicator rates by the mean number of secondary diagnoses among patients at risk as (previously presented by Saskia Drösler and Patrick Romano and discussed in 2009/10), Make recommendations for the continuation of the development work on PSIs (agenda item 4). Make recommendations on which patient safety indicators should be published in the chapter on Quality of Care in Health at a Glance 2011 (agenda item 5).