Setting the Standard: NPCR and SEER Join Forces to Establish Data Quality Benchmarks

Size: px

Start display at page:

Download "Setting the Standard: NPCR and SEER Join Forces to Establish Data Quality Benchmarks"

Mark Banks
5 years ago
Views:

1 Setting the Standard: NPCR and SEER Join Forces to Establish Data Quality Benchmarks Serban Negoita, Clara Lam, Rebecca Ehrenkranz, Amy Solis, Reda Wilson, Manxia Wu, Vicki Benard June 12, 2018

2 NCI SRP and CDC CSB: Joint Project on Data Quality Objectives: Minimize work duplication, optimize the use of resources Develop benchmarks to be applied consistently independent of the funding agency Create reference points to assess the effects of 2018 NAACCR standards implementation Develop infrastructure for real-time monitoring of data quality!2

3 Benchmarks - Definition Benchmark: A slang or jargon term, usually meaning a measurement or point of reference taken at the beginning of a survey or project, used for comparison with subsequent measurements of the same variable; sometimes it means the best or most desirable value of the variable. Alternatively, an acceptable standard in evaluation (e.g., of air quality, of performance). Dictionary of Epidemiology, 2014 edition point of reference most desirable acceptable standard!3

4 Benchmark Development: Grade (Differentiation) and Special Grade Systems Grade data element suggested by the CDC NPCR based on the results of the NPCR quality audits Data element frequently used in research Reference points for grade necessary: significant changes in 2018 for Grade data collection standards See NAACCR Grade Coding Instructions and Tables Manual!4

5 Phase I: Planning/ Protocol Development Select period of observation, tumor behavior, histology criteria, etc. Select a two-grade system tumor site: urothelial cancers Select a three-grade system tumor site: stomach Select a four-grade system tumor site: colon and rectum Select cancer type-specific grade variables: Gleason Score for prostate cancer (SSF 6,8,10) Bloom - Richardson Score for breast cancer (SSF 7)!5

6 Pre-analytic: SEER Program Coding and Staging Manual - Grade History Timeline SEER Program Coding and Staging Manual: Grade History Timeline: Gleason Score, Prostate Gleason's Score Grading Gleason's Score Grading SEER Value Gleason's Score Grading SEER Value I, well I, well differentiated 2--4 I, well differentiated 1 differentiated 1 II, moderately II, moderately II, moderately differentiated 5,6 differentiated 2 differentiated 2 III, poorly III, poorly differentiated III, poorly differentiated 3 differentiated 3!6

7 Pre-analytic Phase: Collaborative Stage - Grade History Timeline Collaborative Stage: Grade History Timeline: Gleason Score, Prostate SSF 6: Gleason's Score SSF 8: Gleason's Score on Needle Core Biopsy/TURP Code Description Code Description Collaborative Stage did not exist 0 Test not done 998 No needle core biopsy/turp performed Gleason's score Gleason's score NA/Unk 988, 999 NA/Unk SSF 10: Gleason's Score on Prostatectomy/Autopsy Code Description 998 No prostatectomy/autopsy performed Gleason's score 988,999 NA/Unk!7

8 Pre-analytic Phase: AJCC - Grade History Timeline AJCC: Grade History Timeline: Gleason Score, Prostate AJCC 5 AJCC 6 AJCC 7 Gleason's Score (Description) Gleason's Score Description Gleason's Score Description 2--4 (well differentiated) 2--4 Well differentiated/slight anaplasia 5,6 (moderately Moderately differentiated/ differentiated) 5--6 moderate anaplasia 7 (moderately poorly Poorly differentiated/ differentiated) undifferentiated/marked (poorly anaplasia differentiated) Well differentiated/slight anaplasia Moderately differentiated/moderate anaplasia Poorly differentiated/undifferentiated/ marked anaplasia X Grade cannot be assessed X Gleason score cannot be processed!8

9 Phase II: Pre-analytic search for grade collection data quality standards Verify if benchmarks for grade collection available: NAACCR members IARC European cancer registries Lit research grade distribution reported by clinical trials and observational studies Gleason Score ranges across all studies in lit review: n>1,000 GS 2-6: 29-63% GS 3+4 = 7: 10-56% GS 4+3 = 7: 3-25% GS 8-10: 1-20%!9

10 Phase III: Analytic Outliers Detection for Gleason Score Selection Analytic dataset provided by CDC team Includes SEER and NPCR funded registries, Gleason score analysis restricted to dx. years Selection criteria: prostate cancer, positive histologic confirmation Gleason score post-prostatectomy: surgery code (confirming prostatectomy) Indicators (proportions) of interest: % cases Gleason Score Unknown (999) % cases Low Gleason Score 2-6 (prognostic group 1, least aggressive) % cases High Gleason Score 9-10 (prognostic group 5, most aggressive)!10

11 Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry 12% SSF 10, Gleason Score = 999, Dx. Years % 6% 3% 0% T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A!11

12 Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry (n = 36) 12% SSF 10, Gleason Score = 999, Dx. Years % Q1: 25 th percentile Q3: 75 th percentile 6% 3% 0% T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A!12

13 Benchmarking Method: IQR Rules for Outliers Benchmarking method: Fences at Q1-1.5*IQR and Q3+1.5*IQR Q1 (25 th percentil e) % Gleason Score Unknown (SSF 10 = 999) Q3 (75 th percentile) IQR Lower Fence Upper Fence 1.3% 2.6% 1.4% -0.8% 4.7% 25 Percentil e % Low Gleason Score (SSF 10 = 2-6) 75 Percentile IQR Lower Fence Upper Fence 21.4% 29.2% 7.8% 9.6% 41.0% 25 Percentil e % High Gleason Score (SSF 10 = 9-10) 75 Percentile IQR Lower Fence Upper Fence 5.9% 7.9% 2.0% 2.9% 10.9%!13

14 Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry 12% SSF 10, Gleason Score = 999, Dx. Years % 6% 3% 0% T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A!14

15 Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year 12% SSF 10, Gleason Score = 999, Dx. Years % 6% 3% 0% T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A!15

16 Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry SSF 10, Gleason Score , Dx. Years % 34% 23% 11% 0% K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C!16

17 Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year SSF 10, Gleason Score , Dx. Years % 34% 23% 11% 0% K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C!17

18 Proportion High Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry 20% SSF 10, Gleason Score , Dx. Years % 10% 5% 0% C S EE DD Q J HH X W G AA N I GG Z U M F II D P B L R JJ CC Y T K FF E H BB O A V!18

19 Proportion High Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year 20% SSF 10, Gleason Score , Dx. Years % 10% 5% 0% 20% C S EE DD Q J HH X W G AA N I GG Z U M F II D P B L R JJ CC Y T K FF E H BB O A V SSF 10, Gleason Score , Dx. Year % 10% 5% 0%!19

20 Conclusions Large number of registries is needed to develop valid benchmarks Pre-analytic phase is important: Subject mater expertise necessary to select the right targets (indicators) Lit. research, research of international standards limited return Numerous statistical methods available additional research necessary to select the optimal method IQR-based fence method relatively easy to implement Immediate applicability it identified outliers in the distribution of Gleason Score for cases diagnosed in 2016 Collaboration really helped, minimized effort to obtain datasets and history timelines Feasible, inexpensive method that resulted in reference points for assessing data completeness and frequency distribution Gleason Score!20

21 Many Thanks to: NCI Division of Cancer Control and Population Sciences Leadership CDC Division of Cancer Control Leadership Analysts at CDC and IMS: Jessica King (CDC) Denise Duran(CDC) MaryBeth Freeman(CDC) Jennifer Stevens (IMS) Registrars and all those contributing to cancer surveillance!21

23 Benchmarks in Cancer Surveillance Cancer system outcomes = cancer data sets Measurable characteristics of the cancer data sets Missing data (% unknown) Precision (% NOS) Validity (trueness) : re-abstraction/ re-coding : observed vs expected distribution!23

24 Quality Indicators Current data quality indicators evaluate: Processes Case ascertainment - completeness Death clearance - DCO rate De-duplication - duplicate rate Data editing - edits-failing cases Quality of output (data sets, data elements) Sex, age, race, county at dx all about demographics Both important, process indicators difficult to interpret and use No quality indicators for clinically-relevant data elements!!24

25 Population-Based High-Quality Clinically-Relevant Data Reference points (benchmarks) define quality Methods to compare against reference points (benchmarking) Measure and track improvement Infrastructure/ institutional support (standards setters) Select the reference point and measurement methods Integrate tracking/monitoring in registry operations Use benchmarks to promote population-based cancer data sets!25

26 Institutional support needed to develop benchmarks for cancer surveillance!26

27 Providing High-Quality Cancer Data How does the cancer surveillance community: Defines high-quality? Measures improvement toward high-quality? Makes optimal resource allocation decisions to achieve high-quality? How to inform cancer data users on the high-quality of population-based surveillance data Differentiate our data products from other data sources Substantiate the need for additional resources Accomplish cancer control and research mission!27

28 Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry SSF 10, Gleason Score , Dx. Years % 34% 23% Lowest % based on lit. research 11% 0% K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C!28

29 Complexity for Benchmark Selection Select reference points and measurement methods Prioritization decisions on data items that need benchmarks Treatment, stage, prognostic factors, cancer identification? Requires research, analysis, interpretation of existing data and publications Are there any existing benchmarks use by registries or oncology organizations, domestically or internationally? Statistical methods: Outlier detection Trend-based forecasting Measures of dispersion Observed versus expected categorical tests Other?!29

30 Methods for Benchmark Selection Use of external datasets to select benchmarks: Administrative/Claims data for treatments Patterns of care: treatment, stage, prognostic factors Re-abstraction/recoding studies: Manual re-abstraction/re-coding Large sample size Multiple registrars Adjudications of best answers Natural Language Processing used for re-abstractions, automated recoding/re-consolidation NLP algorithms need validation!30

31 Project Timeline NCI-CDC team meetings started December 2017 Multiphase project Planning/ Protocol development Pre-Analytic Assessment Analysis evaluation of statistical methods Detection of outliers Trend-based forecasting Interpretation and recommendations Implementation in program activities Implementation in registry systems dashboards!31

Reactive Quality Audit Pilot and Results: Quality Assurance Strategies for Cancer Surveillance

Reactive Quality Audit Pilot and Results: Quality Assurance Strategies for Cancer Surveillance Rebecca Ehrenkranz, MPH; Valentina Petkov, MD; Clara Lam, PhD; Amy Solis; Steve Friedman; Steven Cheng, PhD;