Moderator & Speaker. Speakers FORUM. Working Group Background Cancer population selection using secondary data sources in the oncology literature

Transcription

1 FORUM Moderator & Speaker A CHECKLIST FOR POPULATION SELECTION IN ONCOLOGY OUTCOMES RESEARCH USING RETROSPECTIVE DATABASES TUESDAY, MAY 24, 2011 Working Group Background Cancer population selection using secondary data sources in the oncology literature Kathy Schulman, MA Co-Chair, ISPOR Oncology Good Outcomes Research Practices WG, Principal, Outcomes Research Solutions, Inc. Bolton, MA, USA Speakers Summary of Validated Algorithms Jonas de Souza, MD Postdoctoral Fellow, Section of Hematology/Oncology, The University of Chicago Medical Center, Chicago, IL, USA A Draft Checklist for Population Selection in Oncology Outcomes Research Using Retrospective Databases Karina Berenson, MPH Co-Chair, ISPOR Oncology Good Outcomes Research Practices WG, Associate Director, Covance Market Access Services Inc., Gaithersburg, MD, USA Co-Chairs: Karina Berenson, MPH, Associate Director, Covance Market Access Services, Inc Kathy L. Schulman, MA, Principal, Outcomes Research Solutions, Inc. Leadership Group: Vijayveer Bonthapally, Ph.D. Manager Oncology, Abbott Laboratories; Jonas de Souza, MD, Oncology Clinical Fellow, University of Chicago; Donatus U. Ekwueme, PhD, MA, Senior Health Economist, CDC; Arijit Ganguli, MBA, Manager Oncology, Abbott Laboratories; Ed Kim MD, MBA, Executive Director, HEOR, Novartis; Lenka Kellerman, MBA, Managing Director, Oncology Information Services; Lois Lamerato, PhD, Epidemiologist, Dept Public Health Sciences, Henry Ford Hospital; Ya Chen (Tina) Shih, PhD, Associate Professor, Director, Program in Economics of Cancer, Section of Hospital Medicine, University of Chicago; Alex Shteynshlyuger MD, Fellow, Urologic Oncology, Washington University in St Louis; Samuel Wagner, PhD, RPh, MSc, BSc, Executive Director, HEOR, Bristol Myers Squibb Objective: Develop standards for oncology outcomes research using secondary data Defining population selection criteria Identification of the primary tumor type Clinically relevant treatment subset Line of treatment Future topics/activities Defining chemotherapy regimens Defining the measurement of clinical outcomes Identification of economic and cost measures Describing methods for handling bias Why Start With Primary Tumor Identification? Secondary data often depend on standard code sets These systems may lack clinical detail Diagnostic and procedural information may reflect the primary objective of the data source, i.e. to facilitate reimbursement; to reflect the diagnostic process; to meet the objectives of a local disease registry A diagnosis on a single claim or record in the absence of confirmatory treatment and without additional restriction will likely result in enrollment of patients without the disease But how big a problem is this? 1

2 Pubmed search criteria Published between Jan Dec 31, 2010 English language Abstracts Reviewed N=863 Potentially Eligible, Full Article Review N=321, 37.2% Ineligible N=542, 62.8% Secondary data (claims, hospital discharge, EMR, registry, administrative ) Eligible N=294, 91.6% Ineligible N=27, 8.4% Mesh "Neoplasms Search yielded 863 abstracts N=179, 31.5% N=169, 29.7% N=198, 34.8% N=23, 4.0% Reasons for Rejection Analytic sample not a cancer population (cancer either a study covariate or outcome) Not a secondary data source Not a research study (e.g. guidelines, reviews, meta-analyses etc.) Other 22.1% 5.1% 21.8% Registry Only Claims Only Registry + Claims Other Data Sources All Data Sources N, % N, % N, % N, % N, % All Studies 15, 100% 64, 100% 150, 100% 65, 100% 294, 100% Outcomes - clinical 6, 40.0% 11, 17.2% 69, 46.0% 17, 26.2% 103, 35.0% 51.0% Registry Only Claims Only Registry + Claims Other Data Source Treatment/Practice patterns 4, 26.7% 16, 25.0% 54, 36.0% 26, 40.0% 100, 34.0% Other 4, 26.7% 10, 15.6% 12, 8.0% 17, 26.2% 43, 14.6% Cost burden 0, 00.0% 12, 18.8% 11, 7.3% 4, 6.1% 27, 9.2% Outcomes -cost 1, 6.7% 15, 23.4% 4, 2.7% 1, 1.5% 21, 7.1% 73.1% of all studies conducted in the US, representing 90.6% of claims only studies, 84.6% of claims/registry combination studies, 46.7% of registry only studies and 35.4% of other studies 11.9% of all studies were conducted in Canada; representing 10.7% of claims/registry combination studies, 33.3% of registry only studies and 21.5% of other studies. There were no claims only studies. Other countries with >2 studies include Taiwan, Korea, France, Denmark, England, France, Italy, Australia, Japan Registry Only Claims Only Registry + Claims Other Data Sources Total N, % N, % N, % N, % N, % Any Cancer 1, 6.7% 8, 12.5% 5, 3.3% 13, 20.0% 27, 9.2% Breast 2, 13.3% 34, 53.1% 46, 30.9% 12, 18.5% 94, 32.0% Colorectal 7, 46.7% 13, 20.3% 45, 30.2% 10, 15.4% 75, 25.5% Leukemia 1, 6.7% 7, 10.9% 7, 4.7% 2, 3.1% 17, 5.8% Lymphoma 1, 6.7% 12, 18.8% 14, 9.4% 1, 1.5% 28, 9.5% Lung 2, 13.3% 19, 29.7% 27, 18.1% 14, 21.5% 62, 21.1% Multiple Myeloma 1, 6.7% 5, 7.8% 5, 3.4% 1, 1.5% 12, 4.1% Prostate 2, 13.3% 10, 15.6% 40, 26.8% 5, 7.7% 57, 19.4% Other 7, 46.7% 20, 31.3% 45, 30.0% 26, 40.0% 98, 33.3% 2

3 48.4% of studies relied solely on diagnosis codes for population selection; 35.5% of these studies did not report requiring more than one claim with a cancer diagnosis as a pre-requisite for study entry 27.4% used a restricted data source (i.e. only inpatient or outpatient claims) to select the population 12.9% of studies reported using a previously published algorithm, 6.5% a validated algorithm 4.8% discussed the implications of their selection criteria on study results Secondary data sources are widely used for oncology research Population identification still largely relies on registry or chart based systems No standards exist for data sources which rely on ICD 9 based coding systems (claims, discharge datasets) There is little reported use of existing algorithms and almost no discussion about the implications of selection criteria on results Summary of Validated Algorithms Jonas A. de Souza, M.D. Leader, ISPOR Oncology Good Outcomes Research Practices WG Postdoctoral Fellow, Section of Hematology/Oncology, The University of Chicago Medical Center, Chicago, IL, USA Medical claims are designed for administrative purposes Issues in identifying prevalent versus incident cases Issues in identifying cancer stage, appropriateness of therapy Issues on primary vs. secondary cancer sites Issues on rule-out diagnoses used in clinical care Issues on solid organ cancers vs. non-solid organs (e.g. a lymphoma of the small bowel is not a small bowel tumor, it is a lymphoma) Validation Usually linking claims data to gold-standard (registry, charts) Breast cancer - most commonly validated and studied Matching & regression mechanisms 2 most common methods Sensitivity Proportion of cases in the gold-standard that were identified from the claims database e.g., 1992 SEER breast cancers that were identified from the 1992 Medicare data (based on the 1992 SEER data and among those who resided in SEER sites) Gold-Standard (Registry, charts) Specificity Proportion of non-cases in the gold-standard that were identified as non-cases in the claims database e.g., women who did not have breast cancer (based on the 1992 SEER data and among those who resided in SEER sites) who were identified as non-cancer cases in the 1992 Medicare claims data Gold-Standard (Registry, charts) Claims to be Validated Claims to Be Validated Specificity: D/B+D 3

4 Positive Predictive Value (PPV) Claims to be Validated Negative Predictive Value (NPV) NPV: D/(C+D) Gold-Standard (Registry, charts) Validation Algorithms Matching methods Regression-based methods Implications of definitions and number or criteria used Wash-out period Common reasons for incorrect identification of patients Registries as gold-standards Validation Studies Regression-based Algorithms Validation Studies Example of A Matching-based Algorithm Example of a matching validation method in selecting incident breast cancer cases Setoguchi, et al claims-based definitions to identify incident breast cancer Subjects aged > 65 ( ) from Pennsylvania Medicare & drug benefit data linked to state cancer registry. Required all subjects to have at >1 claim for any service and a prescription during each 6-month period until subjects die, or until the study period ends Required all subjects to be enrolled, and have no cancerrelated claims during the 6 months before 1 January 1997 to exclude subjects currently undergoing treatment for cancer 4

5 Validation Studies Example of A Regression-based Algorithm Freeman, et al 2000 In general, more stringent definitions increase PPV but decrease sensitivity In other words, those captured more likely to be true cases However, method also misses more true cases (not captured) Validation Algorithms Matching methods Regression-based methods Implications of definitions and number or criteria used Wash-out period Common reasons for incorrect identification of patients Registries as gold-standards Setoguchi, 2007 Solin et al, 1994 Hotes et al, 2004 Discrepancy between definitions in goldstandards To determine breast cancer case counts, on a given set, using SEER and IARC rules Applied to a dataset provided by the North American Association of Central Cancer Registries from

6 There are several validated methods in breast cancer, with reported sensitivities and PPV s There is no one right method (or a perfect study) Know your question, the disease, the codes and the population you want to select Input from medical coders, billing staff, clinicians Know the method you are choosing, advantages and disadvantages and disclose them A Draft Checklist for Population Selection in Oncology Outcomes Research Using Retrospective Databases Develop a draft checklist that focuses on issues that are specific to selection of an oncology population from a secondary data source Karina Berenson, MPH Co-Chair, ISPOR Oncology Good Outcomes Research Practices WG Associate Director, Covance Market Access Services Inc. Gaithersburg, MD, USA 6

7 Assist decision makers in evaluating the quality of published studies Provide researchers a list of methodological issues for consideration when conducting oncology outcomes research studies in a secondary data source The working group has developed a DRAFT checklist and this presentation is intended to prompt discussion and feedback This disease-specific checklist is intended to complement other analysis guidelines (e.g. ISPOR checklist to assess retrospective database studies, ISPOR CER good research practices) The checklist should serve as a general guide, as not every element of the checklist will apply for all studies Tumor registry alone Tumor registry with linkage to claims data Claims data alone Other combinations of inpatient/outpatient administrative data, claims data, EMR or paper chart review 1. Research the cancer under study 2. Choose an appropriate data source 3. Review existing literature and any published algorithms 4. Develop the population selection criteria/algorithm a) Registry-based or EMR/Chart-based b) Encounter/Claims Code-based 5. Consider validation 6. Report the population selection criteria/algorithm 1. Research the cancer under study Issues to consider: Assess how the biology, natural history and etiology of the specific cancer may impact study population selection Identify biomarkers or other prognostic factors that influence choice of treatment Are there exclusionary conditions that need to be eliminated (e.g. non-solid tumor types in a study of solid tumors) Review recommended and commonly used treatment regimens (e.g. NCCN guidelines) 1. Research the cancer under study Issues to consider: Review the codes that could be used to identify the specific cancer population Is coding for the cancer specific to clinical subtype of interest? (e.g. ICD-9 codes for lung cancer can t be used to select patients with NSCLC) Should in situ codes be included or excluded? Identify any changes in practice over time (e.g. use of new diagnostic or interventional procedures during the study period) 7

8 2. Choose an appropriate data source Issues to consider: Is the level of clinical detail sufficient to identify the cancer population of interest? Is follow-up of cancer patients in the data source sufficient i to address the study question? Cancers are chronic conditions/high mortality Cancer patients are seen across treatment settings Cancer patients may be more likely to go on longterm disability or have a change in employment status 2. Choose an appropriate data source Issues to consider - registries: Is the level of clinical detail sufficient to identify population of interest? Captures clinical detail only at registration of the tumor May be missing lab/histology data (e.g. KRAS mutation for CRC) Unless linked to another data source, there likely won t be longterm follow-up and there may or may not be accurate data on mortality. Are the data collected and reported according to registry standards? Are ICD-O codes used and if so what version(s)? Is a staging system (e.g. AJCC) used and if so what version(s)? If linking registry data or other sources, has the linkage process been described and/or previously validated (e.g. SEER-Medicare)? Is the registry centralized or hospital-specific? 2. Choose an appropriate data source Issues to consider claims data: Is the clinical detail sufficient to identify population of interest? Are lab values or histology data available? If available, are data complete for population of interest? Will follow up be captured for patients who: Have a change in employment status or go on long-term disability? Are seen across treatment settings (Readmission? Followup in the outpatient setting?) Are referred to specialists/hospitals outside the health system? Are there accurate mortality data? Are there reimbursement issues that may affect claims completeness(e.g. Medicaid vs. Medicare)? 3. Review existing literature and any published algorithms Issues to consider: Validity of published algorithm Potential for selection or misclassification bias Comparable data sources If using SEER/Medicare for a time period that differs from the validation study, will the algorithm apply? If using an algorithm validated in a different data source, is it appropriate to use the algorithm as is or are there reasons to adapt it? 4. Develop the population selection criteria/algorithm a. Registry-based or EMR/chart-based algorithm Issues to consider: If using EMR text-searching for cancer, there is potential for misclassification if search text refers to rule-out diagnosis and context is not examined If using registry data, determine whether there is a time period allowed for recoding/finalization of the tumor type 4. Develop the population selection criteria/algorithm b. Encounter/Claims Coding-based algorithm Issues to consider: Physicians differential diagnosis may be recorded in claims codes (rule outs) Require more than one diagnosis code for tumor of interest, or include additional codes to eliminate false positives Distinguish incident and prevalent cases (wash-out period) Distinguish diagnostic and interventional procedure codes if using cancer-related procedures to select population A biopsy code following a breast cancer ICD-9 code may not be reliable indicator of cancer Validate selected tumor type is primary tumor (ICD-9 codes for metastasis are not always used): Breast cancer that spreads to the lung may be coded as primary lung cancer; patient could be misclassified as lung cancer patient instead of metastatic breast cancer patient. 8

9 Selection Approach Examples Impact A diagnosis code with a second component Use of a single diagnostic code Patients with an ICD-9 Selecting patients with a breast cancer code and a single ICD-9 code for mastectomy code pancreatic cancer Patients with an ICD-9 breast cancer code and Herceptin codes Probable breast cancer patient High sensitivity, but low specificity (PPV of 38%) (source: Friedlin 2010) 5. Consider validation/sensitivity analyses Issues to consider: Is there considerable potential for misclassification of the particular tumor type (e.g. leukemia)? Diagnostic certainty of data source Availability of data for validation (e.g. chart/emr review) Assess specificity in addition to sensitivity to understand the PPV, NPV 6. Report the population selection criteria/algorithm Issues to consider: Provide a detailed description of the data sources - be careful when using the term administrative to describe the data source Provide a detailed description of population selection/algorithm - ensure that other researchers could replicate your population selection methodology Discuss any implications of population selection for the research findings Describe validation/sensitivity analyses performed Future work for the checklist will include the addition of factors to consider for: Selection of disease stage Selection of treatment line Clinical sub-groups Questions? Feedback? Our Contact Information Kathy.Schulman@orsolutionsinc.com Karina.Berenson@covance.com Jonas.DeSouza@uchospitals.edu Thank you 9