Content Validity: Establishing and Reporting the Evidence in Newly-Developed Patient-Reported Outcome (PRO) Instruments for Medical Product Evaluation - Good Research Practices Donald L. Patrick PhD, MSPH, Laurie B. Burke RPh, MPH, Chad J. Gwaltney PhD, Nancy Kline Leidy PhD, Mona L. Martin RN, MPA, Lena Ring PhD Elizabeth Molsen RN Liaison to ISPOR PRO Task Force Initiatives
Task Force established in March 2009. Task Force Background Donald Patrick Built on previous work of a previous Task Force: Evaluating and Documenting the Content Validity of Patient- Reported Outcome Instruments to Support Medical Product Labeling and Promotional Claims in the United States: Report of the ISPOR PRO Existing Instruments & Their Modifications Task Force(Rothman et. al., 2009). Authors represent a range of perspectives on PRO instruments: academia, government, research organization, and industry. More than 50 ISPOR members submitted written comments during 2 rounds of comments, in addition to those received orally during the Atlanta ISPOR presentation.
ISPOR PRO GRP Task Force Reports The 3 ISPOR PRO GRP reports published in 2009 were downloaded 5,964 times from April 2010 to March 2011. This is an average of166 times per month each. Content Validity of Existing Instruments Changing Modes of Administration Translation & Linguistic Validation
Appreciation to Reviewers We would like to thank all the reviewers of this paper for their time in reviewing and sending helpful comments for these papers. We addressed each suggestion individually and discussed all as group.
Content Validity: Establishing and Reporting the Evidence in Newly-Developed Patient-Reported Outcome (PRO) Instruments for Medical Product Evaluation Good Research Practices Part I: Eliciting Concepts for a New PRO Instrument Part II: Assessing Respondent Understanding
Paper 1: Eliciting Concepts Five Steps 1. Determine the context of use. 2. Develop the research protocol for qualitative concept elicitation and analysis. 3. Conduct the concept elicitation interviews and focus groups. 4. Analyze the qualitative data. 5. Document concept development and elicitation methodology and results.
Step 1: Determine the context of use Laurie Burke Develop a hypothesized disease model based on literature, experts, and patients. Name and define the concept within the context of a clinical trial end-point model. Select and define the target population including cultural\language groups.
Step 1: Determine the context of use continued Make preliminary decisions on the range of instrument content and structure. Select a theoretical approach to guide qualitative methods for content validity. Develop an hypothesized conceptual framework for the proposed instrument.
Clinical Trial Endpoint Model A diagram or description of the hierarchy of hypothesized relationships among all endpoints in a clinical trial Example: Concepts Indication: Reduce asthma exacerbations Endpoints Primary: Asthma Exacerbation Secondary FEV1 Asthma symptom score
Context of Use A comprehensive and clear statement that describes the manner and purpose of use and plans for interpretation of a clinical outcome assessment (COA) Concept measured Target claim Target population Intrinsic and extrinsic sources of heterogeneity considered Type of treatment Type of trial (endpoint model)
Figure 2: Example conceptual framework for a PRO evaluating the concept of pain quality Aching Pain quality Deep pain Dull Itchy Surface pain Numb Tingling
Step 2: Develop research protocol for concept elicitation & analysis Chad Gwaltney Define the target sample characteristics. Clinical and demographic characteristics Sample size/saturation Select the data collection method - focus groups, individual interviews, both. Determine the setting and location for data collection. Develop the interview guide Draft, pilot, revise. Not a script stimulate discussion Draft, pilot, revise.
Step 3: Conduct the concept elicitation interviews & focus groups Mona Martin Obtain IRB approval. Recruit and train sites. Recruit participants; monitor sample characteristics to assure representation. Select and train interviewers. Conduct interviews implement quality control measures. Record or videotape interviews. Transcribe and clean transcripts.
Step 4: Analyze the qualitative data Lena Ring Analyze qualitative data according to theoretical approach used. Establish preliminary coding framework; update as data are coded. Establish coding procedures and train coders. Organize data using a qualitative research software. program Assess saturation. Interpret results.
Figure 3: From coding framework to coding dictionary Hypothesized Conceptual Framework CODING FRAMEWORK CODING DICTIONARY PRO Instrument Review Target Product Profile Interview Guide Starting Structure for Organizing Codes Structure Expanded to add more concepts expressed by patients during coding process Contains all codes assigned, grouped by concept, and linked to patient language (quotes) Literature Clinical Experts All Code Assignments from transcripts Patient Interview Results
Figure 4: Example Saturation Table for Concept Elicitation Interviews (Example) Symptom Saturation Table CONCEPT CODES Shortness of Breath Difficult to Breathe Not Enough Air Gasping Shallow/Quick Breathing Wheezing Coughing Chest Tightness Chest Pain Dizziness/Lightheadedness NUMBER OF NEW CONCEPT CODES APPEARING IN EACH TRANSCRIPT GROUP % of TOTAL NEW CONCEPT CODES (Total = 10) Transcript Group Where Concept First Appeared Transcript Group 1 (N=5 transcripts) X X X X X X X Transcript Group 2 (N=4 transcripts) X X Transcript Group 3 (N=5 transcripts) X Transcript Group 4 (N=5 transcripts) 7 2 1 0 70% 20% 10% 0%
Figure 5: Example Multi-Vectored Approach to Understanding Qualitative Data Patient language Relevance of Concept Attribute to Measure Necessary Coverage (degree of importance to patient) INTERPRETATION & MEANING Presenting the relation of qualitative results to concepts, and wording items with the most meaningful language appropriate to measurement design Essential Coverage (degree of importance clinically and to measurement strategy) Variability of occurrence Degree of Difficulty to Patient Degree of Bother to Patient
Step 5: Document concept development & elicitation methodology & results Donald Patrick Provide target claims and any other context for use. Describe target population. Provide hypothesized and revised disease model and any input from content experts. Provide endpoint model. Provide conceptual framework and revisions made from preliminary to revised.
Step 5: Document concept development & elicitation methodology & results continued Provide study methods via protocols and guides. Provide summary of results, including evidence of saturation. Provide transcripts of interviews and focus group. Track origin and derivation of concepts captured in the PRO instrument. Summarize qualitative data. Provide key references.
Table 4: Documentation of Concept Elicitation
Part II: Assessing Patient Understanding Five Steps 1. Develop items based on findings from concept elicitation. 2. Design cognitive interview process for the planned context of use. 3. Conduct cognitive interviews. 4. Make decisions to revise the PRO instrument. 5. Document cognitive interview results for evaluation of content validity.
Step 1: Develop items based on findings from concept elicitation Chad Gwaltney Develop criteria for item development and evaluation. Choose concepts to measure. Select recall period and modes of administration. Draft Instructions
Step 1: Develop items based on findings from concept elicitation continued Develop items and match to response scales Assess readability Determine order and sequence
Sample item criteria The item captures the concept that is intended. The item is worded in a manner consistent with the expressions used by patients. The item represents a single concept, rather than a multidimensional concept. The content of the item is appropriate for the mode of administration
Step 2: Design cognitive interview process for the planned context of use Nancy Kline Leidy Identify population Design cognitive interview process Develop Protocol and Cognitive Interview Guide
Step 3: Conduct cognitive interviews Mona Martin o Train interviewers o Train subject to think aloud o Use verbal probes o Monitor interview quality o Record and transcribe o Prepare result summaries
Step 4: Make decisions to revise the PRO instrument Nancy Kline Leidy Employ an iterative process. Reduce ambiguity in item language. Assess saturation. Balance respondent input with principles of item construction and decisions on conceptual framework.
Step 5: Document cognitive interview results for evaluation of content validity Lena Ring Complete Item Tracking Matrix including final item, final response scale, any preliminary domain assignment, description of intent of item, and patient quotes supporting item intent.
Figure 2: Example Item Tracking Matrix
Next Steps and Manuscript Submission Donald Patrick Submission to Value in Health in June. Questions or comments, please email: donald@u.washington.edu or pro@ispor.org