Introduction Patient-reported outcome (PRO) data play an important role in the evaluation of new medical products. PRO instruments are included in clinical trials as primary and secondary endpoints, as support for product reimbursement, and for use in publication and communication strategies. In December 2009, the US Food and Drug Administration (FDA) released its guidance for industry titled "Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims" (FDA 2009). The formalization of this guidance has helped to facilitate the inclusion of PRO instruments as endpoints in clinical trials. A review of US and EU therapeutic area-specific regulatory guidelines and product labels issued between January 1st 2006 and 16th November 2010 (Marquis et al. 2011) found that of 38 FDA draft or final guidance documents, 15 (39.5%) contained recommendation for the inclusion of PRO endpoints in clinical trials. Furthermore, of 432 total FDA approvals in the same time period, 93 (21.5%) had a PRO instrument listed as an endpoint in the label. A similar trend is reflected in Europe, where 35 of 95 (35.8%) EMA draft or final guidance documents contained recommendations for the inclusion of PRO endpoints in clinical trials. A PRO instrument was listed as an endpoint in the labeling of 54 of 248 (21.8%) EMA approvals. Alongside the increasing use of PRO instruments as endpoints in clinical trials, there has been recognition of the numerous advantages of electronic data capture. While most patient-reported outcome measures were originally developed to be administered via paper-and-pencil, the collection of PRO data via electronic devices is expanding. Electronically adapted PRO instruments have the advantages of easier implementation of skip patterns, avoidance of secondary data entry errors, and more accurate and complete data (e.g., Bloom 1998; Taenzer et al. 1997; Tourangeau and Smith 1996; Stone et al. 2002). Hence, the advent of electronic patient-reported outcome (epro) instruments has the ability to improve the quality and the usefulness of the data collected in clinical studies. With the rise in the use of epro instruments, study teams are now presented with considerations not previously of concern with paper data capture. Specifically, there is now the opportunity of requiring subjects to provide a response to an item before allowing the subject to proceed to the next item, in an effort to minimize missing data. This is in contrast to completing a PRO instrument on paper, where the respondent may not provide a response to an item, either accidentally or actively, resulting in missing data. This missing data may be problematic in a clinical trial, since there is limited scope for study teams to retrospectively query the subject about any missing data. The impact of forcing subjects to respond to electronic instruments is not well understood; however, it is a fundamental property of the study design that requires consideration. While the ability to require subjects to respond to items in a measure would seem to guarantee a complete data set, it does raise questions about the conditions under which it is appropriate to require subjects to respond to the items in an instrument. The purpose April 2014 Page 1
of this document is to provide guidance on the circumstances under which allowing a subject to opt-out of responding to an item, or items, may be appropriate. The discussion that follows assumes that rigorous care has been taken by a study team to select welldeveloped measures, with evidence of validity and reliability that are relevant to the study population (see Patrick 2011a and 2011b). The FDA s PRO Guidance (FDA 2009) and the draft guidance for the Qualification Process for Drug Development Tools (DDTs) (FDA 2010) highlight the importance of selecting concepts and measures that are appropriate for the target populations and context of use. Critically reviewing instruments before including them in a study will help identify those that may raise issues in regards to sensitive (e.g. questions of a sensitive nature) or nonsensical (e.g. questions about work when the patient is not employed) items; however, these issues are beyond the scope of this manuscript. The focus of the document is on issues relating to the collection of epro data in clinical studies; however, the discussion may also be considered applicable to the collection of epro data in other settings. There are three main scenarios that will be discussed in this document: 1. Requiring subjects to complete all items in all the instruments in the study; 2. Requiring subjects to complete all items used as key endpoints in the study, and allowing the subject to opt-out of responding to some, or all, other items (including sensitive items); 3. Allowing subjects to opt-out of responding to all items in the study. Note: For any of the above scenarios allowing the subject to opt out of responding to an item, the use of programmed edit checks are highly recommended to confirm that the subject intended to skip, or opt-out, of the item. This ensures that at the end of the study the database contains an explicit data point indicating when a patient actively decided to skip a question, which may assist in how to treat the missing data. Requiring Subjects to Complete All Items While requiring the subject to complete all items contained in a PRO instrument would seem to overcome the issues relating to missing data and ensure a more complete dataset, there a number of key issues that must be considered. The primary concern with requiring subjects to respond to an item is the possibility of obtaining poor quality data. For instance, if the subject does not wish to provide a response to an item, and is not allowed to skip the item and to move onto the next item, they might not provide a genuine, or reliable, response. On paper PRO instruments, such a scenario would typically result in a missing response. However, by requiring the subject to choose a response on the electronic April 2014 Page 2
platform before the subject can proceed, there is the risk of the subject providing inaccurate data that may add noise to analysis. Hence, there is a trade-off between complete but potentially inaccurate data, and the possibility of missing data points occurring within a dataset that may contain, overall, more accurate data. An extreme consequence of requiring subjects to respond to all items is the risk of a participant refusing to complete the remaining instruments for that study visit, or dropping out of the study altogether. To mitigate these possibilities, the subjects should be told at the time of study consent that they will be required to complete all of the items in the questionnaire. Further, there also is the possibility that ethics committees could raise concerns about not allowing subjects to opt-out of items. However, with the growing acceptance of electronic data capture this is not foreseen to be a significant issue. Requiring subjects to complete all items used as endpoints in the study, and allowing the subject to opt-out of responding to some, or all, other items (including sensitive items) An alternative to requiring completion of all items, is to require completion of those items used as primary or secondary endpoints in the study, and allow subjects to opt-out of some, or all, other items. This approach may overcome the concerns raised above about requiring completion of all items or instruments. If subjects are allowed to opt-out of items, a soft opt-out system should be implemented, meaning that the electronic system should ask the participant to actively confirm that they are choosing to skip the item. The system should capture the active opt-out and then allow the subject to advance to the next item. If allowing the subject to opt-out of all other items (beyond the items needed for analyses of the primary and secondary endpoints) is not acceptable, an alternative is to attempt to identify items that might prove problematic (e.g., sensitive or potentially inapplicable items), and allow participants to optout of responding only to these items, while requiring completion of all other items. This approach assumes that all potentially troublesome items can be identified beforehand, which may not be possible; however, it should be considered that participants may be presented with items for which they do not wish to respond, or which has no appropriate response. This is another case where requiring the subject to respond could result in inaccurate data being collected. However, this is predominately, but not exclusively, a limitation of poorly designed measures, which ideally would not be included in a properly designed study. While a well-designed informed consent document can inform subjects that they may be presented items of a sensitive nature, it cannot be assumed that participants will fully understand the type of items they may be asked to respond to until they are confronted with the items. For example, participants in an erectile dysfunction study may not be surprised if asked about sexual functioning; however, subjects may not expect items about sexual functioning in a depression study. It should be noted that electronic April 2014 Page 3
data capture might overcome some of the subject s concerns about reporting sensitive topics as it provides a private and secure data collection format, unlike paper. An added complication of trying to identify potentially inapplicable or sensitive items can occur in global studies run in unfamiliar cultures. Items that seem relevant or innocuous in one s native culture can prove to be problematic in other cultures. While well-conducted linguistic validation of a PRO instrument should identify previously undetected issues of this nature, requiring the completion of all items may result in the collection of inaccurate data. Allowing subjects to opt-out of responding to all items The final approach regarding the completion of an electronic measure is to allow the subject to opt-out of responding to all items. This approach may avoid many of the risks related to requiring completion of items identified in the previous sections. In this scenario, participants have the opportunity to skip any item they wish. However, this approach risks undermining one of the key strengths of electronic data capture namely, more complete data. This approach requires consideration of the allowable missing responses that permit a score to be produced on the instrument. As noted in the previous section, the electronic system should confirm that the subject intended to skip the item, capture the active opt-out in the dataset, and allow the subject to advance to the next item. This is an important element of allowing subjects to skip an item, as it has implications for the way that missing values are handled statistically, as described in the next section. Statistical Considerations One consideration in determining whether to allow subjects to opt-out of responding to items is the acceptability of missing data, versus potentially inaccurate data. While a complete and accurate dataset is obviously preferred, partial data on a small subset of participants may still be acceptable. As stated previously, if a subject is required to provide answers to an item, for which they may wish to skip, there is a risk that the subject may enter inaccurate and unreliable data. If the participant is provided the option of actively skipping the item, the data is likely sound even though missing data points may appear in the analytical dataset. Missing data can have a serious impact on the inferences drawn from a clinical trial. If enough data are missing, a clinical trial endpoint may not be evaluable. Missing data tend to be driven largely by discontinuation and loss to follow up; hence, Little and colleagues (2012) suggest limiting the burden and inconvenience of data collection on the subjects to make the study experience as positive as possible. It should be noted that the FDA s PRO Guidance (FDA 2009) and the draft guidance for the April 2014 Page 4
Qualification Process for Drug Development Tools (DDTs) (FDA 2010) stress the importance of selecting concepts relevant to the target population and choosing measures appropriate for the context of use. It is important that data missing due to an active skip by the participant be handled appropriately, most likely as data missing at random (MAR) or data not at random (MNAR). While there are a number of ways for dealing with missing data, these methods are outside of the scope of this paper but should be considered by the study team prior to implementing any of the approaches discussed previously. Furthermore, some PRO instruments have scoring algorithms specifically designed to account for missing responses that also should be considered when deciding on the best approach. Conclusions Complete and accurate data is the cornerstone of any clinical trial. Electronic data capture has provided study teams a powerful tool for collecting high-quality PRO data. One of the key strengths of electronically implemented PRO instruments is the ability to require subjects to provide a response to an item before they can advance to the next item. While this increases the likelihood of complete PRO instrument data at the close of the study, it can result in inaccurate, or unreliable, data being collected when subjects are confronted with items they are unwilling, or unable, to answer. In this situation, there is no method available for the study team to know this has occurred and that the data may be compromised. Further, the subject may refuse to proceed with responding to the instrument, or in extreme cases, may dropout of the study completely. This paper discusses three potential alternatives to requiring completion of all items in a PRO instrument: requiring completion of those items used as primary or secondary endpoints in the study, and allowing subjects to opt-out of some, or all, other items; and, allowing subjects to opt-out of responding to all items. While the first alternative protects the completeness of key data, it may result is some missing data for other items. This approach also assumes that potentially troublesome items can be successfully identified a priori. The second alternative provides the most freedom for subjects to opt-out of responding to any item, but again, at the risk of missing data. Study teams need to weigh the pros and cons of the various approaches outlined in this document carefully, being particularly mindful of the importance of specific aspects of the data needed to analyze the study endpoints. It is important to note that all of the items discussed in this document assume a well-developed and appropriate instrument or instruments are being used in the study. Failing to do so is a fundamental threat to the quality of a study, more so than issues around missing data. Regardless of the approach taken, it is strongly recommended that if some form of opt-out is allowed, that the electronic system be programmed such that the subjects actively confirm their intent to skip an April 2014 Page 5
item. While it is likely that the issue of allowing subjects to opt-out of responding to items is realistically of limited importance across the course of a clinical trial, the risk of missing data should still be considered carefully. Conversely, if requiring subjects to respond to every item (i.e., not allowing optout), the possibility and impact of the collection of inaccurate and unreliable data should be carefully considered. April 2014 Page 6
References 1. Bloom DE. Technology, experimentation, and the quality of survey data. Science 1998;280:847-848. 2. Little RJ, D Agostino R, Cohen ML, et al. The Prevention and Treatment of Missing Data in Clinical Trials. N Engl J Med 2012;367:1355-1360. 3. Marquis P, Caron M, Emery MP, et al. The Role of Health-Related Quality of Life Data in the Drug Approval Processes in the US and Europe: A Review of Guidance Documents and Authorizations of Medicinal Products from 2006-2010. Pharm Med 2011;25(3):147-160. 4. Patrick DL, Burke LB, Gwaltney CJ et al. Content validity - Establishing and reporting the evidence in newly-developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 1 - Eliciting concepts for a new PRO instrument. Value in Health 2011;14:967-977. 5. Patrick DL, Burke LB, Gwaltney CJ, et al. Content validity - Establishing and reporting the evidence in newly-developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: Part 2 Assessing respondent understanding. Value in Health 2011;14:978-988. 6. Stone AA, Shiffman S, Schwartz JE, et al. Patient noncompliance with paper diaries. BMJ 2002;324:1193-1194. 7. Taenzer PA, Speca M, Atkinson MJ, et al. Computerized quality-of-life screening in an oncology clinic. Cancer Practice 1997;5:168-175. 8. Tourangeau R, Smith TW. Asking sensitive questions: the impact of data collection mode, question format, and question context. Public Opinion Quarterly 1996;60:275-304. 9. US Food and Drug Administration. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims, December 2009. Available at: http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/uc M193282.pdf 10. US Food and Drug Administration. Guidance for Industry: Qualification Process for Drug Development Tools, October 2010. Available at: http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/uc M230597.pdf April 2014 Page 7