ISPOR Good Research Practices for Retrospective Database Analysis Task Force

ISPOR Good Research Practices for Retrospective Database Analysis Task Force II. GOOD RESEARCH PRACTICES FOR COMPARATIVE EFFECTIVENESS RESEARCH: APPROACHES TO MITIGATE BIAS AND CONFOUNDING IN THE DESIGN OF NON-RANDOMIZED STUDIES OF TREATMENT EFFECTS USING SECONDARY DATA SOURCES - Report of the ISPOR Retrospective Database Analysis Task Force Part II Comments received from Reviewer/Leadership/ISPOR Membership : Respondent #1 Thank you for the opportunity to review and comment on the documents pertaining to good research practices for nonrandomized studies of treatment effects using secondary databases. This is very important work and the authors have put together very thoughtful and useful information. I will comment on the documents generally and then each document separately. Overall comments It would benefit the audience if the authors could first define their key terms such as effectiveness research comparative effectiveness research as secondary data sources. observational data epidemiologic studies and to use these terms and definitions consistently across the three documents. Providing common definitions would also be extremely helpful to the field. It s not clear whether these series is limited to comparative effectiveness or effectiveness. The authors seem to use observational research and epidemiologic research interchangeably. In discussing the relative merits of observational studies versus RTCs, the authors should acknowledge that most evidence hierarchies consider RTCs the gold standard of evidence and rank case control and cohort studies further done on the evidence later. The main reason for this is that unmeasured or unobservable confounding brings into question the validity of observational studies. This does not mean that observational studies are not useful, it just acknowledges how they are currently being viewed and why. The role of unobservable confounding and the limited clinical information to control for confounding should be acknowledged early on in the documents. II.Comments Specific to Good Research Practices for Designing Non-Randomized Studies of Treatment Effects Using Secondary Databases. The definition of comparative effectiveness provided in the abstract sounds more like effectiveness than comparative effectiveness, for example, see the IOM definition of comparative effectiveness clinical information on the relative merits or outcomes of one intervention in comparison to one or more others; benefits or harms (Institute of Medicine. 2007), Additionally, in some places of the manuscript the definition of secondary data seems limited to insurance claims data, in other places it seems to include medical records. Are hospital discharge abstracts such as HCUP and registries included in the definition? The document by Berger and colleague seems to include these types of data in the definition of secondary data. The paper covers a lot of ground but could benefit from an overall organizing viewpoint that is laid out in the introduction so that the reader can anticipate what will be presented and how it all fits together. 1

Much of the discussion focused on measurement error in claims data. The presumption is that there is less error in primary data collection studies, which seems to be a simplification. Some items, for example, expenditures and utilization, will be much more accurately measured in claims than through primary data collection. An important limitation in secondary data, perhaps more important than the discussion of measurement error, is the lack of clinically data on secondary data. These may include laboratory tests, pathology results, radiology reports, symptoms scales, measures of functioning, etc. Although EMR data sets have more clinically rich data, many of these data elements will be missing in various systems. Thus a key limitation of secondary data analysis, that could be discussed, is the fact that you are working of the data that you have rather than the data that you would ideally want to control for confounding or measure outcomes. A key problem with claims and EMR data that may deserve more discussion is censoring bias. Secondary data is both subject to left and right censoring. For example, incident users are only defined as incident to the extent that patients can be followed retrospectively and most studies do not include more than a 3 or 6 month clean period. On the other hand, outcomes are often measured only over continuously enrolled patients which may lead to the exclusion of patients who dropped out because of serious illness or death. Respondent #3 Overall- it would be nice to have a short commentary/executive summary talking about all of the papers together and outlining the recommendations from them. Paper2 Focused on administrative claim and EMR data- I think much of the information would pertain to registry data as well as secondary analyses of a completed prospective study- maybe worth mentioning. In the abstract it mentions that compared to RCT, observational studies have issues with validity. May want to specify internal validity. On page 4, when talking about prescription claim data, it may be worth mentioning that although it is a wealth of informationit still doesn t tell us whether or not they actually took the medicine. Casual graphs are discussed but colliders or effect modifiers are not mentioned. I understand that the focus is on confounding but I think it should at least be mentioned. Box highlighting the overall recommendations Respondent #6 Thank you for providing me the chance to read these documents. All the topics are interesting and provide careful insight to the respective subject matter. However in the attachment: Comments on II. GOOD RESEARCH PRACTICES FOR DESIGNING NON-RANDOMIZED STUDIES OF TREATMENT EFFECTS USING SECONDARY DATABASES: Report of the ISPOR Retrospective Database Analysis Task Force Part II This report has good points. It seems that the recommendations should be numbered and listed in a table. There is some overlap between this report and the first one by Berger et al. Recommend that the domains for each be established and points managed in a single document. 2

Line 441 Patients with the Same Treatment Indication: Alternative Drug Users. The authors could perhaps include a mention that some studies may want to identify different indications in order to compare outcomes and remove the impact of the underlying disease. For example, some side effects may be due to the drug or to the disease or to both (interaction). Using a group exposed to the drug but not having the disease is a distinct advantage here. Respondent #8 Thanks for this important work! It is very helpful for evaluating routine data. I plan to use it as a sort of checklist so I won t forget important questions. You should take into account that a lot of work with this type of data is never published because it is used for decision making or negotiations in settings which are not very transparent. There it is of high priority to answer questions in time and fitting the system you are working in. Scientific accuracy in most cases is not so important because results are not published. Publication only occurs in transparent settings or if some people think it supporting for their career. So it would be helpful to have these reports published so they can be referred to. Respondent #11 II. Good Research Practices for Designing Non-randomized Studies of Treatment Effects Using Secondary Databases Need some practical explanantions. Respondent #12 3. To check reliability of the study, grading on the scale of 10 should be done. 4. Unpublished and published case series can be helpful for the preparation of the databases. Pulling data from different case series (may be from same region) can form a bigger sample size. 5. Stratification of case series, for a particular disease can be performed to target a research question. 6. Addition of the drug utilization studies into the task can be helpful to make a database to find out cost effective treatment for a disease. Respondent #13 I appreciated the opportunity to review the three working papers from the ISPOR Good Research Practices - Retrospective Database Analysis Task Force. I found all three well written and informative. My one comment would be to encourage this ISPOR task force to consider cross trial comparisons as part of your mandate to recommend good research practices for non-randomized studies of treatment effects using secondary databases. As you know, cross trial comparisons are routinely conducted by Industry and HTA agencies such as NICE and DERP to explore both issues of comparative and cost -effectiveness. Whilst the original clinical trial may have met the definition of a RCT, when they are aggregated for these cross trial comparisons the database (a collection of clinical trials) is no longer a randomized trial and face many of the same methodological challenges you discuss in your task force papers as a simple example, the publication may not contain sufficient information to determine that patients are comparable at baseline. Using placebo controlled trials to make head-to-head comparisons certainly qualifies the analysis as secondary. A white paper that offers guidance on designing and conducting scientifically robust cross trial comparisons would greatly benefit the field. Respondent #14 The drafts look very good. I just have a few comments. 3

2. Report II (1) the table below line 145, is it worth mentioning negative binomial as an error distribution for count data? Negative binomial regression can be used when there are signs of overdispersion in Poisson regression. Respondent #15 Thank you very much for the opportunity to briefly comment on these excellent reports. As more and more transnational research is performed, and in the light of increasingly limited resources, a comment on prespecified country-specific secondary analyses of such studies - if performed in a multinational setting - may be of help for country-specific decision making, however: - Justification for such secondary country-specific analyses needs to be provided a priori - Meaningful, pre-defined minimum patient numbers from these countries need to be reached (conditio sine qua non) and CID should be determined - Country-specific results need to be put into perspective, i.e., discussed in the light of the results of the entire cohort Respondent #17 Comments for Report II Authors did a great job in describing main challenges of using data from non-randomized studies for epidemiological or effectiveness analysis. This review identifies principal possible sources of biases in misclassification and confounding, as well as possible problems in measurement of exposure and outcome. The review has valuable recommendations for researchers on how to deal with these possible challenges to provide valid and evidence based conclusions. Below are comments on some of the parts of the review. They provide additional perspectives to the issues described in the review. First a few typos: Line 73.It is measure Line 74.It is inaccuracies Line 81.Tables need to be numbered. Line 86. Largely because it is designed for billing purposes, prescription claim data has important information on payments, including payment by insurers, co-payments and deductibles. This economic data can affect the exposure and the outcome of the drug therapy. The claims data often has information on pharmacies where the drug was sold and this information on pharmacies can be used to deal with potential confounding factors such as: accessibility, accessibility within health insurance coverage and competitiveness of the prices in the local pharmacy market. ECONOMIC DATA AS A PART OF CLAIM INFORMATION CAN BE IMPORTANT SOURCE TO IDENTIFY CLASSIFICATION BIAS AND POSSIBLE CONFOUNDING FACTORS. Line 119. In Table X, it is important that the fact of use or being exposed to the drug based on medication claims can be biased. The claim provides evidence of the fact that the drug was purchased but some of these drugs may not be used or may be used for other purposes. For example, a patient can purchase antibiotics, but take it for shorter period of time than it was 4

prescribed. FILLING PRESCRIPTION AT THE PHARMCY (AS IT IS EVIDENCED BY THE CLAIM) DOESN T NECESSARULY IMPLY THAT THE PATIENT IS EXPOSED TO THE MEDICATION AND IT CAN BE THE SOURCE OF POTENTIAL BIASE. Line 170 and 192.It is important to adjust not only for eligibility but also for the benefit plan design because having different benefit plan designs or changing it can affect compliance and therefore contribute to the classification bias. For example, a switch to the less expensive health plan but with higher out-of-pocket payments for medications can cause the patient to stop taking medication (especially, when it is not life-threatening condition). SO NOT ONLY ELIGIBILITY MUST BE CONTROLLED BUT ALSO BENFIT PLAN DESIGN OF HEALTH CARE PLANS. Line 215. A follow-up period should be long enough to obtain important outcomes measures and at the same time short enough to have sufficient sample size of continuously enrolled patients with characteristics of interest. Line 237. Whenever possible one should exclude from the study individuals enrolled in capitated health plans. Line 241. Misclassification of diagnosis and procedures are not rare in the claims data. As authors correctly pointed out for some diseases the source lies in clinical misclassification. For example, asthma, COPD and some other respiratory conditions are often misdiagnosed, especially when not all available tools for diagnosis are used, like spirometry. Most importantly, researchers must make clear from the beginning what case definition for the condition they use and underscore possible limitations in generalizability of the results. Line 407. Adherence can cause classification bias of exposure and can be a confounding factor in the analysis. Adherence is affected among other factors by the level of the risk aversion of the individual, in other words by the degree of how risky the person is. Another factor to access the adherence to the certain medication is whether there is a substitute for this drug. For example, when the study is about the effect of controller medication to prevent asthma patients from exacerbations, asthma patient may have low adherence to the controller medications because of willingness to take this risk and reliance on reliever medications. Respondent #19 On document 2, designing: There seems to be no appropriate introductory section describing the overall need to create operational definitions of exposures and outcomes that will keep the limitations of each dataset in mind -- this section is critical, along with illustrative examples. There are bits and pieces throughout, but it's really a central theme that should be displayed up front. The authors should be careful NOT to describe confounding as a bias, because it is not one -- it is not a systemic design or data collection issue, it is a reality that must be addressed and mitigated. I am not sure of the rationale for describing anything but a "new user" design as the basis for treatment-effect studies. This seems like the best document for describing demographic and/or propensity matching as well as IV designs. Why was this not included? On document 3, analyzing: Stratification's benefits extend beyond epidemiology; would suggest replacing the term with "observational study". Missing from the regression section is the notion of "common support". It is not enough to include covariates that may or may not differ on average between two treatment groups. Scatter-plotting must also be done to make sure the distribution of each variable has enough overlap between groups -- if these distributions occupy very different areas of the covariate space, a bias will be imparted to any regression model (Dehejia R, J Economics 2005; Heckman JJ, Rev Econ Stud 1997) Respondent #23 5

I am afraid that I could not reply to this request in time. ISPOR Good Research Practices for Retrospective Database Analysis Task Force Comments If I may add a few comments for these great papers, I would like to point out to add a section how to integrate patient reports. I think patients' self-reported dataset which will fill in the bias of secondary datasets, mostly likely to be claims based information. Respondent #24 Thank you for considering me to make comments on these documents. In general, I think that they are well written and understandable. I just make few comments and I hope they can be useful. Comments to the following documents: Good research practices for designing non-randomized studies of treatments effects using secondary data bases In the section that talks about eligibility, it may be worth to deep on how the lack of eligibility can be controlled or corrected in the study? In the paragraph that talks about the questions that we need to consider to address the issue of confounding. It may be appropriate to specify another question before to ask about the way to control confounding. It may be worth to know to what extent confounding can affect the outcomes of the study. If we describe the magnitude of the effect of confounding in the study, it may be easy to describe the more appropriate approach to tackle this issue. Therefore the comprehension of this point may be follow easily when reading about this issue. This is because, I think that confounding is not an easy topic to follow, especially if you are beginning to know about it. It may lead to several confusions; therefore it should be clear for the reader the origin, its importance and the potential harmful for the study, as well as the consequences of not addressing this point adequately. Respondent #25 II. GOOD RESEARCH PRACTICES FOR DESIGNING NON-RANDOMIZED STUDIES OF TREATMENT EFFECTS USING SECONDARY DATABASES: Report of the ISPOR Retrospective Database Analysis Task Force Part II Line 301 The assertion that traditional textbook techniques are inadequate in controlling for simultaneity might be inaccurate. Indeed, 2SLS or instrumental variables techniques and dynamic panel data models can be used in analyzing/addressing the issue of simultaneity. Line 336 A brief commentary on whether the confounding being described in the report is individualspecific and how to handle such a situation may be helpful. Line 357 Since restrictions tends to be based on observables, I believe confounding attributable to unobservables may still plague the analysis. Overall, the report could benefit from some proofing (grammar, punctuation,etc.) III. GOOD RESEARCH PRACTICES FOR THE ANALYSIS OF NON-RANDOMIZED STUDIES OF TREATMENT EFFECTS USING SECONDARY DATABASES: Report of the ISPOR Retrospective Respondent #26 Please find comments as requested below: II. Good Research Practices for Designing Non-randomized Studies of Treatment Effects Using Secondary Databases 6

This paper gives very clear information up until the section of causal graphs (line 312). At this point issues are addressed with the guidance of the causal graph. I found myself having to re-read these sections a couple of times to grasp this idea. I have read these three papers as a consumer looking for guidance and have found them very useful. I hope these comments will in some way be help. Respondent #27 Comments for report 2: Results and conclusion derived from this study are more helpful to understand the different aspects about drug exposure and medication. Points discussed under the heading of "classification" and "new user design" are very impressive. Lot of efforts have been taken by authors to propose this valuable and very useful study. Respondent #31 Overall- it would be nice to have a short commentary/executive summary talking about all of the papers together and outlining the recommendations from them. Paper2 Focused on administrative claim and EMR data- I think much of the information would pertain to registry data as well as secondary analyses of a completed prospective study- maybe worth mentioning. In the abstract it mentions that compared to RCT, observational studies have issues with validity. May want to specify internal validity. On page 4, when talking about prescription claim data, it may be worth mentioning that although it is a wealth of information- it still doesn t tell us whether or not they actually took the medicine. Casual graphs are discussed but colliders or effect modifiers are not mentioned. I understand that the focus is on confounding but I think it should at least be mentioned. Box highlighting the overall recommendations Respondent #32 II. Good Research Practices for Designing Non-randomized Studies of Treatment Effects Using Secondary Databases Line 74: "in accuracies" should be replaced by "inaccuracies" Line 81: Table x - I hope the x would be replaced by an actual number later? Line 96-97: "...NDCs codes is used" should be replaced by "...NDC codes are used". Line 152: "...someone as drug exposed" be replaced by "...subjects as being exposed to drug" Line 179: The requirement of continuous health plan enrollment often reduce the study sample significantly - may talk about the trade-off between the sample size and longer period of continuous health plan enrollment. Line 193-195: I am not sure how readily the information like prior authorization be available in administrative claims databases. Line 208-209: I think measuring exposure to OTC medication using administrative claims data is not a good idea to begin with. Line 229-230: "...that considered clinical..." should be replaced by "...what is considered as clinically..." Line 286: I could not find where unmeasured (residual) confounding factors discussed in this report. 7

Line 407: Please note that socio-economic variables such as "educational status" will not be available in a typical claims database. One can, of course, link the claims database to other databases to capture SES data. Line 415: I will include the following sentence after the full stop: Therefore, it is recommended that for comparisons involving a new drug, the study should ideally include only new patients. Line 438: I would drop "unmeasured" or qualify its use in some other way as it is impossible to determine the unmeasured risk factors between the treated and the comparison cohorts. 8