SECONDARY DATA ANALYSIS: Its Uses and Limitations. Aria Kekalih

Transcription

1 SECONDARY DATA ANALYSIS: Its Uses and Limitations Aria Kekalih

2 it is always wise to begin any research activity with a review of the secondary data (Novak 1996).

3 Secondary Data Analysis can be literally defined as second-hand analysis. is the analysis of data or information that was either gathered by someone else or for some other purpose than the one currently being considered or often a combination of the two (Cnossen 1997) provide a cost-effective way of gaining a broad understanding of research questions helpful in designing subsequent primary research can provide a baseline with which to compare your primary data collection results

4 Kinds of Research can be Conducted with Anything but randomized trials By discipline: Secondary Data Outcomes research Epidemiology Health services research By question: Descriptive Comparative Causal

5 They are not primary data! Efficiency: fast and cheap No regrets Scale and scope Secondary data: Its Uses Size and detail not otherwise feasible for individual research team Generalizable Novel and creative research questions

6 Secondary data: Its Uses can be carried out rather quickly when compared to formal primary data gathering and analysis exercises. Where good secondary data is available, researchers save time and money by making good use of available data Depending on the level of data disaggregation, secondary data analysis lends itself to trend analysis as it offers a relatively easy way to monitor change over time. It informs and complements primary data collection, saving time and resources often associated with overcollecting primary data. Persons with limited research training or technical expertise can be trained to conduct a secondary data review (Beaulieu 1992; University of Cincinnati 1996).

7 Secondary data: Its Limitations Secondary data helps us understand the condition or status of a group, but compared to primary data they are imperfect reflections of reality. Without proper interpretation and analysis they do not help us understand why something is happening. The person reviewing the secondary data can easily become overwhelmed by the volume of secondary data available, if selectivity is not exercised. It is often difficult to determine the quality of some of the data in question. Sources may conflict with each other. Because secondary data is usually not collected for the same purpose as the original researcher had, the goals and purposes of the original researcher can potentially bias the study.

8 Secondary data: Its Limitations Because the data were collected by other researchers, and they decide what to collect and what to omit, all of the information desired may not be available (Israel 1993) Much of the data available are only indirect measures of problems that exist in countries and regions (University of Cincinnati 1996) Secondary data can not reveal individual or group values, beliefs, or reasons that may be underlying current trends (Beaulieu 1992)

9 CHALLENGES AND PITFALLS 1. Data mining/overfitting When the analysis precedes the question Does urine cortisol predict Catholicism? 2. Causal inference Inherently limited with observational data But does not preclude quasi-experimental designs to recover causal effects

10 CHALLENGES AND PITFALLS 3. Validity of measures Beware of assumptions Problems: coding, reporting, recall biases Solutions: direct validation in subgroup or another data source, literature review, sensitivity analyses 4. Complexity of file structure Row in dataset may not be unit of analysis Skip patterns, proxy respondents

11 CHALLENGES AND PITFALLS 5. Representativeness of Sample External validity (generalizability) Internal validity (selection bias) Example: comparing outcomes for insured and uninsured patients using hospital discharge data Must be hospitalized to enter sample Not only limits generalizability (to outpatients) But inferences about the sample may be wrong Sample would need to include uninsured who would have been hospitalized if insured

12 STATISTICAL CONSIDERATIONS: 1. MISSING DATA Sources Non-response: unit and item Variability in data collection (e.g. across states or over time, collected on subset due to expense) Incomplete linkages Approaches Listwise deletion, complete case Imputation Mean imputation (biased standard errors) Multiple imputation Weighting techniques Random effects models

13 STATISTICAL CONSIDERATIONS: Complex survey designs Example multistage probability sample (NAMCS): US divided into PSUs (counties / MSAs) sample of PSUs selected within each PSU, stratify MDs by specialty sample of MDs within each stratum quasirandom sample of patients seen by each MD Survey design 2. ANALYZING SURVEY DATA Clustering: convenience, precision Stratification: bad sample) Oversampling:

14 STATISTICAL CONSIDERATIONS: 2. ANALYZING SURVEY DATA Survey weights: affect point estimates Individuals may have unequal selection probabilities Need to apply weights to recover representativeness W = 1/p(selection) = # people represented W s reflect sampling design, adjustments to match to census totals, non-response Survey strata, clusters: affect se s Need variance estimators that account for correlated data Most statistical packages able to handle

15 References Korn, E.L. and Graubard, B.I. (1999). Analysis of Health Surveys. New York: John Wiley. (Must read) John Ayanian, MD, MPP, Ellen McCarthy, PhD, Research with Large Databases, Harvard School of Public Health Michael Steinman, MD. Using Secondary Data Analysis for Outcomes Research. Division of Geriatri. VA M. Katherine McCaston, HLS Advisor June Updated from M.Katherine McCaston (1998) -Partnership & Household Livelihood Security Unit. Tips for Collecting, Reviewing, and Analyzing Secondary Data. CARE