THE DEPARTMENT OF VETERANS AFFAIRS HEALTH SERVICES RESEARCH & DEVELOPMENT SERVICE 810 Vermont Avenue Washington, DC 20420

Transcription

1 HSR&D Service Document # THE DEPARTMENT OF VETERANS AFFAIRS HEALTH SERVICES RESEARCH & DEVELOPMENT SERVICE 810 Vermont Avenue Washington, DC June 1989 Common Methodological Problems in Investigator Initiated Research Proposals Abstract This document presents an analysis of scientific critiques or research proposals submitted to the Health Services Research & Development (HSR&D) Service during three submission review cycles from March 1987 through January 1988 under the Investigator Initiated Research (IIR) Program. Proposals submitted to the HSR&D Service for funding are reviewed by a primary and two secondary reviewers selected from the Scientific Review & Evaluation Board (SREB) of the HSR&D Service. When considered appropriate, a collateral review of the proposal may also be sought from a subject-matter expert. The reviewers present their reviews in a meeting of the full SREB. Discussions of the reviews during these meetings are reflected in a summary statement prepared by the Executive Secretary. The summary statement as well as the individual reviews (without the identity of the reviewer) are provided to the investigators submitting the proposals. The individual reviews and summary statements on 103 IIR projects were used for this analysis. Deficiencies identified by the reviewers are ranked In order of the frequency in which they occurred among the critiques analyzed. Major methodological problems (in rank order) In IIR proposals submitted to the HSR&D Service relate to: 1) statistical analysis strategy, 2) logistics of data collection, 3) adequacy of research design, and 4) validity of measures. Information in this report is provided to increase investigators' awareness of methodological weaknesses commonly observed In IIR proposals so that they can improve the scientific merit of their proposals submitted for peer review. For More Information Contact: Jane Gulko, Ph.D. IIR Review Program Manager HSR&D-Special Projects Office VAMC Perry Point, MD 21902

2 Common Methodological Problems in Investigator Initiated Research Proposals* The VA Health Services Research and Development Service receives approximately one hundred and forty research proposals annually. These proposals, submitted by investigators throughout the VA system, are peer reviewed by the Scientific Review & Evaluation Board (SREB). As in any system of this type, the quality of the proposals varies widely. In an effort to assist investigators in enhancing the quality of their proposals, the current project was undertaken to identify recurrent methodological problems identified by the peer reviewers. It is hoped that a list of criteria which reviewers use to evaluate the scientific merit of proposals could be used by investigators as a reminder to address important conceptual and methodological and analytical issues in their proposals. Even if some problems do not have obvious solutions, if the investigators are aware of them, they can consider/discuss alternate solutions, and provide a rationale for the methodological decisions they make. Method In the VA review process, each proposal is assigned to a primary reviewer and two secondary reviewers who prepare written reviews for presentation at the meetings of the SREB. Each member of the review committee serves as the primary and secondary reviewer on several proposals. The principal components of the reviews are: description, critique (significance and originality, methodology and analysis, data source, organization, and management of the project), investigator qualifications, human subjects, facilities and resources, budget, importance and impact, and recommendations. At the SREB meeting, the written reviews are read and discussed by the members. Following the meeting, a summary review is prepared by the Executive Secretary which summarizes the written comments and the discussion of the proposal by the SREB. The written peer reviews and summary reviews of 103 VA research proposals from the VA review committee meetings of March 9-10, 1987, July 27-29, 1987, and January 11-13, 1988, were examined by the authors. Design, analysis, administrative, and * This is an edited version of a paper developed by Richard R. Bootzin, Lee Sechrest, Anne Scott, Maureen Hannah from the University of Arizona and is based on the analysis conducted by them on request by the VA HSR&D Service. methodological issues identified by reviewers were catalogued for each proposal. Proposals were considered the units of analysis, not reviewers. In fact, information identifying the reviewers was removed by the HSR&D Service before the reviews were sent to the authors. A methodological issue was listed for a proposal if it was brought up anywhere in any of the reviews regarding that proposal. It should be noted that the views of some reviewers may be represented more often than others by virtue of having had more proposals to review. However, each proposal was reviewed by three reviewers and discussed by the entire panel. Thus, the methodological issues identified reflect the collective judgment of a group of experts rather than the judgments of an individual reviewer.

3 Table 1 provides the frequency and percentage of proposals in which issues were mentioned either as part of the discussion (signifying their importance in the review) or as a criticism in the written reviews. Table 1 Frequency (Percent) of Occurrence of Methodological Issues In Proposal Reviews Mentioned Criticism Background Literature review 31 (30.1%) 25 (24.3%) Pilot research 17 (16.5%) 10 ( 9.7%) Internal Validity Adequacy of design 30 (29.1%) 40 (38.8%) Confounding 6 (5.8%) 39 (37.9%) Attrition 5 (4.9%) 23 (22.3%) Whether experimenters or 1 (1.0%) 5 (4.9%) subjects are blind to experimental conditions Construct Validity Adequacy of operational 7 (6.8%) 39 (37.9%) definitions Specification errors 1 (1.0%) 8 (7.8%) Statistical Conclusion Validity Adequacy of statistical 15 (14.6%) 53 (51.5%) analysis strategy Unit of analysis 4 (3.9%) 13 (12.6%) Power 19 (18.4%) 35 (34.0%) External Validity Representativeness of the 3 ( 2.9%) 38 (36.9%) sample Sample size 6 ( S.8%) 11 (10.7%) Measurement Issues Reliability of measures 17 (16.5%) 35 (34.)%) Validity of measure 12 (11.7%) 40 (38.8%) Appropriateness of measures 14 (13.6%) 27 (26.2%) Respondent burden 4 ( 3.9%) 33 (32.0%) Logistic/Administrative Issues Recruitment of subject 6 ( 5.8%) 21 (20.4%)s Data Collection 21 (20.4%) 4 (43.7%) Data management 29 (28.2%) 17 (16.5%) Training of staff 2 ( 1.9%) 4 ( 3.9%)

4 Work plan 16 (15.5%) 17 (16.5%) Cost Analyses 4 ( 3.9%) 33 (32.0%) Qualifications of Investigator Team Statistical qualifications 6 ( 5.8%) 24 (23.3%) Methodological qualifications 19 (18.4%) 30 (29.1%) Social science research 7 ( 6.8%) 17 (16.5%) qualifications Appropriateness of time 11 (10.7%) 16 (15.5%) allocations Budget Problems 18 (17.5%) 31 (30.1%) Significance 31 (30.1%) 29 (28.2%) Background In this section of the proposal, the applicant reviews the relevant literature, indicates how the current proposal is an extension of prior research, and may describe some pilot research. Whether a proposal had an adequate literature review was noted in over half of the proposals. Positive comments were made on 30% of the proposals while 24% were criticized for leaving out important and relevant areas or not being thorough enough. Approximately 10% of the proposals were criticized for not having conducted pilot research which, in some cases, would be a prerequisite for determining the feasibility of the proposed study. Internal Validity The adequacy of the experimental design is one of the most important issues. If the proposal had good design features, reviewers frequently mentioned them with approval. However, more than one-third of the proposals were criticized for having a poor design. The most frequently identified design problem had to do with the confounding of the experimental or comparison group characteristics with other variables. For example, geographic location or severity of the problem might be confounded with treatment conditions. Another important problem, affecting more than one in five proposals, was potential attrition. If differential attrition among treatment conditions were likely, the groups would no longer be equivalent. Reviewers particularly expected a discussion of attrition for long term studies in which discussion about how subjects were to be motivated to remain in the study was often missing. Attrition may also be a problem if it appears likely that its magnitude may affect-the statistical power of the study. Construct Validity Listed in this category are problems relating to the adequacy of definitions for the independent variables and errors of specification between different treatment conditions. The issues raised by reviewers regarding the adequacy of measures of dependent variables are compiled under measurement issues. More than one-third of the proposals were criticized for having inadequate operational definitions of treatment conditions. This included inadequately described treatment conditions (e.g., amount and length of

5 treatment), vague operationalization of constructs, and little rationale for interventions. Statistical Conclusion Validity The single most frequent criticism, applying to half of the proposals, dealt with the statistical analysis strategy. The proposals often did not include enough detail about the proposed analyses or the reviewers had reservations about the analyses proposed. Reviewers thought that proposals could have benefited from statistical consultation and considered such consultants important members of the investigative team. Generally, direct involvement of statistical and methodological experts was considered better than reliance on occasional consultants. A common consideration among statistical issues was power. Reviewers commented positively if power calculations were included and were critical if they were omitted. One-third of the proposals were criticized for either lacking a power analysis, providing incorrect power analyses, or for having inadequate power to test the hypotheses. A less common, but important, statistical problem was the unit of analysis. Proposals were criticized if they did not discuss what the appropriate unit of analysis should be and how that might affect the overall analysis strategy. External Validity The primary issue regarding external validity concerned the representativeness of the sample. This was seen as a problem in one-third of the proposals. Examples of the types of concerns identified by reviewers included selection problems that would result in poor generalizability, lack of discussion of important subject variables, and unspecified sampling plans. Measurement Issues The reliability and validity of measures of the dependent variables were recurrent issues identified by reviewers. If the measures were developed by the investigators, data about reliability and validity were expected. However, reliability and validity issues were often entirely omitted from the proposals. Even when standard measures were proposed, the inadequacy or absence of demonstrated appropriateness of the measures for the questions being investigated was frequently criticized. Sometimes it was noted that measures seemed to be included more because of the ease of data collection than because they provided an adequate test of the hypotheses. Over one-fourth of the proposals were criticized because the measures were not considered appropriate such as when there was an exclusive reliance on self report measures or if information could be obtained more reliably from medical records than directly from the subjects. Another common concern of the reviewers was related to the amount of respondent burden. Almost one-third of the proposals were criticized for excessive respondent burden and related issues such as underestimating practical problems posed by the measures. Logistic/Administrative Issues The most recurrent logistic problem related to data collection. Over 40% of the proposals were identified as having such problems. Reviewers often commented that the proposal gave no attention to data completeness or quality control and that there was

6 questionable reliability of data collection. Logistic problems relating to data management and difficulties associated with the recruitment of subjects were also frequently identified. Twenty percent of the proposals were criticized for inadequate specification of a work plan including personnel sufficient to complete necessary activities, their training, and a clearly articulated description of their responsibilities. Cost Analyses Almost one-third of the proposals were criticized for issues regarding costbenefit analyses. As was the case with statistical analyses, reviewers thought that the use of a consultant or other involvement of an expert would have been beneficial for those proposals in which cost-benefit analyses were proposed. Frequently, cost analyses seemed to have been "tacked on" without adequate consideration for their centrality to the problem under study or their complexity and without giving due consideration to collecting appropriate data for a clearly specified analytical plan. Qualifications of Investigators The most commonly identified problems regarding the qualifications of the research team had to do with methodological, social science research, and statistical expertise. For some proposals, the research team had little experience with research in the area of the proposal. Reviewers often noted the number of previous publications of the investigators if they were seen as insufficient. Consultants retained with limited time commitments were not seen as a solution to this problem. Many proposals were criticized because of inadequate time commitments by important members of the research team. Budget Problems Thirty percent of the proposals were criticized for budget problems. Many of these had to do with insufficient justification. The budget was examined carefully by reviewers for detail, thoughtfulness, and adequate justification. Sometimes the reviewers even commented that the proposed budget was not large enough for the project. Budgets that were either too large or too small raised questions of credibility about the research team's ability to manage the study. Significance Reviewers are required to comment on the significance of the project to the VA and in advancing the state of knowledge in health services. Over one-fourth of the proposals were criticized for not making a significant contribution to health services research. Issues identified in Table 1 that were most frequently cited by the reviewers are listed in rank order in Table 2. Table 2 Rank Order of Top l5 Methodological Issues Criticisms Overall (Positive & Negative Comments)

7 1 Statistical analysis strategy 1 Adequacy of design 2 Logistics of data collection 2 Statistical analysis strategy 3.5 Adequacy of design 3 Logistics of data collection 3.5 Validity of measures 4 Significance 5.5 Confounding 5 Background literature 5.5 Constructs adequately defined 6 Statistical power 7 Representativeness of sample 7.5 Reliability of measures 8 Statistical power 7.5 Validity of measures 9 Reliability of measures 9.5 Methodological qualifications 10.5 Cost analysis 9.5 Budget 10.5 Respondent burden 11.5 Constructs adequately defined 12 Budget 11.5 Logistics of data management 13 Methodological qualifications 13 Confounding 14 Significance 4.5 Representativeness of sample 15 Appropriateness of measures 14.5 Appropriateness of measures The fifteen most highly ranked criticisms could serve as a checklist against which investigators should check their proposals to ensure that they have dealt adequately with common methodological foci of the reviewers. Two issues deserving mention are those which deal with the general conceptual framework of research. First, the conceptual and theoretical integrity of the research seemed to be of paramount importance. Investigators would be well advised to put more thought into what they want to find out before they specify how to find it out. Many of the methodological problems could be resolved more easily if the investigators developed rigorously stated formal hypotheses emanating from a clearer conceptual framework for their research. Second, reviewers responded positively to evidence of programmatic research and a continuing commitment to a problem area. Proposals placed within a broader, programmatic and theoretical context seem to provide a better perspective for the reviewers to assess the significance of the proposed research and its potential contribution. Two issues are much higher on the overall list than they are on the list of criticisms, i.e., significance and background literature. Both are elements of the review outline provided to reviewers and, in as much as most reviewers made some comment relating to these, investigators would substantially strengthen their proposals if they developed these sections well. Summary This analysis suggests that while the problems involved in meeting review committee expectations are not simple, they are also not intractable. In preparing proposals, investigators need to ask themselves on which of the points suggested by this analysis might their proposed work be vulnerable. The material presented here could certainly serve as a checklist, a starting point. Investigators need to ask themselves whether they have dealt with all the issues and, if not, what steps are required to forestall criticism. Even if no completely satisfactory solutions are available for dealing with some issues, such as differential attrition in study groups, the investigators should, at a minimum, demonstrate their awareness of the problem and provide justification for their approach to its resolution. Finally, investigators need to provide assurance to review committees that the methods that are proposed will be sufficient to provide sound information justifying the types of conclusions they expect to draw. That

8 assurance will be in some proportion to the clarity of thinking and definiteness of plans that are put forth. These elements of a proposal are under the complete control of the investigators and can be effectively used by them for improving their proposals and the outcomes of their review.