Surveying clinical surveys

Transcription

1 DOI: /j x Commentary Surveying clinical surveys PFW Chien Department of Obstetrics and Gynaecology, Ninewells Hospital and Medical School, Dundee, UK Correspondence: Dr PFW Chien, Department of Obstetrics and Gynaecology, Ninewells Hospital and Medical School, Dundee, DD1 9SY, UK. Accepted 27 May Please cite this paper as: Chien P. Surveying clinical surveys. BJOG 2009;116: Although survey research originally was mainly employed in applied social research, it is increasingly used in biomedical studies. 1 In this issue of BJOG, two such studies have been selected for publication. The first study investigates the preferences of pregnant women and healthcare professionals on either karyotyping or rapid aneuploidy detection in prenatal diagnosis in four Dutch hospitals 2 and the second study explores the experience and views of obstetricians in eight European countries on late termination of pregnancy. 3 The BJOG is interested to publish such studies but only on the condition that they meet the highest standards in the conduct and reporting of such studies. This type of research approach has previously been regarded as an easy method of conducting medical research but like all other types of research, high quality studies employing rigorous research methodology are required to generate internally valid results and conclusions. 4 Guidelines and checklists on good practice in conducting survey research are now available 4,5 similar to those available for randomised controlled trials, 6 observational studies, 7 diagnostic accuracy studies 8 and systematic reviews and meta-analyses, 9 qualitative studies, 10 cost-effective analyses, 11 economic evaluations, 12 case series 13 and case reports. 14 The reader is also referred to the EQUATOR Network website ( which is a collaboration aimed at raising awareness and to improve the conduct and reporting of medical research, including survey research. Furthermore, there is also a published checklist for reporting on internet-based surveys 15 in this website. The rest of this commentary aims to provide a summary of the salient features of high quality study design and reporting for survey research. Definition of survey research Surveys are generally considered to be descriptive studies, whereby data are obtained usually by questionnaires or interviews from a sample of a pre-determined population. 5 It is best suited to clinical studies, whereby the main aim is to evaluate the viewpoints or satisfaction level of healthcare consumers and/or providers. attempt is usually made to control for, or manipulate, any variables nor is there any allocation of participants to any groups or therapeutic treatments received. Research question Like all types of health service research, it is important to have a clear and explicit research question as this will provide a focus to drive the study design, method of population sampling, data collection and analysis. 4 If the aim of the study is to gather information at a certain time point, then a cross-sectional survey design should be employed 5 and this is the case for the two surveys published in this issue on the methods of preferred prenatal diagnosis for chromosomal abnormalities 2 and obstetricians experience of late termination of pregnancies. 3 On the other hand, if the aim of the study is to illuminate the direction of observed associations, then data should be collected at more than one time point on the same study population and a longitudinal study design would be more appropriate. 5 Sampling method Sample selection can be either random or deliberate. 16 Random sampling includes simple random sampling (when every individual in the general population has an equal chance of being included into the study sample), systematic random sampling (when the investigator randomly selects a starting point and then selects individuals systematically at a pre-specified sampling interval), stratified random sampling (whereby potential participants are organised into strata or groups and then randomly sampled within the groups to ensure that specific groups of interest are ª 2009 The Author Journal compilation ª RCOG 2009 BJOG An International Journal of Obstetrics and Gynaecology 1285

2 Chien represented within the sampled population) or cluster sampling(whereby the general population is divided into clusters and the clusters (or individuals within the clusters) are then sampled in a stepwise manner. 4 This is the sampling method employed in the survey on obstetricians experience of late termination of pregnancies. 3 Deliberate sampling is employed when the probability of a participant for inclusion into the study cannot be estimated. This method of sampling is employed when the researchers intend to study participants that are difficult to be recruited e.g. in the survey on the preference of either karyotyping or rapid aneuploidy detection in prenatal diagnosis of chromosomal abnormalities highlighted in this issue of BJOG, the target population was women undergoing amniocentesis in four Dutch hospitals. Because there are relatively few women undergoing amniocentesis for prenatal diagnosis of chromosomal abnormalities, recruitment was targeted in these four Dutch hospitals to ensure that a reasonable sample size can be feasibly achieved. This method of study design will naturally limit the generalisability of the results. Deliberate sampling can either be purposive sampling (individuals selected on the basis of meeting a pre-specified criteria), or quota sampling (researchers targeting a specific number of participants with a particular qualities), chunk sampling (whereby participants are selected based on their availability) or snowball sampling (researchers identifying participants meeting specific criteria, who in turn then identify other potential participants meeting the same criteria. 4 Sample size estimation and response rate Generally, there is no definitive answer the appropriate sample size for a survey study. The chosen target sample size is usually dependent on the research question and the level of statistical testing required for the study. 5 Formulas for calculating sample size for descriptive survey studies have been reported elsewhere. 4 For qualitative surveys, the sample size required will generally be smaller compared with that for quantitative surveys. 5 Once the sample size is determined a priori, the final sample with analysable data is then determined by the response rate in the survey. A high response rate not only increases the precision of any estimate, but it also reduces the risk of any selection bias and improves the internal validity of the results. 4 Generally, a response rate in excess of 70% is desirable although rates between 60% and 70% and sometimes <60% (for controversial study topics) may be acceptable. 20 It has been reported that reminders e.g. second and subsequent mailing in postal surveys will improve the response rate. 4,5 Questionnaire design In designing the questionnaire, a balance between including of all the potentially relevant items and the exclusion of irrelevant questions (to ensure that it is as concise as possible) need to be achieved. Some researchers will employ the Delphi process to select or eliminate questions for the questionnaire and then categorising the selected questions into domains within the questionnaire to ensure that the layout is easily readable by the study participants. It will also allow statistical testing to be more focused to items within the domains and ensuring that all the relevant questions pertaining to the research topic has been included into the questionnaire. 19,21 Each question should be focused on a single construct and the response should be either an open (free text) or closed (structured) format. 4 Open response format is generally employed for qualitative surveys whereas closed formats are either dichotomous (yes/no), ordinal (ranked responses like Likert scales) or continuous variables. 4 If statistical testing is required for the data analysis, then closed responses should be employed in the questionnaire. Openended responses are generally analysed by content analysis to generate themes (and their frequencies), preferably by at least two independent coders. 22 The questionnaire then has to be assessed for its reliability (ability to generate responses in a reproducible fashion) 23 and validity (it actually measures what it is purported to measure). 24 Reliability should be assessed for its test retest reliability (ability to yield the same response when the same questions are posed to the same participant at different time points), inter-rater reliability (ability to generate similar response when the same question is posed to participants with similar viewpoints) and internal consistency (the correlation between different items tapping into the same construct). 4 For test retest and inter-rater reliability, the statistical tests employed are the kappa statistic for dichotomous data, Spearman rho correlation coefficient for ordinal data and Pearson correlation coefficient for continuous variables. 4,25 If there are multiple raters employed in the assessment of reliability for continuous variables, then the intraclass correlation coefficient can be used. 4,25 The statistical test usually employed to assess the internal consistency of a questionnaire is Cronbach alpha for ordinal data. 4 The types of validity assessed in questionnaires include face (whether the questionnaire measures what it intends to measure during clinical testing), content (whether the questionnaire accurately measures all the fundamental aspects of the topic), construct (whereby specific criteria cannot be identified to adequately define the construct being measured) and criterion validity (whereby the responses to the survey items are compared with a gold standard ) ª 2009 The Author Journal compilation ª RCOG 2009 BJOG An International Journal of Obstetrics and Gynaecology

3 Surveying clinical surveys The common methods of conducting the questionnaire are postal questionnaires, face-to-face interviews, telephone interviews and online surveys. 5 Postal questionnaires have the advantage of being able to be delivered to a larger target sample over a wider geographical area, but suffer from a lower response rate. The lack of any personal contact between the researcher and the study participant tends to generate a lower response rate and this is usually compensated for by the use of a larger sample size and the use of reminders. Face-to-face interviews generally provide a higher response rate but are time consuming and more expensive to conduct. Telephone interviews and on-line surveys are cheaper and quicker to conduct than face-toface interviews, but the refusal rate is usually higher. Piloting Prior to launching the survey, researchers should pilot their questionnaire to ensure that it is comprehensible by the intended study participants, generate responses in a consistent manner and highlight whether all the relevant response categories are included in the closed format questions. 4,5 Table 1. Checklist for reporting for survey research 4 with the quality of reporting for the studies on the preferences of pregnant women and healthcare professionals on antenatal testing for chromosomal abnormalities 2 and the experience and views of obstetricians on late termination of pregnancy 3 Section Question Survey on views on antenatal chromosomal testing 2 Survey on views on late termination of pregnancy 3 Abstract Is the objective clearly stated? Is the design of the study stated? Is the study setting well described? Is the survey population described? Is the response rate reported? Are the outcome measures identified? Are the main results clearly reported? Are the conclusions appropriate? Introduction Is the problem clearly stated? Is the pertinent literature cited and critically appraised? Is the relevance of the research question explained? Is the objective clearly stated? Methods Is the study design appropriate to the objective? Is the setting clearly described? Are the methods described clearly enough to permit duplication of study? Is the survey sample likely to be representative of the population? Is the questionnaire described adequately? Have the validity and reliability of the questionnaire been established? Was the questionnaire administered in a satisfactory way? Are the statistical methods used appropriately? Results Do the results address the objective? Are all the respondents accounted for? Are the results clearly and logically presented? Are the tables and figures appropriate? Are the numbers consistent in the text and the tables? Discussion Are the results succinctly summarised? Are the implications of the results stated? Are other interpretations considered and refuted? Are the limitations of the study and its results explained? Are the appropriate conclusions drawn? From Burns et al. 4 which is modified from Huston et al. 29 ª 2009 The Author Journal compilation ª RCOG 2009 BJOG An International Journal of Obstetrics and Gynaecology 1287

4 Chien The questionnaire may need to be reviewed and revised further at this stage. The conduct of the questionnaire at this stage will allow for logistic problems in the execution of the questionnaire to be highlighted. Ethics and covering letter In general, the collection of any identifiable data from patients or where patients are approached directly for data collection will require formal ethics approval. If there is any doubt, the opinion of the local ethics review board should be sought. In accordance with good research governance, all participants should be provided with a covering letter to provide information on the aims of the study, the justification in the participant selection, contact details of the researcher and the manner in which the data collected will be stored and reported. 4,5 Reporting Checklists for items of reporting with survey research have been published elsewhere 4,5 and they include most of the issues discussed above. Table 1 shows the items of the published checklist and the quality of reporting for the two published articles on the preferences of pregnant women and healthcare professionals on antenatal testing for chromosomal abnormalities 2 and the experience and views of obstetricians on late termination of pregnancy 3 in this issue. In addition, the BJOG will also require that researchers to report on any potential conflict of interest, source(s) of financial support and individual contribution to authorship for each manuscript submitted for publication. It is important to be aware that surveys, particularly those conducted with deliberate sampling suffer from lack of generalisability. 4 In the two examples of such studies provided in this issue, the views on prenatal chromosomal assessment in the four Dutch hospitals 2 and late termination of pregnancy in the eight European countries 3 reported may not be representative of that out with the study population. Indeed, the data from the study on preferences by patients and healthcare providers on either karyotyping or rapid aneuploidy detection in prenatal diagnosis in the Dutch hospitals 2 showed disparate results between preferences of patient and healthcare workers and the preferences of patients in United Kingdom. 26,27 It is therefore important for researchers to report and discuss these differences in the manuscript. Conclusions Clearly, survey research can provide important information to guide clinical practice. However, it needs to be designed, conducted and reported to a sufficiently high rigour as for any other type of research work. This will require the provision of sufficient resources and expertise to achieve these goals. 28 Disclosure of interests Although Dr Chien is an Editor of BJOG, he has not been involved with the editorial process of this manuscript. Contribution to authorship Dr Chien was invited to write this commentary by BJOG. Details of ethics approval t Applicable. Funding ne. j References 1 Vernon A. A Quaker Businessman: Biography of Joseph Rowntree ( ). London: Allen & Unwin, Boormans EMA, Birnie E, Bilardo CM, Oepkes D, Bonsel GJ, van Lith JMM. Karytotyping or rapid aneuploidy detection in prenatal diagnosis? The different views of users and providers of prenatal care. BJOG 2009;116: Habiba M, Da Frè M, Taylor DJ, Arnaud C, Bleker O, Lingman G et al. Late termination of pregnancy: a comparison of obstetricians experience in eight European countries. BJOG 2009;116: Burns K, Duffett M, Kho M, Meade M, Adhikari N, Sinuff T, et al. A guide to the design and conduct of self administered survey of clinicians. CMAJ 2008;179: Kelley K, Clark B, Brown V, Sitzia J. Good practice in the conduct and reporting of survey research. Int J Qual Health Care 2003;15: Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134: von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Ann Intern Med 2007;147: Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. Clin Chem 2003;49: Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 1999;354: Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19: Siegel JE, Weinstein MC, Russell LB, Gold MR. Recommendations for reporting cost-effectiveness analyses. Panel on Cost-Effectiveness in Health and Medicine. JAMA 1996;276: Vintzileos AM, Beazoglou T. Design, execution, interpretation, and reporting of economic evaluation studies in obstetrics. Am J Obstet Gynecol 2004;191: ª 2009 The Author Journal compilation ª RCOG 2009 BJOG An International Journal of Obstetrics and Gynaecology

5 Surveying clinical surveys 13 Jabs DA. Improving the reporting of clinical case series. Am J Ophthalmol 2005;139: Sorinola O, Olufowobi O, Coomarasamy A, Khan KS. Instructions to authors for case reporting are limited: a review of a core journal list. BMC Med Educ 2004;4:4. 15 Eysenbach G. Improving the quality of web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res 2004;6:e Aday LA, Cornelius LJ. Designing and Conducting Health Surveys. A Comprehensive Guide, 3rd edn. San Francisco: Jossey-Bass, Henry RC, Zivick JD. Principles of survey research. Fam Pract Res J 1986;5: Rubenfeld GD. Surverys: an introduction. Respir Care 2004;49: Passmore C, Dobbie AE, Parchman M, Tysinger J. Guidelines for constructing a survey. Fam Med 2002;34: Sierles FS. How to do research with self-administered surveys. Acad Psychiatry 2003;27: Ehrlich A, Koch T, Amin B, Liewehr DJ, Steinberg SM, Turner ML, et al. Development and reliability testing of a standardised questionnaire to assess psoriasis phenotype. J Am Acad Dermatol 2006;54: Weber PR. Basic Content Analysis, 2nd edn (Sage University Paper Series on Quantitative Applications in the Social Sciences, ). California: Sage Publications, Khan KS, Chien PFW. Evaluation of a clinical test. Part I: assessment of reliability. BJOG 2001;108: Chien PFW, Khan KS. Evaluation of a clinical test. Part II: assessment of validity. BJOG 2001;108: Hirsh RP, Riegelman RK. Statistical First Aid. Interpretation of Health Research Data, 1st edn. Boston: Blackwell Scientific Publications, Grimshaw GM, Szczepura A, Hultén M, MacDonald F, Nevin NC, Sutton F, et al. Evaluation of molecular tests for prenatal diagnosis of chromosomal abnormalities. Health Technol Assess 2003;7: Ryan M, Diack J, Watson V, Smith N. Rapid prenatal diagnosis testing for Down syndrome only or longer wait for full karyotype: the views of pregnant women. Prenat Diagn 2005;25: Thompson AGH. Questioning practices in health care research: the contribution of social surveys to the creation of knowledge. Int J Qual Health Care 2003;15: Houston P. Reporting on surveys: information for authors and peer reviewers. CMAJ 1996;154: ª 2009 The Author Journal compilation ª RCOG 2009 BJOG An International Journal of Obstetrics and Gynaecology 1289