EMCDDA CONFERENCE 20 YEARS OF MONITORING AND COMMUNICATING EVIDENCE ON DRUGS

Transcription

1 EMCDDA CONFERENCE 20 YEARS OF MONITORING AND COMMUNICATING EVIDENCE ON DRUGS Workshop 10. Adapting to the modern world: the opportunities and challenges posed by new technologies; the challenge of online and virtual world LISBON 22 SEPTEMBER 2015 COMPARISON OF PAPER-AND-PENCIL VERSUS WEB ADMINISTRATION ON SELF-REPORTS OF SENSITIVE AND NON-SENSITIVE BEHAVIORS AMONG ITALIAN STUDENTS Sabrina Molinaro, PhD Institute of Clinical Physiology, National Research Council of Italy (CNR)

2 BACKGROUND : Advantages and disadvantages of web-survey over paper and pencil REDUCED COSTS OF THE DATA COLLECTION AVOIDANCE OF DATA ENTRY ERRORS BY AUTOMATICALLY FILLING ANSWERS INTO AN ELECTRONIC DATABASE ACCESSIBILITY OF DATA IN A SHORT TIME WEB QUESTIONNAIRES ARE DEPENDENT ON THE INTEGRITY OF PARTICIPANT SCHOOLS INTERNET CONNECTION ( SELECTION BIAS) INCREASING DIFFICULTY IN SCHEDULING DATA COLLECTION (COMPUTER LABS NEED TO BE RESERVED IN ADVANCE) NEED TO ENSURE ONLY SELECTED STUDENTS PARTICIPATE AND SUBMIT ONLY ONE SURVEY EACH, PRESERVING THEIR ANONYMITY Comparison of paper-and-pencil versus Web administration of the Youth Risk Behavior Survey (YRBS): Participation, data quality, and perceived privacy and anonymity, Denniston, M. M., Brener, N. D., Kann, L., Eaton, D. K., McManus, T., Kyle, T. M., Roberts, A. M., Flint, K. H., Ross, J. G., Computers in Human Behaviour, 26, 5 pp , 2010 Database ScienceDirect The impact of computer versus paper pencil survey, and individual versus group administration, on self-reports of sensitive behaviors, Bates, S. C., Cox, J. M., Computers in Human Behaviour, 24, 3, pp Year 2008

3 Aim of the study : To investigate the comparability of a wide range of data concerning sensitive and non-sensitive behaviors obtained via Web, versus data obtained via P&P questionnaires in a national representative sample of students (ESPAD Italia ).

4 Design and Sample 35 schools were randomly drawn from the ESPAD Italia 2014 sample ( 405 Schools ). A representative sub-sample of Italian secondary schools. Two full courses of study (from 1st grade to 5th grade) were randomly selected and assigned to P&P or Web condition. 53 schools refused to participate were replaced with others randomly drawn with the same characteristics (geographical and school type). Not specified problems 19% WEB-related problems 32% Missing courses 21% Time-related problems (eg involved in other projects) 28%

5 Design and Sample P&P Web N=2223 N=1937 pvalue Gender (male) Age (years) 16.9 ± ± Missing answers for gender and birth year. No answer to 50% of the core questions. Inconsistencies: answers reporting prevalence of use in the last year but not in lifetime, or cases in which last month use is reported but not last year and/or lifetime use. Max use (40 or more) for all substances. SENSITIVE: tobacco, alcohol, intoxication, cannabis, amphetamine, ecstasy, GHB, MDMA, cocaine, crack, LSD, mushroom, ketamine, heroin, heroin smoked, energy drinks, steroids, inhalants NON-SENSITIVE: fruits and vegetables consumption.

6 Design and Sample 31 Schools participating WEB (1937 students ) NO SENSITIV SENSITIV E ITEMS E ITEMS P&P (2223 students ) NO SENSITIV SENSITIV E ITEMS E ITEMS MISSING DATA INCONGRUENCES PREVALENCES

7 Results (I/IV) Missing answers for P&P (%) Web (%) N=2223 N=1937 pvalue Gender Age (missing) P&P (%) Web (%) N=2223 N=1937 pvalue Missing 50% questionnaire p<0.001 Max use all substances Deleted questionnaires (%)

8 Results (II/IV) % INCONSISTENCIES IN "NOT CLEANED" DATA P&P (%) Web (%) N=2223 N=1937 pvalue Tobacco Alcohol Intoxication Cannabis Amphetamine Ecstasy GHB MDMA Cocaine Crack LSD Mushroom Ketamine Heroin Heroin (smoked) Energy drinks Steroids Inalants Total inconsistences

9 Results (III/IV) LAST YEAR prevalence NOT CLEANED DATA P&P (%) IC(95%) Web(% ) IC(95%) pvalu e OR (web) Tobacco Alcohol Intoxication Cannabis Amphetamine Ecstasy GHB MDMA Cocaine Crack LSD Mushroom Ketamine Heroin Heroin (smoked) Energy drinks Steroids Inalants Fruit Vegetable

10 Results (IV/IV) LAST YEAR CLEANED PREVALENCE substance national prevale nce IC(95%) p&p prevalen ce IC(95%) web prevalen ce IC(95%) Two-sample proportion test naz vs p&p naz vs web p&p vs web % % % % % % % % % p-value p-value p-value Tobacco Alcohol Intoxication Cannabis Amphetam Ecstasy GHB MDMA Cocaine Crack LSD Mushroom Ketamine Eroine Eroine (smoked) Energy drinks Steroids Inalants Fruit Vegetable

11 In summary IN THE WEB STUDY IT WAS FOUND THE HIGHER PROPORTION OF THE MISSING 50% QUESTIONNAIRES (4,3% VS 0.9%) AND THE DELETED QUESTIONNAIRES (10.2% VS 7.9%). ADMINISTRATION MODE WAS ASSOCIATED SIGNIFICANTLY WITH THE REPORTING OF 4 RISK BEHAVIORS. IN GENERAL P&P MODE PRODUCED THE HIGHER PREVALENCE ESTIMATES. NO DIFFERENCES WERE FOUND FOR NON-SENSITIVE QUESTIONS. OVERALL, COMPARING WITH NATIONAL PREVALENCE, THE SUB-SAMPLE PARTICIPATING IN THE COMPARISON STUDY PRODUCED LOWER PREVALENCE ESTIMATES.

12 CONCLUSIONS A MODE EFFECT WAS OBSERVED AMONG ITALIAN STUDENTS, AS WELL AS A SELF-SELECTION OF THE SCHOOLS ABLE TO PARTICIPATE IN WEB-BASE ADMINISTRATION. BEFORE MAKING ANY CHANGES IN THE ADMINISTRATION MODE OF AN ONGOING NATIONAL SURVEILLANCE SYSTEM, IT IS CRITICAL TO CONSIDER THE POTENTIAL EFFECTS THAT SUCH CHANGES MIGHT HAVE ON: - SCHOOL PARTICIPATION, - RESPONDENTS PARTICIPATION, - DATA QUALITY - PREVALENCE ESTIMATES CHANGES IN THE DATA COLLECTION MODE COULD AFFECT THE ABILITY TO ASSESS TRENDS OVER TIME.

13 CONCLUSIONS SINCE IN MANY COUNTRIES THERE HAVE BEEN NO SIGNIFICANT DIFFERENCES BETWEEN THE DIFFERENT ROUTES OF ADMINISTRATION, WE BELIEVE THAT THE CULTURAL FACTOR HAS A VERY IMPORTANT ROLE CONSIDERING THAT IN THE COMING YEARS THE PROBABILITY OF MOVING FROM P&P TO THE WEB IS VERY HIGH, WE STRONGLY RECOMMEND TO CONDUCT METHODOLOGICAL STUDIES WITH THE AIM TO MEASURE THE ERROR WE'RE GOING TO INTRODUCE IN THE DATA OF OUR SURVEY IN ORDER TO ENSURE COMPARABILITY OF DATA

14 Acknowledgement Emanuela Colasante Valeria Siciliano Loredana Fortunato Elisa Benedetti Cristina Doveri Gabriele Trivellini Sabrina Molinaro Thank you for your attention!