Institute of Justice POLAND. Final Report Quality Report

Transcription

1 Institute of Justice POLAND Translating and testing a victimisation survey module JLS/2007/ISEC-FPA/050 Final Report Quality Report Authors: Andrzej Siemaszko Jacek Czabański Beata Gruszczyńska Marek Marczewski Katarzyna Drapała Roman Kulma Paweł Ostaszewski Anna Więcek-Durańska Warsaw, February

2 Contents 1. Response rates Organisational aspects of testing Description of difficulties encountered Comparison of the results with data from other crime surveys conducting in Poland Comparison of the results with data from national administrative sources Reference period preferred for the analysis of results and reason for doing so Breakdown of responses for each question An assessment of the results achieved for each question and the utility of the question An estimate of the cost of implementing such a module at national level Conclusions and recommendations for further actions Final comments Annexes Annex 1. Comments to the questionnaire Annex 2. Breakdowns for Parts A, B, C, E, F and R Annex 3. Breakdowns for Part D Annex 4. Breakdowns for Part G (CAPI only).... 2

3 1. Response rates In Poland, in line with the assumptions adopted for testing the victimisation module, two interviewing techniques, CATI and CAPI, were used. CATI interviewing was performed on a nationwide random sample of 500 persons aged 16 and more. The operand for drawing nation-wide, representative samples in CATI type surveys is the register of housing resources run in the Central Statistical Office (GUS). On the basis of those data, proportional layer drawing of measuring points was performed (technical aspects of drawing the sample were described in Report no. 2). The number of interviews, refusals and respondents unavailability is as follows: - Total number of attempts: Number of completed interviews: Number of interrupted interviews: 33 - Number of refusals: Not done for other reasons (wrong number, no one at home, etc.): 3450 That information served as a basis for calculating the response rates - gross and net. The gross response rate was 8.75%. It was calculated as a quotient of the number of interviews completed, and the total number of telephone contacts. The net response rate was 22.1%. It was also calculated as a quotient of the number of interviews completed, and the total number of successful telephone contacts (i.e. excluding unsuccessful attempts: wrong telephone number, respondent s absence at home). CAPI survey was carried out on a sample of 100 persons aged The selection of a random route type was used. This selection technique consists of a random drawing of start addresses and then selecting from within their direct environment, according to strictly prescribed rules, the foreseen number of further addresses. 3

4 In the survey discussed, each drawn address was a starting point to choose only one address. 1.2 Duration of interviews Along with the response rates, an important piece of information is that on interviewing duration. In CATI exercise, the average interview duration was 26 minutes (minimum 11 minutes and maximum 84 minutes). In CAPI, the average duration was 41 minutes (minimum 13 and maximum 153 minutes i.e. 2 hours and 33 minutes). It should be mentioned that the CAPI questionnaire was longer. This is discussed in detail in the section about the fieldwork below. Frequency distributions by interview duration time in CATI and CAPI techniques parts are presented in Figure 1 and Figure 2. 4

5 Figure 1. Duration time of CATI interviews Number of interviews Duration time in minutes Figure 2. Duration time of CAPI interviews Number of interviews Duration time in minutes 5

6 2. Organisational aspects of testing 2.1. Organisational issues The pilot study of the module consisted of several phases. In the first one, the questionnaire was translated and the cognitive test carried out. Then the necessary modifications of the questionnaire, as well as of some questions in particular, were done. In the second phase, the Institute of Justice team elaborated the pilot study concept and the design of sample drawing. In the next stage, the field work was carried out by TNS OBOP (a survey company which was awarded the contract for fieldwork) under the supervision of the Institute of Justice. Before launching the fieldwork, the Institute of Justice thoroughly analysed the questionnaire and reported several observations and doubts (see Annex 1. Comments to the questionnaire). In collaboration with TNS-OBOP, some questions in the questionnaire were adopted to Polish circumstances (see point 2.2 below). Moreover, the Institute of Justice team took part in instructing interviewers as to the following issues: 1) description of victimization surveys, the aim of the survey concerned and the way the questionnaire is arranged; 2) detail presentation of the contents of individual questions; 3) analysis of observations, uncertainties and doubts as to the questionnaire, as reported by the interviewers; 4) discussion over the remarks voiced by the interviewers; 5) detail description of organizational aspects of the survey. This training was videoed and then the tapes sent out to the survey coordinators. 6

7 2.2 Modification of the questionnaire before the fieldwork Before the launch of the fieldwork, the questionnaire was modified. The following modifications were introduced: In both CATI and CAPI questionnaires questions on the respondent and his or her household (questions A4 A14 and A16 A18) were moved to the end of the questionnaire (but were put before Part G concerning being victim of violence). Moreover, in the CATI questionnaire, which was very lengthy, having the TNS OBOP consulted, it order to avoid respondents withdrawals it was decided to shorten Part G. Therefore, in the CATI questionnaire, only the following questions from the part concerning violence were included: Q9 Q28; Q31 Q38 and Q40 Q64, as well as Q111 Q117. In the English version of the questionnaire, answers like Don t know, No answer or Don t like to answer were not included in many instances. In the Polish version such answers were added in all questions requiring them (particularly in Part G). 3. Description of difficulties encountered In spite of our suggestions to modify some questions and/or make them shorter, especially in Part G, the questionnaire was not amended and, as a consequence, we faced serious difficulties during the fieldwork. A long and complicated questionnaire with many filters, and several meticulous questions with enormously developed configurations of answers caused serious difficulties in understanding by respondents and might have led to errors and misunderstandings. Also, the length of the questionnaire was a significant reason for achieving a low completion rate (numerous refusals). Thus, due also to the organizational and financial constraints, only 100 CAPI interviews were carried out successfully. 7

8 4. Comparison of the results with data from other crime surveys conducting in Poland The CAPI and CATI pilot study results were compared with the Polish Crime Survey and the International Crime Victim Survey (ICVS): The Polish Crime Survey (PCS). PCS was a victimisation survey financed by Polish National Police Headquarters. The survey was carried out in collaboration with Institute of Justice (in January of each year) in , on the random sample of 17,000 inhabitants. In total, 51,000 respondents were interviewed. The PCS questionnaire took into account six crime types, namely: theft of car, theft from car, theft generally, burglary (into apartment, garage, cellar etc.), robbery, and assault. The respondents were asked on being victimized in the year preceding that of the survey (a yearly victimization). A simple and concise questionnaire allowed to interview huge numbers of respondents, with few refusals. The International Crime Victim Survey (ICVS 04/05). ICVS is an international victimization survey carried out each fourth year, starting from Poland participated in all the hitherto ICVS sweeps. In Poland, the last edition was carried out on the nation-wide random sample of 5013 inhabitants aged 16 and over. In the ICVS, a standard questionnaire is being used to ask questions concerning a dozen or so criminal offences: theft of car, theft from car, vandalism to car, theft of motorcycle, moped, scooter, theft of bicycle, personal property theft, household burglary, attempted burglary, robbery, assault (and threat of assault), sexual incident, consumer fraud, and corruption. Questions on reporting such crimes to the police were related to a five year period directly preceding that of the fieldwork. It is to be stressed that comparability of those data is very limited due to differences in definitions, sample design patterns etc., as well as interviewing techniques. Both the ICVS and PCS took the form of face to face interviews. Also, small samples used in the pilot study of the victimization module (CATI 500, CAPI 100) practically render comparisons with other surveys impossible. In Table 1, the yearly victimisation rates of Module CATI, CAPI (2008), PCS (2008), ICVS (2004) are compared. 8

9 Table 1. One year victimisation rates in Victimisation module, PCS and ICVS. Percentage of victims CRIME TYPES MODULE CATI MODULE CAPI PCS 09 (Polish crime survey) ICVS 04 1 THEFT OF CAR (% of owners) * THEFT FROM CAR (% of owners) * CAR DAMAGE (% of owners) n.a. n.a. 4 THEFT OF MOTORCYCLE, SCOOTER, MOPED (% of owners) n.a BICYCLE THEFT (% of owners) n.a HOME BURGLARY ** OTHER BURGLARIES (second house) (% of owners) n.a PROPERTY DAMAGE n.a. 1.1*** 9 ROBBERY THEFT * In PCS % of all respondents ** In PCS the question covered house, second house, garage, basement. *** In ICVS question covered burglaries and attempts. Leaving the aforementioned limitations of comparisons aside, below we try to highlight the most important differences of rates revealed in the surveys. CATI vs. CAPI From among 10 crimes involved, within the four (theft of car, theft of motorcycle, burglary, robbery) no victim was revealed in the sample drawn for the CAPI method (probably owing to a small number of respondents). On the other hand, no one of 9

10 CATI respondents had his or her motorcycle stolen. As far as six other crime types are concerned, in relation to car damage, bicycle theft and theft the victimization rates in CATI were higher than those in CAPI. On the other hand, within household burglary and property damage, CAPI rates were higher than those of CATI. Only the theft from car rate was identical. CATI vs. PCS and/or ICVS The comparison embraces the five crime types: theft of car, theft from car, domestic burglary, robbery and theft. The victimization indexes are higher in the case of theft of car and theft from car in Victimisation Module. Home burglary victimisation rate was in CATI almost four times lover than in PCS and somewhat lower than in ICVS. The robbery victimisation rate was in CATI slightly higher than in ICVS, but lower than in PCS. The theft victimization rate was in CATI much lower than in ICVS, but somewhat higher than in PCS. 5. Comparison of the results with data from national administrative sources The police data are arranged by the Criminal Code definitions which, most often, differ from those used in the victimisation survey questionnaire. Moreover, the police data do not cover crimes which have not been reported by victims and registered by the police (the dark number of crime) while the victimisation survey results take account of all the crimes and thus are much closer to the actual picture of crime. Hence, in our table we included only crimes whose definitions are similar. In Table 2 victimisation rates are presented, i.e. numbers of incidents of the same type per 100 respondents, as well as crime rates per 100 inhabitants. 10

11 Table 2. Incident rates (per 100 respondents) and crime rates (per 100 inhabitants) CRIME TYPES MODULE CATI (2008) MODULE CAPI (2008) Police data (2008) Number of incidents Incident rate (per 100 respondents) Number of incidents Incident rate (per 100 respondents) Number of offences Rate (per 100 inhabitants) 1 THEFT OF CAR THEFT FROM CAR CAR DAMAGE n.a. n.a. 4 THEFT OF MOTOR CYCLE, SCOOTER, MOPED BICYCLE THEFT HOME BURGLARY OTHER BURGLARIES (second house) PROPERTY DAMAGE (Police data include property damage and car damage ROBBERY THEFT Summing up the data presented above, it may be said that in all the crime types the victimisation rates are several times higher than those stemming from the police statistics. It proves once again that official statistics are severely inadequate and do not reflect the real scope of crime. 11

12 6. Reference period preferred for the analysis of results and reason for doing so On the basis of Polish experience with victimisation surveys, it may be stated that most fruitful are questions on yearly victimisation, and that such questions should be asked in the first quarter of the next year the most favourable period for such an exercise. Such a solution reduces errors resulting from the so called telescoping. Questions on the five-year victimisation should be of only supplementary character. 7. Breakdown of responses for each question Breakdowns for Parts A, B, C, E, and F, as well as for R have been presented in the original version of the questionnaire (in English language), separately for CAPI and CATI. They form the Annex 2 to this Report. Percentages of answers for Part D have been elaborated in Excel tables, separately for CATI and CAPI (Annex 3). Percentage of answers for Part G (only in CAPI version) have been presented in SPSS format (Annex 4). In part G, in tables devoted to separate questions, numbers of categories of answers have been inserted. It should be mentioned that to some questions the additional categories of answers had to be added: Don t know, Don t like answer, No answer (Annex 4). 8. An assessment of the results achieved for each question and the utility of the question The usefulness of questions contained in the questionnaire has been evaluated on the basis of their detailed analyses, as well as remarks of interviewers. The most important findings are presented in Table 3. 12

13 Table 3. Most important remarks concerning usefulness of questions contained in the questionnaire Number of question Contents of the question Commentary Questions that do not produce important information A3 B7 D2A D2B What is your month of birth? Why do you feel unsafe in your home at night? In which month did the most recent < CRIME TYPE > happen? In which month did the previous< CRIME TYPE > before the most recent incident happen? D2C In which month did the < CRIME TYPE > before the second most recent incident happen? Too detailed question. In our opinion this information is useless. Breakdown of answers to Q. B 6 proves that only 2% respondents (both CAPI and CAPI) feel unsafe in their homes at night. Inquiring into reasons of feeling unsafe directed to such a slight percentage of respondents seems to be redundant. These questions are too complicated. We should not ask about precise dates of incidents that took place up to 5 years ago. We suggest to cross these questions out. Anyway, in any crime category there will be not enough data to get the time distribution (if that was the rationale for these questions). D9 Who did the stolen (damaged) property belong to? Question does not provide important information Excessively detailed questions D10 E37 What item or items were actually stolen? Bribery Module. In which country did this incident Too many categories of answer. Their number should be reduced. Breakdown of answers indicates that there 13

14 happen? was none bribery incident in another country. We suggest to give up this question Sensitive questions A16-A18 F2 Household income questions Is your home protected by: According to the interviewers, the respondents were reluctant to answer. According to the interviewers, the respondents were reluctant to answer. Questions difficult to the respondents A11 How old is each member of the household excluding yourself, are they male or female, and what is the relation of each household member to you? First, according to the interviewers, some respondents lost their orientation while answering. They needed explanations and had to be guided. Second, the very purpose of this question is not quite clear. Questions causing other troubles D22 D33 Can you tell me why you reported this incident to the police? As a result of what happened have you or your household taken any action to try to prevent it happening again? The interviewers complained that the list of possible answers is too long. Finding the category chosen by the respondent was a time consuming exercise. Moreover, the pilot study revealed that the answers should be improved, since the interviewers faced difficulties trying to match the respondents actual answers and those proposed. It is further proved by a relatively high percentage of the answer Other. Categories of answers are not tailored to different crime types. Thus, they should be analysed from this viewpoint. More comments to the questionnaire are presented in Annex 1 to the Report. 14

15 9. An estimate of the cost of implementing such a module at national level Victimisation surveys costs are the function of several factors, including the length of the questionnaire, interviewing time, number of open-ended questions and those with multi-answers. Thus, the costs are difficult to be estimated in general. However, the CATI method is obviously much less expensive than both the CAPI, and face-to-face modes. Below we present approximate, indicative costs of the survey for two options: CATI and CAPI, while taking the following assumptions: average interviewing time - 35 minutes sample size 2000 or 5000 interviews Variant I CATI method interviewing: Average interviewing time Sample size Cost in EURO 35 minutes * + VAT (22%) 35 minutes * + VAT (22%) * Exchange rate: 1 EURO = 4.05 PLN Variant II CAPI method interviewing: Sample drawing in CAPI technique: sample of households drawn by lots from GUS data base, using the random-route technique, the person in a household to be interviewed is randomly chosen according to the last birthday. Average interview time Sample size Cost in EURO 35 minutes * + VAT (22%) 35 minutes * + VAT (22%) * Exchange rate: 1 EURO = 4.05 PLN 15

16 10. Conclusions and recommendations for further actions 10.1 General recommendations (preferred victimisation survey module) While planning the future victimisation survey module for the EU Member States, the results of pilot studies conducted in several countries should be taken into account as well as experience learned in conducting international victimisation surveys in last years, for example ICVS. To develop the module of European victimisation survey, particular attention should be paid to the following considerations: regularity of surveys, uniform schemes of drawing samples, carrying out the fieldwork in essentially the same months, the use of similar procedures, training and instructing interviewers, as well as similar constructions of data bases and of basic indicators. Most concisely it may be said that the best would be a series of short but regular surveys (each year or each second year), on large samples, covering a basic catalogue of crimes, the police response, evaluation of the police work, and the inhabitants sense of security. More in-depth surveys of chosen victimisation types (e.g. domestic violence, corruption, stalking, mobbing) should be carried out less frequently (each third or fifth year) rather with the use of CAPI method. In a survey the introduction (preamble) should be elaborated with great care - to present the purpose of the survey, and its international (European) character. Also, the anonymity should be safeguarded and exposed in the preamble, along with pointing out there that the survey is a very important tool of investigating crime threat across the European Union Drawing schemes and samples sizes The adoption of sample minimum size to be used in each country is important. The following elements should be common: Sample drawing unit (a person or a household) Respondents age brackets Interviewing technique Deadline for completion 16

17 In the case of CATI technique, a possible inclusion of mobile phones to the sample should be considered Questions contained in the questionnaire The most important finding of the pilot study is that the number of questions should be much lower. The questionnaire was simply too lengthy what causes many problems in practice. Respondents after some time did not pay much attention to questions. They also hurried up to finish the interview as quickly as possible. Therefore, the reliability of answers was severely undermined. Some questions were unnecessary. For example, there were too many questions about households, too many questions about feeling safe. Questions should also not be too long, as respondents got confused on their true meaning. Sensitive questions deserve particular attention. In the Polish pilot study, the most sensitive questions as perceived by respondents, were not those on sexual violence (as one might presume) but the questions on: house protection, family income, and family household characteristics. As has been mentioned above, it is necessary to prepare introduction to the questionnaire that would explain the goal and methodology of the survey. All involved parties and a survey company should be named. Respondents should be assured that their answers will be used for scientific purposes only. And there should be thank you at the end of the questionnaire, the lack of which was yet another shortcoming of the pilot survey. It is a serious mistake to provide a lengthy (several pages) set of questions on social and demographic characteristics at the beginning. Such questions should be asked as the last ones, since respondents should not be deterred beforehand by dull questions. 17

18 10.4 How to make the questionnaire short In order to substantially shorten the questionnaire, two approaches may be adopted: either the survey will ask on many offences with few details, or on few offences with many details. We recommend adopting the first approach, i.e. asking about many typical crimes, while not going into much detail. We also strongly advise to limit the number of unrelated/not necessarily required questions (see below).some sensitive questions should be eliminated: income, household composition, possession of security devices are not of main concern in a victimisation survey, but often make respondents uncomfortable during the interview. We think that Part G should be abandoned all together, while moving questions on battery to Part C. Main focus should be on the last year victimisation, with only additional 5-year period. The question on a life-time victimisation should be eliminated Many offences - few details We recommend to adopt the aim of the EU Victimisation module as a reliable source for providing the comparable European victimisation rates and trends. That means that the survey should be conducted frequently, and that the scope of victimisation is important. By providing common offence definitions, the comparable rates will be obtained, circumventing the well-known limitations of police statistics. The set of most common offences should be chosen. However, there is no need to frequently ask about many details of crime. Instead, rotating panels may be used for specific topics: details of particular crime or new crime type, if required by policy needs Unnecessary details There were numerous unnecessary details in the pilot survey. For example, the exact month of the crime is of no importance. While some other details might be important for criminological analysis, they are nevertheless not useful for 18

19 international comparisons. This includes such questions as the number of offenders, number of witnesses, alcohol and/or drug use by offenders and/or victims, offenders age, relationship to the victim, place of the offence, victim support assessment. Such factors also do not tend to change swiftly, so there is no need to ask the questions about them every 2 or 3 years. Final comments We suggest to shape the European victimisation survey as a concise questionnaire (interviewing time up to 15 minutes), being in use frequently (once a year or once each two years). Such an approach would enable us to continually monitor crime trends in Member States. In order to study particular phenomena in more detail (obtaining more information on particular crime types, or measuring the scope of emerging new crimes), we suggest to introduce one or two rotating panels, whose contents might be adjusted. In comparative surveys the most important is to preserve, as far as feasible, the same questions in the span of many years, and, at the same time, to be able to add questions on current problems. On the other hand, it is not necessary to regularly monitor factors that are relatively constant (such as life-time victimisation, or circumstances of crimes, e.g. place, number of perpetrators, modus operandi). To achieve such goals, the present questionnaire should be significantly redrafted and, first of all, shortened. It is of utmost importance to consider and analyse all the remarks (both those of countries and of the experts) in order to eliminate basic errors the task that unfortunately was neglected during the pilot study before the launch of the future European survey. 19