Using the cumulative deviation method for cross-institutional benchmarking in the Berlin progress test

Size: px

Start display at page:

Download "Using the cumulative deviation method for cross-institutional benchmarking in the Berlin progress test"

Claire Lyons
6 years ago
Views:

1 20; 32: Using the cumulative deviation method for cross-institutional benchmarking in the erlin progress test STEFAN SHAUER & ZINE M. NOUNS harité Universitätsmedizin erlin, Germany Abstract The erlin Progress Test has grown to a cooperation of 13 universities. Recently, comparisons between the participating schools became an area of high interest. Muijtjens et al. [Muijtjens AM, Schuwirth LWT, ohen-schotanus J, Thoben AJNM, van der Vleuten PM. 2008a. enchmarking by cross-institutional comparison of student achievement in a progress test. Med Educ 41(1):82 88; Muijtjens AM, Schuwirth LWT, ohen-schotanus J, van der Vleuten PM. 2008b. Differences in knowledge development exposed by multi-curricular progress test data. Adv Health Sci Educ 13: ] proposed a method for cross-institutional benchmarking based on progress test data. Progress testing has some major advantages as it delivers longitudinal information about student s growth of knowledge. y adopting the procedure of Muijtjens et al. (2008a, b), we were able to replicate the basic characteristics of the cumulative deviation method. esides the advantages of the method, there are some difficulties as errors of measurement are not independent, which violates the premises of testing statistical differences. Introduction The erlin Progress Test is carried out at 13 medical schools in Germany and Austria. The project started in 1999 as a student initiative and has since grown to a widespread cooperation. Twice a year about 9000 medical students take part in the progress test. In all participating institutions, the test is provided as strictly formative assessment of medical knowledge. In medical education research, progress test data have been used for evaluating a wide range of questions concerning the differential development of medical knowledge (Albers et al. 1989; van Leeuwen et al. 1995; Verhoeven et al. 2002; ohen- Schotanus et al. 2008). Some studies addressed the question of systematic differences between schools. One of the first studies using progress test data for comparisons between different faculties was published by Albano et al. (1996). These studies used diverse statistical methods and cross-sectional designs. However, Muijtjens et al. (2008a, b) proposed a method for inter-institutional comparisons which integrates the longitudinal character of progress testing. In brief, Muijtjens et al. (2008a) were able to show that their method can reveal smooth, stable patterns of between-school differences. In general, progress tests deliver a rich source of information for school comparisons. One important aspect of progress test data is that the longitudinal process of knowledge acquisition is reflected. As a consequence, consecutive information on both the students and institutional achievement can be included in the analysis. In contrast, results of single-point measurements for example national exams merely display the students knowledge at that particular moment. It has been Practice points. The cumulative deviation method is a valuable approach to longitudinal between-school comparisons.. If certain assumptions are valid, it delivers stable patterns of between-school differences. shown that such single-point benchmarking is prone to discard valuable information (Muijtjens et al. 2008a). As cross-institutional comparisons became an area of major interest within our cooperation, we aimed to transfer the cumulative deviation method to erlin Progress Test data. The focus of our study was to replicate the key characteristics of Muijtjens et al. s approach with the data of the erlin Progress Test. We furthermore evaluated to what extent the method can deal with the different characteristics of our test. Method Although sharing key features, there are some differences between the erlin and Maastricht Progress Tests. First, there are major differences in test administration: the erlin Progress Test is carried out only twice a year. Some institutions have been participating irregularly or just once per academic year. Second, the erlin Progress Test is carried out as a formative assessment tool. Therefore, students are not faced with negative consequences concerning their results. This could be an explanation for the occurrence of unserious test takers (cf. rauns 2007). On the other hand, there is no pressure to orrespondence: S. Schauber, Assessment/Progress Test Medizin, harité-universitätsmedizin erlin, haritéplatz 1, 117 erlin, Germany. Tel: ; fax: ; stefan.schauber@charite.de ISSN X print/issn X online// ß 20 Informa Healthcare Ltd. 471 DOI:.39/ X

2 S. Schauber & Z. M. Nouns Table 1. Sample sizes given under each combination of school, academic year and measurement moment. A A A perform in the test and therefore the results give an ad hoc picture of the spontaneously recallable level of medical knowledge. Instrument The erlin Progress Test consists of 200 multiple-choice questions in single best answer format selected from an item database and matching a blueprint. The students are asked to take the test in about 3 h. The individual student s test score is obtained by negative marking of incorrect answers, whereas choosing a don t know option has no effect on the individual score. Students are encouraged to make use of the don t know option in order to provide more reliable feedback. Sample We restricted our analysis to three schools and eight measuring occasions within an academic year including students from the first to the fourth year. The eight measuring occasions in a year equate to the data of two tests delivered to four cohorts. The schools were randomly selected from our database. Students classified as unserious test takers were excluded by the person conformity index (rauns 2007) resulting in a sample size of N ¼ We also excluded participants who took less than 50 min to complete the test. This pragmatic criterion is justified by the assumption that the questions are hardly solvable in a shorter period of time. Therefore, our final sample consisted of a total number of N ¼ 8965 student responses. Table 1 shows the number of observations in each combination of grouping factors. Analysis All data were analysed using the R language for statistical computing (R Development ore Team 2009). A dataset storing all observations since 1999 was imported and a subset was specified according to the restrictions stated previously. In a first step, we applied single-point benchmarking. After that, we used the cumulative deviation method for longitudinal benchmarking to compare the schools. 472 Single-point benchmarking was carried out by following the procedure introduced by Muijtjens et al. (2008a). The overall mean score for each year group on one test was calculated across the participating institutions. The individual school mean at each measurement moment was then subtracted from the overall mean. As a result, the obtained deviations reflect the relative position to the average institutional score. Therefore, negative values reflect a performance below the average of the other schools. Positive scores reflect higher-than-average performances. The longitudinal benchmarking method goes beyond the single-point benchmarking because the information of consecutive tests is integrated in every single measurement moment within one academic year. First, individual schools deviation scores are calculated according to the single-point benchmarking method. Next, the deviation scores are summed for each measurement moment including all deviations up to that particular point. Then, the average cumulative deviation score at one measurement moment is calculated by dividing the summed deviations by the number of consecutive measurements. As a consequence, the last measurement moment within an academic year reflects the average overall performance of an institution across that period. Results Single-point benchmarking The single-point benchmarking method can be understood as a special case of the longitudinal method. The two major differences are as follows: (1) just one academic year is analysed and (2) growth curves are not smoothed by cumulating the deviation scores. Graphical representations of the results are presented in Figures 1 3. Figures 1 and 2 display results from single-point benchmarking. Figure 1 shows the school means at each measuring point. Figure 2 illustrates each school s deviations from the overall mean. Like the findings of Muijtjens et al. (2008a), point-to-point fluctuations of school means and deviations result in noisy patterns obscuring between school differences.

3 Using the cumulative deviation method mean Deviation from grand mean Figure 1. Mean achievement of schools for three academic years. A A Figure 2. Deviations of the individual schools from the overall mean. Longitudinal benchmarking Figure 3 offers the results from longitudinal benchmarking. Although the schools annual achievement curves are smoothed and noise is reduced, one can see the importance of longitudinal benchmarking. As school A and stay quite consistent regarding their achievement, school decreases severely in their relative performance. Significance We determined the statistical significance according to the procedure described by Muijtjens et al. (2008b). Year group sizes per school are given in Table 1. The SDs of the overall test scores varies from 9.4 to 24.8 for measurement moments 1 8. The standard error of the mean (SE m ) ranges from 0.04 to Hence, the SE of the deviation scores (SE d ) for a set of three schools and eight measurement moments varies from 0.02 to 0.4. As a result, cumulative deviation values 40.8 are at least significant at a level of 5%. Discussion The basic characteristics of the results of Mujitiens et al. (2008b) could be replicated with our data. We were able to show that the advantages of the cumulative deviation method can be transferred from summative to formative progress 473

4 S. Schauber & Z. M. Nouns Average cummulative deviation 5 0 A 5 testing. The cumulative deviation method is a valuable procedure for cross-institutional comparisons. It delivers smoothed representations of institutional achievement in successive progress tests; hence, interpretation of differences is simplified. Another important aspect is the integration of consecutive data. onsistent with Muijtjens et al. (2008a), we could show that information is lost by neglecting the longitudinal character of the progress testing. A positive side effect of cross-institutional benchmarking studies may be the facilitation of competition between schools. However, according to Albanese (2008), this may also be an obstacle for future studies because unintended motivations might play a role in students motivation to take the test. This aspect might be particularly relevant for formative progress testing without any further negative consequences for the students. However, there are certain limitations which are especially important for comparisons based on formative progress tests. The results of formative assessment have no negative consequences for the individual student. This circumstance could be a reason for at least a part of the fluctuations of sample size within subsets of the data, given in Table 1. Furthermore, there are irregular fluctuations of test-participation, not only on the individual student level, but also on the institutional level. As the cumulative deviation method focuses on the longitudinal integration of quite strictly successive measurement moments, it should be modified for institutions with irregular testparticipation. As an example, a school taking part just once within an academic year has to be treated in a different way because the deviations for the other point would be missing. Independent of the underlying data, another problem might occur as the cumulative deviation method is a mixture of longitudinal and cross-sectional analysis so that students are repeatedly assessed within an academic year. ut the longitudinal character of data which has been collected during a period of more than 1 year is not being taken into account. 474 Figure 3. This fact leads to an impact of an excellent progress in 1 year to all the subsequent analyses for other years, assuming a certain continuity of learning. Furthermore, observations also consist of dependent samples across academic years insofar as individuals move on from 1 year group in one academic year to the subsequent year group in the following year. Hence, errors of measurement are not independent. This violates the premises of testing statistical differences using the approach described here. In order to fully integrate the longitudinal information, which has been delivered by consecutive progress tests, differences between schools could be modelled using multilevel analysis (Hox 2002; Singer & Willett 2003). Such models take the dependency of measuring moments into account and are a widely used approach for data such as this. In addition, incomplete group-data can be integrated without additional assumptions. This would result in changing the view to the data. y changing to a cohort-sequential perspective, some of the methodological concerns can be avoided. As a consequence, further research questions can be addressed. For example, the process of knowledge acquisition or specific changes in the curriculum can be evaluated. onclusion umulative deviation of the school means. The cumulative deviation method could be applied successfully to our data and revealed smooth and stable patterns of between-school deviations. The advantages mostly concern the interpretation of these patterns and the highlighting of systematic differences. On the other hand, the method ignores the partly dependent structure of the data. Therefore, statistically significant differences should be interpreted with caution. If there are inhomogeneous student-cohorts, alternative methods should be considered for inter-institutional comparisons.

5 Using the cumulative deviation method Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article. Notes on contributors STEFAN SHAUER holds a master s degree in psychology and works as a scientific staff at the Progress Test Medizin, harité-universitaetsmedizin erlin. ZINE MIRIAM NOUNS (MD) works at harité Universitätsmedizin erlin on progress testing for the German speaking medical schools in Europe. esides working in medical education, she is in training for her clinical education in gynaecology and obstetrics. References Albanese M enchmarking progress tests for cross-institutional comparisons: Every road makes a difference and all of them have bumps. Med Educ 42:4 7. Albano MG, avallo F, Hoogenboom R, Magni F, Majoor G, Manenti F, Schuwirth L, Stiegler I, van der Vleuten An international comparison of knowledge levels of medical students: The Maastricht progress test. Med Educ 30: Albers W, Does RJMM, Imbos TJ, Janssen MPE A stochastic growth model applied to repeated tests of academic knowledge. Psychometrika 54(3): rauns K Entdeckung von musterkreuzern beim progress test medizin. Unpublished PhD thesis. Humboldt Universität zu erlin, Germany. ohen-schotanus J, Schönrock-Adema J, ouwkamp-timmer T, Terwisscha van Scheltinga GR, Kuks JM One-year transitional programme increases knowledge to level sufficient for entry into the fourth year of the medical curriculum. Med Teach 30: Hox J Multilevel analysis. New York: Psychology Press. Muijtjens AM, Schuwirth LWT, ohen-schotanus J, Thoben AJNM, van der Vleuten PM. 2008a. enchmarking by cross-institutional comparison of student achievement in a progress test. Med Educ 41(1): Muijtjens AM, Schuwirth LWT, ohen-schotanus J, van der Vleuten PM. 2008b. Differences in knowledge development exposed by multicurricular progress test data. Adv Health Sci Educ 13: R Development ore Team R: A language and environment for statistical computing. R Foundation for Statistical omputing, Vienna, Austria. Available from: Singer JD, Willett J Applied longitudinal data analysis: Modeling change and event occurrence. Oxford, UK: Oxford University Press. van Leeuwen YD, Mol SSL, Pollemans M, Drop MJ, Grol R, van der Vleuten PM hange in knowledge of general practitioners during their professional careers. Fam Pract 12: Verhoeven H, Verwijnen GM, Scherpbier AJJA, van der Vleuten PM Growth of medical knowledge. Med Educ 36:

Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO. M. Ken Cor Stanford University School of Education.

The Reliability of PLATO Running Head: THE RELIABILTY OF PLATO Investigating the Reliability of Classroom Observation Protocols: The Case of PLATO M. Ken Cor Stanford University School of Education April,