NPC-BASED GLOBAL INDICATORS FOR TEACHING UNIVERSITY ASSESSMENT. Carrozzo E. a Joint with Arboretti R. b

Transcription

1 Dealing with Complexity in Society: From Plurality of Data to Synthetic Indicators NPC-BASED GLOBAL INDICATORS FOR TEACHING UNIVERSITY ASSESSMENT Carrozzo E. a Joint with Arboretti R. b a Department of Management and Engineering, University of Padova b Department of Civil, Architectural and Environmental Engineering, University of Padova September 17 th and 18 th,

2 1. Preliminary notion The problem 2. Conditions Properties 3. Extension of NPC ranking for categorical variables The teaching university assessment 4. OUTLINE Carrozzo et al. Dealing with Complexity in society 2

3 Preliminary notions Let us consider a set of k informative ordered categorical variables representing judgments on specific quality aspects under evaluation. Let us denote the responses as a k-dimensional variable: where each marginal variable can assume m ordered discrete scores and large values of h correspond to higher satisfaction rates. Variables may have different (non-negative) degrees of importance: Carrozzo et al. Dealing with Complexity in society 3

4 to the problem The problem: to find a global satisfaction index or global ranking of N statistical subjects starting from k variables or dependent rankings on the same N subjects, each representing a specific aspect under evaluation. Some main points to be considered: i. The search of a suitable combining function of two or more variables or rankings; ii. The consideration of extreme units of the global ranking;, the principle that being ranked lowest does not immediately equate with genuinely inferior performance should be recognized and reflected in the method of presentation of ranking. (Bird, S.M. et al. (2005)) Carrozzo et al. Dealing with Complexity in society 4

5 1. Preliminary notion OUTLINE The problem 2. Conditions Properties 3. Extension of NPC ranking for categorical variables The teaching university assessment 4. Carrozzo et al. Dealing with Complexity in society 5

6 Conditions The nonparametric combination (NPC) of dependent rankings (Arboretti R., Pesarin F., Salmaso L., 2005) provides a solution for problem i) Starting from component variables Y i, i=1,,k, we whish to construct a global index or combined ranking: We introduce a set of minimal reasonable conditions related to variables Y i : a. for each of the k informative variables a partial ordering criterion is well established, that is to say large is better ; b. Regression relationships within the k informative variables are monotonic (increasing or decreasing); c. The marginal distribution of each informative variable is nondegenerate. Notice that we don t need not to assume the continuity of Y i so that the probability of exequo can be positive. Carrozzo et al. Dealing with Complexity in society 6

7 Properties The combining real function is chosen from a class of combining functions satisfying the following minimal properties: 1. must be continuous in all 2k arguments, in that small variations in any subset of arguments imply small variation in the - index; 2. must be monotone non decreasing in respect to each argument: 3. must be symmetric with respect to permutation of the arguments, in that if for instance is any permutation of 1,,k then: Carrozzo et al. Dealing with Complexity in society 7

8 1. Preliminary notion OUTLINE The problem 2. Conditions Properties 3. Extension of NPC ranking for categorical variables The teaching university assessment 4. Carrozzo et al. Dealing with Complexity in society 8

9 Extension of NPC ranking for categorical variables For problem ii) we propose an extension of the NPC ranking method to the case of ordered categorical variables based on extreme satisfaction profiles defined apriori on a hypothetical frequency distribution of variable Y i, i=1,,k. Each marginal variable can assume m ordered discrete scores ; Large values of h correspond to higher satisfaction rates, f hj is the relative frequency of category h for variable Y i. Carrozzo et al. Dealing with Complexity in society 9

10 Extension of NPC ranking for categorical variables The strong satisfaction profiles The maximum satisfaction is obtained when The minimum satisfaction is obtained when The weak satisfaction profiles The maximum satisfaction is obtained when The minimum satisfaction is obtained when Carrozzo et al. Dealing with Complexity in society 10

11 Extension of NPC ranking for categorical variables In order to include the extreme satisfaction profiles in the analysis, we transform the original values For values of h corresponding to a judgement of satisfaction, i.e. the last the transformed values of h are defined as: For values of h corresponding to a judgment of dissatisfaction, i.e. the first the transformed values of h are defined as: Carrozzo et al. Dealing with Complexity in society 11

12 Extension of NPC ranking for categorical variables The transformation of values weighted by relative frequencies is applied to observed values Let us denote the transformed values of by. In this setting, we can consider the following transformations: where and are obtained accordingly to an extreme satisfaction profile. represents the preferred value for each variable, and it is obtained when satisfaction is at its highest level accordingly to the extreme satisfaction profile. represents the smallest value, and it is obtained when satisfaction is at its lowest level accordingly to the extreme satisfaction profile Carrozzo et al. Dealing with Complexity in society 12

13 Extension of NPC ranking for categorical variables In order to synthesize the k partial rankings based on scores by means of the NPC ranking method, we use a combining function: Many are the possible combining function: Liptack s combining function, Tippet s combining function, Truncated combining function, Blalock etc.) One of the most popular combining function is the Fisher s combing function defined as In order the global index varying in the interval [0,1] we put j=1,,k. Carrozzo et al. Dealing with Complexity in society 13

14 Extension of NPC ranking for categorical variables Where and and are obtained accordingly to the estreme satisfaction profiles: represents the unpreferred value of the satisfaction index; represents the preferred value of the satisfaction index; and are reference values in order to evaluate the distance of the observed satisfaction values from the situation of highest satisfaction defined accordingly to an extreme satisfaction profile. Carrozzo et al. Dealing with Complexity in society 14

15 The teaching university assessment Our proposal has been used in order to evaluate the global satisfaction of the students of the School of Engineering at the University of Padova for the academic year 2013/2014. The composite indicator has been computed for each teaching, of a specific degree course and teacher. In particular we are studying the relation between the answer to the overall satisfaction question and our composite indicator, and how our proposal can help to better understand the satisfaction related to the partial aspects (questions in the evaluation questionnaire). Carrozzo et al. Dealing with Complexity in society 15

16 The teaching university assessment The partial aspects considered for the construction of the composite indicator are: D01: At the beginning of the course the aims and the contents were clearly presented? D02: The examination procedures were clearly defined? D03: The times of teaching activities were complied with? D09: The recommended course material was appropriate? D07: The teacher encouraged/motivated the interest in the subject? D08: The teacher set out the topics clearly? D10: The professor was during his office hours for clarifications and explanations? D11: Workshops, tutorials and seminars, if any, were appropriate? Organizational Aspects Teaching Activity Carrozzo et al. Dealing with Complexity in society 16

17 The teaching university assessment An application example: comparing different teaching courses (A, B, C) separately for the different areas. (Strong satisfaction profile Fisher s combining function) Organizational Aspects Teaching activity Global Index Global Index A B C A B C 95% CI: [0.47, 0.52] [0.40, 0.46] [0.35, 0.40] [0.30, 0.33] [0.30, 0.36] [0.24, 0.30] Carrozzo et al. Dealing with Complexity in society 17

18 The teaching university assessment What happen considering only the answer of overall satisfaction of the questionnaire? As a trivial analysis we performed a linear regression model in order to find which are the partial aspects which mostly impact on the overall satisfaction, for each single teaching course. Overall satisfaction ~ D1 + D2 + D3 + D9 + D7 + D8 + D10 + D11 What emerges is that in many cases only one aspect (or few aspects) are significantly related with the overall satisfaction. Some other kind of non-linear regression models maight be used to better identify also possible latent relations between the response and the covariates Carrozzo et al. Dealing with Complexity in society 18

19 The teaching university assessment Differences in distribution between overall satisfaction and the composite indicator (on the 0-1 scale). Overall Satisfaction Overall Satisfaction Composite Indicator Satisfaction Global Index Composite Indicator Carrozzo et al. Dealing with Complexity in society 19

20 1. Preliminary notion OUTLINE The problem 2. Conditions Properties 3. Extension of NPC ranking for categorical variables The teaching university assessment 4. Carrozzo et al. Dealing with Complexity in society 20

21 Aiello F., Attanasio M. (2004). How to transform a batch of simple indicators to make up a unique one? Atti della XLII Riunione Scientifica della Società italiana di Statistica, CLEUP, Padova, Arboretti Giancristofaro, R., Pesarin, F., Salmaso, L., (2005). Nonparametric approaches for multivariate testing with mixed variables and for ranking on ordered categorical variables with an application to the evaluation of PhD programs, in Real Data Analysis, American Educational Research Association, S. Sawilowsky Editor, Age Publishing (to appear). Bird, S.M., Cox, D., Farewell, V.T., Goldstein, H., Holt, T., Smith, P.C. (2005). Performance indicators: good, bad and ugly, Journal of the Royal Statistical Society, Series A, 168, 1, Fayers, P.M., Hand D. J. (2002). Causal variables, indicator variables and measurement scales: an example from quality of life. Journal of the Royal Statistical Society, Series A, 165, 2, Pesarin, F. Salmaso L. (2010) Permutation Tests for Complex Data. Theory, Applications and Software, Wiley, Chichester. Carrozzo et al. Dealing with Complexity in society 21