COMPARISON OF DIFFERENT SCALING METHODS FOR EVALUATING FACTORS IMPACT STUDENTS ACADEMIC GROWTH

Transcription

1 International Journal of Innovative Management Information & Production ISME International c 2014 ISSN Volume 5, Number 1, March 2014 PP COMPARISON OF DIFFERENT SCALING METHODS FOR EVALUATING FACTORS IMPACT STUDENTS ACADEMIC GROWTH MINGCHUAN HSIEH National Academy for Educational Research, Taiwan Taipei, 23703, Taiwan hm7523@hotmail.com ABSTRACT. Many factors could be included in the questionnaires as investigating factors, but it is not possible to include all of them. How to select the adequate factors is difficult. The main purpose of this study is to identify the factors which are logically related to the student s academic growth. This study compared two approaches for selecting the factors-t test and successive intervals scaling, the differences between these two approaches are discussed. Keywords: Questionnaire Development; Successive Intervals Scaling; Equal Appearing Intervals 1. Introduction. The goal of this study was to investigate the constructs which could impact the student s academic growth. Based on the past studies, many factors should be included, but it is not possible to include all of them. The questionnaires are administered to elementary school students, the number of questions need to be as short as possible. It is also desired to know the relative importance of the factors since it can help shorten the questionnaires. Therefore, the purpose of this study is not only to identify the factors which are logically related to the student s academic growth, but to order the relative importance of these factors. This study compared two approaches for selecting the factors-t test and successive intervals scaling, the differences between these two approaches are discussed. There are many studies on the topic relate to student s academic growth. Rather, the intent of the author is to provide readers sufficient background information to understand the various factors used by large scales assessment and their main finding. Moreover, a discussion of psychometric scaling, in particular the successive intervals scaling, will be undertaken to explain how and why this type of metric was used to evaluate the variables in this study. The discussion will begin with the review of the factors which could impact student academic achievement Factors Impacting the Academic Performance. Student s academic performance is impacted by three main elements, including parents, schools, and students. For the student variable, the possible impacting factors included the self educational expectation, learning attitudes, gender and race. Based on the finding from Taiwan Education Panel Survey (TEPS), it is found that the self expectation and learning attitude could have impact on

2 COMPARISON OF DIFFERENT SCALING METHODS FOR EVALUATING FACTORS 63 academic performance (You, 2009). Students with more positive learning attitude toward study usually have better academic performance. In addition, the academic achievement of male students is usually better than the female students (Kao, 2007; Chen, 2004, Tseng, 2004). This conclusion is controversial, though (Liu, 2006, Hsieh, 2004). It is also found that minority population in Taiwan, such as immigrated or indigenous people, usually have difficulties in some subject areas. However, Lin (2007) claimed that, their low performance may cause by the culture difference or low socio-economic status of those minority population. The student performance attribute to the family is without questions, and there is an increasing awareness of the importance of the parents role in the student academic growth. Barton (2003) indicated that parental involvement in their children's education, how much parents read to young children, how much TV children are allowed to watch and how often the students transfer schools has very high impact on student academic performance. The last factor related to school variables. For the student s teacher and the classmates, it is found that the relationship between the teacher and student are usually reciprocal. Teacher s expectation and motivation of teaching significantly influences student s performance (Atkinson, 2000). Peer relationships have positive correlations with student s academic performance, and it is found that students failing in the school are usually those rejected students by their classmates. As for the school environment area, the factors include teacher preparation for class and teaching experience, as well as the design of school curriculum. Other possibility impacting factors include the usage of technology in the classroom, class size and school safety (Barton, 2003) Equal Appearing Intervals and Successive Intervals Scaling. Psychometric scales can be regarded as calibrated instruments to interrogate concepts (Wright, 1980). In the measurement field, it often needs to scale a set of stimuli or objects into a psychological continuum when the relative positions of the set of stimuli on a corresponding physical continuum are unknown. The scale construction is the process of quantifying the abstracts constructs. A popular way of obtaining the psychological scale values of stimuli is the method of pair comparisons, which is developed by Thurstone (1927). However, one drawback of the method of pair comparisons is the number of judgment could grow quite fast as the number of stimuli increases. Thurstone further developed the successive intervals and equal appearing intervals method (Thurstone, 1927, 1929; Thurstone and Chave, 1929). Comparatively, these two methods do not require as many judgments and easier to implement. For the method of equal appearing intervals, after the items have been developed, they are submitted to a group of experts acting as judges. Each judge need to evaluate the statements in terms of the degree of favorableness/unfavorableness they indicate toward the object or phenomenon being investigated. They were asked to sort the items into 11 categories, the first category represented the least favorable or negative feeling toward the object, the middle category represent the neutral sentiments, and the highest category representing the most favorable attitudes or positive feeling. In short, the judges are given each of the statement from 1 (least favorable) to 11 (most favorable) for all the statements developed.

3 64 MINGCHUAN HSIEH Two statistics information are used to select the number of items for the final scale. The median for each of the items is used as its scale value. The lower the median, the less favorable the judges consider the statement to be an important factor under investigation; the higher the median, the more favorable the expressed intent of the item. Moreover, the interquartile range is computed for each item. The statistics is used to measure the spread of the middle 50% of the judgments. The higher agreement among the judges concerning the location of the item on the scale, the smaller the interquartile range would be. On the other hand, a larger interquartile range indicates there is substantial disagreement among the judges toward this item. The following criteria outlined by Mciver and Carmines (2010) are applied for selecting the final items. First, the items are chosen with median values distributed along the full range of the scale and these items could span the entire attitudinal continuum. Second, for the items with very high median values, those with the smallest interquartile ranges are selected because they reflect less disagreement among the judges concerning this item. Because the disagreement should be considered as a signal of ambiguity, this criterion is used to reduce the pool of items to those that are least ambivalent in representing a specific group of judge s attitude. Finally, the items are chosen to fall at as many equal appearing intervals along the scale as possible. The successive intervals method is quite similar to the equal appearing intervals method, experts also sort the items into fixed categories spaced along a continuum, ranging from the extreme less favorable, through neutral, to extreme favorable. However, the mathematics calculation of the successive intervals method is more complicated. To implement successive categories method, first need to construct a n r matrix, where n is the number of stimuli and r is the number of rating categories. And the element of the matrix is P jk, which is the proportion of judges rating a given stimulus j in the k th category or below. The corresponding normal deviate X jk can be found from a table of the standard normal distribution. Base on P jk and X jk, the scale value for each stimuli can be determined. Reader who are interested in more detail of the method of successive intervals (SI) can consult with Saffir (1937), Attenave(1949), Edwards (1952, 1957), Guilford (1954), Edwards & Thurstone, (1952), Green (194) and Torgerson (1958). 2. Method. The first step was to find the possible factors which might impact the student academic achievement. In addition to search the related references, we also collect several large-scale questionnaires from united states and Taiwan, including The Programme for International Student Assessment (PISA), Trends in International Mathematics and Science Study (TIMSS), National Education Longitudinal Study of 1988 (NELS:88), National Assessment of Educational Progress(NAEP), Taiwan Education Panel Survey(TEPS), and TASA. The constructs in the family, school and student questionnaires were collected and compared. The constructs appear more than three assessments were regarded as important constructs for large scale survey. After identify those factors, the next step was to have some field scholars to further examine and revise these factors. The revised list of factors was written into the scaling questionnaire. In the questionnaire, the 34 variables are listed in the rows and the 11 points scale is given as 11 columns along the top. The directions for completing the scale were

4 COMPARISON OF DIFFERENT SCALING METHODS FOR EVALUATING FACTORS 65 included in the questionnaires. Description of the 34 variables is designed to be short and straight-forward. The scaling questionnaire was mailed out to 79 senior teachers. A total of 54 of the scales were returned completed. This represents a 68.35% return rate. The teachers are consisted of 22 males (40.74%) and 32 females (59.26%). Moreover, 13 teachers have college degree, 40 teachers have master degree and 1 teacher has PhD degree. Each of them has at least 5 years teaching experience and was nominated as excellent teacher by the Department of Education in Taiwan and considered as experts in field of academic achievement. The average teaching experience of these teachers is 18 years. Table 1 lists the subjects which the teachers taught at school. It shows that the teachers sampled for this study are well represented in each content area, except the content area of Human Right Education. TABLE 1. The returning rates for each content area teacher sampled in this study. Subject Mailing out N Receive N Returning Rates Chinese Native language English Health and Sports Mathematics Life Curriculum Social Science Arts Science Synthesis Activities Gender Equity Education Human Right Education Total The judges were asked to rate each factors on a 1 to 11 scale in terms of the importance for each factor. One (1) represented an extremely least important factors impacting the student academic achievement and eleven (11) represented an extremely important factors impacting the student academic achievement. As mentioned previously, these 34 factors were mainly extracted from several large scale questionnaires. Besides the primary scale, the judges were also asked to add variables that he/she felt were indicative of the possible factors that had been omitted on our scale. Several judges added variables and the results are also presented. Since the sample size was too small and the number of factors is large, it is not suitable to scale the factors using the Thurstone s pair comparison or item response theory method. Thus, the equal appearing intervals and successive interval scaling method was chosen for this study. For each statement, the authors computed the median and the inter-quartile range. The first quartile is the value below 25% of the cases fall and the third quartile is the value above which 75% of the cases fall. The inter-quartile range is the difference between the third and first quartiles. A variable with higher median and a small inter-quarter range

5 66 MINGCHUAN HSIEH would show stronger agreement of favorability among the judges and vice versa for less agreement of favorability. The further examine the relative importance of these factors, the researchers also computed T value and successive interval scale values. The T value was computed by comparing the individual mean versus the overall average mean of all factors. If T value is positive and the probability is less than 0.05, it means the given factor has strong agreement of importance among the judges and vice versa for less agreement of importance. The data were also entered into Excel to compute the scale values. 3. Results. Table 2, the statistical results using median and inter-quartile Range, is presented. Table 2 depicts the 34 variables along the rows and the median, quartile 1, quartile 3, the inter-quartile range and variable significance in each column. A median of 8.71 and an inter-quartile range of 4 or less made this variable significant. An inter-quartile range of 2.5 was chosen because it represented the overall mean of all 34 inter-quartile ranges. An increasing median and a decreasing inter-quartile range represent a more significant variable. As shows in Table 2, there are 17 variables met the test of significance and are identified in the variable significance category with Yes and an asterisk. A median of less than 8.71 or the inter-quartile range greater than 2.5 made this variable insignificant. Seventeen variables fell into this category and are identified by No in the variable significance category. Even though the Thurstone Scale Method is generally interpreted using the median and inter-quartile range, the authors felt that this way is not precise enough since the data could have been obtained by chance. Thus, the data were further analyzed using T test. In addition, the successive intervals scaling method was used to scale all the factors surveyed in this study. These factors were scaled on a psychological continuum and the relative positions of each factor are graphed. Table 3, the statistical results using mean, standard deviation, T value, probabilities for the significance level, and the scales values calculated using the successive intervals scaling methods are presented in the columns. An increasing T value represents a more significant variable. Because T was calculated using the individual mean minus the overall mean of all factors, the positive and significant T values are regarded as variables with greater impact on student academic achievement compared to the variables with negative or non-significance T values. Thirteen of the variables met the test of significance and are identified in the variable significance category with Yes and an asterisk. T values which are less than 0 or the significance level greater than 0.05 would make the variable insignificant. Twenty-one of the variables fell into this category and are identified by No in the variable significance category.

6 COMPARISON OF DIFFERENT SCALING METHODS FOR EVALUATING FACTORS 67 TABLE 2. Statistical results using median and inter-quartile range Factor/Variable Inter-Quartile Median Variable Range Q1 Q3 8.71>is 2.5 Significance significant or<significant Language usage at home No Parent s background No The working status of family members No Parents occupation No Parents education Level No Parents involvement of student s study Yes* Student participates in extracurricular activities No Number of books at home Yes* Student s likeness for the subject areas Yes* The amount of time students spend on homework No Student s future education plan No Student s attendance for class No The amount of time students spend on study Yes* Student s reading habits Yes* Student s self evaluation Yes* School climate and safety in school No Student s interpersonal relationship Yes* The occurrence of student s misbehavior Yes* Parents age No Family religion No Parents marital status No Household income No The education resources providing from parents Yes* Family s living environment No Parents employment status Yes* Parents working condition No Parents involved in children s interpersonal relationships Yes* Parents and children spend time together Yes* Parent and children s interaction Yes* Parents effort for children's education Yes* Parents expectation on children s education Yes* The family expense on education Yes* Parents interaction with school No Parents attitude toward school Yes*

7 68 MINGCHUAN HSIEH TABLE 3. Statistical results for T value and successive interval scale values Factor/Variable Mean Std T Prob Variable significance Scale Value Language usage at home No 1.58 Parent s background No 3.61 The working status of family members No 3.67 Parents occupation No 3.19 Parents education Level No 3.32 Parents involvement of student s study Yes* 4.68 Student participates in extracurricular activities No 3.45 Number of books at home Yes* 4.42 Student s likeness for the subject areas Yes* 4.83 The amount of time students spend on homework No 3.02 Student s future education plan No 3.44 Student s attendance for class No 3.82 The amount of time students spend on study No 3.90 Student s reading habits Yes* 4.27 Student s self evaluation Yes* 4.42 School climate and safety in school No 3.65 Student s interpersonal relationship No 3.84 The occurrence of student s misbehavior Yes* 4.19 Parents age No 1.94 Family religion No 1.07 Parents marital status No 3.30 Household income No 3.10 The education resources providing from parents Yes* 4.56 Family s living environment No 3.41 Parents employment status Yes* 4.17 Parents working condition No 3.17 Parents involved in children s interpersonal relationships Yes* 4.11 Parents and children spend time together Yes* 4.59 Parent and children s interaction Yes* 4.27 Parents effort for children's education Yes* 4.69 Parents expectation on children s education Yes* 4.30 The family expense on education No 3.85 Parents interaction with school No 3.29 Parents attitude toward school No 3.73 The last column of Table 3 shows the scale value which calculated through successive interval scaling methods. It shows that the factor student s likeness for the subject areas

8 COMPARISON OF DIFFERENT SCALING METHODS FOR EVALUATING FACTORS 69 has the highest scale value (4.83), which indicated that this factor has the largest impact on student s academic performance. And the factor family religion has the least scale value (1.07), which indicated that family religion may have least impact on student s academic performance compared to other factors. Figure 1 further put these 34 factors on the same graph. It shows that most of the factors scale values are quite close to each other. The significant dropping points occur at the parents age, language usage at home and family religion. 6 5 Scale Value Student's likeness for the subject areas Parent's effort for children's education Parent's involvement of student's study Parents and children spend time together The education resources providing from Number of books at home Student's self evaluation Parent's expection on children's education Student's reading habits Parent and children's interaction The occurance of student's misbehavior Parents' employment status Parents involved in children's The amount of time students spend on The family expense on education Student's interpersonal relationship Student's attendence for class Parent's attitude toward the school The working status of family members Factors School climate and safety in school Parent's background Student participates in extracurricular Student's future education plan Family's living environment Parent's education Level Parent's matital status Parent's interaction with school Parent's occupation Parent's working condition Household income The amount of time students spend on Parent's age Language usage at home Family religion FIGURE 1. The successive interval scale values of 34 factors 4. Conclusions and Recommendations. The purpose of this study is to investigate the factors which might impact student s academic performance. Based on the survey data obtained from the 54 senior teachers, this is found that the following 13 factors have significant impact across all of the statistical methods. In addition, based on the successive interval scaling method, the importance of these factors can be ordered as follows: (1) student s likeness for the subject areas (2) Parent s effort for children s education (3) Parent s involvement of student s study (4) Parents and children spend time together (5) The education resources providing from parents (6) Number of books at home (7) Student s self evaluation (8) Parent s expectation on children s education (9) Student s reading habits (10) Parent and children s interaction (11) The occurrence of student s problematic behavior (12) Parents employment status (13) Parents involved in children s interpersonal relationships

9 70 MINGCHUAN HSIEH And the following four variables are not as definitively indicated across all statistical methods, but could be the important factors impacting student s performance, these include: (1) The amount of time students spend on study (2) Interpersonal relationship (3) The family expense on education (4) Parent s attitude toward the school The following 17 factors are not significantly indicated in the statistical methods used in this study, which could infer that they may not have great impact on student academic performance: (1) Language usage at home (2) Parent s background (3) The working status of family members (4) Parent s occupation (5) Parents education level (6) Student participates in extracurricular activities (7) The amount of time students spend on homework (8) Student s future education plan (9) Student s attendance for class (10) School climate and safety in school (11) Parent age (12) Family religion (13) Parent s marital status (14) Household income (15) Living environment (16) Parent s working condition (17) Parent s interaction with school Besides these factors indicated above, additional factors submitted by the expert judges indicated that many other factors may also have impact on student s academic performance, those factors included: Parent s attitude toward children learning, the family atmosphere of learning, the academic achievement of student s siblings the interaction with teachers, the school curriculum and teacher s teaching style, the education resources at school, additional learning opportunities (such as go to cram school after class), Students have their own study space, Family violence, Student s health condition etc. Some of these factors from these variables were also selected to comprise the questionnaires that the surveyed samples could complete in 40 minutes. The questionnaires will be implemented along with the achievement test to obtain the validity evidence. Overall, after summarized the possible factors questionnaires in several large-scale achievement, this study used three different methods to identify the factors which could impact the student academic performance. The first method is to compute the median and the inter-quartile range to test the relative importance of factors. The second method is to compare the mean of the single factor with the overall mean of all factors based on the judges decisions using T test. The third method is to use the successive intervals scaling methods to scale the importance of the factors. It is found that there are more factors were identified at the significance level using the first method (17 factors) than the second

10 COMPARISON OF DIFFERENT SCALING METHODS FOR EVALUATING FACTORS 71 method (13 factors). But the results are quite consistent. The 13 factors identified using the T test are exactly overlapped with the factors identified using the first method. Moreover, those overlapped 13 factors are also the 13 highest factors using the successive scaling method. However, there are some limitations of this study. The method of equal appearing intervals or the successive intervals scaling assumed that the items have determinate scale positions that are the same for different judges (Scott, 1968); however, it is sometime not the case. As Scott (1968) indicated, the model requires that differences in judged location of a particular item are random and do not depend on systematic characteristics of the judges, however, it is usually found that judges with extremist attitudes toward the phenomenon, either positive or negative, do not discriminate effectively among moderate items. It is often unrealistic to assume that the judge s own attitudes are independent of his item judgments. Thus, similar study should be replicated to test the robustness of the conclusion obtained in this study. REFERENCES [1] A. L. Edwards (1952), The scaling of stimuli by the method of successive intervals, Journal of Applied Psychology, vol.36, pp [2] A. L. Edwards (1957), Techniques of attitude scale construction, New York: Appleton-Century-Crofts. [3] A. L. Edwards and L. L. Thurstone (1952), An interval consistency check for scale values determined by the method of successive intervals, Psychometrika, vol.17, pp [4] B. D. Wright (1980), The objective construction of scales, Paper presented at the National Workshop on Research Methodology and Criminal Justice Program Evaluation, Baltimore, Md., March. [5] B. F. Green (1954), Attitude measurement. In G. Lindzey (Ed.), Handbook of social psychology, vol.1, pp [6] E. S. Atkinson (2000), An investigation into the relationship between teacher motivation and pupil motivation, Educational Psychology, vol.20, no.1, pp [7] F. Attneave (1949), A method of graded dichotomies for the scaling of judgments, Psychological Review, vol.56, pp [8] J. P. Guilford (1938), The computation of psychological scale values from judgments in absolute categories, Journal of Experimental Psychology, vol.22, pp [9] J. P. Guilford (1954), Psychometric methods, New York: McGraw-Hill. [10] J. Y. Lin (2007), The impact of individual-family and school factors on students academic achievement: To analysis the educational equality and the relevant issues of junior high school level in terms of SEM, Unpublished PhD dissertation, National Kaohsiung Normal University. [11] J. Y.You, M.Y.Chen, C. H. Tseng, H. C. Lee (2009), The mathematics performance of Taiwanese students and its finding in TEPS, In Service Education Bulletin, vol.25, no.5, pp [12] L. L. Thurstone (1927), A law of comparative judgment, Psychological Review, vol.34, pp [13] L. L. Thurstone (1929), Fechner s law and the method of equal appearing intervals, Journal of Experimental Psychology, vol.12, pp [14] L. L.Thurstone (1929), Theory of attitude measurement, Psychological Review, vol.36, pp

11 72 MINGCHUAN HSIEH [15] L. L.Thurstone and E. J. Chave (1929), The Measurement of attitudes, Chicago: University of Chicago Press. [16] M. A. Saffir (1937), A comparative study of scales constructed by three psychophysical methods, Psychometrika, vol.2, pp [17] Mciver and Carmines (2010), Unidimensional scaling, Sage University Paper Series on Quantitative Applications in the Social Sciences, pp [18] P. E. Barton (2003, October), Parsing the achievement gap: Baselines for tracking progress classroom (Policy Information Report), Princeton, NJ: Educational Testing Service, Policy Information Center, Obtained from [19] W. A. Scott (1968), Attitude measurement, In G. Lindzey (Ed.), Handbook of social psychology, vol.1, pp [20] W. S. Torgerson (1958), Theory and methods of scaling, New York: Wiley.