ASSESSMENT OF RELIABILITY AND STATISTICAL SIGNIFICANCE BY P- VALUE OF PRE AND POST SURVEYS ON INDUSTRIAL BUYING BEHAVIOR ON A 5 POINT LIKERT SCALE

Transcription

1 ASSESSMENT OF RELIABILITY AND STATISTICAL SIGNIFICANCE BY P- VALUE OF PRE AND POST SURVEYS ON INDUSTRIAL BUYING BEHAVIOR ON A 5 POINT LIKERT SCALE Mahesh M. Kulkarni Research Scholar, Pacific University, Udaipur Dr. Nitin Joshi Director, V N Bedekar College of Management Studies ABSTRACT In most of the educational and social science research, the most frequently used psychometric tool is 5 Point or 7 point Likert scale. Numerous studies revealed the importance of Likert type scales in assessing consumer behavior patterns with respect to variables affecting the decision making process. Some of the research carried out in the past also reflected the analysis and inclusion of points on the scale. After reviewing through various literatures available and then understanding the significance of the responses collected, this research paper aims at providing a simple methodology for assessment of reliability and statistical significance of Pre and Post surveys carried out on a set of 5 Point Likert Scale. The analysis starts with the necessity of psychometric tools like 5 Point Likert scale and its variants and focuses on validity, reliability and significance of the scale. These responses are measured on five independent variables such as Perceived Risk, Perceived service Quality, environmental uncertainty, Interpersonal Relationship and Perceived Brand Ranking. All data were collected using online surveys and Personal Interviews during the Pre survey and the same respondents were interviewed during the post surveys phase. The survey is conducted across various functions and positions in the organization in order to get dispersed data for this research purpose. A random sample was drawn. Total 80 Survey forms were circulated in the pre-survey phase but due to some inevitable reasons 20 respondents were not available in postsurvey phase. Due to this reason these 20 Pre-surveys are excluded from the analysis. The Pre and Post responses approach transforms the Likert responses into paired comparison responses between the items. All the 80 sets of data were reliable where p-value is less than 0.05 for the entire group of data collected. This means that the Pre and Post survey questionnaires are reliable and significant and not due to chance. Key Words Reliability, Likert scale, statistical significance, Industrial buying behavior, T-test, Pre and post surveys, etc.

2 Introduction In research on Industrial buying behaviour, independent variables such as Perceived risk, perceived service quality, Interpersonal relationship, environmental uncertainty and perceived brand ranking which leads to purchase decision process is difficult to capture. This is due to the simple reason that there are more than one person and function involved in the decision making process and the perception of each individual if different depending on the situation they are in. A common method for data-collection is to use questionnaires and they are often of Likert scale type either 5 point scale or 7 point scale. It is important to check the reliability and statistical significance of this questionnaire in advance before circulating it to the larger sample size. This analysis provides the ability to perform statistical tests such as T-test to determine the significance of mean differences between the Pre and post responses received from various respondents. In connection to the analysis of such data we want to test the reliability and statistical significance of the questionnaire and do necessary alterations, if needed. During the literature survey of earlier studies, there is no specific case found on Industrial buying behaviour and the constructs mentioned above. This has directed the researcher to carry out some simple techniques such as Pre and post surveys and discuss; The response to the questionnaire by the same respondent in pre and post survey phase is reliable and not due to chance. Simple techniques of calculating mean, difference and p-value can help is evaluation of Pre-Post surveys. It is also important to note that how we interpret the data and level of agreement for each survey question and record the same on a scale of 1 to 5. (1 Strongly Disagree and 5 strongly agree). To make sure what are appropriate analysing methods of the data one has to be aware of the type of the data that has been collected. When developing questions to assess psychological attributes, it has been strongly suggested that the end points of preference response scales should be words or phrases (Jones and Thurstone, 1955). Furthermore, it need to investigate the semantic properties of commonly used

3 scale point descriptors to make sure that they possess the required properties and also carry meaning that is as clear as possible to subject that represent the researcher s population of interest. The aim of the research is to develop simple methodology to check the response and questionnaire reliability on a Likert scales. Literature Survey A Likert scale (Likert, 1932) is a popular instrument to measure psychological behaviour, attitudes, preferences, perceptions, opinions, conceptions, etc., in general (Göb, McCollin, & Ramalhoto, 2007; Wu, 2007). There are three kinds of concepts emerging about Likert scales; these are a Likert scale, known as well as a Likert (summated) scale, a Likert item, and a Likert-type item. Often researchers use Likert scale when speaking about a Likert item and consider these as synonyms. Sometimes the researcher end up applying these notions to constructs which are nonrelevant to the studies. John S. Uebersax (2006) points out two reasons why an understanding of the differences between these constructs, and the adequate use of them, is important. Firstly, mutual agreement on definitions of the construct is essential, so that we can more clearly understand the exact meaning of the responses. Secondly, the appropriate statistical methods on terms for making a progress on the responses received from various respondents and then analyse the variables (Uebersax, 2006). The most important feature of a Likert scale is the way we construct the scale. First, a series of questions are defined and the construct is designed in such a way that the answer/response leads to the feature in question. Then the questionnaire is administered to a small or representative sample size and analysed for reliability and statistical significance. Method of calculating Cronbach Alpha is one way to check the reliability. But, researcher in this paper has also designed p-value calculation by t-test method for any small sample size. Once the reliability and statistical significance is assessed, then these items are administered to a large group of respondents. Based on the answers of each respondent an index of selectivity is calculated, and finally, the items with the highest selectivity constitute the scale. For the purpose of this research, following scale is used. 1 = Strongly disagree 2 = Disagree

4 3 = Neutral/Indifferent 4 = Agree 5 = Strongly agree Therefore, a Likert scale is never an individual item; it is always a set of items (Uebersax, 2006), or in other words, a series of statements (Leder & Forgasz, 2002). Within the research any data for management studies and marketing profess, there is a growing concern for the quality of survey data. As problems become more complex, collection of large information and more complex data appears desirable. Though surveys can be (and are being) designed to satisfy this demand for more information, researchers have begun to recognize that such "information-dense" surveys may not only be inefficient (i.e., because of low response rate), but also may compromise the quality of data because of their length and complexity (Childers and Ferrell 1979). New approaches for data collection such as computerized or online interviewing method, personal interviews at right place and right time, etc. have been proposed to address this problem. It has been observed that the computerized/digital surveys method is reported to be more "efficient" and "interesting" to respondents. For most research involving attitudinal constructs, researchers have no choice but to have respondents respond to all items on a predesigned questionnaire. Data Collection and Analysis As the survey has been conducted by personal interview and digital survey method, it may be certain that the statements in our study were one of the basic considerations for finding appropriate statistical methods for analyzing the data. The questionnaire is administered to 80 participants in Presurvey phase and the responses are collected and recorded. I. Table 1 Questionnaire Design (Refer Table I Here) The Data consists of respondents answer to the question of interest, their age, type of Industry, purchasing process, department, number of years of experience, etc. Then the actual constructs describing Perceived risk, Perceived service quality, perceived brand ranking, etc. will follow. A Likert scale itself is comprised of Likert items or Likert-type items because it was rated as Strongly disagree, disagree, Neutral/indifferent, Agree, and Strongly Agree.

5 Once the responses are collected then each numerical coding system is followed as 1 5 e.g. strongly agree is 5 and strongly disagree is 1 so that a higher score reflects a high level of agreement of each item. This is important to code the responses in right manner because this will further need to calculate an average or mean score for the whole sample size and for each survey question. It is of vital importance how you interpret the level of agreement got each survey questions will depend on each question asked. II. Table II Pre-survey Responses (Refer Table II Here) III. Post-survey Responses Table III Post-survey Responses The same respondents were contacted during post survey phase and the responses were recorded. A deliberate attempt is made during the post-survey phase to contact the respondents during at the same place and time in order to get the authentic responses. For this purpose, all the respondents in Pre and Post-surveys are contacted during office lunch time. (Refer Table III Here) RESULTS AND DISCUSSIONS Table IV Results (Refer Table IV Here) The p-value is a numerical estimate calculated for a set of 4 questions in each of the above variables mentioned. This p- value estimate of the reliability of the assumption that the responses received during the Pre and Post survey phase are real and not by chance. 1. As the p-value calculated for each question in the constructs result as 0.10 or less except for question 3 of environmental uncertainty which is This means that 90% surety of the questions and the responses in Pre and post surveys are real and not by chance. 2. For majority questions the p-values is less than 0.05 which means that the level of confidence for these questions in pre and post survey phase in more than 95% and the researcher may say that the data is

6 reliable and statistically significants for a p-value of less than For p-value of 0.10 or above, researcher may take a call to reframe the question in order to make it reliable and statistically significant when administering the questionnaire to larger sample size. CONCLUSION Although, there is no common methodology used by researchers for evaluating Likert scale date and the methods which are the most appropriate one. Researchers need to explain the concepts one uses or to justify the methods one chooses to use. In this research paper, the evaluation of reliability and statistical significance is carried out for the pre and post surveys conducted with the same set of respondents. Pilot survey conducted in the assessment of Industrial buying behaviour by pre and post survey method, results obtained by t- test confirms that the data is reliable and statistically significant at a p-value of 0.05 or less than The differences in the mean scores for each of the questions in the construct are significant, rather than due to chance. Acknowledgement We would like to thank you all the respondents who has contributed during the Pre and Post Survey phase of this research work. My sincere thanks to colleagues who were instrumental in conducting and arranging the interviews. Special thanks to Ashok Konduskar and Amit Bogam for fruitful discussions and suggestions about carrying out the Post surveys at right time and right place. References - Adams, E., Fagot, R. F., & Robinson, R. E. (1965). A theory of appropriate statistics. Psychometrika, 30 (2), Andrews, F. M., Klem, L., Davidson, T. N., O Malley, P. M., & Rodgers, W. L. (1981). A guide for selecting statistical techniques for analyzing social science data. Ann Arbor: Institute for Social Research, University of Michigan. Bond, T., & Fox, C. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, New Jersey: Lawrence Erlbaum Associates. Clason, D. L., & Dormody, T. J. (1994). Analyzing data measured by individual

7 Likert-type items. Journal of Agricultural Education, 35 (4), Cronbach, L.J (1987). Statistical tests for moderator variable: Flaws in analyses recently proposed. Psychological Bulletin, 102, Fiedler, Fred E., Terence R. Mitchell, and Harry C. Triadis. "The Culture Assimilator: An Approach to Cross Cultural Training," Journal of Applied Psychology, 55 (1971), Göb, R., McCollin, C., & Ramalhoto, M. F. (2007). Ordinal methodology in the analysis of Likert scales. Quality & Quantity, 41, Haley, Russell J. "Benefit Segmentation: A Decision-Oriented Research Tool," Journal of Marketing, 32 (July 1968), Hart, M. C. (1996). Improving the discrimination of SERVQUAL by using Magnitude Scaling. In G. K. Kanji (Ed.), Total quality management in action. London: Chapman and Hall. Jean-Marie Blinf and Joe A. Dodsonj, The Relationship between Attributes, Brand Preference, & Choice: A Stochastic View, Management Science, Vol 26, 1980 PP Johnston, W.J. and Lewin, J.E. (1996), Organizational buying behavior: toward an integrative framework, Journal of Business Research, Vol.35, pp Moriarty, R.T. and Spekman, R.E. (1984), An empirical investigation of the information sources used during the industrial buying process, Journal of Marketing Research, Vol. 21 No. 2, pp Kislenko, K. (2007a). Structuring students' beliefs in mathematics: A Norwegian case. In K. Hoskonen, and M. S. Hannula (Eds.), Current State of Research on Mathematical Beliefs XII. Proceedings of the MAVI-12 Workshop, May 25-28, University of Helsinki. Department of applied Sciences of Education. Research Report. Kislenko, K. (2007b). A study of Norwegian students beliefs in mathematics. In B. Jaworski, A. B. Fuglestad, R. Bjuland, T. Breiteig, and B. Grevholm (Eds.), Læringsfellesskap i

8 matematikk Learning Communities in Mathematics (pp ). Bergen: Caspar Forlag. Leder, G. C., & Forgasz, H. J. (2002). Measuring mathematical beliefs and their impact on the learning of mathematics. In G. C. Leder, E. Pehkonen, and G. Törner (Eds.), Beliefs: A hidden variable in mathematics education? (pp ). Dordrecht: Kluwer Academic Publishers. Likert, R. (1932). A technique for the measurement of attitudes. NY: Archives of Psychology. Lodge, M. (1981) Magnitude scaling: Quantitative measurements of opinions. Sage University Paper series on quantitative applications in the social sciences, , Beverly Hills CA: Sage Publications. Lord, F. M. (1953). On the statistical treatment of football numbers. American Psychologist, 8, Ma, X. (1997). Reciprocal relationships between attitude toward and achievement in mathematics. Journal of Educational Research, 90, Nurmi, A, Hannula, M. S., Maijala, H., & Pehkonen, E. (2003). On pupils selfconfidence in mathematics: gender comparisons. In N. A. Pateman, B. J. Dougherty, and J. Zillox (Eds.), Proceedings of the 27 th conference of the international group for the psychology of mathematics education (pp ). Hawaii: University of Hawaii. Pehkonen, E. (1994). On differences in pupil s conceptions about mathematics teaching. The Mathematics Educator, 5 (1), Pehkonen, E. & Törner, G. (2004). Methodological considerations on investigating teachers beliefs of mathematics teaching and learning. Nordic Studies in Mathematics Education, 9 (1), Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103, Streitlien, Å., Wiik, L., & Brekke, G. (2001). Tankar om matematikkfaget hos elever og lærarar. Oslo: Læringssenteret.

9 Uebersax JS. Likert scales: dispelling the confusion. Statistical Methods for Rater Agreement website Available at: es/jsuebersax/likert.htm. Accessed: January 10, Vale, C. M., & Leder, G. C. (2004). Student views of computer-based mathematics in the middle years: does the gender make a difference? Educational Studies in Mathematics, 56, Wu, C. H. (2007). An empirical study on the transformation of Likert-scale data to numerical scores. Applied Mathematical Sciences, 1 (58),

10 List of Tables: Table I: Questionnaire Design

11 Table II: Pre-survey Responses Table III: Post-survey Responses

12 Table IV: Results Prisk Perceived Risk Squal Service Quality Prel Personal relationship Uncer Environmental Uncertainty BrandR Brand Ranking