Product Interest and Engagement Scale, Beta (PIES-beta): Initial Development

Similar documents
Making a psychometric. Dr Benjamin Cowan- Lecture 9

Personality Traits Effects on Job Satisfaction: The Role of Goal Commitment

College Student Self-Assessment Survey (CSSAS)

The Development of Scales to Measure QISA s Three Guiding Principles of Student Aspirations Using the My Voice TM Survey

Original Article. Relationship between sport participation behavior and the two types of sport commitment of Japanese student athletes

Identifying or Verifying the Number of Factors to Extract using Very Simple Structure.

International Conference on Humanities and Social Science (HSS 2016)

Running head: CFA OF STICSA 1. Model-Based Factor Reliability and Replicability of the STICSA

Chapter 9. Youth Counseling Impact Scale (YCIS)

Psychometric Evaluation of the Major Depression Inventory at the Kenyan Coast

VALIDATION OF TWO BODY IMAGE MEASURES FOR MEN AND WOMEN. Shayna A. Rusticus Anita M. Hubley University of British Columbia, Vancouver, BC, Canada

Packianathan Chelladurai Troy University, Troy, Alabama, USA.

Modeling the Influential Factors of 8 th Grades Student s Mathematics Achievement in Malaysia by Using Structural Equation Modeling (SEM)

Running head: CFA OF TDI AND STICSA 1. p Factor or Negative Emotionality? Joint CFA of Internalizing Symptomology

The Impact of Visualization and Expectation on Tourists Emotion and Satisfaction at the Destination

Basic concepts and principles of classical test theory

Development and Psychometric Properties of the Relational Mobility Scale for the Indonesian Population

Research Brief Convergent and Discriminate Validity of the STRONG-Rof the Static Risk Offender Need Guide for Recidivism (STRONG-R)

ASSESSING THE UNIDIMENSIONALITY, RELIABILITY, VALIDITY AND FITNESS OF INFLUENTIAL FACTORS OF 8 TH GRADES STUDENT S MATHEMATICS ACHIEVEMENT IN MALAYSIA

Must be the music: Validation of a theory-based survey

Exploratory and Confirmatory Factor Analysis: Developing the Purpose in Life test-short

Self-Oriented and Socially Prescribed Perfectionism in the Eating Disorder Inventory Perfectionism Subscale

Emotional Intelligence and Leadership

Examining the efficacy of the Theory of Planned Behavior (TPB) to understand pre-service teachers intention to use technology*

John Roodenburg Krongold Centre, Faculty of Education Monash University, Victoria 3800 Australia Esther M.

Testing the Multiple Intelligences Theory in Oman

Factor Analysis. MERMAID Series 12/11. Galen E. Switzer, PhD Rachel Hess, MD, MS

Evaluation: Scientific Studies. Title Text

The Ego Identity Process Questionnaire: Factor Structure, Reliability, and Convergent Validity in Dutch-Speaking Late. Adolescents

Survey research (Lecture 1) Summary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2015 Creative Commons Attribution 4.

Survey research (Lecture 1)

Connectedness DEOCS 4.1 Construct Validity Summary

Structural Validation of the 3 X 2 Achievement Goal Model

Internal structure evidence of validity

The MHSIP: A Tale of Three Centers

Understanding Social Norms, Enjoyment, and the Moderating Effect of Gender on E-Commerce Adoption

A Short Form of Sweeney, Hausknecht and Soutar s Cognitive Dissonance Scale

Critical Evaluation of the Beach Center Family Quality of Life Scale (FQOL-Scale)

Item Analysis & Structural Equation Modeling Society for Nutrition Education & Behavior Journal Club October 12, 2015

Paul Irwing, Manchester Business School

Unit outcomes. Summary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2018 Creative Commons Attribution 4.0.

Unit outcomes. Summary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2018 Creative Commons Attribution 4.0.

An Empirical Study of the Roles of Affective Variables in User Adoption of Search Engines

Analysis of the Reliability and Validity of an Edgenuity Algebra I Quiz

Decision process on Health care provider A Patient outlook: Structural equation modeling approach

An Assessment of the Mathematics Information Processing Scale: A Potential Instrument for Extending Technology Education Research

Design and Preliminary Validation of an Instrument to Measure Medical Student Attitudes Toward the Homeless

RELIABILITY OF THE DENISON ORGANISATIONAL CULTURE SURVEY (DOCS) FOR USE IN A FINANCIAL INSTITUTION IN SOUTH AFRICA CHRISSTOFFEL JACOBUS FRANCK

Assessing the Reliability and Validity of Online Tax System Determinants: Using A Confirmatory Factor Analysis

PSYCHOLOGY, PSYCHIATRY & BRAIN NEUROSCIENCE SECTION

The Validity And Reliability Of The Turkish Version Of The Perception Of False Self Scale

The measurement of media literacy in eating disorder risk factor research: psychometric properties of six measures

School Administrators Level of Self-Esteem and its Relationship To Their Trust in Teachers. Mualla Aksu, Soner Polat, & Türkan Aksu

Pushing the Right Buttons: Design Characteristics of Touch Screen Buttons

validscale: A Stata module to validate subjective measurement scales using Classical Test Theory

The Psychometric Properties of Dispositional Flow Scale-2 in Internet Gaming

Measuring Perceived Social Support in Mexican American Youth: Psychometric Properties of the Multidimensional Scale of Perceived Social Support

A Factorial Validation of Internship Perception Structure: Second-Order Confirmatory Factor Analysis

Developing Academic Identity Statues Scale (AISS) and Studying its Construct Validity on Iranian Students

Reliability and Validity checks S-005

Psychometric Details of the 20-Item UFFM-I Conscientiousness Scale

AUTISM SPECTRUM RATING SCALES (ASRS )

Running head: CPPS REVIEW 1

System and User Characteristics in the Adoption and Use of e-learning Management Systems: A Cross-Age Study

Re-Assessing the Usability Metric for User Experience (UMUX) Scale

Confirmatory Factor Analysis of the BCSSE Scales

Verification of the Structural Model concerning Selfesteem, Social Support, and Quality of Life among Multicultural Immigrant Women

The Youth Experience Survey 2.0: Instrument Revisions and Validity Testing* David M. Hansen 1 University of Illinois, Urbana-Champaign

Summary & Conclusion. Lecture 10 Survey Research & Design in Psychology James Neill, 2016 Creative Commons Attribution 4.0

Development of self efficacy and attitude toward analytic geometry scale (SAAG-S)

UNDERSTANDING THE BLOGGERS CONTINUANCE USAGE: INTEGRATING FLOW INTO THE EXPECTATION-CONFIRMATION THEORY INFORMATION SYSTEM MODEL

IDEA Technical Report No. 20. Updated Technical Manual for the IDEA Feedback System for Administrators. Stephen L. Benton Dan Li

A 2-STAGE FACTOR ANALYSIS OF THE EMOTIONAL EXPRESSIVITY SCALE IN THE CHINESE CONTEXT

Understanding University Students Implicit Theories of Willpower for Strenuous Mental Activities

External Variables and the Technology Acceptance Model

Personal Style Inventory Item Revision: Confirmatory Factor Analysis

Teachers Sense of Efficacy Scale: The Study of Validity and Reliability

Psychometric evaluation of the self-test (PST) in the responsible gambling tool Playscan (GamTest)

Dimensionality, internal consistency and interrater reliability of clinical performance ratings

Oak Meadow Autonomy Survey

Doing Quantitative Research 26E02900, 6 ECTS Lecture 6: Structural Equations Modeling. Olli-Pekka Kauppila Daria Kautto

Instrument equivalence across ethnic groups. Antonio Olmos (MHCD) Susan R. Hutchinson (UNC)

UNIVERSITY OF THE FREE STATE DEPARTMENT OF COMPUTER SCIENCE AND INFORMATICS CSIS6813 MODULE TEST 2

Electronic Supplementary Material. for. Pride, Personality, and the Evolutionary Foundations of Human Social Status

AN EVALUATION OF CONFIRMATORY FACTOR ANALYSIS OF RYFF S PSYCHOLOGICAL WELL-BEING SCALE IN A PERSIAN SAMPLE. Seyed Mohammad Kalantarkousheh 1

Family Assessment Device (FAD)

Survey Research. We can learn a lot simply by asking people what we want to know... THE PREVALENCE OF SURVEYS IN COMMUNICATION RESEARCH

Background. Workshop. Using the WHOQOL in NZ

Author s response to reviews

Measuring the User Experience

The New Zealand Mental Health Commission has defined recovery as. The Wellness Recovery Action Plan (WRAP): workshop evaluation CONSUMER ISSUES

Psychometric Instrument Development

Psychometric Instrument Development

2/2/2014. Psychometric Reanalysis of Happiness, Temperament, and Spirituality Scales with Children in Faith-Based Elementary Schools.

MEASUREMENT, SCALING AND SAMPLING. Variables

The Relationship of Competitiveness and Achievement Orientation to Participation in Sport and Nonsport Activities

Towards an instrument for measuring sensemaking and an assessment of its theoretical features

Taylor University s Intercultural Inventory Analysis. Presented by Natalie Nunes August 5, 2009

Factorial Validity and Reliability of 12 items General Health Questionnaire in a Bhutanese Population. Tshoki Zangmo *

Before we get started:

Transcription:

Product Interest and Engagement Scale, Beta (PIES-beta): Initial Development Christopher N. Chapman Microsoft Corporation Microsoft Way (cchap) Redmond, WA 98052 USA chris.chapman@microsoft.com Michal Lahav Sakson & Taylor Consulting 09 N 36th St. Seattle, WA 9803 USA michalahav@gmail.com Edwin Love Western Washington University Dept. of Finance and Marketing 56 High St. Bellingham, WA 98225 USA Edwin.Love@wwu.edu James L. Alford Volt Information Sciences 26 Willows Rd. NE, Ste. 200 Redmond, WA 98052 USA jalford0974@yahoo.com Copyright is held by the author/owner(s). CHI 2009, April 4 9, 2009, Boston, MA, USA ACM 978--60558-246-7/09/04. Abstract We report a work in progress: development and initial validation of the Product Interest and Engagement Scale (PIES), a short assessment instrument measuring consumer interest in technology products. PIES reflects an explicitly multidimensional, hierarchical, and extensible model of product interest. It assesses consumer product interest in terms of an overall interest scale plus three subscales assessing interest in features and choices, personal image as affected by a product, and interest in optimizing one s choice with regards to a product. We report factor structure in a sample of N=225 US consumers and replication with N=80 US consumers. The results establish reliability of the overall 2-item scale and subscales in a broad consumer sample (Cronbach s alpha = 0.89 overall, 0.82-0.88 for subscales). Validity measures in the validation sample demonstrate convergent and discriminant validity with product ownership and product involvement measures. We regard PIES as currently being in beta form (PIES-beta). It is suitable for usage now but will undergo further revision in 2009. Keywords Product interest, product involvement, evaluation, psychometrics, scale development. ACM Classification Keywords H.m. Information systems. Miscellaneous. K.. Computing milieu. Computer industry. Markets.

2 Introduction An important factor in applied human-computer interaction (HCI) research is the degree to which users are interested in or engaged with a given product or service, i.e., the extent to which users are interested to think about a product, its capabilities, features, importance in their lives, and so forth. A traditional market research term is product involvement, [3,9,0] although we adopt a more expansive view that may be described as product interest and engagement (PIE). PIE is important for user-centered product design and development for several reasons. A product s audience is related to interest: more-interested consumers are more likely to use a product [3,]. A high PIE product may have different user requirements than a low PIE product (for instance, rather than just deliver base functionality, it might support a consumer image ). HCI practitioners may use an assessment of PIE to gain a better understanding of their customers, products, and the dimensions of product interest. Existing product involvement measures are insufficient for HCI usage for several reasons. The problems include (a) respondent difficulty with concepts for instance, one scale [0] rates products on a scale of mundane fascinating, which is a difficult conceptual pair; (b) potential construct problems in the technology space (e.g., pairs such as mundane/ fascinating may not be opposites for a product such as a PC interface device); and (c) concepts may be inapplicable e.g., a product may be needed without being interesting (cf. scales [9,0]). Most importantly, existing product involvement scales are generally unidimensional [9,0]. This reflects scale consistency and reliability, but excludes the opportunity to measure different kinds of product interest that are related yet distinct. A unidimensional scale gives little insight into why a user is interested in a given product. We have designed a new product interest scale (PIE scale, or PIES) that is intended to be: () multidimensional and able to differentiate kinds of interest; (2) psychometrically strong and reliable; (3) consistent with expected validity measures; (4) useful across a range of technology products; (5) applicable to a broad consumer population. PIES does not replace traditional product involvement measures but extends them with a multidimensional framework that may be preferable for some products and research goals. PIES is a work in progress. We report here on initial development comprising item selection, factorial modeling, subscale construction, validity measures, and replication with two products in two samples. Future efforts will include scale refinement and application to additional products, as discussed below. Construct Definition and Item Generation We began by defining the product interest and engagement (PIE) construct: PIE is the degree of interest that a consumer shows to think about and use products from a specific product category. PIE is related to product involvement and also includes other aspects of product interest. These include interest in features, in attributes important to product choice, and the contribution of a product to social interaction such as

3 one s image. PIE is hierarchical; it is valid to consider both overall interest and discrete aspects of interest. On the basis of literature review and construct brainstorming, the authors wrote 38 initial items that reflect general areas of product interest. Each item refers to one potential aspect of product interest with a placeholder for a specific product. For example, item 2: The differences among are very important. The initial 38 items were piloted with N=3 consumer respondents in Redmond, WA. Following that review, several items were revised for clarity, 3 were cut, and was added. This yielded a list of 36 items for initial testing: PIES-0 (available from first author). Development Samples and 2 PIES-0 was administered online to a sample of N=405 adult consumer respondents in the US, with quotas to ensure broad sampling by age, gender and technology product ownership. The products assessed were PC keyboards and ipods/mp3 players. Each PIES statement was presented in random order as a Likerttype agreement item with a 7-point scale including neutral, from strongly disagree to strongly agree. PIES-0 was given twice to each respondent (once per product), along with the 0-item Zaichkowsky scale [0] for ipods/mp3 players, in randomized order. The N=405 sample was randomly sampled into N=225 Sample for model development and N=80 Sample 2 for validation. The two PIES-0 product response sets, PIES-0-Keyboard and PIES-0-MP3, combine as 2 cases per respondent to give PIES-0-ALL. Data analysis was performed with R version 2.8. [2,5,6,7]. Factor Analysis We initially investigated construct coherence through exploratory factor analysis (EFA) [8]. Our EFA approach comprised three iterated steps: () identifying the number of factors through parallel analysis, scree plot, and interpretability; (2) factor extraction followed by oblique rotation (oblimin []), consistent with the theorized hierarchical structure; (3) item selection based on factor loading and interpretability. Space does not permit full description of the EFA, but we recap here. Parallel analysis [6,7,8] demonstrated 4 factors for PIES-0-ALL (36 items) in Sample. 2 items were removed due to negative loading, and 8 were removed for cross-loading. The trimmed PIES-0-ALL (26-item) dataset exhibited 4 clear factors in Sample. One of those factors concerned immediate purchase intent orthogonal to PIE so its items were removed. The remaining three EFA factors represented clear factorial constructs: interest in the choices and features that are available ( choice interest ); interest in optimizing a choice rather than just selecting one that is good enough ( optimize ); and interest in a product s contribution to one s personal image ( image ). For each of those factors, 4 items were selected that maximized item-factor loading with content differentiation. These 2 items then underwent a new EFA with Sample 2 (holdout data). Consistent with Sample, EFA in Sample 2 yielded three factors with items loading in the same pattern as in Sample. EFA loadings for Sample 2 are shown in Table. There were three relatively simple factors with high itemfactor loading. Factor correlations were F/F2: r=0.47

4 (p<0.0); F/F3: r=0.28 (p<0.0); F2/F3: r=0.52 (p<0.0). Thus, the factors were related as expected. Item Factor Factor 2 Factor 3 i8 0.677 i9 0.240 0.60 i2 0.597 i3 0.783 i5 0.773 i20 0.780 i26 0.855 i27 0.226 0.584 i28 0.875 i33 0.88 i34 0.896 i35 0.869 Table : Factor loadings for 2-item version in Sample 2 (validation sample). Loadings < 0.20 are not shown. Subscale Construction and Reliability Given the clear item-factor pattern, we proceeded to construct subscales. A subscale was defined for each factor comprising the four items with highest loadings in the 2-item EFA. Each subscale score is the sum of the individual items 7-point scores (reversed for some items), while the overall score is the sum of all items. The scale and subscales are shown in Table 2. Reliability was assessed by examining the EFA structure replication from Sample to Sample 2 (as above) and by Cronbach s alpha reliability index [7]. For PIES-0- ALL in Sample 2, alpha=0.89 for the overall PIES scale, alpha=0.84 for the Choice Interest subscale, alpha = 0.82 for Optimize subscale, and alpha=0.88 for Image subscale. This demonstrates strong scale and subscale consistency in our consumer validation sample. Choice interest scale (derived from Factor 2) I find some to be exciting products (i9) The differences among are very important (i2) There are some features that matter a lot to me (i26) In choosing a I would look for some specific features or options (i28) Optimize scale (derived from Factor 3) It doesn't matter much which I might choose (i3r) When selecting a I would probably just choose the first one that was good enough (i5r) One is about as good as another (i20r) I don't care much about the specific features of (i27r) Image scale (derived from Factor ) It is important which a person owns (i8) When other people see my they form an opinion of me (i33) The that I use says a lot about me (i34) You can learn a lot about someone by seeing what kind of they use (i35) Table 2. PIES-beta with subscales. Parentheses show item numbers (of 36 items). R indicates reverse scored items. The result of this initial scale development process was a proposed 2-item measure with 3 subscales of 4 items each, which we term PIES-beta. The complete PIES-beta scale is shown in Table 2.

5 Construct Validity Construct validity of PIES-beta was assessed first by looking at convergent and discriminant correlations among the PIES-beta results across products and with a standard product involvement scale. We used the 0- item Zaichkowsky product involvement scale [0] with MP3 players as the comparison (hereafter Z-MP3 ). In Sample 2, the correlation of PIES-beta-MP3 with Z-MP3 was r=0.63 (df=78, p<.0). This shows that PIESbeta is strongly related to product involvement but is not identical with it. PIES-beta-Keyboard correlated with PIES-beta-MP3 with r=0.50 (df=78, p<.0). This demonstrates that people high in interest for one technology product are likely to be interested in another. The correlation of PIES-beta-Keyboard with Z- MP3 was r=0.22 (df=78, p<.0), where the lower correlation shows discriminant validity for PIES-beta vs. product involvement when products differ. People who have more interest in a product should be more likely to own it. We tested this for ownership of MP3 players and third-party (not original) PC keyboards. For MP3 players, PIES-beta-MP3 was higher for owners than non-owners, while for Keyboards, PIES-beta-Keyboard was higher for owners than nonowners. In both, PIES-beta for one product showed no significant difference for owners/non-owners of the other product. Results are shown in Table 3, indicating validity of PIES-beta with regard to product ownership. Consistency of the hierarchical model of three factors and an overall second-order factor was assessed with a hierarchical structural equation model (SEM) [2,4,8] using Sample 2 data. The tested model and resulting path coefficients are shown in Figure. Keyboard PIES Keyboard Ownership MP3 Ownership Owner KB PIES mean: 4.05 KB PIES mean: 3.74 Non-Owner KB PIES mean: 3.5 KB PIES mean: 3.65 t-test t=3.02, df=28, p<.0 t=0.50, df=48, p=.62 MP3 PIES Keyboard Ownership MP3 Ownership Owner MP3 PIES mean: 4.37 MP3 PIES mean: 4.55 Non-Owner MP3 PIES mean: 4. MP3 PIES mean: 3.90 t-test t=.49, df=08, p=.4 t= 4.2, df=63, p<.0 Table 3. t-tests for PIES-beta scores by product ownership. Briefly, the overall SEM model fit for the hierarchical model was good. Model chi-square=9, df=55; GFI=0.926 (good > 0.90); NFI=0.94 (good > 0.90); RMSEA= 0.083 with 90% CI=0.070-0.096 (good < 0.0); and SRMR=0.097 (good < 0.0). (See [4] for details on SEM measures and their interpretation.) PIES.83.92 F: Choice interest F2: Optimize F3: Image.8.90.77.02.02.98.98.72.03 i9 i2 i26 i28 i3 i5 i20 i27 i8 i33 i34 i35 some [are] exciting differences are important some features matter a lot look for specific features doesn t matter which I choose (r) would select good enough (r) one is as good as another (r) don t care about features (r) is important which one owns when see, others form an opinion the I use says a lot about me can say a lot about someone Figure. SEM confirmatory factor model for PIES-beta. Unstandardized coefficients are shown. Items are abbreviated.

6 Conclusion, Discussion and Future Directions The proposed 2-item PIES-beta scale and subscales show strong reliability and validity in a broad consumer sample as a measure of the PIE construct. We believe that PIES-beta is suitable now for exploratory usage for some technology products. PIES-beta may assist HCI practitioners better to understand their products, customers, and dimensions of product interest. There are areas for further development of PIES. Some items are perhaps unnecessarily confounded with product ownership (e.g., Image items in Table 2). Some items would benefit from rewording (e.g., item 35 uses a singular they ). The Optimize subscale comprises entirely reverse-scored items; it would be preferable to reduce the potential confound with scoring direction by including positive items. Assessment with additional products would confirm the scale s breadth of applicability. We also plan to compare the proposed hierarchical factor model to alternative SEM models. It is important to note that the PIE construct is intentionally non-exhaustive with regards to product interest and is intended to be extensible. PIES may be extended with new factors or subscales if warranted. The next steps for PIES development, to occur in the first half of 2009, are to revise the existing items and add new potential items; to field the revised items with a new development sample; to revise the subscales as needed and confirm the model; and to validate the results in a final sample. In parallel with this scale development effort, the authors will begin to use PIESbeta in applied research. (For updates on PIES, address correspondence to the first author.) Citations [] Bernaards, C.A. and Jennrich, R. I. (2005) Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement: 65, 676-696, 2005. [2] Fox, J. sem: Structural equation models. R package ver. 0.9-4. http://www.socsci.mcmaster.ca/jfox/, 2008. [3] Kappelman, L.A. Measuring user involvement. SIGMIS Database, 26:2-3, 65-86, 995. [4] Kline, R.B., Principles and Practice of Structural Equation Modeling (2 nd ed). Guilford, 2004. [5] R Development Core Team (2008). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. www.r-project.org, 2008. [6] Raiche, G. nfactors: Non-graphical solution to the Cattell scree test. R package ver. 2., 2007. [7] Revelle, W. psych: Procedures for personality, psychometric, and psychological research. R package ver..0-58, http://personality-project.org/r, 2008. [8] Worthington, R.L. & Whittaker, T.A. Scale development research. The Counseling Psychologist, 34, 806-838, 2006. [9] Zaichkowsky, J.L. Measuring the involvement construct. J. of Consumer Research, 2, 34-352, 985. [0] Zaichkowsky, J.L., The personal involvement inventory reduction, revision, and application to advertising, J. of Advertising, 23:4, 59-70, 994. [] Zins, A.H., and Bauernfiend, U. Explaining online purchase planning experiences with recommender websites. In Frew, A.J. (ed), Information and Communication Technologies in Tourism 2005, Springer, 2005.