Measure twice and cut once: the carpenter s rule still applies

Size: px

Start display at page:

Download "Measure twice and cut once: the carpenter s rule still applies"

Colin Terry
5 years ago
Views:

1 Mark Lett (2015) 26: DOI /s x Measure twice and cut once: the carpenter s rule still applies Wagner A. Kamakura Published online: 27 April 2014 # Springer Science+Business Media New York 2014 Abstract In a lead article published by the Journal of Marketing Research in 2007, Bergkvist and Rossiter (Journal of Marketing Research 44:175 84, 2007) recommend that for the many constructs in marketing that consist of a concrete singular object and a concrete attribute, such as A Ad or A Brand, single-item measures should be used (page 175). This conclusion is based on empirical analyses correlating single-item and multiple-item scales measuring attitudes towards advertisements and the advertised brands, collected simultaneously in a single survey instrument. Finding no statistically significant differences between the correlations obtained with the single and multiple items, the authors conclude that there is no loss in predictive validity with the use of single items, which is the basis for their recommendation. Obviously, their recommendation produces substantial savings in data-gathering costs. Consequently, their article has been highly cited (over 600 cites as of September 2013), by authors justifying their use of single-item measures. In this note, I revisit well-known concepts of psychometric theory to demonstrate that this practice is illadvised. First, I argue that repeated measures are necessary not only to improve the validity of some measurement instruments but also, more importantly, to make it possible to assess and correct measurement instruments for random (non-systematic) measurement errors. Second, I argue that rather than testing for predictive validity, the authors actually tested for concurrent validity, using a common survey instrument, thereby confounding their results with spurious correlations due to common-methods biases. Third, I conduct a true predictive validity test using two attitudinal scales and consumption behavior as a predictive criterion to show that, once corrected for measurement errors, multiple-item scales consistently outperform their single-item equivalents. Keywords Measurement. Psychometrics. Validity. Reliability Even though classic measurement theory has been known for more than half a century, it remains relevant to marketers, because psychometrics plays a critical role in their W. A. Kamakura (*) Jones Graduate School of Business, Rice University, 6100 Main Street (MS 531), Houston, TX 77005, USA kamakura@rice.edu

2 238 Mark Lett (2015) 26: quest for understanding consumer behavior. Consequently, the property of surveybased measurement instruments has been the subject of a long debate among marketing researchers. In a recent Journal of Marketing Research (JMR) article, Bergkvist and Rossiter (2007) investigated the relative merits of multiple-item and single-item measurement scales and, based on theoretical considerations and empirical evidence, concluded that for the many constructs in marketing that consist of a concrete singular object and a concrete attribute, such as A Ad or A Brand, single-item measures should be used (page 175). This conclusion suggests that when measuring a concrete singular object on a concrete attribute, a single measurement should suffice, thereby contradicting the old carpenter s rule( measure twice, cut once ). My own behavior is often more consistent with the carpenter s rule than to the one prescribed by the recent JMR article; any time I need to measure the length (concrete attribute) of a piece of wood (concrete object), I have a tendency to repeat the measurement at least once before cutting it. Other than being obsessive, what leads me to this seemingly wasteful behavior, according to Bergkvist and Rossiter (B&R heretofore)? I know the object being measured does not change between measures, and I also know that the measurement scale is valid. What I do not know is the scale s precision, particularly because of operator error ; repeated measures provide me with information about the precision of my measurements. Obviously, with a single measure, there is no way to know anything about measurement precision until it is too late (and another portion of a precious tree is put to waste). Repeated measures are essential for assessing the precision of a measuring instrument. Because asking exactly the same question multiple times is rarely feasible, social scientists use multiple items in lieu of repeated measurements to assess measurement error. This has been the recommended practice in our literature (Perdue and Summers 1986; Zhao et al. 2010). The now popular JMR article recommends otherwise. This conflict between common wisdom and the practice prescribed by B&R invites further scrutiny of their arguments and empirical evidence, because their conclusion can be potentially used by some researchers to justify reducing costs in their measurements. In other words, their article can be misused as an excuse for reducing data collection costs by measuring important constructs with single items, thereby preventing the proper error assessment of these measurements. B&R present two main arguments for the use of multiple items in measurement scales. The first is that multiple-item measures are inherently more reliable because they enable computation of correlations between items (page 176). The use of multiple items does indeed enable the assessment of reliability, which is not possible without repeated measures. However, the main benefit is not that the measurement scales are automatically more reliable, but that reliability can be assessed and therefore items can be selected to obtain a potentially more reliable measurement instrument. The use of repeated measures makes it possible to assess the precision of a measurement instrument. An intuitive way to understand the value of replicates is to consider each of them as random draws from the same distribution of measurements centered around the true value (assuming no systematic bias), with a standard deviation that accounts for random measurement error. Multiple measurements produce multiple draws from this distribution, thereby allowing the researcher to assess measurement error from the standard deviation of these replicates, which decreases with the number of replicates. With a single item, this possibility is lost. Even for concrete attributes of concrete

3 Mark Lett (2015) 26: objects, such as temperature, distance and weight, multiple measures are essential to assess the precision of a measurement instrument, regardless of its validity; this is not a psychometric requirement but a common practice across disciplines. In their discussion on page 176, B&R equate reliability to validity, for example by using alpha reliability (an assessment of measurement error) as a synonym for alpha validity (B&R, page 178). The second argument for using multiple items is to cover the entire domain of the construct being measured. B&R acknowledge that some constructs might be multifaceted and therefore multiple items might be necessary to cover the entire domain. However, even when the construct is relatively simple and unidimensional, multiple items might be necessary to cover the entire range of the construct. In the title and abstract, B&R claim to have tested the predictive validity of multipleitem versus single-item measures of the same two constructs. In reality, predictive validity was not tested in their study, because both A Ad or A Brand measurements were collected from the same subjects in the same survey at the same occasion. In a typical predictive validity test, test scores are collected first and the criterion measure is collected at a later time. An alternative way for testing the predictive validity of a measurement instrument is to relate it to the measurement of another construct, obtained from an independent source, such as eye-movement camera, pupilometer, and galvanic skin response, so that there is real prediction involved from the measurement to the criterion. While prediction is often cited in the article, the only evidence presented in it is of an association or correlation between two concurrent measures gathered within the same (survey) instrument. Given that B&R did not assess the predictive validity of A Ad or A Brand, what does the empirical evidence presented in their article show? Correlation between two measurements obtained at the same time provides evidence of concurrent or convergent validity (Cronbach and Meehl 1955), if the two constructs being measured are expected to be correlated; if the two constructs are not expected to be correlated, correlation among the two measurements provides evidence of poor discriminant validity. However, since the two measurements were collected via perceptual items at the same occasion, the correlation between the two measurements contains a strong spurious component due to well-known common-methods biases (Podsakoff et al. 2003), such as halo effect, response styles, and self-generated validity (Simmons et al. 1993). B&R imply that this spurious correlation could occur between two single-item measures if an identical format is used for both, which is true, but does not preclude biases when the items have different formats, since the various forms of common-methods bias will still bias any concurrent measurements based on selfreported items. As a validity test for multiple and single-item scales, B&R posit three hypotheses that are stated in terms of correlations among measurements (H1 H3, page 176). Given that the main research interest is on the correlation among constructs rather than measurements, it would seem more reasonable to state the hypotheses in terms of the constructs being measured. In other words, concurrent/convergent validity refers to the convergence or association of constructs rather than measurements. For multiple-item scales, this distinction is more than merely semantic, because the researcher can assess the reliability of multiple-item measurements, which have direct implications on the inferred correlations among the constructs. Because multiple-item scales produce

4 240 Mark Lett (2015) 26: measurements with a known expected error, the observed correlations between these error-prone measurements and any criterion are attenuated by these measurement errors (Nunnaly and Bernstein 1994). Again, the intuition here is also fairly simple. If two constructs A and B have a correlation of r, and both are measured with error, then the correlation between their multiple-item measurements will be attenuated by the fact that these measurements combine the true construct values with random error. And as one would expect, the greater the measurement errors on both scales, the more attenuated the observed correlation will be. Fortunately, the extent of this attenuation can be estimated from the reliabilities of the two scales and properly corrected for a better assessment of the association between the two underlying constructs. This can be done with Spearman s correction for attenuation (Nunnaly and Bernstein 1994): r AB ¼ p r ffiffiffiffiffiffiffi XY s X s Y,wherer AB is the corrected correlation between the two constructs A and B, r XY is the observed correlation between scale X (measuring A) and Y (measuring B), and S X and S Y are their respective alpha reliabilities. 1 Given that the alpha reliabilities of the multiple-item scales reported by B&R ranged between 0.85 and 0.93 (B&R page 180), I assume a reliability of 0.9 in the following discussion. Once Spearman s correction for attenuation is taken into account for the multiple-item measures, the validity, assessed at the construct (rather than measurement) level, leads to different conclusions from those reported by B&R. The correlation between the A Ad and A Brand constructs are clearly larger for the multipleitem scale than for the single-item equivalent, contrary to what B&R concluded. Table 1 replicates portions of Tables 3, 4, and 5 in B&R, using the construct correlations involving the multiple-item measures corrected for attenuation. In all tables and products (with a single exception, jeans in Table 1), the validity correlations based on multiple-item scales are clearly larger than those obtained with the single-item measurement. Obviously, without access to the raw data (or information on the higher moments of the respective variables), it is not possible to test for the statistical significance of the now larger differences, but the consistency of the results across the four products and three tables lends support for the higher concurrent validity of the multipleitem measure, despite the fact that common-methods biases pull these spuriously high correlations together. Obviously, these differences will be even greater if the actual reliabilities for the specific multi-item scales are lower than the assumed value of 0.9. The correction for attenuation is not only relevant in assessing the concurrent validity of the constructs inferred from the measurements but is also particularly relevant in testing H4 H6, because these hypotheses are already (correctly) stated in terms of the constructs (rather than measurements), as the other set of hypotheses should also have been stated. Again, the corrected correlations obtained with the multiple-item scales are clearly higher than those obtained with single items, raising questions about spurious effects in the concurrent correlations. These results highlight an important distinction between concurrent and predictive validity. While the former is based on the association of concurrent measurements, the latter calls for real predictions, or independent measures, at the very least. 1 All concepts discussed here are in the common domain of classic measurement theory, normally taught in the first measurement course in the social sciences. A classic reference on this topic is Nunnaly and Bernstein (1994) Psychometric Theory, New York: McGraw Hill, or any of its newer editions.

5 Mark Lett (2015) 26: Table 1 Concurrent validity correlations (corrected for attenuation) for multiple- and single-item measurement scales Advertised product Painkillers Coffee Pension plan Jeans Predictors of A Brand(L) A Ad3 (multi-item) A Ad(G) (single-item) Predictors of A Brand3 A Ad3 (multi-item) A Ad(G) (single-item) Predictors of A Brand(G) A Ad3 (multi-item) A Ad(G) (single-item) A Brand3 three-item scale measuring attitude towards the advertised brand, A Brand(L) single-item measure of attitude towards the advertised brand, A Ad3 three-item scale measuring attitude towards the advertisement, A Ad(G) single-item measurement of the attitude towards the advertisement To compare the predictive validity of multiple- and single-item measures, I utilize survey and diary data from the ZUMA Consumer Panel, ( based on GfK s ConsumerScan Household Panel. The purchase diary data contains all grocery purchases made by the 3,680 panelists during the year of The survey data contains a battery of 74 Likert-type items covering a broad range of attitude and value orientations, collected at some point before collection of the purchase data. The main purpose of this exercise is not to test any theory regarding the simple constructs being measured but merely to highlight the distinction between predictive validity and concurrent validity and the need to consider measurement error in this type of test. For the purposes of this predictive validity test, I selected attitudinal items measuring the panelists concerns about weight and eating natural, preservative-free foods. A factor analysis of these items, displayed in Table 2 with paraphrased English translations of the original German items, clearly shows two dimensions, the first measuring Table 2 Rotated factor loadings for nine attitudinal items Item Weight Natural Concern about being slim Worry about weight Pay attention to figure Pay attention to calories Pay attention to low fat nutrition Prefer natural products Prefer to buy additive-free food Dislike preservatives in products Will not purchase environmentally unfriendly food

6 242 Mark Lett (2015) 26: the respondents concern about weight loss/control and the second related to eating natural, preservative-free foods. These two factors explain 69 % of the original variance, with a third factor adding 7 %, thereby justifying the two-factor solution and identifying two multiple-item scales: weight, a five-item unweighted scale measuring concern about weight loss/control, and natural, a four-item unweighted scale measuring an orientation towards eating natural foods; both scales are clearly unidimensional. Alpha reliability is assessed as 0.85 for the weight scale and 0.87 for the natural scale. The fact that the 3,680 consumers in this panel also report their consumption of milk during the same year makes it possible to test the predictive validity of multiple-item and single-item versions of the scales for the two constructs, by relating these attitudes to the subsequent purchase of low-fat and organic milk throughout the year. In other words, rather than assessing the concurrent validity of perceptual measures obtained at the same time in the same survey as done by B&R, I assess the predictive validity, by relating the multiple- and single-item measurements to measures of related constructs obtained through other means at a different time, which avoids the common-methods biases inherent in concurrent validation of attitudinal measures, as discussed earlier. Given the nature of the two attitudinal measures, one should expect weight and its single-item equivalents to be correlated to the actual percentage of low-fat milk out of all milk purchases (in Euros) while natural and its single-item equivalents to be correlated to the actual percentage of organic milk purchases. These correlations are reported in Table 3. However, one should take into consideration that predictive validity refers to the correlation between the construct inferred from the measurement scale and the external criterion. In other words, I expect the construct concern about weight loss/control to be correlated to the percentage of low-fat milk purchases and therefore Table 3 Predictive validity pairwise correlations between single- and multiple-item scales and observed purchase behavior Measurement Percentage of lowfat milk Percentage of organic milk Weight scale Multi-item Corrected for attenuation Single items Pay attention to figure Pay attention to low fat nutrition Worry about weight Pay attention to calories Concern about being slim Natural scale Multi-item Corrected for attenuation Single items Dislike preservatives in products Prefer to buy additive-free food Will not purchase environmentally unfriendly food Prefer natural products Sample size=3,680 panelists

7 Mark Lett (2015) 26: must correct the correlations between the measure (weight) and the criterion for attenuation. With only one exception, the observed correlations between the multipleitem measurements and the external criteria are equal or higher than the single-item equivalents (the researcher could have selected that single item for his/her single-item measure, but that would require prescience). Once these correlations are corrected for attenuation, the superiority of the multiple-item scales in terms of predictive validity becomes even more obvious, thereby contradicting the results and conclusions reported by B&R for true predictive validity. In conclusion, validity (as claimed by B&R) or reliability is not the main reason why one would want to use multiple measures, even when measuring concrete objects on concrete attributes. My sole motivation in writing this short note is to remind readers of old but still useful concepts from classic measurement theory that seem to have been forgotten or ignored and to caution authors who feel tempted to cut corners in their measurement scales, citing the B&R article to support it. Repeated measures are used across disciplines because they allow an assessment of measurement error; researchers need replicates to assess measurement error, regardless of the nature of the construct being measured because measurement error cannot be assessed with a single measure. This is not to deny that when properly built and tested, multiple-item scales may also become more reliable and valid. Moreover, it is not clear that B&R tested for the predictive validity of the multiple-/ single-item scales as claimed in the title and content of their article. At most, they provided evidence for concurrent validity using correlations which are unfortunately confounded with spurious correlations from common-methods biases, which do occur even when constructs are measured with a single item, contrary to what B&R imply in their Table 1. Based on the arguments presented earlier and the results from the predictive validity tests on two constructs shown in Table 3, the reader might want to think carefully before ignoring the carpenter s rule: measure twice, cut once! References Bergkvist, L., & Rossiter, J. R. (2007). The predictive validity of multiple-item versus single-item measures of the same constructs. Journal of Marketing Research, 44(May), Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), Nunnaly, J. C., Jr., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: Mc-Graw Hill. Perdue, B. C., & Summers, J. O. (1986). Checking the success of manipulations in marketing experiments. Journal of Marketing Research, 23(4), Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), Simmons, C. J., Bickart, B., & Lynch, J. G., Jr. (1993). Capturing and creating public opinion in survey research. Journal of Consumer Research, 20(September), Zhao, X., Lynch, J. G., & Chen, Q. (2010). Reconsidering Baron and Kenny: myths and truths about mediation analysis. Journal of Consumer Research, 37(2),

Appropriate use of single-item measures is here to stay

DOI 10.1007/s11002-014-9325-y Appropriate use of single-item measures is here to stay Lars Bergkvist # Springer Science+Business Media New York 2014 Abstract In their article, Bergkvist and Rossiter (Journal