Regression Modeling of Reader s Emotions Induced by Font Based Text Signals

Transcription

1 Regression Modeling of Reader s Emotions Induced by Font Based Text Signals Dimitrios Tsonos 1, Georgios Kouroupetroglou 1, and Despina Deligiorgi 2 1 Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece {dtsonos,koupe}@di.uoa.gr 2 Department of Physics, National and Kapodistrian University of Athens, Greece despo@phys.uoa.gr Abstract. In this work we presented a mathematical model for the readers emotional state responses triggered by font style, type and color. It is based on multiple regression analysis of the repeated measures from 45 students and for 35 textual stimuli using the Self-Assessment Manikin test. Based on the dimensional theory of emotions, we propose a model on how emotional dimensions Pleasure, Arousal, and Dominance vary according to the typographic text signals: font style, font type and font/background color combinations. We observe that Pleasure dimension is affected negatively by font type ( Arial and Times New Roman ) and positively by color brightness difference of font/background color combinations. Arousal and Dominance are affected only by color brightness difference (negative correlation). According to the proposed model, font type Arial elicits more pleasant emotional state than Times New Roman. The results can be applied to augment user interface experience or to add expressivity in Text-to-Speech systems and provide accessibility of typography induced text signals. Keywords: document accessibility, text signals, reader s emotions, Text-to-Speech, Self-Assessment Manikin test. 1 Introduction Universal Accessibility of printed and/or electronic documents is an open research topic [1]. Users require documents in alternative formats, depending on her/his interaction preferences - needs and/or device specifications. Text-to-Speech (TtS) is a common software technology that converts in real-time any electronic text into speech [2], that can provide document accessibility through acoustic modality. A document is the medium in which a message (information) is communicated [3]. The term signal is introduced as the writing device that emphasize aspects of a text s content or structure without adding to the content of the text [4]. It attempts to pre-announce or emphasize content and/or reveal content relationship [5-6, 23-26]. The title, heading, typographic cues are considered as signals. Also, input enhancement is C. Stephanidis and M. Antona (Eds.): UAHCI/HCII 2013, Part II, LNCS 8010, pp , Springer-Verlag Berlin Heidelberg 2013

2 Regression Modeling of Reader s Emotions Induced by Font Based Text Signals 435 an operation whereby the saliency of linguistic features is augmented through e.g. textual enhancement for visual input (i.e. bold) and phonological manipulations for aural input (i.e. oral repetition) [7]. Tsonos and Kouroupetroglou [1, 18] categorize the signals, focusing on visual presentation of documents, into three layers: logical, layout and typographic: (i) Logical layer: it associates content with architectural elements such as headings, titles/subtitles, chapters, paragraphs, tables, lists, footnotes, and appendices. (ii) Layout layer: it associates content with architectural elements relating to the arrangement on pages and areas within pages, such as margins, columns, and alignment. (iii) Typography layer: it includes font (type, size, color, background color, etc.) and font style such as bold, italics, underline. The term plain used in this work indicates text of any font, but without font style. These three layers are complementary and not independent. Typography can be applied to both the logical and the layout layers of a document. For example, a footnote (Logical layer) could be in italics or in smaller font size than the body of the text. The vertical space in a text block, called leading (Layout layer), can be affected by the font type. Moreover, typography can be applied to the body of the text directly, for example, a word in bold can be used either for the introduction of a new term or to indicate a person s name. All the text signaling devices, either mentioned as signals or layers: a) share the goal for directing the reader s attention during reading, b) facilitate specific cognitive process occurring during reading, c) ultimate comprehension of text information, d) may influence memory on text and e) direct selective access between and within texts [4]. Current Text-to-Speech (TtS) systems do not include effective provision of the semantics and the cognitive aspects of the visual (such as typography) and non-visual (such as logical structure) text signals [1-2-22]. In order to achieve the automated metadata s rendition in visual, acoustic and/or haptic modality, it is mandatory to mathematically formulate, either semantic or cognitive responses of user during their interaction with the documents. Emotion variations during reading process are cognitive information that fails to be rendered into acoustic modality. A proposed methodology presented in [1] tries to overcome this obstacle proposing the mathematical formulation of reader s emotional state response and convey this information into acoustic and/or haptic modality. Focusing on TtS systems, it is proposed the combination of reader s emotional state response with the mapping rules of existing Expressive Speech Synthesis models e.g. [8]. Thus, we can acoustically map document s typographic alteration into prosodic variations, namely pitch, rate and volume. This paper proposes a mathematical model of readers emotional state response triggered by font style, type and color. It is based on multiple regression analysis of the repeated measures, using the Self-Assessment Manikin test [9]. Based on the dimensional theory of emotions [10], we propose a model on how emotional dimensions (namely Pleasure, Arousal, Dominance) vary according to typographic signals (font style, type and font/background color combinations). We apply multiple

3 436 D. Tsonos, G. Kouroupetroglou, and D. Deligiorgi regression analysis on the derived results (based on the methodology described in [11]). A set of three equations describe the relation between the three emotional dimensions (dependent variables) on typographic signals (independent variables). 2 The Experimental Procedure Following the dimensional approach of emotions [10], Lang [9] introduced the Self- Assessment Manikin test (SAM) to assess the emotional states Pleasure, Arousal, and Dominance of the participants (known also as the PAD test ). Synonyms are used for the expression of the PAD dimensions. Pleasure can be replaced by valence or evaluation, Arousal by activation or activity, Dominance by power or potency. The participants can choose between at least five selections of manikins. There are no verbal expressions to assess their emotional states. For the emotional state of Pleasure the rating spans from a happy (smiling) manikin to an unhappy (frowning) one. For the Arousal dimension the one pole is represented by a highly energetic manikin and the other by a relaxed with eyes-closed one, while for Dominance the controlled and in-control poles are represented by a small and large manikin, respectively (Figure 1). Fig. 1. The manikins of SAM Test as they are displayed during the experimental procedure

4 Regression Modeling of Reader s Emotions Induced by Font Based Text Signals Participants Forty-five students participated in the experiments, ranging from 18 to 36 years old (mean age 24.4 years, SD=4.61). They were 23 male and 22 female undergraduate or postgraduate students and their native language was Greek. All of them have selfreported normal color vision and normal or corrected-to-normal visual acuity. 2.2 Stimuli Thirty five combinations of font type, style and color attributes were investigated using Greek text (a paragraph with approximately 46 words) from which any content; emotion, and/or domain dependent information were excluded. All stimuli were displayed in a random sequence on a 17-inch LCD display with resolution, in full screen mode, using 32 bit color depth. The 35 combinations are (abbreviations explained in Table 1): YU, WB, WU, RG, BG, BW, GY on Times New Roman using Plain, Bold and Bold-Italics YU, WB, WU, RG, BG, BW, GY on Arial using Bold and Bold-Italics Table 1. The text signals and their corresponding attributes used in the experimental procedure Font/Background Color Font Style Font Type Yellow Blue (YU) White on Black (WB) White on Blue (WU) Red on Green (RG) Black on Grey (BG) Black on White (BW) Green on Yellow (GY) Plain Bold Bold-Italics Times New Roman Arial 2.3 The Experimental Procedure At the beginning of the experiment, the participants were asked to read carefully the exact instructions provided by the IAPS guidelines [9] and ask the instructor for possible clarifications. Then, they were asked to fill in the electronic form with their demographic information and a declaration that they agree to take part in the experiment. Afterwards, they participated in a demo version of the experiment to be familiarized with the procedure. Each stimulus was displayed for 10 seconds, and after that, participants were asked to assess their emotional state on a 9-point PAD scale using the manikins provided by the SAM test (Figure 1).

5 438 D. Tsonos, G. Kouroupetroglou, and D. Deligiorgi 3 Multiple Regression Analysis In multiple linear regression, the value of a variable Y can be predicted using multiple predictor variables (X 1, X 2, X 3, ). The mathematical notation is: (1) where a 1, a 2, a 3 a n are the regression coefficients for the respective variable and b is the regression constant. In this study we refer to Y variable as the dependent variable and X n variables as the independent. There are many ways to perform multiple (or single) regression analysis, e.g. least squares, means regression, generalized least squares [12]. In this work we use weighted means regression analysis. The responses are averaged on each stimulus over subjects (along with their respective standard error - SE) and then weighted least squares regression on means was applied. According to [12] and [13] this method: a) provides unbiased estimators, b) is simple and fast and c) it is easy to manipulate the regression approach that is described in the following section. For the analysis we used the statistical software Origin [14]. 3.1 The Variables The dependent variables used in this experimental design are Pleasure, Arousal and Dominance and their values are in the [-1, 1] continuous space. Each variable consist a separate regression analysis. The independent variables are: Font Style Variables. We have selected two dummy variables, namely Plain and Bold. These two variables can be assigned with boolean values (0 or 1). The value 1 denotes the existence of the typographic attribute. E.g. if the text is plain then Plain variable is 1 and Bold variable is 0. Similar it is in the case of bold text. We also want to represent Bold-Italics. To obtain this we assign the value 0 to both Plain and Bold variables. Font Type. This variable is also a dummy one. The value 1 corresponds to Times New Roman font type attribute and value 0 to Arial. Color Brightness Difference (DBrightness). We have selected the color difference between font and background color. The value of the variable can be calculated using equations (2) and (3) [15] by applying the corresponding RGB scale for the selected colors. The variable is continuous ranging [0, 255]. h = h h (2) h =( ) /1000 (3) Note: The independent variables are little or no correlated.

6 Regression Modeling of Reader s Emotions Induced by Font Based Text Signals Predictor Variable Determination The multiple regression analysis can be performed in several ways (e.g. forward or backward) [11]. In the present study we follow the backward approach (full model), namely that at the beginning of the regression analysis we add all the independent variables that we want to observe, and the significance level for each predictor determines which variable is excluded. In detail, the next steps were followed: Determine the regression model Determine R 2 Determine whether the multiple regression is statistically significant Determine the significance of the predictor variables Recalculate the entire solution without the dropped predictor(s). Table 2 presents the multiple regression results including all the independent variables. The R-value is 0.78 and adjusted-r 2 is The overall ANOVA regression is statistical significant (F=11.651, p<0.05). Only Font and DBrightness variables are statistical significant. Table 2. Results of the Multiple Regression analysis on the Pleasure dimension Pleasure Value Standard Error p Intercept Plain Bold Font DBrightness Overall Statistics Overall ANOVA R Value F p Adj. R-Square Regression Recalculation We observe that Plain and Bold variables are statistical insignificant (Table 2), thus they are excluded/dropped during the next step of multiple regression recalculation (Table 3). The final equation is: = h (4) Where: Pleasure [-1, 1], Font (Boolean, 0/1), DBrightness [0, 255]

7 440 D. Tsonos, G. Kouroupetroglou, and D. Deligiorgi Table 3. Results of Multiple Regression Recalculated on the Pleasure dimension Pleasure Value Standard Error p Intercept Font DBrightness Overall Statistics Overall ANOVA R Value F p Adj. R-Square All the independent variables significantly contribute to the model and the overall ANOVA is also significant. The adjusted-r 2 is higher than in the initial calculation denoting that the recalculation process optimized the regression model. We observe that Pleasure dimension is negative correlated with Font variable which means that changing from Arial to Times New Roman the dimension is decreasing. Also, the color brightness difference is positive correlated to Pleasure, thus increasing color brightness difference between font color and background color increases this dimension. Similar to Pleasure dimension the results for Arousal and Dominance are presented in Tables 4 and 5 respectively. All regression ANOVAs are statistical significant and the only independent variable that significantly contributes to both models is color brightness difference. Thus, the results for Arousal and Dominance are summarized in Equations (5) and (6). = h (5) = h (6) Where: Arousal and Dominance [-1, 1], DBrightness [0, 255] Table 4. Results of Multiple Regression Analysis for the Arousal Dimension Arousal Value Standard Error p Intercept DBrightness Overall Statistics Overall ANOVA R Value F p Adj. R-Square

8 Regression Modeling of Reader s Emotions Induced by Font Based Text Signals 441 Table 5. Results of the Multiple Regression Analysis for the Dominance dimension Dominance Value Standard Error p Intercept DBrightness Overall Statistics Overall ANOVA R Value F p Adj. R-Square Conclusions In this study we presented a model of readers emotional state response on typographic signals, based on multiple linear regression. We used Weighted Regression of Means (WRM) as a simple, fast and easy to manipulate approach that provides unbiased estimators [13]. We observe that Pleasure dimension is negative affected by font type ( Arial and Times New Roman ) and positive by color brightness difference of font/background color combinations. Arousal and Dominance are affected only by color brightness difference (negative correlation). According to the proposed model, font type Arial elicits more pleasant emotional state than Times New Roman. The results are consistent with those presented in [16]. Color brightness difference is positive correlated with Pleasure meaning that high color differences elicit more pleasant emotional state (similar to one presented in [17]). The results are also similar to one presented in [18], that also uses SAM Test. The major drawback of the results presented in [18] is that following ANOVA interpretation of the results, we can have only qualitative results and not quantitative. Namely, that we can observe which text signals influence emotions during reading process and not how they affect them. Regression analysis, as it is mentioned above, is a simple, fast and easy to manipulate data that derive from the experimental procedure, and observe how signals affect emotions either in qualitative and/or quantitative approach. Also, participants in the experimental procedure [18] were less than those in present study (30 against 45 students). We observe that the results provide emotional state prediction. There are a few limitations in the present study, such as the small number of text signals combinations and their corresponding values, especially for the font/background color combinations. Of course the main disadvantage of WRM is that it is prone to inflate Type I error. But, it is an appropriate method to obtain estimates of regression coefficients [13]. There are several ways to analyze repeated measurement, towards a quantitative analysis that reduces Type I error such as Generalized Least Squares (GLS). The results of the current study can be incorporated in systems for the automated typography-derived emotional extraction and annotation of printed and/or electronic documents [19] using existing standards such as EmotionML [20], or for the

9 442 D. Tsonos, G. Kouroupetroglou, and D. Deligiorgi multimodal document presentation, focusing on the acoustic modality [1]. The creation and the estimation of such model can be used in the automated extraction of reader s emotional state response [19] either using a quantitative model or using a qualitative descriptive approach. The automated extraction can be utilized to augment user interface experience or to add expressivity in TtS systems and provide accessibility of typography to print disabled users [21]. Acknowledgements. This research has been co-financed by the European Union (European Social Fund ESF) and Greek national funds through the Operational Program "Education and Lifelong Learning" of the National Strategic Reference Framework (NSRF) under the Research Funding Project: THALIS-University of Macedonia- KAIKOS: Audio and Tactile Access to Knowledge for Individuals with Visual Impairments, MIS References 1. Kouroupetroglou, G., Tsonos, D.: Multimodal Accessibility of Documents. In: Pinder, S. (ed.) Advances in Human-Computer Interaction, pp I-Tech Education and Publishing, Vienna (2008) 2. Fellbaum, K., Kouroupetroglou, G.: Principles of Electronic Speech Processing with Applications for People with Disabilities. Technology and Disability 20(2), (2008) 3. McLuhan, M., Fiore, Q.: The Medium is the Message. Gingko Press, Berkeley (2005) 4. Lorch, R.F.: Text-Signaling Devices and Their Effects on Reading and Memory Processes. Educational Psychology Review 1(3), (1989) 5. Spyridakis, J.H.: Signaling effects: A review of the research Part I. Journal of Technical Writing and Communication 19(3), (1989) 6. Lemarié, J., Eyrolle, H., Cellier, J.M.: Visual signals in text comprehension: How to restore them when oralizing a text via a speech synthesis? Computers in Human Behavior 22(6), (2006) 7. Han, Z.H., Park, E.S., Combs, C.: Textual Enhancement of Input: Issues and Possibilities. Applied Linguistics 29(4), (2008) 8. Schröder, M.: Expressing degree of activation in synthetic speech. IEEE Trans. on Audio, Speech and Language Processing 14(4), (2006) 9. Lang, P.J., Bradley, M., Culthbert, B.: International affective picture system (IAPS): instruction manual and affective ratings. Technical Report A-6, The Center for Research in Psychophysiology, University of Florida, USA (2005) 10. Mehrabian, A.: Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament. Current Psychology 14(4), (1996) 11. Hinkle, D.E., Wiersma, W., Jurs, S.G.: Applied Statistics for the Behavioral Sciences, 5th edn. Wadsworth Publishing (2002) 12. Holsclaw, T.N.: Investigation of repeated measures linear regression methodologies. Master Thesis, Faculty of the Department of Mathematics, San Jose State University (2007) 13. Lorch Jr., R.F., Myers, J.L.: Regression analyses of repeated measures data in cognitive research. J. Exp. Psychol. Learn. Mem. Cogn. 16(1), (1990) 14. OriginLab,