Different styles of modeling

Different styles of modeling Marieke Timmerman m.e.timmerman@rug.nl 19 February 2015 Different styles of modeling (19/02/2015) What is psychometrics? 1/40

Overview 1 Breiman (2001). Statistical modeling: the two cultures Summary 2 Different cases - different cultures? 1. Causal inference 2. Suppporting a theory 3. C & W s criteria for stochastic models 4. Prediction - exploratory assessing relevant predictors 5. Accurate prediction from predictor set 6. Distributional modeling, e.g., growth curves 3 Exemplary cases Randomized clinical trial Measuring developmental level Identifying personality traits using the lexical approach Different styles of modeling (19/02/2015) What is psychometrics? 2/40

1. Statistical modeling: the two cultures Breiman, L. (2001). Statistical modeling: the two cultures. Statistical Science, 16, 199-231. Goals in data analysis Prediction: What will responses be, given certain observed variables (e.g., input variables)? Information: Extract knowledge on the nature of the relationships (e.g., of input and response variables) Two cultures Data modeling Algorithmic modeling Different styles of modeling (19/02/2015) What is psychometrics? 3/40

Data Modeling Stochastic data model Examples response variables = f (observed input variables, parameters, noise) response variables = f (latent variables, parameters, noise) Model validation Breiman: yes-no goodness-of-fit tests, residual examination Different styles of modeling (19/02/2015) What is psychometrics? 4/40

Algorithmic Modeling core goal: predict outcome measurement from set of features core focus on... properties of algorithms, rather than data modeling prediction accuracy Different styles of modeling (19/02/2015) What is psychometrics? 5/40

Algorithmic Modeling Prediction accuracy check: Typically via some form of cross-validation from: http://cse3521.artifice.cc/classification-evaluation.html Different styles of modeling (19/02/2015) What is psychometrics? 6/40

Problems in Data Modeling focus on the model, rather than the question / sample lack-of-fit often not detected via goodness-of-fit tests / residual examination good fit does not imply that model represents reality Different styles of modeling (19/02/2015) What is psychometrics? 7/40

Lessons from Breiman Rashomon: Multiplicity of models: different models, with different interpretations, with about equal fit for a single data set In data modeling, e.g., regression Also in algorithmic modeling reduce problem, i.e., improve prediction accuracy, by aggregating over large number of competing models Occam: Conflict between simplicity and accuracy Bellman: Dimensionality - curse or blessing? Different styles of modeling (19/02/2015) What is psychometrics? 8/40

2. Different cases - different cultures? 1. Causal inference 2. Supporting a theory 3. Cox & Wermuth s criteria for stochastic models 4. Prediction - exploratorily assessing relevant predictors 5. Accurate prediction from predictor set 6. Distributional modeling (e.g., growth curves) Different styles of modeling (19/02/2015) What is psychometrics? 9/40

1. Causal inference Experimental design, randomized clinical trial Potential outcome model (Rubin, 1974) requirements (Sagarin et al., 2014): effect of a cause is always relative to another cause each observation unit (e.g., patient) is potentially exposable to any one of the causes a cause must temporally precede its effect viewed in terms of prediction: What would happen if a future patient would receive treatment A, idem treatment B? Sagarin, B.J., West, S.G., Ratnikov, A. Homan, W.K., Ritchie, T.D., Hansen, E.J. Treatment noncompliance in randomized experiments: Statistical approaches and design issues. Psychological Methods, Vol 19(3), Sep 2014, 317-333. http://dx.doi.org/10.1037/met0000013 Different styles of modeling (19/02/2015) What is psychometrics? 10/40

2. Suppporting a theory To test a theory with free parameters is to... (Roberts & Pashler, 2000): determine how the theory constrains possible outcomes (i.e., what it predicts) assess how firmly actual outcomes agree with those constraints determine if plausible alternative outcomes would have been inconsistent with the theory, allowing for the variability of the data Roberts, S., & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review, 107, 358-367. Different styles of modeling (19/02/2015) What is psychometrics? 11/40

3. Cox & Wermuth s Criteria for [Stochastic] models (1996, section 1.8) Link with underlying substantive knowledge Link with previously published work, for comparison Some indication or pointer towards a process that might have generated the data Model must be rich enough to capture central features of interest (i.e., include key parameters, e.g., treatment effect)... Model should be consistent with data under analysis Cox, D.R. & Wermuth, N. (1996). Multivariate Dependencies - Models, Analysis and Interpretation. London: Chapman & Hall. Different styles of modeling (19/02/2015) What is psychometrics? 12/40

4. Prediction - exploratory assessing relevant predictors Aim: to accurately predict treatment success treatment-subgroup interaction, subgroup-treatment effect interaction, or treatment-covariate interaction Question: How to identify good and useful predictors? Different styles of modeling (19/02/2015) What is psychometrics? 13/40

4. Prediction - Qualitative interaction trees to identify qualitative treatment-subgroup interactions Dusseldorp, E., & Van Mechelen, I. (2014). Qualitative interaction trees: a tool to identify qualitative treatment-subgroup interactions. Statistics in medicine, 33(2), 219-237. Different styles of modeling (19/02/2015) What is psychometrics? 14/40

4. Prediction - Qualitative interaction trees Case: large number of moderators, absence of clear a priori hypotheses Problems in identifying moderators: multiplicity and spurious interactions that cannot be replicated in follow-up studies QUINT (Dusseldorp & Van Mechelen, 2014): 1 Check on presence of qualitative interaction 2 Identify combinations of dichotomized moderators that are most important for qualitative treatment-subgroup interactions 3 Result: binary tree, partitioning the total sample in three groups: Patients for whom... treatment A is better than treatment B... treatment B is better than A... it does not make any difference Different styles of modeling (19/02/2015) What is psychometrics? 15/40

4. QUINT Start: group of N patients randomly assigned to one of two treatments A and B Before treatment: set of baseline variables, i.e., categorical and/or continuous background characteristics (e.g., severity of disease), and possibly outcome variable After treatment: one primary continuous outcome variable Outcome variable before and after treatment: use change score, or slope of response over time, or time to an event can be used as outcome for QUINT Different styles of modeling (19/02/2015) What is psychometrics? 16/40

4. QUINT - conditions Aim: find the best partition all patients using the baseline variables into two or three mutually exclusive and exhaustive subgroups (i.e., partition classes) p 1 : A B; p 2 : A B; p 3 : A B Conditions 1 Difference in treatment outcome component: In p 1 and p 2, the difference in outcome between treatments A and B should be as large as possible 2 Cardinality component: p 1 and p 2 should comprise as many patients as possible Partitioning criterion ensures maximization of the conjunction of the two components Different styles of modeling (19/02/2015) What is psychometrics? 17/40

4. Stepwise binary splitting procedure At each step: Maximize partitioning criterion After each split: all leaves re-assigned afresh to subgroups (i.e., nonrecursive procedure) Different styles of modeling (19/02/2015) What is psychometrics? 18/40

4. Sequential partitioning algorithm Algorithm: 1 Stepwise procedure to optimize criterion C 2 Stopping criterion: when C is maximized, and 4 boundary conditions are met 3 Pruning: reduce tree resulting after 2., to ensure a well-fitting tree for future data as well (using bootstrap procedure) Different styles of modeling (19/02/2015) What is psychometrics? 19/40

4. Quint in action - Improvement in depression Background characteristics and Outcome Different styles of modeling (19/02/2015) What is psychometrics? 20/40

4. Quint in action - Improvement in depression Final Tree Different styles of modeling (19/02/2015) What is psychometrics? 21/40

5. Accurate Prediction from set of predictors Aim to accurately predict. Dot. Relationships predictors - criterion is a black box Example: Movieweb - individual tailored suggestion for other movies of interest Different styles of modeling (19/02/2015) What is psychometrics? 22/40

6. Models for the Distribution of criterion scores, as a function of predictors Growth curve (e.g., length), as a function of age and gender Norming (i.e., test scores), as a function of age from: http://cse3521.artifice.cc/classification-evaluation.html Different styles of modeling (19/02/2015) What is psychometrics? 23/40

Example of norming Van Wiechen scheme: each Dutch child is assessed during 8 visits at Child Health Care Center Developmental scores (D-score) as a function of age from: Jacobusse G.W., Buuren S. van, Verkerk P.H. (2006). An interval scale for development of children aged 0-2 years. Statistics in Medicine, 25(13), 2272-2283. Different styles of modeling (19/02/2015) What is psychometrics? 24/40

3. Exemplary cases Randomized clinical trial (RCT): Effectiveness of treatments to panic disorder (Van Apeldoorn et al., 2010) Treatments: Cognitive behavioral therapy (CBT), medication (SSRI), or both (CBT+SSRI) Research questions: Which treatment is most effective, in the short-term and long-term? Are there any differential effects ( what works for whom )? Different styles of modeling (19/02/2015) What is psychometrics? 25/40

RCT - Design pre-test 9 months treatment 18 sessions CBT 9 months treatment 9 sessions SSRI 9 months treatment 18 sessions CBT + 9 sessions SSRI 3 months treatment 3 sessions CBT post-test I 3 months treatment 3 sessions SSRI (taper-off) 3 months treatment 3 sessions CBT + 3 sessions SSRI post-test II 6 months after post-test II: follow-up I 12 months after post-test II: follow-up II C_7A 9 Different styles of modeling (19/02/2015) What is psychometrics? 26/40

RCT - Patient flow Different styles of modeling (19/02/2015) What is psychometrics? 27/40

RCT - Panic disorder treatment Outcome measures: Indicators of Anxiety, Depression, Quality of life Predictors treatment, time duration of ilness, level of agoraphobia (no,..., severe), benzodiazapine use (yes, no) patient type (completer, no-taper, drop-out) Different styles of modeling (19/02/2015) What is psychometrics? 28/40

RCT - Model based expected scores Different styles of modeling (19/02/2015) What is psychometrics? 29/40

RCT - Questions Which approach appears to be most reasonable: Data modeling or algorithmic modeling? Or somewhere in between... Causal inference Supporting a theory C & W s criteria for stochastic models Prediction - exploratorily assessing relevant predictors Accurate prediction from predictor set Distributional modeling (e.g., growth curves) The authors used a random coefficient model, using a data modeling approach. Limitations? Alternative ideas for modeling? Different styles of modeling (19/02/2015) What is psychometrics? 30/40

Measuring developmental level Van Wiechen scheme: each Dutch child is assessed during 8 visits at the Child Health Care Center Examples of items: from: Jacobusse G.W., Buuren S. van, Verkerk P.H. (2006). An interval scale for development of children aged 0-2 years. Statistics in Medicine, 25(13), 2272-2283. Different styles of modeling (19/02/2015) What is psychometrics? 31/40

Rasch model P(X ijt θ it, δ j ) = exp(θ it δ j ) (1+exp(θ it δ j )), with θ it the developmental level of child i at age t and δ j the difficulty of item j Empirical pass rates as a function of age from: Jacobusse G.W., Buuren S. van, Verkerk P.H. (2006). An interval scale for development of children aged 0-2 years. Statistics in Medicine, 25(13), 2272-2283. Different styles of modeling (19/02/2015) What is psychometrics? 32/40

Fit P(X ijt θ it, δ j ) = exp(θ it δ j ) (1+exp(θ it δ j )), with θ it the developmental level of child i at age t and δ j the difficulty of item j Empirical pass rates as a function of D-score from: Jacobusse G.W., Buuren S. van, Verkerk P.H. (2006). An interval scale for development of children aged 0-2 years. Statistics in Medicine, 25(13), 2272-2283. Different styles of modeling (19/02/2015) What is psychometrics? 33/40

Measuring developmental level - Questions Which approach appears to be most reasonable: Data modeling or algorithmic modeling? Or somewhere in between... Causal inference Supporting a theory C & W s criteria for stochastic models Prediction - exploratorily assessing relevant predictors Accurate prediction from predictor set Distributional modeling (e.g., growth curves) The authors used a random coefficient model, using a data modeling approach. Limitations? Alternative ideas for modeling? Different styles of modeling (19/02/2015) What is psychometrics? 34/40

Identifying personality traits using the lexical approach Psycholexical research: use trait taxonomies Around 30 trait taxonomies are known, in different languages, e.g., Albanian, Arabic, English, Indian, Japanese Taxonomies: from a dictionary, all trait-descriptive words (i.e., adjectives) are selected; leads to large number of adjectives (range 300-600) Typical approach: perform principal component analysis on sample ratings Outcome: trait dimensions, e.g., extraversion, agreeableness Key question: which traits are pancultural? Different styles of modeling (19/02/2015) What is psychometrics? 35/40

The search for a Pancultural trait structure Included ratings from 11 taxonomies in different languages 1. translate all non-english adjectives into English 2. identify adjectives with same meaning - yielded 1993 unique trait variables, of which 1071 occurred only in a single language retained only 922 trait variables that occurred in minimally two languages only 2 trait variables in all 11 languages (impulsive and sentimental), and another 13 trait variables in 10 languages (e.g., ambitious, conscientious, consistent, creative, emotional, generous, industrious...) Data: 11 languages (sample size range 369-991), total 7104 participants, on 922 trait variables, with structural missing data (trait variable not present in taxonomy) 3. Perform Simultaneous Component Analysis on the data Different styles of modeling (19/02/2015) What is psychometrics? 36/40

Simultaneous Components - across 11 countries A, agreeableness; C, conscientiousness; Dyn, dynamism;... from: De Raad, B. et al. (2014). Towards a Pan-cultural Personality Structure: Input from 11 Psycholexical Studies. European Journal of Personality, 28(5), 497-510. doi:10.1002/per.1953 Different styles of modeling (19/02/2015) What is psychometrics? 37/40

Fit - variance accounted for (VAF ) per country, for SCA and PCA Different styles of modeling (19/02/2015) What is psychometrics? 38/40

Fit - Ratio (VAF SCA /VAF PCA ) and Difference (VAF SCA VAF PCA ) Different styles of modeling (19/02/2015) What is psychometrics? 39/40

The search for a Pancultural trait structure - Questions Which approach appears to be most reasonable: Data modeling or algorithmic modeling? Or somewhere in between... Causal inference Supporting a theory C & W s criteria for stochastic models Prediction - exploratorily assessing relevant predictors Accurate prediction from predictor set Distributional modeling (e.g., growth curves) The authors used a random coefficient model, using a data modeling approach. Limitations? Alternative ideas for modeling? Different styles of modeling (19/02/2015) What is psychometrics? 40/40