The MHSIP: A Tale of Three Centers
|
|
- Scott Blair
- 5 years ago
- Views:
Transcription
1 The MHSIP: A Tale of Three Centers P. Antonio Olmos-Gallo, Ph.D. Kathryn DeRoche, M.A. Mental Health Center of Denver Richard Swanson, Ph.D., J.D. Aurora Research Institute John Mahalik, Ph.D., M.P.A. Jefferson Center for Mental Health Presented at the Organization for Program Evaluation in Colorado Annual Meeting, May 15,
2 Presentation Overview Accountability in mental health Description and intended use of the MHSIP Review of constructs of measurement Purpose and Methods Results of the psychometric investigation Reliabilities Measurement invariance Differential item functioning Discussion of results Future directions for accountability in mental health 2
3 Accountability in Mental health 3
4 Accountability in Mental Health More and more prevalent in the MH field This is a good thing, if it helps centers to improve their services But for accountability to be used to improve quality, the approach needs to meet two criteria: 1. The feedback provided to the centers must be useful and, 2. The yardstick must be the same for all centers. 4
5 How does accountability work in MH? Accountability has changed from Formative- to more Summative-oriented Grant funding (Federal, Private) requires that outcomes be demonstrated (NOMS, GPRA) State-based requirements (CCAR, MHSIP, YSSF) Stakeholders are more in-tune with accountability
6 Description and Intended Uses of the MHSIP What is the MHSIP? What is it used for? 6
7 What is the MHSIP? Mental Health Statistics Improvement Program (MHSIP) Consumer survey Annual administration to a stratified random sample of consumers Colorado Version: 28 items designed to measure 5 constructs Annual administration of the MHSIP is a result of President s New Freedom Commission on Mental Health (2003) focusing on consumer informed performance measurement 7
8 What is the MHSIP Used For? Designed to assess the performance of mental health services inside and across individual centers Mental health centers are compared according to their MHSIP scores, which assess the center s performance for the current year MHSIP results are reported, and can be viewed by all mental health centers and their stakeholders 8
9 Domains of the MHSIP Overall Performance of Mental Health Centers Access (4 items) Quality/ Appropriateness (8 items) Participation in Service/ Treatment Planning (2 items) Consumer Perception of Outcomes (7 items) General Satisfaction (3 items) Note: Participation is not considered a domain for many centers at the national or state level, but it is used in Colorado, therefore it was included in the analysis 9
10 How can Mental Health Centers Use the MHSIP results? Excerpt from MHSIP Consumer Survey Technical Report 2007 (DMH) This information [MHSIP numerical scores] can be used to inform future change within individual centers and can provide a catalyst for more indepth study of particular domains at the center level. (page 1) In 2006, due to low scores, we conducted a study to see how we could improve Participation in services The results did not match what the MHSIP found, nor it provided any explanation for why our scores were low 10
11 So we pondered, how can this happen? We were wondering how is the MHSIP measuring centers, Is it possible that measurement artifacts may be influencing the quantitative results? The remainder of the presentation describes our investigation into the psychometrics of the MHSIP for its intended use 11
12 Measurement Constructs 12
13 Psychometrics Properties The focus of the analysis is based around two questions: 1) What do scores from the MHSIP mean? and 2) How should we interpret the scores produced from the MSHIP to assist us in quality improvement? To investigate the meaning of a numeric score, we need to review the psychometric properties of a survey: Reliability- Does the survey produce the same score for consumers with the same true opinions about their mental health center (as defined by the MHSIP)? Validity- Does the survey measure performance of a mental health center as it is intended to do? Or does it measure some other trait, as well as performance? The underlying premise of psychometrics is to examine how a numeric score from the MHSIP survey captures the true opinions about consumers satisfaction. 13
14 Reliability of the MHSIP Reliability estimates contain two critical components, including the true score or true opinions of performance and error or anything beside the true score, in measuring performance. (According to Classical Test Theory) True score of consumer satisfaction Error (anything beside the true score) Numeric Score produced by the MHSIP Where the reliability is the ratio of true scores to error 14
15 What are we comparing? To be able to compare centers regarding their MHSIP scores, all centers should have similar reliabilities (or a similar percentage of error) This means that each center in the analysis should have similar ratios of true score vs. error in their MHSIP scores (i.e. invariance across centers) Mental Health Center A: True score (80%) Error (20%) Numeric Score Reliability.80 Mental Health Center B: True score (50%) Error (50%) Numeric Score Reliability.50 In this example, the numeric Scores from Center A cannot be compared to Center B, because the two scores have different meanings 15
16 Rasch Modeling Perspective In terms of Rasch modeling (type of IRT model) theory, the numeric score also consists of item difficulty, or how hard or easy it is to agree with an item. For example, Q16 was one of the more difficult-to-endorse items Q16: Staff respected my wishes about who is, and is not to be given information about my treatment True score of consumer satisfaction Error (anything beside the true score) Item Difficulty (how hard or easy it is to agree with the item) Numeric Score produced by the MHSIP 16 16
17 Purpose and Methods Participants, Procedures, and Data Analysis 17
18 Purpose of the Investigation Are the MHSIP numeric scores measuring the same construct across centers? We plan to accomplish this through three different statistical measurement techniques 1. Do the reliabilities among sub-domains of the MHSIP vary across mental health centers? (CTT) 2. Is the structure, and error, of the MHSIP the same across centers (invariance testing- structural equation modeling)? (CTT) True score Error Numeric Score Reliability 3. Are the characteristics of the items similar across mental health centers? Do consumers interpret the items in the same manner? (Rasch- Differential Item Functioning-DIFF) True score of consumer satisfaction Error (anything beside the true score) Item Difficulty (how hard or easy it is to agree with the item) Numeric Score produced by the MHSIP 18
19 Participants MHSIP surveys collected during the State Fiscal Year 2006 (July 1, 2005-June 30, 2006) Three mental health centers from the State of Colorado (in alphabetical order: Aurora, Jefferson, MHCD)* Center 1 n=137 Center 2 n= 101 Center 3 n= 148 * For this presentation, centers will not be identified 19
20 Procedures The MHSIP is administered annually to a stratified (Medicaid/non-Medicaid) random sample. Consumers were sampled from an unduplicated file of FY Colorado Client Assessment Record (CCAR) Narrowed to those who had a recorded encounter with the mental health system in the latter half of FY
21 Psychometric Examination of the MHSIP Reliability, Measurement Invariance, and Differential item Functioning 21
22 Comparing Subscales Initially, we analyzed the reliability estimates of the five MHSIP subscales within the three centers during 2007 Reliability estimates range from 0 to 1, with 1 representing no error or perfect measurement of centers performance, and 0 representing all error Estimates of 0.70 or higher considered acceptable. 22
23 Reliability Estimates in 2007 among Subscales and Centers Access Quality Outcomes Satisfaction Participation Center 1 = 0.77 Center 2 = 0.65 Center 3 = 0.73 (Q4 0.75) Center 1 = 0.84 Center 2 = 0.80 Center 3 = 0.80 (Q ) Center 1 = 0.85 (Q ) Center 2 = 0.85 (Q ) Center 3 = 0.75 (Q ) Center 1 = 0.84 Center 2 = 0.79 Center 3 = 0.88 Center 1 = 0.47 Center 2 = 0.50 Center 3 = 0.28 *Note: A Q# suggested that an alpha-if-item-delete value higher than the actually reliability value, suggesting that deletion of that question could increase the reliability of the scale 23
24 Reliability Summary All subscales produced acceptable reliability, expect for participation Only contains 2 items (reliability increases as the number of items in scale increase) We cannot infer meaning from the scores for the participation domain In the Outcomes domain, reliability estimates would have increased among all centers with the removal of Q26 ( I do better in school and/or work ) All other items deal with concepts associated with mental health treatment (i.e., decreasing symptom interference, relationships, control of their life ) Notice that many consumers have good outcomes without participating in school or work (resiliency factor) 24
25 Invariance Testing Across Centers 25
26 Confirmatory Factor Analysis A model with all five domains could not be fit Some of the parameters could not be estimated (Variance-Covariance matrix may not be identified) Exploratory analyses using only Outcomes and Participation showed that Outcomes was the major culprit
27 Outcomes/Participation X 2 (26) = RMSEA = 0.13 GFI = 0.79 CFI = 0.85
28 Invariance with 3 domains We tested invariance on three domains only: Satisfaction, Access and Quality We ran separate models for every center to have an idea up-front of their similarities/differences Trouble can be expected based on the fit Center 2 had the worst fit, Center 3 had a notso-bad fit; Center 1 was in between the other two centers
29 Center 1 Standardized scores n = 137 X 2 (87) = RMSEA = 0.11 GFI = 0.82 CFI = 0.87
30 Center 2 Standardized Scores n = 101 X 2 (87) = RMSEA = 0.15 GFI = 0.73 CFI = 0.68
31 Center 3 Standardized scores n = 148 X 2 (87) = RMSEA = 0.09 GFI = 0.85 CFI = 0.90
32 Measurement Invariance Whether or not, we can assert that we measured the same attribute under different conditions If there is evidence of variability, any findings reporting differences between individuals and groups cannot be interpreted Differences in average scores can be just as easily interpreted as indicating that different things were measured Correlations between variables will be for different attributes for different groups
33 Factorial Invariance One way to test measurement invariance is FACTORIAL INVARIANCE The main question it addresses: Do the items making a particular measuring instrument work the same across different populations (e.g., Males and Females)? The measurement model is group-invariant Tests for Factorial Invariance (in order of difficulty):
34 Steps in Factor Invariance testing 1 Equivalent Factor structure Same number of factors, items associated with the same factors (Structural model invariance) 2 Equivalent Factor loading paths Factor loadings are identical for every item and every factor
35 Steps in Factor Invariance testing (cont) 3 Equivalent Factor variance/ covariance Variances and Covariances (correlations) among factors are the same across populations 4 Equivalent Item reliabilities Residuals for every item are the same across populations
36 Results Factorial Invariance Model χ 2 df χ 2 df RMSEA GFI CFI 1) Number of factors invariant ) Model (1) with pattern of factor loadings held invariant (Lambda-X Invariant) 3) Model (2) with factor variances and covariances held invariant (PHI- Invariant) 4) Model (3) with factor invariance of item-pair reliabilities (Theta-Delta- Invariant) * # Not run * p < 0.05 # p < 0.01
37 Conclusions Factorial Invariance The model does not provide a good fit for the different centers Most of the discrepancy is centered on loadings and how the domains interact with each other (variance-covariance) Since the model is incremental, (later tests are more challenging than early ones), we did not run equivalent item reliabilities (the most stringent test)
38 Differential Item Functioning (DIF) 38
39 Differential Item Functioning Rasch analysis separates the item characteristics from participants scores It assumes that some items can be more difficulty to agree with than others. DIFF- examines and tests (statistically) whether the item characteristics (difficulty scores) vary across centers Since difficulty is an item characteristic, if difficulty scores vary among mental health centers, then it can be assumed that the items measure the centers differently (opposed to actually being a true difference in their scores) 39
40 0.6 Access DIF Plot Difficulty Scores Q4 Q5 Q6 Q7 ITEM 40
41
42 42
43
44 1 General Satisfaction Difficulty Q1 Q2 Q3 ITEM 44
45 Summary of DIFF Analysis The Quality/Appropriateness, Participation, and General Satisfaction subscales measure equally across mental health centers In the Access and Outcomes subscales, there are 6 questions that produced significant DIFF s meaning that characteristics of the measurement changes across centers Q4- The locations of service was convenient Q6- Staff returned my phone calls within 24 hours Q22- I am better able to control my life Q23- I am better able to deal with crisis Q24- I am getting along better with my family Q26- I do better in school and/or work Regarding these 6 questions, variations across centers may be due to differences in measurement, as opposed to true differences in consumer satisfaction 45
46 Discussion 46
47 What did we learn about the MHSIP? Some items and subscales (domains) do not seem to measure equally across centers Therefore comparing centers using these items/domains may not reflect true differences in performance It is more likely that they reflect differences in measurement (including error, difficulty, reliability) 47
48 Some domains are reliable, some are not Satisfaction was Ok from all 3 perspectives Quality had some good characteristics, but some items were bad Participation is not very reliable (only two items; but the items were good) Outcomes is overall, a real bad domain (bad items, lots of cross-loading, correlated errors) Employment/education may not be a desired outcome for all consumers
49 Discussion Despite the fact that the samples may not be appropriate (biases, sampling frameworks that can be improved), the data at hand suggests that there are some intrinsic problems with the MHSIP But the analyses also suggest some very specific ways to improve it 49
50 Suggestions Revise the Outcomes Scale (differentiate between recovery/resiliency) Add items to participation scale Some items in Access need to be reviewed (Q4 and Q6) How do we deal with all these cross-loading factors? Is it one domain (satisfaction) that we artificially broke into many domains (outcomes, access, )? How does the factor structure for the entire sample (EFA included in the annual report) holds up for individual centers? More research is needed in this area
51 More suggestions Sampling Suggestions: Attempt to Stratify the sample by Consumer s needs level At MHCD, we have developed a measure of consumer s recovery needs level (RNL) Equating Suggestions: Use some form of equating procedures to equate scores across centers Using Item Response Theory techniques: IRT could help learn more about how the MHSIP measures satisfaction/performance within/among mental health centers
52 More suggestions Mixed Method Design: Conducting focus groups at each center would provide a cross-validation to quantitative measurement This would also enhance the utilization of the results for quality improvement Include in the annual reports the psychometrics (reliability) for every center Helps to know how much confidence we should have in the scores
53 Questions??? Contact Information: Antonio Olmos, Kate DeRoche, Richard Swanson, John Mahalik, 53
54 χ2 (Chi-Square): in this context, it tests the closeness of fit between the unrestricted sample covariance matrix and the restricted (model) covariance matrix. Very sensitive to sample size: The statistic will be significant when the model fits approximately in the population and the sample size is large. RMSEA (Root Mean Square Error of Approximation): Analyzes the discrepancies between observed and implied covariance matrices. Lower bound of zero indicates perfect fit with values increasing as the fit deteriorates. Suggested that values below 0.1 indicate a good fit to the data, and values below 0.05 indicate a very good fit. It is recommended not to use models with RMSEA values larger than 0.1 GFI (Goodness of Fit Index): Analogous to R2 in that it indicates the proportion of variance explained by the model. Oscillates between 0 and 1 with values exceeding 0.9 indicating a good fit to the data. CFI (Comparative Fit Index): Indicates the proportion of improvement of the overall fit compared to a null (independent) model. Sample size independent, and penalizes for model complexity. It uses a 0-1 norm, with 1 indicating perfect fit. Values of about 0.9 or higher reflect a good fit
Instrument equivalence across ethnic groups. Antonio Olmos (MHCD) Susan R. Hutchinson (UNC)
Instrument equivalence across ethnic groups Antonio Olmos (MHCD) Susan R. Hutchinson (UNC) Overview Instrument Equivalence Measurement Invariance Invariance in Reliability Scores Factorial Invariance Item
More informationThe Psychometric Development Process of Recovery Measures and Markers: Classical Test Theory and Item Response Theory
The Psychometric Development Process of Recovery Measures and Markers: Classical Test Theory and Item Response Theory Kate DeRoche, M.A. Mental Health Center of Denver Antonio Olmos, Ph.D. Mental Health
More informationDoing Quantitative Research 26E02900, 6 ECTS Lecture 6: Structural Equations Modeling. Olli-Pekka Kauppila Daria Kautto
Doing Quantitative Research 26E02900, 6 ECTS Lecture 6: Structural Equations Modeling Olli-Pekka Kauppila Daria Kautto Session VI, September 20 2017 Learning objectives 1. Get familiar with the basic idea
More informationTechniques for Explaining Item Response Theory to Stakeholder
Techniques for Explaining Item Response Theory to Stakeholder Kate DeRoche Antonio Olmos C.J. Mckinney Mental Health Center of Denver Presented on March 23, 2007 at the Eastern Evaluation Research Society
More informationConfirmatory Factor Analysis of Preschool Child Behavior Checklist (CBCL) (1.5 5 yrs.) among Canadian children
Confirmatory Factor Analysis of Preschool Child Behavior Checklist (CBCL) (1.5 5 yrs.) among Canadian children Dr. KAMALPREET RAKHRA MD MPH PhD(Candidate) No conflict of interest Child Behavioural Check
More informationMultilevel Techniques for Quality Control Charts of Recovery Outcomes
Multilevel Techniques for Quality Control Charts of Recovery Outcomes INFORMS Annual Meeting 2009 San Diego, CA October, 11 th, 2009 Linda Laganga, PhD* (Linda.Laganga@mhcd.org) CJ McKinney, MA Kate DeRoche,
More informationDATE: 8/ 1/2008 TIME: 3:32. L I S R E L 8.71 BY Karl G. J reskog & Dag S rbom
DATE: 8/ 1/2008 TIME: 3:32 L I S R E L 8.71 BY Karl G. J reskog & Dag S rbom This program is published exclusively by Scientific Software International, Inc. 7383 N. Lincoln Avenue, Suite 100 Lincolnwood,
More informationBasic concepts and principles of classical test theory
Basic concepts and principles of classical test theory Jan-Eric Gustafsson What is measurement? Assignment of numbers to aspects of individuals according to some rule. The aspect which is measured must
More informationPersonality Traits Effects on Job Satisfaction: The Role of Goal Commitment
Marshall University Marshall Digital Scholar Management Faculty Research Management, Marketing and MIS Fall 11-14-2009 Personality Traits Effects on Job Satisfaction: The Role of Goal Commitment Wai Kwan
More informationExamining the efficacy of the Theory of Planned Behavior (TPB) to understand pre-service teachers intention to use technology*
Examining the efficacy of the Theory of Planned Behavior (TPB) to understand pre-service teachers intention to use technology* Timothy Teo & Chwee Beng Lee Nanyang Technology University Singapore This
More informationDevelopment of self efficacy and attitude toward analytic geometry scale (SAAG-S)
Available online at www.sciencedirect.com Procedia - Social and Behavioral Sciences 55 ( 2012 ) 20 27 INTERNATIONAL CONFERENCE ON NEW HORIZONS IN EDUCATION INTE2012 Development of self efficacy and attitude
More informationMultifactor Confirmatory Factor Analysis
Multifactor Confirmatory Factor Analysis Latent Trait Measurement and Structural Equation Models Lecture #9 March 13, 2013 PSYC 948: Lecture #9 Today s Class Confirmatory Factor Analysis with more than
More informationRunning head: CFA OF TDI AND STICSA 1. p Factor or Negative Emotionality? Joint CFA of Internalizing Symptomology
Running head: CFA OF TDI AND STICSA 1 p Factor or Negative Emotionality? Joint CFA of Internalizing Symptomology Caspi et al. (2014) reported that CFA results supported a general psychopathology factor,
More informationThe following lines were read from file C:\Documents and Settings\User\Desktop\Example LISREL-CFA\cfacommand.LS8:
DATE: 8/ 1/2008 TIME: 2:34 L I S R E L 8.52 BY Karl G. J reskog & Dag S rbom This program is published exclusively by Scientific Software International, Inc. 7383 N. Lincoln Avenue, Suite 100 Lincolnwood,
More informationApplications of Structural Equation Modeling (SEM) in Humanities and Science Researches
Applications of Structural Equation Modeling (SEM) in Humanities and Science Researches Dr. Ayed Al Muala Department of Marketing, Applied Science University aied_muala@yahoo.com Dr. Mamdouh AL Ziadat
More informationStructural Validation of the 3 X 2 Achievement Goal Model
50 Educational Measurement and Evaluation Review (2012), Vol. 3, 50-59 2012 Philippine Educational Measurement and Evaluation Association Structural Validation of the 3 X 2 Achievement Goal Model Adonis
More informationChapter 9. Youth Counseling Impact Scale (YCIS)
Chapter 9 Youth Counseling Impact Scale (YCIS) Background Purpose The Youth Counseling Impact Scale (YCIS) is a measure of perceived effectiveness of a specific counseling session. In general, measures
More informationContents. What is item analysis in general? Psy 427 Cal State Northridge Andrew Ainsworth, PhD
Psy 427 Cal State Northridge Andrew Ainsworth, PhD Contents Item Analysis in General Classical Test Theory Item Response Theory Basics Item Response Functions Item Information Functions Invariance IRT
More informationTHE PROCESS OF MENTAL HEALTH RECOVERY/RESILIENCY IN CHILDREN AND ADOLESCENTS
THE PROCESS OF MENTAL HEALTH RECOVERY/RESILIENCY IN CHILDREN AND ADOLESCENTS Kate DeRoche Erica Gosselin Antonio Olmos Riley Rhodes Mental Health Center of Denver Presented at the American Evaluation Association,
More informationAssessing Measurement Invariance in the Attitude to Marriage Scale across East Asian Societies. Xiaowen Zhu. Xi an Jiaotong University.
Running head: ASSESS MEASUREMENT INVARIANCE Assessing Measurement Invariance in the Attitude to Marriage Scale across East Asian Societies Xiaowen Zhu Xi an Jiaotong University Yanjie Bian Xi an Jiaotong
More informationRunning head: CFA OF STICSA 1. Model-Based Factor Reliability and Replicability of the STICSA
Running head: CFA OF STICSA 1 Model-Based Factor Reliability and Replicability of the STICSA The State-Trait Inventory of Cognitive and Somatic Anxiety (STICSA; Ree et al., 2008) is a new measure of anxiety
More informationThe Psychometric Properties of Dispositional Flow Scale-2 in Internet Gaming
Curr Psychol (2009) 28:194 201 DOI 10.1007/s12144-009-9058-x The Psychometric Properties of Dispositional Flow Scale-2 in Internet Gaming C. K. John Wang & W. C. Liu & A. Khoo Published online: 27 May
More informationOak Meadow Autonomy Survey
Oak Meadow Autonomy Survey Patricia M. Meehan, Ph.D. August 7, 214 1 Contents Contents 3 List of Figures 3 List of Tables 3 1 Introduction 4 2 Data 4 3 Determining the Number of Factors 5 4 Proposed Model
More informationAnalysis of the Reliability and Validity of an Edgenuity Algebra I Quiz
Analysis of the Reliability and Validity of an Edgenuity Algebra I Quiz This study presents the steps Edgenuity uses to evaluate the reliability and validity of its quizzes, topic tests, and cumulative
More informationScale Building with Confirmatory Factor Analysis
Scale Building with Confirmatory Factor Analysis Latent Trait Measurement and Structural Equation Models Lecture #7 February 27, 2013 PSYC 948: Lecture #7 Today s Class Scale building with confirmatory
More informationInternational Conference on Humanities and Social Science (HSS 2016)
International Conference on Humanities and Social Science (HSS 2016) The Chinese Version of WOrk-reLated Flow Inventory (WOLF): An Examination of Reliability and Validity Yi-yu CHEN1, a, Xiao-tong YU2,
More informationBipolar items for the measurement of personal optimism instead of unipolar items
Psychological Test and Assessment Modeling, Volume 53, 2011 (4), 399-413 Bipolar items for the measurement of personal optimism instead of unipolar items Karl Schweizer 1, Wolfgang Rauch 2 & Andreas Gold
More informationUsing Analytical and Psychometric Tools in Medium- and High-Stakes Environments
Using Analytical and Psychometric Tools in Medium- and High-Stakes Environments Greg Pope, Analytics and Psychometrics Manager 2008 Users Conference San Antonio Introduction and purpose of this session
More informationRunning head: NESTED FACTOR ANALYTIC MODEL COMPARISON 1. John M. Clark III. Pearson. Author Note
Running head: NESTED FACTOR ANALYTIC MODEL COMPARISON 1 Nested Factor Analytic Model Comparison as a Means to Detect Aberrant Response Patterns John M. Clark III Pearson Author Note John M. Clark III,
More informationManifestation Of Differences In Item-Level Characteristics In Scale-Level Measurement Invariance Tests Of Multi-Group Confirmatory Factor Analyses
Journal of Modern Applied Statistical Methods Copyright 2005 JMASM, Inc. May, 2005, Vol. 4, No.1, 275-282 1538 9472/05/$95.00 Manifestation Of Differences In Item-Level Characteristics In Scale-Level Measurement
More informationPresented by Paul A. Carrola, Ph.D., LPC S The University of Texas at El Paso TCA 2014 Mid Winter Conference
Presented by Paul A. Carrola, Ph.D., LPC S The University of Texas at El Paso TCA 2014 Mid Winter Conference Discussion Outline My Background Culture of Correctional Mental Health Study Overview Rational/Research
More informationVALIDATION OF TWO BODY IMAGE MEASURES FOR MEN AND WOMEN. Shayna A. Rusticus Anita M. Hubley University of British Columbia, Vancouver, BC, Canada
The University of British Columbia VALIDATION OF TWO BODY IMAGE MEASURES FOR MEN AND WOMEN Shayna A. Rusticus Anita M. Hubley University of British Columbia, Vancouver, BC, Canada Presented at the Annual
More informationPsychometric Evaluation of the Major Depression Inventory at the Kenyan Coast
Psychometric Evaluation of the Major Depression Inventory at the Kenyan Coast INDEPTH Scientific Conference Addis Ababa 10 th - 13 th November 2015 Mark Otiende IHTAS INDEPTH Healthy Transitions to Adulthood
More informationDifferential Item Functioning
Differential Item Functioning Lecture #11 ICPSR Item Response Theory Workshop Lecture #11: 1of 62 Lecture Overview Detection of Differential Item Functioning (DIF) Distinguish Bias from DIF Test vs. Item
More informationObjective. Life purpose, a stable and generalized intention to do something that is at once
Validation of the Claremont Purpose Scale with College Undergraduates Objective Life purpose, a stable and generalized intention to do something that is at once meaningful to the self and of consequence
More informationThe Development of Scales to Measure QISA s Three Guiding Principles of Student Aspirations Using the My Voice TM Survey
The Development of Scales to Measure QISA s Three Guiding Principles of Student Aspirations Using the My Voice TM Survey Matthew J. Bundick, Ph.D. Director of Research February 2011 The Development of
More informationAnumber of studies have shown that ignorance regarding fundamental measurement
10.1177/0013164406288165 Educational Graham / Congeneric and Psychological Reliability Measurement Congeneric and (Essentially) Tau-Equivalent Estimates of Score Reliability What They Are and How to Use
More informationMeasurement Invariance (MI): a general overview
Measurement Invariance (MI): a general overview Eric Duku Offord Centre for Child Studies 21 January 2015 Plan Background What is Measurement Invariance Methodology to test MI Challenges with post-hoc
More informationValidity of the Risk & Protective Factor Model
Validity of the Risk & Protective Factor Model The Use in Evaluation Vicki Schmitt Bruce Frey Michelle Dunham Carol Carman School Program Evaluation and Research (SPEaR) University of Kansas Background
More informationFactor Analysis. MERMAID Series 12/11. Galen E. Switzer, PhD Rachel Hess, MD, MS
Factor Analysis MERMAID Series 2/ Galen E Switzer, PhD Rachel Hess, MD, MS Ways to Examine Groups of Things Groups of People Groups of Indicators Cluster Analysis Exploratory Factor Analysis Latent Class
More informationConsumer Perception Survey (Formerly Known as POQI)
Department of Behavioral Health Consumer Perception Survey (Formerly Known as POQI) CPS Comparison May 2017 On a semi-annual basis the County of Fresno, Department of Behavioral Health (DBH) conducts its
More informationAll reverse-worded items were scored accordingly and are in the appropriate direction in the data set.
PSYC 948: Latent Trait Measurement and Structural Equation Modeling Homework #7 (Total 10 Points) Due: Wednesday, April 10, 2013 at 11:59pm. Homework Problems: The homework problems are based a simulated
More informationPersonal Style Inventory Item Revision: Confirmatory Factor Analysis
Personal Style Inventory Item Revision: Confirmatory Factor Analysis This research was a team effort of Enzo Valenzi and myself. I m deeply grateful to Enzo for his years of statistical contributions to
More informationUsing Generalizability Theory to Investigate the Psychometric Property of an Assessment Center in Indonesia
Using Generalizability Theory to Investigate the Psychometric Property of an Assessment Center in Indonesia Urip Purwono received his Ph.D. (psychology) from the University of Massachusetts at Amherst,
More informationA Modification to the Behavioural Regulation in Exercise Questionnaire to Include an Assessment of Amotivation
JOURNAL OF SPORT & EXERCISE PSYCHOLOGY, 2004, 26, 191-196 2004 Human Kinetics Publishers, Inc. A Modification to the Behavioural Regulation in Exercise Questionnaire to Include an Assessment of Amotivation
More informationAdult Consumer and Family Member Perceptions of Care 2012: Findings from the Annual Survey of Pennsylvania Behavioral Health Service Recipients
Adult Consumer and Family Member Perceptions of Care 2012: Findings from the Annual Survey of Behavioral Health Service Recipients CONTENTS INTRODUCTION.. 1 METHODOLOGY.. 2 ADULT SURVEY FINDINGS.. 4 to
More informationThe Ego Identity Process Questionnaire: Factor Structure, Reliability, and Convergent Validity in Dutch-Speaking Late. Adolescents
33 2 The Ego Identity Process Questionnaire: Factor Structure, Reliability, and Convergent Validity in Dutch-Speaking Late Adolescents Koen Luyckx, Luc Goossens, Wim Beyers, & Bart Soenens (2006). Journal
More informationI. Introduction and Data Collection B. Sampling. 1. Bias. In this section Bias Random Sampling Sampling Error
I. Introduction and Data Collection B. Sampling In this section Bias Random Sampling Sampling Error 1. Bias Bias a prejudice in one direction (this occurs when the sample is selected in such a way that
More informationPaul Irwing, Manchester Business School
Paul Irwing, Manchester Business School Factor analysis has been the prime statistical technique for the development of structural theories in social science, such as the hierarchical factor model of human
More informationImpact and adjustment of selection bias. in the assessment of measurement equivalence
Impact and adjustment of selection bias in the assessment of measurement equivalence Thomas Klausch, Joop Hox,& Barry Schouten Working Paper, Utrecht, December 2012 Corresponding author: Thomas Klausch,
More informationEMOTIONAL INTELLIGENCE skills assessment: technical report
OnlineAssessments EISA EMOTIONAL INTELLIGENCE skills assessment: technical report [ Abridged Derek Mann ] To accompany the Emotional Intelligence Skills Assessment (EISA) by Steven J. Stein, Derek Mann,
More informationInstrument Validation Study
Instrument Validation Study REGARDING LEADERSHIP CIRCLE PROFILE By Industrial Psychology Department Bowling Green State University INSTRUMENT VALIDATION STUDY EXECUTIVE SUMMARY AND RESPONSE TO THE RECOMMENDATIONS
More informationPurpose of Workshop. Faculty. Culturally Sensitive Research. ISOQOL Workshop 19 Oct 2005
Introduction to ing Health-Related Measures in Diverse Populations Pre-conference Workshop 2 The International Society for Quality of Life Research Center for Aging in Diverse Communities: A Resource Center
More informationFactorial Validity and Consistency of the MBI-GS Across Occupational Groups in Norway
Brief Report Factorial Validity and Consistency of the MBI-GS Across Occupational Groups in Norway Astrid M. Richardsen Norwegian School of Management Monica Martinussen University of Tromsø The present
More informationThe Modification of Dichotomous and Polytomous Item Response Theory to Structural Equation Modeling Analysis
Canadian Social Science Vol. 8, No. 5, 2012, pp. 71-78 DOI:10.3968/j.css.1923669720120805.1148 ISSN 1712-8056[Print] ISSN 1923-6697[Online] www.cscanada.net www.cscanada.org The Modification of Dichotomous
More informationCBT+ Measures Cheat Sheet
CBT+ Measures Cheat Sheet Child and Adolescent Trauma Screen (CATS). The CATS has 2 sections: (1) Trauma Screen and (2) DSM5 sx. There are also impairment items. There is a self-report version for ages
More information1. Evaluate the methodological quality of a study with the COSMIN checklist
Answers 1. Evaluate the methodological quality of a study with the COSMIN checklist We follow the four steps as presented in Table 9.2. Step 1: The following measurement properties are evaluated in the
More informationRevised Motivated Strategies for Learning Questionnaire for Secondary School Students
19 Revised Motivated Strategies for Learning Questionnaire for Secondary School Students Woon Chia Liu, Chee Keng John Wang, Caroline Koh, Stefanie Chye, Bee Leng Chua, and Boon San Coral Lim National
More informationNonparametric DIF. Bruno D. Zumbo and Petronilla M. Witarsa University of British Columbia
Nonparametric DIF Nonparametric IRT Methodology For Detecting DIF In Moderate-To-Small Scale Measurement: Operating Characteristics And A Comparison With The Mantel Haenszel Bruno D. Zumbo and Petronilla
More informationReliability and Validity of the Divided
Aging, Neuropsychology, and Cognition, 12:89 98 Copyright 2005 Taylor & Francis, Inc. ISSN: 1382-5585/05 DOI: 10.1080/13825580590925143 Reliability and Validity of the Divided Aging, 121Taylor NANC 52900
More informationCollege Student Self-Assessment Survey (CSSAS)
13 College Student Self-Assessment Survey (CSSAS) Development of College Student Self Assessment Survey (CSSAS) The collection and analysis of student achievement indicator data are of primary importance
More informationAuthor s response to reviews
Author s response to reviews Title: The validity of a professional competence tool for physiotherapy students in simulationbased clinical education: a Rasch analysis Authors: Belinda Judd (belinda.judd@sydney.edu.au)
More informationMeasures of children s subjective well-being: Analysis of the potential for cross-cultural comparisons
Measures of children s subjective well-being: Analysis of the potential for cross-cultural comparisons Ferran Casas & Gwyther Rees Children s subjective well-being A substantial amount of international
More informationMEASURING MEANING AND PEACE WITH THE FACIT-SP: DISTINCTION WITHOUT A DIFFERENCE?
MEASURING MEANING AND PEACE WITH THE FACIT-SP: DISTINCTION WITHOUT A DIFFERENCE? Eboni Hedgspeth, B.A. Charlie Reeve, Ph.D. Amy Peterman, Ph.D. University of North Carolina at Charlotte ACKNOWLEDGEMENTS
More informationPackianathan Chelladurai Troy University, Troy, Alabama, USA.
DIMENSIONS OF ORGANIZATIONAL CAPACITY OF SPORT GOVERNING BODIES OF GHANA: DEVELOPMENT OF A SCALE Christopher Essilfie I.B.S Consulting Alliance, Accra, Ghana E-mail: chrisessilfie@yahoo.com Packianathan
More informationA methodological perspective on the analysis of clinical and personality questionnaires Smits, Iris Anna Marije
University of Groningen A methodological perspective on the analysis of clinical and personality questionnaires Smits, Iris Anna Mare IMPORTANT NOTE: You are advised to consult the publisher's version
More informationOn Test Scores (Part 2) How to Properly Use Test Scores in Secondary Analyses. Structural Equation Modeling Lecture #12 April 29, 2015
On Test Scores (Part 2) How to Properly Use Test Scores in Secondary Analyses Structural Equation Modeling Lecture #12 April 29, 2015 PRE 906, SEM: On Test Scores #2--The Proper Use of Scores Today s Class:
More informationThe Influence of Psychological Empowerment on Innovative Work Behavior among Academia in Malaysian Research Universities
DOI: 10.7763/IPEDR. 2014. V 78. 21 The Influence of Psychological Empowerment on Innovative Work Behavior among Academia in Malaysian Research Universities Azra Ayue Abdul Rahman 1, Siti Aisyah Panatik
More informationA Short Form of Sweeney, Hausknecht and Soutar s Cognitive Dissonance Scale
A Short Form of Sweeney, Hausknecht and Soutar s Cognitive Dissonance Scale Associate Professor Jillian C. Sweeney University of Western Australia Business School, Crawley, Australia Email: jill.sweeney@uwa.edu.au
More informationHanne Søberg Finbråten 1,2*, Bodil Wilde-Larsson 2,3, Gun Nordström 3, Kjell Sverre Pettersen 4, Anne Trollvik 3 and Øystein Guttersrud 5
Finbråten et al. BMC Health Services Research (2018) 18:506 https://doi.org/10.1186/s12913-018-3275-7 RESEARCH ARTICLE Open Access Establishing the HLS-Q12 short version of the European Health Literacy
More informationThe Bilevel Structure of the Outcome Questionnaire 45
Psychological Assessment 2010 American Psychological Association 2010, Vol. 22, No. 2, 350 355 1040-3590/10/$12.00 DOI: 10.1037/a0019187 The Bilevel Structure of the Outcome Questionnaire 45 Jamie L. Bludworth,
More informationSaville Consulting Wave Professional Styles Handbook
Saville Consulting Wave Professional Styles Handbook PART 4: TECHNICAL Chapter 19: Reliability This manual has been generated electronically. Saville Consulting do not guarantee that it has not been changed
More informationANOVA. Thomas Elliott. January 29, 2013
ANOVA Thomas Elliott January 29, 2013 ANOVA stands for analysis of variance and is one of the basic statistical tests we can use to find relationships between two or more variables. ANOVA compares the
More informationAssessing the Validity and Reliability of a Measurement Model in Structural Equation Modeling (SEM)
British Journal of Mathematics & Computer Science 15(3): 1-8, 2016, Article no.bjmcs.25183 ISSN: 2231-0851 SCIENCEDOMAIN international www.sciencedomain.org Assessing the Validity and Reliability of a
More informationDevelopment and Psychometric Properties of the Relational Mobility Scale for the Indonesian Population
Development and Psychometric Properties of the Relational Mobility Scale for the Indonesian Population Sukaesi Marianti Abstract This study aims to develop the Relational Mobility Scale for the Indonesian
More informationConnectedness DEOCS 4.1 Construct Validity Summary
Connectedness DEOCS 4.1 Construct Validity Summary DEFENSE EQUAL OPPORTUNITY MANAGEMENT INSTITUTE DIRECTORATE OF RESEARCH DEVELOPMENT AND STRATEGIC INITIATIVES Directed by Dr. Daniel P. McDonald, Executive
More informationUsing Your Brain -- for a CHANGE Summary. NLPcourses.com
Using Your Brain -- for a CHANGE Summary NLPcourses.com Table of Contents Using Your Brain -- for a CHANGE by Richard Bandler Summary... 6 Chapter 1 Who s Driving the Bus?... 6 Chapter 2 Running Your Own
More informationThe measurement of media literacy in eating disorder risk factor research: psychometric properties of six measures
McLean et al. Journal of Eating Disorders (2016) 4:30 DOI 10.1186/s40337-016-0116-0 RESEARCH ARTICLE Open Access The measurement of media literacy in eating disorder risk factor research: psychometric
More informationMHSIP Consumer Survey Technical Report. Fiscal Year 2012
MHSIP Consumer Survey Technical Report Fiscal Year 2012 A Report from the Colorado Department of Human Services Division of Behavioral Health This report was prepared by: Angie Lawson, Ph.D., Division
More informationValidity and reliability of physical education teachers' beliefs and intentions toward teaching students with disabilities (TBITSD) questionnaire
Advances in Environmental Biology, 7(11) Oct 201, Pages: 469-47 AENSI Journals Advances in Environmental Biology Journal home page: http://www.aensiweb.com/aeb.html Validity and reliability of physical
More informationBy Hui Bian Office for Faculty Excellence
By Hui Bian Office for Faculty Excellence 1 Email: bianh@ecu.edu Phone: 328-5428 Location: 1001 Joyner Library, room 1006 Office hours: 8:00am-5:00pm, Monday-Friday 2 Educational tests and regular surveys
More informationSelf-Oriented and Socially Prescribed Perfectionism in the Eating Disorder Inventory Perfectionism Subscale
Self-Oriented and Socially Prescribed Perfectionism in the Eating Disorder Inventory Perfectionism Subscale Simon B. Sherry, 1 Paul L. Hewitt, 1 * Avi Besser, 2 Brandy J. McGee, 1 and Gordon L. Flett 3
More informationIssues That Should Not Be Overlooked in the Dominance Versus Ideal Point Controversy
Industrial and Organizational Psychology, 3 (2010), 489 493. Copyright 2010 Society for Industrial and Organizational Psychology. 1754-9426/10 Issues That Should Not Be Overlooked in the Dominance Versus
More informationThe Youth Experience Survey 2.0: Instrument Revisions and Validity Testing* David M. Hansen 1 University of Illinois, Urbana-Champaign
The Youth Experience Survey 2.0: Instrument Revisions and Validity Testing* David M. Hansen 1 University of Illinois, Urbana-Champaign Reed Larson 2 University of Illinois, Urbana-Champaign February 28,
More informationIntroduction to Multilevel Models for Longitudinal and Repeated Measures Data
Introduction to Multilevel Models for Longitudinal and Repeated Measures Data Today s Class: Features of longitudinal data Features of longitudinal models What can MLM do for you? What to expect in this
More informationConfirmatory Factor Analysis of the Procrastination Assessment Scale for Students
611456SGOXXX10.1177/2158244015611456SAGE OpenYockey and Kralowec research-article2015 Article Confirmatory Factor Analysis of the Procrastination Assessment Scale for Students SAGE Open October-December
More informationOriginal Article. Relationship between sport participation behavior and the two types of sport commitment of Japanese student athletes
Journal of Physical Education and Sport (JPES), 17(4), Art 267, pp. 2412-2416, 2017 online ISSN: 2247-806X; p-issn: 2247 8051; ISSN - L = 2247-8051 JPES Original Article Relationship between sport participation
More informationComprehensive Statistical Analysis of a Mathematics Placement Test
Comprehensive Statistical Analysis of a Mathematics Placement Test Robert J. Hall Department of Educational Psychology Texas A&M University, USA (bobhall@tamu.edu) Eunju Jung Department of Educational
More informationConstruct Invariance of the Survey of Knowledge of Internet Risk and Internet Behavior Knowledge Scale
University of Connecticut DigitalCommons@UConn NERA Conference Proceedings 2010 Northeastern Educational Research Association (NERA) Annual Conference Fall 10-20-2010 Construct Invariance of the Survey
More informationTwo-Way Independent ANOVA
Two-Way Independent ANOVA Analysis of Variance (ANOVA) a common and robust statistical test that you can use to compare the mean scores collected from different conditions or groups in an experiment. There
More informationAN EVALUATION OF CONFIRMATORY FACTOR ANALYSIS OF RYFF S PSYCHOLOGICAL WELL-BEING SCALE IN A PERSIAN SAMPLE. Seyed Mohammad Kalantarkousheh 1
AN EVALUATION OF CONFIRMATORY FACTOR ANALYSIS OF RYFF S PSYCHOLOGICAL WELL-BEING SCALE IN A PERSIAN SAMPLE Seyed Mohammad Kalantarkousheh 1 ABSTRACT: This paper examines the construct validity and reliability
More informationAssessing Measurement Invariance of the Teachers Perceptions of Grading Practices Scale across Cultures
Assessing Measurement Invariance of the Teachers Perceptions of Grading Practices Scale across Cultures Xing Liu Assistant Professor Education Department Eastern Connecticut State University 83 Windham
More informationPsychometrics in context: Test Construction with IRT. Professor John Rust University of Cambridge
Psychometrics in context: Test Construction with IRT Professor John Rust University of Cambridge Plan Guttman scaling Guttman errors and Loevinger s H statistic Non-parametric IRT Traces in Stata Parametric
More informationTesting the Multiple Intelligences Theory in Oman
Available online at www.sciencedirect.com ScienceDirect Procedia - Social and Behavioral Sciences 190 ( 2015 ) 106 112 2nd GLOBAL CONFERENCE on PSYCHOLOGY RESEARCHES, 28-29, November 2014 Testing the Multiple
More informationMeasurement and Descriptive Statistics. Katie Rommel-Esham Education 604
Measurement and Descriptive Statistics Katie Rommel-Esham Education 604 Frequency Distributions Frequency table # grad courses taken f 3 or fewer 5 4-6 3 7-9 2 10 or more 4 Pictorial Representations Frequency
More informationValidating Measures of Self Control via Rasch Measurement. Jonathan Hasford Department of Marketing, University of Kentucky
Validating Measures of Self Control via Rasch Measurement Jonathan Hasford Department of Marketing, University of Kentucky Kelly D. Bradley Department of Educational Policy Studies & Evaluation, University
More informationExamination of the factor structure of critical thinking disposition scale according to different variables
American Journal of Theoretical and Applied Statistics 2015; 4(1-1): 1-8 Published online August 30, 2015 (http://www.sciencepublishinggroup.com/j/ajtas) doi: 10.11648/j.ajtas.s.2015040101.11 ISSN: 2326-8999
More informationOrganizational readiness for implementing change: a psychometric assessment of a new measure
Shea et al. Implementation Science 2014, 9:7 Implementation Science RESEARCH Organizational readiness for implementing change: a psychometric assessment of a new measure Christopher M Shea 1,2*, Sara R
More informationThe CSGU: A Measure of Controllability, Stability, Globality, and Universality Attributions
Journal of Sport & Exercise Psychology, 2008, 30, 611-641 2008 Human Kinetics, Inc. The CSGU: A Measure of Controllability, Stability, Globality, and Universality Attributions Pete Coffee and Tim Rees
More informationDevelopment and validation of a patient-reported outcome measure for stroke patients
Luo et al. Health and Quality of Life Outcomes (2015) 13:53 DOI 10.1186/s12955-015-0246-0 RESEARCH ARTICLE Open Access Development and validation of a patient-reported outcome measure for stroke patients
More information