Measuring Nonspecific Factors in Treatment: Item Banks that Assess the Healthcare

Transcription

1 Measuring Nonspecific Factors in Treatment: Item Banks that Assess the Healthcare Experience and Attitudes from the Patient s Perspective Supplemental Material Online Only This supplement contains details concerning study participants and psychometric methodology. Methods PROMIS instrument development methodology overview. The initial stages of the PROMIS instrument development process include qualitative strategies utilizing patient focus groups, clinician interviews, and identification of existing conceptual material and questionnaires through extensive literature review. Banks of items are then calibrated using item response theory, which results in optimal precision and minimal patient burden. Focus group information Patients discussed their views of healthcare and healing in 6 focus groups of 6-9 participants each. Topics included characteristics of providers, healthcare offices, and patients own attitudes that contribute to health improvement. There were two focus groups each of integrative medicine clinic patients, conventional medicine clinic patients, and patients from the general community (N = 46; mean age = 40, ranging from 20-75; 28% male, 41% minority race, and reading level ranging from 3 rd grade to post-high school). Cognitive interviews. Forty-two patients from integrative medicine and conventional medicine clinics (mean age = 47, range = 19 84; 40% male; 33% minority; and reading level ranging from 6 th grade to post-high 1

2 school) participated in cognitive interviews, during which patients think aloud while reviewing one item at a time with a trained interviewer. Each of the 359 items was reviewed by at least 6 patients, representing a broad range of reading levels, both genders, and minority and majority races. Patients provided feedback on the clarity of the item, vocabulary, and appropriateness of the response scale. Based upon participant feedback, 63 items (17.5%) were rewritten or removed. All revised items were subsequently reviewed again by patients in further cognitive interviews. The item pool contained 296 items following the cognitive interview phase. The 296 HEAL items retained following the cognitive interviews represented several domains relevant to contextual aspects of healing: Patient-Provider Connection (61 items), Perceptions of the Healthcare Environment (35 items), Treatment Expectancy (43 items), Optimistic Attitudes (50 items), Locus of Control/Self-Efficacy (35 items), Spirituality (38 items), and Health and Wellness attitudes (34 items). Calibration Sampling for calibration of items We field tested the items on two samples of patients who received conventional or integrative medicine treatments for a medical or mental health condition. The first set of patients was an internet community sample of 1,400 persons (average age = 48, 55% female, 10% African American, 13% Hispanic, 33% with high school or less education) provided through the internet survey company, YouGov.com. Patients who reported receiving conventional medicine or integrative medicine treatments within the past year were eligible, and they completed the HEAL item banks and demographic questions at a single computer assessment. The second sample included 257 patients (age = 46, 78% female, 20% African American, 4% Hispanic, and 14% having high school or less education) at the University of Pittsburgh Medical 2

3 Center (UPMC) who had recently started a new integrative medicine (n = 127) or conventional medicine (n = 130) treatment for a medical or psychiatric condition. The participants in this clinical sample completed the HEAL item banks, demographics, legacy instruments, and PROMIS health status questions (PROMIS 29). The clinical sample repeated the HEAL computerized assessment 6 weeks later. At the 6 week follow-up, the clinical sample also completed the single item Clinical Global Impression (CGI)[24] scale to indicate improvement or worsening of symptoms since beginning treatment. Calibration Sample details Internet sample The most common conventional medicine treatments were for hypertension (20%), pain (18%), and diabetes (14%). Integrative medicine treatments included chiropractic (49%), massage or other body work (17%), and acupuncture (9%), and pain was the most commonly reported reason for treatment (70%). Clinical sample at the University of Pittsburgh Medical Center (UPMC) Chiropractic (15%), acupuncture (14%) and meditation classes (11%) were the most frequently reported integrative medicine treatments, and primary care (27%) and specialist care (18%) the most frequent convention medicine treatments. The most frequently reported reason for treatment among the clinical sample was pain (35%) and mental health/stress (23%). Legacy instruments completed by Clinical sample at UPMC in order to provide clues to concurrent validity. A focused validity study is underway; however, the local calibration sample completed measures of constructs similar to HEAL and a simple global assessment of change (CGI) item 3

4 in order to assess initial validity of HEAL item banks. These legacy measures included: the Ambulatory Care Experiences Survey (ACES) [1], which assesses perceptions of the patientprovider encounter, the clinical site, and organization of care in 24 items, the Credibility Expectancy Questionnaire [2] which measures beliefs about current treatment in 6 items, the Life Orientation Test Revised [3] which measures optimism in 10 items, the General Self- Efficacy Scale [4, 5] which measures self-confidence in problem solving in 10 items, the Ironson-Woods Spirituality-Religiosity Index [6], a 25-item measure of spiritual beliefs and religious practices, and the 17-item Complementary and Alternative Medicine Beliefs Inventory [7]. PROMIS 29 completed by Clinical sample To further determine concurrent and discriminant validity information on HEAL, the clinical sample also completed the PROMIS 29, a health status profile that includes 4-item scales for 7 domains: anxiety, depression, fatigue, physical functioning, pain interference, sleep disturbance, and satisfaction with social roles. The final item is a 10-point rating of pain intensity. Psychometric data analyses Classical Test Theory analyses: Factor analysis. The entire calibration sample, both internet and clinical, was randomly divided into two subsamples. The first subsample was used for exploratory factor analysis (EFA, n = 799), and the second was used for subsequent confirmatory factor analysis (CFA, n = 858) [25]. Due to the blocking procedures used in the internet sample, the valid sample for EFA was 450. EFA and CFA were conducted using Mplus 4.21 with Promax rotation [8].Factor loadings, scree plots, and eigenvalues were evaluated. For evidence of unidimensionality in each of the HEAL domains, we considered ratios of >4 for the first 2 eigenvalues, significant factor loadings on the 4

5 primary factor, and small residual correlations [9]. Following EFA, CFAs were performed. Items with low factor loadings (<.5) were dropped. Cross loading, in which an item had a greater than.40 loading on more than 1 factor, occurred with one Spirituality item (cross-loading on factor 4 Health and Wellness Attitudes), and 8 Treatment Expectancy items (cross-loading on factor 1, which described perceptions of the provider and treatment environment). These items were dropped. Following EFA and CFA, the 250 retained items were evaluated and refined using item response theory methods. Item Response Theory (IRT) Advantages of IRT are, first, IRT provides item-parameter estimates that are applicable across samples and populations, and also provides θ estimates for individual participants. For example, an individual s responses can be used to precisely estimate his or her magnitude of treatment expectancy (or other HEAL domain of interest) relative to the population. Secondly, IRT itemparameter estimates are on the same θ scale as those of the individuals who complete the questionnaire, meaning that expectancy (or other domain) items are represented along the same spectrum of intensity as those of individuals. The IRT model that is most commonly used for polytomous items (i.e., those with 3 or more ordinal response categories) is the two-parameter graded response model (GRM) [10]. The GRM has a slope parameter and n-1 threshold parameters for each item, where n is the number of response categories. The slope parameter measures item discrimination, or how well the item differentiates between higher and lower levels of the trait (or θ). Items are deemed to be useful if they have large slope parameters. In the two parameter graded response model, items are allowed to have different discrimination parameters. Threshold parameters measure item 5

6 difficulty, or the ease versus difficulty of endorsing different response options for an item. As an example, the first of the n-1 threshold parameters tells us where along the θ scale a respondent is more likely to endorse a response, for example, of never rather than rarely. Item Selection based on IRT To further refine the HEAL item banks, we used several additional outcomes from IRT analyses and descriptive statistics, such as sparse cell response distributions, item information functions, evidence of differential item functioning, and indicators of local independence. Response distributions Sparse cell Items that have few responses in a particular cell (e.g., items for which few or no respondents answer never ) can be problematic in IRT. It is not possible to obtain reliable estimates of parameters for response categories with very few observations. Therefore, items with sparse cells having fewer than five observations (e.g, few respondents answering never ) were considered for exclusion. Sparse cell items were examined for the combined internet and clinical sample. The numbers of sparse cell items and their HEAL domain were: 3/35 items from Locus of Control, and 3/34 items from Health and Wellness Attitudes. Groups of expert reviewers, consisting of a CAM clinician, conventional medicine clinician, and PROMIS researcher, evaluated the content of each sparse cell item in each of the HEAL domains and made recommendations for retention or removal, based on whether the items content was adequately represented by other items in the same domain. Based on this content review, we did not remove any sparse cell items, preferring to make decisions informed by the CTT and IRT analyses. 6

7 Item parameters and information functions Items with discrimination parameters less than 1.0 were considered poor and evaluated for exclusion. Also, items with information functions that had peaks less than 1.0 were eliminated from the pools. Sixty items were excluded based on discrimination parameters <1.0. Differential item functioning Differential item functioning (DIF) occurs when factors such as age, gender, or education, presumed to be unrelated to the construct of interest, actually do have an effect on measurement. An item is identified as functioning differentially if the item is more (or less) discriminating or more (or less) difficult to endorse in some group compared to a reference group, when the different subgroups have been matched on the latent trait being investigated. We conducted DIF analyses based on gender, age (less than 48 years versus greater than or equal to 48 years), and education (High school or less versus some further education or higher). We focused on these comparisons because the relevant subgroups were adequately represented. Our overall sample included 42% males and 58% females, and 30% of the sample had high school or less education. Other potential DIF comparison groups (e.g., ethnicity and race) were less equally represented. We used two different DIF procedures: the IRT likelihood ratio method [11] and an ordinal logistic regression procedure [12]. Items were considered for removal if they showed significant DIF (p<0.01) by both methods [13]. No items were excluded based upon DIF. Local independence Local independence is an important consideration in IRT. Local independence assumes that the probability of providing a particular response to one item is independent of the probability of providing a specific response to any other item, after controlling for the overall level of the 7

8 underlying trait. If pairs of items are locally dependent, this may lead to overestimation or underestimation of probabilities for specific response patterns [14]. The LDIP computer program [15] was used to calculate local dependence indices based on item-parameter estimates from MULTILOG. The Q3 statistic was used to evaluate local dependence, using a threshold of Two items from Spirituality and had a Q3 index greater than.50: I talk to religious leaders and I pray. Three items from F5 Treatment Expectancy were locally dependent: I am able to follow this treatment program, I expect that this treatment will heal me, and I intend to follow this treatment program. These five items were removed from the item banks. Final Calibrated items. Following refinement based upon IRT, 168 items remained in the item banks. The full item banks are provided in Tables 1 through 5. CAT administration will result in a smaller group of items being presented. We show the full item bank in order to illustrate the breadth of content of the bank. In order to determine a respondent s level or score on a domain, it is typically necessary to answer only 4 8 items. The majority of items use intensity response categories: Not at all, A little, Somewhat, Quite a, Very much. Items with frequency response categories (Never, Rarely, Sometimes, Often, Almost Always) are indicated with *. Reverse scoring is indicated with R. Preliminary validity evidence of HEAL item banks In the clinical sample, we examined concurrent validity between the six HEAL θ scores and legacy instruments measuring similar constructs. The legacy instruments relevant to current treatment were administered at the baseline assessment. Product-moment correlations between HEAL Patient-Provider Connection and Heathcare Environment and the ACES [1] average score were.38 and.39, respectively (p< 0.01), indicating similarity but not complete overlap with the ACES, which assesses various factors in outpatient clinical care. HEAL Treatment 8

9 Expectancy was associated with the CEQ [2] Credibility.71 (p< 0.01) and Expectancy.58 (p<0.01) factors. HEAL Positive Outlook was inversely associated with PROMIS29 depression (-.71, p<0.01) and anxiety (-.54, p<0.01). Legacy measures of trait-like attributes were administered at the 6-week follow-up assessment, in order to minimize participant burden at baseline. The follow-up assessment included 221 of the 257 clinical sample participants. Baseline HEAL spirituality was associated with IW-SR [6] at follow-up.81 (p< 0.01). HEAL Positive Outlook was associated with the LOT-R [3].60 (p<0.01) at follow-up, and HEAL Attitude toward Integrative Medicine was associated with CAMBI [7] Belief in Natural treatments.30 (p< 0.01). Treatment-related HEAL item banks PPC and HCE were unrelated to PROMIS health status measures, providing preliminary support for discriminant validity. To explore predictive validity, baseline HEAL θ scores were compared with follow-up Clinical Global Impression (CGI) ratings in the clinical sample. Correlations (Spearman s Rho), ranged from.36 (p<0.01) for HEAL Treatment Expectancy to.13, (p<0.05) for HEAL Spirituality, indicating that HEAL scores account for some variability in patients perceived improvement with treatment across a broad range of clinical conditions and treatments. Selection of items for short forms In some circumstances, CAT administration is not feasible or indicated, therefore, brief static forms were created, based upon psychometric properties and conceptual or clinical relevance. Short form items are indicated by Bold font in tables 1 5 and are included in the main manuscript. Table 6 provides the short static form, Attitudes toward CAM. The original domain, Health and Wellness Attitudes, was broad in scope and only 6 items pertaining to integrative medicine or CAM loaded together on EFA and CFA. This small number of items is inadequate for CAT administration; therefore we provide the short static form. 9

10 The internal consistency of the short forms was excellent. Alpha coefficients were 0.92 for Healthcare Environment and Positive Outlook, 0.96 for Patient-Provider Connection and Treatment Expectancy, and 0.97 for Spirituality. The correlations between the theta scores derived from the short forms and their corresponding full item banks were high: 0.97 for Positive Outlook, Spirituality, and Treatment Expectancy, 0.96 for Patent-Provider Connection, and 0.93 for Healthcare Environment. Table 1 Calibrated Patient-Provider Connection items Patient-Provider Connection (PPC) Slope Location thresholds (item context) Think of the HCP (Healthcare Provider) who provides your current/ongoing treatment (Discrimi nation) Not at all vs. a little A little vs. somewhat Somewhat vs. quite a I am satisfied with my healthcare provider I trust my healthcare provider My healthcare provider pays attention to my individual needs I trust my healthcare provider's judgment My healthcare provider gives me enough information* Quite a vs. very much My healthcare provider respects me I feel my healthcare provider understands me My healthcare provider gives me support and encouragement I like my healthcare provider My healthcare provider cares about me My healthcare provider has a good plan My healthcare provider listens to me* for helping me* I would recommend my healthcare provider to others My healthcare provider gives his/her best effort* My healthcare provider is sensitive to how I feel My healthcare provider and I agree on what is important for my health* My healthcare provider gives full attention to me* My healthcare provider speaks with me in a positive way My healthcare provider is concerned for my comfort

11 Table 1. Calibrated Patient-Provider Connection items (continued) My healthcare provider is concerned with my overall health My healthcare provider wants me to achieve my goals My healthcare provider accepts me for who I am My healthcare provider gives me hope* My healthcare provider is easy to talk to I can speak freely with my healthcare provider My healthcare provider discusses the treatment plan with me* My healthcare provider is knowledgeable My healthcare provider seems to know what to do* My healthcare provider gets me the care I need* My healthcare provider helps me think more clearly about my health I feel better after seeing my healthcare provider My healthcare provider is honest with me My healthcare provider and I are a team My healthcare provider helps me to understand what to expect from my health issues My healthcare provider helps me make decisions about my treatment Working with this healthcare provider makes a difference in my life My healthcare provider is professional My healthcare provider knows what is important to me My healthcare provider treats the whole person My healthcare provider respects my privacy I like to get advice from my healthcare provider My healthcare provider talks with me about how to follow through with my treatment* My healthcare provider is organized during my visits* My healthcare provider has enough time for me* I understand my health issues much better after seeing my healthcare provider My healthcare provider explains the pros and cons of my treatment* My healthcare provider provides me with choices* My healthcare provider helps me to understand the cause of my health issues

12 Table 1. Calibrated Patient-Provider Connection items (continued) My healthcare provider is flexible about my care My healthcare provider respects my cultural background My healthcare provider is open to my ideas I am involved in decisions as much as I want to be My healthcare provider uses language I can understand* My healthcare provider asks about how my health affects my everyday life* I know what to expect during treatment sessions My healthcare provider gives me advice on a healthy lifestyle* My healthcare provider is aware of my life situation Bold font=item is included in static short form questionnaire *Response categories are: Never, Rarely, Sometimes, Often, Almost always Table 2 Calibrated Healthcare Environment items Healthcare Environment (HCE) Slope Location threshold (item context) Think of the place where you receive your current/ongoing treatment (discrimi nation) Not at all vs. a little A little vs. somewhat Somewhat vs. quite a Quite a vs. very much The staff was respectful The staff was friendly The staff was polite The staff was helpful The staff paid attention to my needs I trust the staff My care was well organized The healthcare provider's office respected my privacy The healthcare provider's office was calm The healthcare provider's office looked Setting up an appointment was easy The healthcare provider's office was clean pleasant The room where I saw my healthcare provider was comfortable I was satisfied with the length of time I spent with the healthcare provider The lighting in the healthcare provider's office was pleasant I could reach my healthcare provider easily* I was satisfied with how long I waited to get an appointment

13 Table 2 Calibrated Healthcare Environment items (continued) It was easy to find my way around the healthcare provider's office The waiting area was comfortable* The healthcare provider's office was flexible with scheduling my appointment I was satisfied with the wait time in the healthcare provider's office The office hours were convenient The healthcare provider's office was quiet Information about health was available at the healthcare provider's office The healthcare provider's office had a pleasant smell Bold font=item is included in static short form questionnaire * Response categories are: Never, Rarely, Sometimes, Often, Almost always Table 3 Calibrated Treatment Expectancy items Treatment Expectancy (TE) slope Location threshold (discrimin ation) Not at all vs. a little A little vs. somewhat Somewhat vs. quite a I am confident in this treatment I believe this treatment will help me This treatment will be successful This treatment will be effective I feel good about this treatment I expect good outcomes from this treatment. This treatment is right for me This treatment is good I am confident that this treatment will help I expect this treatment to help me This treatment will help me feel better I expect to feel better I will feel better because of this treatment I value this treatment This treatment makes sense This treatment is ideal for me I would choose this treatment again I feel comfortable with this treatment This treatment will allow me to get better This treatment will improve my health I would recommend this treatment I expect that this treatment will make me healthier. This treatment fits with my views Quite a vs. very much 13

14 Table 3 Calibrated Treatment Expectancy items (continued) This treatment will help me to manage my health. I want this treatment This treatment allows me to be in control of my health. I expect that I will be healthy Bold font=item is included in static short form questionnaire Table 4 Calibrated Positive Outlook items Positive Outlook (PO) Slope Location threshold (discrimin ation) Not at all vs. a little A little vs. somewhat Somewhat vs. quite a I feel positive about my life I am pleased with my life I am hopeful about my future My future looks good I expect an enjoyable future I am satisfied with my life I am a happy person I believe things will go well for me in the future. I feel confident about myself I like myself I expect good things to happen I look at things in a positive way My future seems hopeless. R I feel useless. R I feel disappointed with my life. R I feel I can cope with my problems I have trouble enjoying life. R I feel in control of my life I am losing confidence in myself. R I accept myself for who I am I feel detached from life. R I am satisfied with my social life I feel rejected. R I know my problems will pass I am grateful for many things in life I expect to succeed at things I try.* I can manage my responsibilities R=reverse scored Bold font=item is included in static short form questionnaire * Response categories are: Never, Rarely, Sometimes, Often, Almost always Quite a vs. very much 14

15 Table 5 Calibrated Spirituality items Spirituality (Sp) slope Location threshold (discrimin ation) 15 Not at all vs. a little A little vs. somewhat Somewhat vs. quite a Spirituality gives me comfort in times of trouble. Spiritual beliefs give meaning to my life. Spiritual beliefs give me hope I find comfort in my faith My spirituality gives me inner strength Prayer is a meaningful part of my life I feel supported by a higher power A higher power guides my life Spirituality is an important part of my health. A higher power is taking care of me I believe that a higher power can heal I find strength in religious services I am a spiritual person I live my life according to spiritual beliefs I spend time thinking about a higher power.* I find peace in a religious place My place of worship is important to me I pray for the well-being of others I experience my spirituality through religious activities. I seek spiritual support from clergy.* My clergy supports me I ask people to pray for me* I attend religious services.* I listen to religious music.* I enjoy reading about religion My life is part of something bigger Bold font=item is included in static short form questionnaire * Response categories are: Never, Rarely, Sometimes, Often, Almost always Quite a vs. very much Table 6 Calibrated Complementary/Alternative Medicine Attitudes short static form items Attitudes toward CAM slope Location threshold (item context) CAM (Complementary and Alternative Medicine) is a nonconventional, holistic, or natural approach to healthcare. Common CAM treatments may include acupuncture, massage therapy, meditation, or herbal remedies. (discrimin ation) Not at all vs. a little A little vs. somewhat Somewhat vs. quite a CAM is effective Quite a vs. very much

16 Table 6 Calibrated Complementary/Alternative Medicine Attitudes short static items (continued) I prefer CAM over conventional medicine It is important to be open to CAM CAM can be used to treat serious illness CAM can prevent health problems I prefer natural remedies

17 References 1. Safran, D.G., M. Karp, K. Coltin, H. Chang, A. Li, J. Ogren, and W.H. Rogers, Measuring patients' experiences with individual primary care physicians. Results of a statewide demonstration project. Journal of General Internal Medicine, (1): p Devilly, G.J. and T.D. Borkovec, Psychometric properties of the credibility/expectancy questionnaire. Journal of Behavior Therapy and Experimental Psychiatry, : p Scheier, M.F., C.S. Carver, and M.W. Bridges, Distinguishing optimism from neuroticism (and trait anxiety, self-mastery, and self-esteem): a reevaluation of the Life Orientation Test. Journal of Personality & Social Psychology, (6): p Schwarzer, R. and A. Born, Optimistic self-beliefs: Assessment of general perceived self-efficacy in 13 cultures. World Psychology, : p Schwarzer, R., Measurement of perceived self-efficacy. Psychometric scales for crosscultural research. 1993, Berlin, Germany: Freie Universitat Berlin. 6. Ironson, G., G.F. Solomon, E.G. Balbin, C. O'Cleirigh, A. George, M. Kumar, D. Larson, and T.E. Woods, The Ironson-woods Spirituality/Religiousness Index is associated with long survival, health behaviors, less distress, and low cortisol in people with HIV/AIDS. Annals of Behavioral Medicine, (1): p Bishop, F.L., L. Yardley, and G. Lewith, Developing a measure of treatment beliefs: the complementary and alternative medicine beliefs inventory. Complementary Therapies in Medicine, (2): p Muthen, L.K. and B.O. Multhen, Mplus User's Guide, 4th ed. 2007, Los Angeles, CA: Mullen & Mullen. 9. Reeve, B.B., R.D. Hays, J.B. Bjorner, K.F. Cook, P.K. Crane, J.A. Teresi, D. Thissen, D.A. Revicki, D.J. Weiss, R.K. Hambleton, H. Liu, R. Gershon, S.P. Reise, J.S. Lai, and D. Cella, Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, (5 Suppl 1): p. S Samejima, F., Estimation of latent ability using a response pattern of graded scores. Psychometrika Mono, Thissen, D., L. Steinberg, and H. Wainer, Detection of differential item functioning using the parameters of item response models., in Differential Item Functioning., P.W. Holland and H. Wainer, Editors. 1993, Lawrence Erlbaum Associates: Hillsdale, NJ. p Zumbo, B.D., A Handbook on the Theory and Methods of Differential Item Functioning (DIF): Logistic Regresion Moedling as a Unitary Framework for Binary and Likert-type (Ordinal) Item Scores , Ottowa, ON: Directorate of Human Resources Research and Education, Department of National Defense. 13. Teresi, J.A., K. Ocepek-Welikson, M. Kleinman, J.P. Eimicke, P.K. Crane, R.N. Jones, J.-S. Lai, S.W. Choi, R.D. Hays, B.B. Reeve, S.P. Reise, P.A. Pilkonis, and D. Cella, Analysis of differential item functioning in the depression item bank from the Patient Reported Outcome Measurement Information System (PROMIS): An item response theory approach. Psychology Science Quarterly, (2): p

18 14. Embretson, S.E. and S.P. Reise, Item Response Theory for Psychologists. 2000, Mahwah, NJ: Lawrence Erlbaum Associates. 15. Kim, S.-H., A.S. Cohen, and Y.-H. Lin, LDIP: A computer program for local dependence indices for polytomous items. Applied Psychological Measurement, : p