Airline choice. Case 1. Australian health care principles. One person s answers. Best Worst Scaling: Principles & History. Who came up With these?

Size: px

Start display at page:

Download "Airline choice. Case 1. Australian health care principles. One person s answers. Best Worst Scaling: Principles & History. Who came up With these?"

Irma Marshall
6 years ago
Views:

Best Worst Scaling: Principles & History Workshop: Practical Experiences with the Use of Best- Worst Scaling in Economic Evaluation ISPOR, Baltimore, May, 2011 Terry N.

Bridges (Johns Hopkins Bloomberg School of Public Health) Christine Poulos (RTI Health Solutions) Random utility theory Humans make errors We rely on these to estimate (correct) statistical models

1 Best Worst Scaling: Principles & History Workshop: Practical Experiences with the Use of Best- Worst Scaling in Economic Evaluation ISPOR, Baltimore, May, 2011 Terry N. Flynn (University of Technology, Sydney) John F.P. Bridges (Johns Hopkins Bloomberg School of Public Health) Christine Poulos (RTI Health Solutions) Random utility theory Humans make errors We rely on these to estimate (correct) statistical models Traditional choice experiments only go so far Tell us about what a respondent likes Need to ask about lots of bad options to get dislikes History Seminal work of Luce & Marley in 1950s and 1960s Invented by Louviere in 1980s in Alberta Return of Marley to the field and explosion in interest in marketing (amongst other fields) Airline choice Case 1 The original object case Number Airline Block Things in each block (set) 1 Air France 1 American Delta US Air 2 American 2 Air France Delta Lufthansa 3 Continental 3 Delta United Virgin 4 Delta 4 Continental Delta Northwest 5 Lufthansa 5 Air France American Continental 6 Northwest 6 American Lufthansa United 7 United 7 American Northwest Virgin 8 US Air 8 Air France US Air Virgin 9 Virgin 9 Lufthansa Northwest US Air 10 Continental United US Air To/From Europe/USA 11 Air France Northwest United 12 Continental Lufthansa Virgin Australian health care principles One person s answers Yes, that means you Principles from the Health Care Reform Report people & family centred equity shared responsibility promoting wellness & strengthening prevention comprehensiveness value for money providing for future generations recognise social & environmental influences shape our health taking the long term view quality & safety transparency & accountability public voice & community engagement a respectful, ethical system responsible spending a culture of reflective improvement & innovation Who came up With these? Whose views do these reflect? What do citizens Think about them? 1

Selected histograms Histograms, 2 Case 1 example International and National

Barri Blauvelt MBA Contact: jbridges@jhsph.

Background Liver cancer is the 6 th leading cause of cancer globally, 3 rd

policies, and practices, with some countries having more advanced in their

global interest to understand and share best practices in HCC, as well as raise

development of comprehensive liver cancer control plans that are consistent with

needs, public policy priorities and emerging technologies impacting (or likely to

2 Selected histograms Histograms, 2 Case 1 example International and National Priorities For Liver Cancer Control John F P Bridges PhD, Gisselle Gallego PhD, Barri Blauvelt MBA Contact: jbridges@jhsph.edu Disclosures/funding: Bristol-Myers Squibb. Background Liver cancer is the 6 th leading cause of cancer globally, 3 rd leading cause of cancer death, and 97% annual mortality Diverse guidelines, policies, and practices, with some countries having more advanced in their knowledge and management of HCC and formulation of clinical guidelines Increasing global interest to understand and share best practices in HCC, as well as raise level of priority for liver cancer control Objectives To facilitate the development of comprehensive liver cancer control plans that are consistent with the values of key stakeholders in medical, policy, and advocacy Identify national needs, public policy priorities and emerging technologies impacting (or likely to impact) liver cancer control Identify the gaps where the medical community, government, and advocacy groups can work together to improve liver cancer prevention, diagnosis, and overall management 2

Data sources Qualitative interviews (n=20) Used to identify issues//technologies Pilot of survey (n=26) Validate and refine survey instrument Survey of key informants (n=200) Different stakeholders

3 Data sources Qualitative interviews (n=20) Used to identify issues//technologies Pilot of survey (n=26) Validate and refine survey instrument Survey of key informants (n=200) Different stakeholders (clinicians, policy makers and patient/disease advocates) 10 Countries (China, Japan, South Korea, Taiwan, Italy, France, Spain, Germany, Turkey, USA Emerging technologies We aimed identify the most important technologies to inform horizon scanning. Based on preliminary research we identified 11 emerging technologies: Molecular targeted therapy, Earlier detection of HCC, Genetic/genomic indicators, Stem cell based technologies, Adjuvant and neoadjuvant therapies, Improved radiology, Improved surgery, Biopsy free HCC detection, transplantation technology, HCV vaccination, Immunomodulation Best-worst scaling case 1 Experimental design We utilized a Balance Incomplete Block Design (BIBD) This generated 11 choice sets A small negative correlation between factors. Coding and analysis We assumed sequential best-worst. Choice set type 1: five technologies and the respondent chooses best Choice set type 2: four remaining technologies and respondent chooses worst Two tricks in coding In choice set types two, code the dummy variables as negatives Use effects coding (i.e. treat the outcomes as attribute levels of a single attribute) Results Emerging Technology Most and Least Counts Conditional logit Most Least Score Coefficient SE Molecular target therapy Earlier detection of HCC Genetic/genomic biomarkers HCV vaccination Interventional radiology Adjuvant/Neo-adjuvant therapies Immunomodulation Transplant technology Stem cell therapy Surgical techniques Biopsy free HCC diagnostics

4 Conclusions BWS Case 1 is a valuable method to identifying important factors It is best to consider these factors as attribute levels of the question in focus (e.g. of future impact on liver cancer control) rather than attributes We have also explored the use of conjoint analysis as a means to rank factors (by looking at strategies for liver cancer control) Example of a conjoint format Question 1: A national liver cancer control plan would consist of a number of different strategies. In this section we will consider competing plans and ask you to identify which one you believe will have the greatest impact in your country. Which national liver cancer plan is better? National Plan A National Plan B Measuring the social burden of liver cancer Organized disease advocacy and public awareness Improved risk assessment and referral by primary care Multidisciplinary management of HCC Increased infrastructure for translational research Transplantation infrastructure and allocation Continuous monitoring of at-risk populations Centers of excellence for liver cancer Education of physicians and hepatologists about HCC Improved access to recommended treatments National standards and guidelines I choose Plan A I choose Plan B Results Strategy OR (95% CI) P-value Continuous monitoring of at-risk populations ( ) <.0001 Multidisciplinary management of HCC ( ) <.0001 National standards and guidelines ( ) <.0001 Education of physicians and hepatologists about HCC ( ) <.0001 Centers of excellence for liver cancer ( ) <.0001 Improved access to recommended treatments ( ) <.0001 Early risk assessment in primary care ( ) <.0001 Increased infrastructure for translational research ( ) <.0001 Measuring incidence, prevalence and burden of liver cancer ( ) <.0001 Organized disease advocacy and public awareness ( ) <.0001 Transplantation infrastructure and allocation ( ) Case 2 The profile (originally called attribute ) case The profile case of BWS Presents profiles one at a time Respondents make choices within a profile, not between whole profiles Choose the attribute that is best and the attribute that is worst (based on levels) It is the profile nature of the task that distinguish this from case 1, hence the name, not attributes An EQ 5D profile task Best Example EQ-5D health state Worst Some problems walking about No problems with self-care Some problems with performing usual activities Moderate pain or discomfort Extremely anxious or depressed THINK explicitly about how you are making your best and worst choices 4

Attributes Investigating Choice Experiments for the Preferences of Older People The ICEpop CAPability Instruments ( O and A) Attachment (love, friendship, affection and companionship) Security

5 An EQ 5D profile task Best Example EQ-5D health state Worst Some problems walking about No problems with self-care Some problems with performing usual activities Case 2 example Moderate pain or discomfort Extremely anxious or depressed Quality of Life: the ICECAP O instrument This profile is based on the EQ-5D health classification system. It is trivially easy! Attributes Investigating Choice Experiments for the Preferences of Older People The ICEpop CAPability Instruments ( O and A) Attachment (love, friendship, affection and companionship) Security (feeling safe and secure, not having to worry and not feeling vulnerable) Role (having a purpose or doing something that makes a person feel valued ) Enjoyment (pleasure and joy, and a sense of satisfaction ) Control (being independent and able to make one's own decisions ) For the next part of the survey you will be asked to consider 16 imaginary quality of life states, which will be presented to you one at a time. Imagine living in the quality of life state presented and decide which of the five things would be best to live with and which would be worst to live with. Here is an example someone has completed. The scores Attachment Security ID # ID # Role Enjoyment Control Please pay attention to the wording of the statements (for example, many, a little etc) as they will change and each quality of life state is different ID # ID #

6 UK scoring (norms) for ICECAP-O Heterogeneity analyses Practical experiences of using Latent Gold Use with care THINK about what s likely going on LOOK at your data USE theory Results E Attachment Security Role Enjoyment Control 31 Case 3 Multiple s Case 3 The multi profile case Presents multiple profiles Extension of discrete choice experiment Respondents choose among profiles Obtain ranking of profiles Through sequential selection of best, worst (from remaining options), best (from remaining options), and so on Attribute 1 Attribute 2 Attribute 3 Which profile do you prefer most? Which of the you prefer least? Which of the two you prefer most? A B C D 34 Case 3 Multiple s Case 3 Multiple s Presents multiple profiles Extension of discrete choice experiment Respondents choose among profiles Obtain ranking of profiles Through sequential selection of best, worst (from remaining options), best (from remaining options), and so on Attribute 1 Attribute 2 Attribute 3 Which profile do you prefer most? Which of the you prefer least? Which of the two you prefer most? A B C D Presents multiple profiles Extension of discrete choice experiment Respondents choose among profiles Obtain ranking of profiles Through sequential selection of best, worst (from remaining options), best (from remaining options), and so on Attribute 1 Attribute 2 Attribute 3 Which profile do you prefer most? Which of the you prefer least? Which of the two you prefer most? A B C D

Case 3 Multiple s Case 3 Literature Presents multiple profiles Extension of discrete choice experiment Respondents choose among profiles

options), best (from remaining options), and so on Ranking: B>D>C>A prefer most? Which of the you prefer least? Which of the two you prefer most?

2008, Journal of Choice Modeling Information from full or partial ranking and optimal design permits the estimation of individual level

Louviere 2008 Results from rank-ordered logit vs. model accounting for sequential selection of best and worst profiles Marley et al.

patients tolerance for risk of fatal side effect associated with the treatment of a chronic degenerative disease Secondary research question: how

BWS Choice Task Case 3 Example: Implementation Features Symptoms in the next 5 years Chance of disease requiring transplant Medicine A Need a

years 1 2 3 4 5 Year 50 cases out of 1000 Medicine C Need a walker Need cane for short 3 years and long walks 2 years 1 2 3 4 5 Year 10 cases out

Experimental Design for BWS: Nearly orthogonal, D-efficient design Precluded dominated alternatives 54 profiles organized into choice sets with 3

7 Case 3 Multiple s Case 3 Literature Presents multiple profiles Extension of discrete choice experiment Respondents choose among profiles Attribute 1 Attribute 2 Attribute 3 Obtain ranking of profiles Which profile do you Through sequential selection of best, worst (from remaining options), best (from remaining options), and so on Ranking: B>D>C>A prefer most? Which of the you prefer least? Which of the two you prefer most? A B C D Issues/themes in existing literature: Louviere et al. 2008, Journal of Choice Modeling Information from full or partial ranking and optimal design permits the estimation of individual level preference models Results of analysis with simpler regression models comparable to results from more complex regression models Lancsar and Louviere 2008 Results from rank-ordered logit vs. model accounting for sequential selection of best and worst profiles Marley et al Very few applications in health Case 3 Example: Benefit-Risk Study Case 3 Example: Main Choice Question Primary research question: patients tolerance for risk of fatal side effect associated with the treatment of a chronic degenerative disease Secondary research question: how does it compare with tolerances for: Risk of side effect requiring transplant Risk of side effect requiring hospitalization Case 3 Example: BWS Choice Task Case 3 Example: Implementation Features Symptoms in the next 5 years Chance of disease requiring transplant Medicine A Need a walker 4 years Need cane for short and long walks 1 year Year 0 cases out of 1000 Medicine B Need a walker Need a wheelchair 3 years 2 years Year 50 cases out of 1000 Medicine C Need a walker Need cane for short 3 years and long walks 2 years Year 10 cases out of 1000 Approach: Discrete choice experiment addresses primary research question Follow-up BWS questions address secondary research questions Experimental Design for BWS: Nearly orthogonal, D-efficient design Precluded dominated alternatives 54 profiles organized into choice sets with 3 alternatives One of 6 blocks of 3 questions was randomly assigned to each subject Chance of disease requiring hospitalization 10 cases out of cases out of cases out of 1000 Data Collection Online survey completed by 645 patients Completed in April

8 Case 3 Example: Results Case 3: Motives for exploring BWS 10 9 Rescaled Preference Weights 1. Explore preferences for additional attributes Prefer rence Weights RPL Clients want preference weights for many attributes. For example: Dimensions of PRO instrument Unique features of comparators 0 Improve in 2 years Improve in 4 years Improve No change Worsen Worsen in 4 years Worsen in 2 years 0% 1% 5% 0% 1% 5% 2. Estimate individual preferences Change in disability level Chance of transplant Chance of hospitalization Summary & Conclusions Case 1 can be used to quantify attitudes, views etc Respondent level scores calculable in Excel No need for frequentist or bayesian analyses Case 2 valuable when between-profile choices difficult THINK about how easy/difficult each profile is Case 3 is just an extension of a choice experiment But more likely to produce robust individual level estimates BWS is easy to do. But practical considerations: Get DESIGN right THINK about how people do the task LOOK at a respondent s data Research Frontier Issues Individual level models for PRO/QALY valuation Do we get same estimates from Case 2 as we would have got from traditional (more difficult!) DCE? Current work suggests we do Are people p equally sure of best and worst choices? No! (Depressed people much more sure of worst than best!) Remember ALL DCEs subject to: ˆ ˆ BWS can help with this! Some key BWS references (1) Some key BWS references (2) Beggs S, Cardell S & Hausman J. (1981) Assessing the potential demand for electric cars. Journal of Econometrics, 16, Helson H. (1964). Adaptation-Level Theory. New York, Harper & Row. Marley, AAJ. (1968) Some Probabilistic Models of Simple Choice and Ranking. Journal of Mathematical Psychology, 5, Finn A & Louviere JJ. (1992) Determining the Appropriate Response to Evidence of Public Concern: The Case of Food Safety. Journal of Public Policy & Marketing. 11(1): p Szeinbach SL, et al. (1999). Using conjoint analysis to evaluate health state preferences. Drug Information Journal. 33, McIntosh E & Louviere JJ. (2002). Separating weight and scale value: an exploration of bestattribute scaling in health economics. HESG meeting, Brunel University, July Marley AAJ & Louviere JJ. (2005). Some probabilistic models of best, worst, and best-worst choices. Journal of Mathematical Psychology, 49, Flynn TN, et al. (2007) Best-Worst Scaling: What it can do for health care research and how to do it. Journal of Health Economics. 26(1): p Lancsar E, Louviere JJ & Flynn TN. (2007). Several methods to investigate relative attribute impact in stated preference experiments. Social Science & Medicine. 64: p Marley AAJ, Flynn TN & Louviere JJ. (2008) Probabilistic models of set-dependent and attributelevel best-worst choice. Journal of Mathematical Psychology, 52, Louviere JJ, et al. (2008). Modelling the choices of single individuals by combining efficient choice experiment designs with extra preference information. Journal of Choice Modelling. 1(1): p Flynn TN. (2010). Using conjoint analysis and choice experiments to estimate quality adjusted life year values: issues to consider. Pharmacoeconomics. 28(9): p Flynn TN. (2010). Valuing citizen and patient preferences in health: recent developments in three types of best-worst scaling. Expert Review of Pharmacoeconomics & Outcomes Research. 10(3): p Flynn TN, Louviere JJ, Peters TJ, Coast J (2010). Using discrete choice experiments to investigate heterogeneity in preferences for quality of life. Variance scale heterogeneity matters. Social Science and Medicine; 70: Louviere JJ & Flynn TN. (2010) Using Best-Worst Scaling Choice Experiments To Measure Public Perceptions and Preferences for Healthcare Reform in Australia. The Patient: Patient-Centered Outcomes Research; 3(4): Marley AAJ & Louviere JJ. Best-Worst Choices for Multiattribute Options, (forthcoming). Louviere JJ, Marley AAJ, Flynn TN. Best-Worst Scaling: Theory, Methods and Applications, CUP (forthcoming) 8

Has Consumer Directed Care improved the quality of life of older Australians? Professor Julie Ratcliffe School of Medicine Flinders University

Has Consumer Directed Care improved the quality of life of older Australians? Professor Julie Ratcliffe School of Medicine Flinders University Acknowledgments: ARC Linkage Project A Health Economics Model