AN ENVIRONMENTAL SCAN

Size: px

Start display at page:

Download "AN ENVIRONMENTAL SCAN"

Dominick Lawrence
5 years ago
Views:

1 How Stage Data Collected From Cancer Centres by the Ontario Cancer Registry (OCR) is Used, Compared to Cancer Care Ontario s (CCO) Corporate and Cultural Assumptions of its Value AN ENVIRONMENTAL SCAN

2 Goal Statement 1. Understand the current use of Regional Cancer Centre (RCC)- submitted TNM staging at CCO. 2. Understand any alternate sources of cancer centre stage used by CCO (e.g. in-house Collaborative Staging) 3. Understand the current (2010 onward) quality and timeliness of RCC-submitted staging. 4. Better define the role of complete all-cancer population staging in the success of CCO corporate goals. 5. Review and clarify assumptions concerning the necessity of RCC stage capture to the goal of population level staging. 2

3 Background Unique Ontario Context Canada is divided into two styles of stage at diagnosis collection English-speaking Canada: Provincial and Territorial cancer registries abstract all their cases as well as staging them There is probably only one fully-fledged hospital cancer registry in the English speaking jurisdictions Princess Margaret in Toronto Quebec does have hospital registries and is in process of including their abstracts into their new provincial system English speaking jurisdictions a matter of scale Except for Ontario, yearly incidence runs between a few thousand to ~24,000 Ontario incidence using the SEER MPS rules is ~75,000 3

4 Background Cancer Care Ontario (CCO) is an arm s length agency of the Ontario Ministry of Health and Long Term Care It s mission statement: Together, we will improve the performance of our health systems by driving quality, accountability, innovation and value. It currently supports all cancer statistics and quality indicators, provincial wait times metrics for all diseases and the provincial renal dialysis registry Cancer Care Ontario corporately decided it could only support the capacity to stage the most common cancers breast, lung, CRC, prostate, and cervix (CCO administers the provincial breast, CRC, and cervical screening programs) based on capacity CCO continues to list staging of all diseases to population level as one of its goals as a necessary element of its mission 4

5 Background Since 2010, the Ontario Cancer Registry (OCR) has staged the top 4 plus cervix to population level amounting to ~40,000 per year Automated case creation, followed by manual curation, 7 staff. Cases transferred every quarter to stage abstracting system (Registry Plus) and Collaboratively Staged by 19 staff. Both sets of staff receive analytic support of ~1 FTE No other data streams into CCO contain stage information except a feed built to authenticate radiation treatment from the 14 Regional Cancer Centres (RCC) important but incomplete set of pop. cancer Acute care hospitals may report for payment of some systemics but not required to provide normal cancer registration data set Other hospitals activity is only represented by discharge summaries and path reports There are ~ 110 hospitals in Ontario 5

6 Background Since 2005, CCO has committed to population level staging for all stagable cancers No plan, past or present is in place to acquire staging from all facilities But the proposed model of RCC and OCR staging was supposed to be the beginning of other, more elaborate engagement plans RCCs were engaged, trained, supported to supply staging to CCO for every newly diagnosed case for cancers not staged by the OCR OCR was to continue this support operationalized as 1.5 FTE subject matter expert and engagement specialist. Those positions were never funded. OCR maintained a pathology SME/engagement specialist, the coding and abstracting staffs and leads and two senior data analysts The RCC and systemic data stream was left with wider CCO Analytics and continues to be used for treatment indicators and funding validation 6

7 Imperative Status quo remained palatable so long as OCR staging and RCC staging were not alike Transition of OCR staging to AJCC TNM put the registry and the RCCs on an equal footing in the eyes of CCO s Clinical and Regional Programs the Business The Analytics and Informatics portfolio supports the business and cannot not tell the programs about the quality of RCC staging Quality of RCC staging ranges from excellent to non-existent A&I wants to understand why quality should be remediated if there is no visible use of the data Programs not articulate about their need for RCC staging except that OCR could begin staging other cancers if RCCs staged all the cancer they saw, including their portion of the OCR cancers Research a known consumer of stage, had no voice 7

8 Scan Methodology Environmental scan, interviews, data analysis, cascading flexible interview-discovery process for internal and external stakeholders. Internal to include: Clinical Programs, Regional Programs, A&I, Prevention and Cancer Control (P&CC, screening and prevention surveillance) and internal researchers External research would include at: Researchers in Public Health Ontario, large non-profit research consortiums (Institute for Clinical Evaluative Sciences -ICES, Queen s Cancer Research Institute), plus individual researchers discovered during interview and CCO Data Disclosure as expressing interest in stage, either at population level or for particular types of cancer. Discovery would lead to additional interviews 8

9 Recruitment Inclusion Criteria User of stage data (CCO employees, public health officials, oncologists, pathologists and researchers) Manager of stage data Expresses interest in stage, either at a population level or for particular types of cancer Directors who utilize stage data for planning purposes Exclusion Criteria Non-user of stage data Director or group manager that is not directly involved with stage except for planning purposes Unfamiliar with stage in any capacity by definition of their position 9

10 Recruitment Participant Recruitment The participants were initially recruited by , and eligibility was assessed through a short, in person/telephone pre-interview about their roles and responsibilities/how they relate to stage data. 10

11 Adding Continuity standard survey Short 18 question Likert-type survey on the use of stage (Appendix 1 for questions) 59 internal and external persons participated in the in-depth interviews Only 37 consented to take the survey Non-consent mostly from clinical and bio-medical researchers or CCO directors Net survey participants were Clinical & Regional Programs managers, A&I and P&CC managers, team leads and data analysts Worth of survey gauge understanding of data by direct users at CCO 11

12 Survey Results Highlights Q1: I am aware of how stage is used at CCO Entirely Agree 8% Mostly Agree 22% Somewhat Agree 42% Neither Agree nor Disagree 3% Somewhat Disagree 14% Mostly Disagree Entirely Disagree 6% 6% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 12

13 Q3: I prefer to use RCC TNM Stage Group when stage information is needed Entirely Agree Mostly Agree Somewhat Agree 0% 0% 3% Neither Agree nor Disagree 76% Somewhat Disagree 5% Mostly Disagree 11% Entirely Disagree 5% 0% 10% 20% 30% 40% 50% 60% 70% 80% Q4: I prefer to have both CSI as well as RCC TNM Stage Group when stage information is needed CSI = Collaborative Stage Entirely Agree 3% Mostly Agree Somewhat Agree 8% 11% Neither Agree nor Disagree 70% Somewhat Disagree 5% Mostly Disagree 0% Entirely Disagree 3% 0% 10% 20% 30% 40% 50% 60% 70% 80% 13

14 Q5: The current quality of stage data is sufficient for my work Entirely Agree 0% Mostly Agree Somewhat Agree 8% 11% Neither Agree nor Disagree 57% Somewhat Disagree 19% Mostly Disagree Entirely Disagree 3% 3% 0% 10% 20% 30% 40% 50% 60% Q7: I would use stage more if it was all of the same type Entirely Agree 3% Mostly Agree Somewhat Agree 16% 14% Neither Agree nor Disagree 54% Somewhat Disagree Mostly Disagree Entirely Disagree 3% 5% 5% 0% 10% 20% 30% 40% 50% 60% 14

15 Q10: Population-level staging is important for my work Entirely Agree 27% Mostly Agree 22% Somewhat Agree 11% Neither Agree nor Disagree 22% Somewhat Disagree 0% Mostly Disagree 5% Entirely Disagree 14% 0% 5% 10% 15% 20% 25% 30% Q11: Stage information is a necessity for my work Entirely Agree 35% Mostly Agree Somewhat Agree 11% 14% Neither Agree nor Disagree 19% Somewhat Disagree 3% Mostly Disagree 0% Entirely Disagree 19% 0% 5% 10% 15% 20% 25% 30% 35% 40% 15

16 Q13: I would use stage at diagnosis more if it were more complete for my cancers of interest Entirely Agree 14% Mostly Agree Somewhat Agree 16% 19% Neither Agree nor Disagree 43% Somewhat Disagree 3% Mostly Disagree 0% Entirely Disagree 5% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Patterns analysts versus Business respondents not discerned Specific analysts and their assigned Business clients responses did align All but four of 18 questions resulted in most common and average answer, Neither agree nor Disagree People were not aware of stage, how it was used, what it was, did they have a preference of RCC TNM or OCR CS But they did have stronger opinions about the importance of stage, mostly that it was important to their work 16

17 Appendix 1 Likert Survey Questions Q1 I am aware of how stage is used at CCO Q2 I prefer to use CSI (CS stage TNM and stage group) when stage information is needed. Q3 I prefer to use RCC TNM Stage Group when stage information is needed. Q4 I prefer to have both CSI as well as RCC TNM Stage Group when stage information is needed. Q5 The current quality of stage data is sufficient for my work Q6 CCO s current use of stage data is sufficient. Q7 I would use stage more if it was all of the same type. Q8 I would use stage at diagnosis more if I could be surer of its quality. Q9 I have a complete understanding of stage, as well as its various classification systems. Q10 Population-level staging is important for my work. Q11 Stage information is a necessity for my work. Q12 I would use stage at diagnosis more if I understood it better. Q13 I would use stage at diagnosis more if it were more complete for my cancers of interest. Q14 A better procedure/system is required for converting earlier staging additions into later ones. Q15 I am confident in the accuracy of stage data. Q16 I am confident in the comparability of stage data. Q17 I am confident in the completeness of stage data. Q18 I would use stage information more if it were reported in a timelier fashion. 17

18 Semi-Structured Interviews, Tiers 1 & 2 NB: Three tiers of questions, but allowing for other volunteered information (Appendix 2 full list of set questions) First Tier Questions, 59 answers Second Tier Questions, 47 answers If you use it, why Rate quality Percent of your work depends on stage Issues RCC vs. CS staging Understanding CS vs. RCC coverage of cancer types If you don t use stage why not Have you asked OCR for interpretation/help What upcoming CCO initiatives might require stage Would you use stage if more complete, better quality, more compatible, better understood Does RCC stage help with quality control issues even if you use RCC reporting for other purposes, e.g. treatment, funding for treatment 18

19 Semi-Structured Interviews, tier 3 Third Tier Questions, 51 answers Have you read the results of CCO Finance s audit of staging, Value for Money What is the value of population level cancer staging Is population level staging necessary in your work What would population level staging allow CCO to accomplish 19

20 Participant Response Profile by Category A & I respondents Regional Programs respondents (Regional Programs organizes 14 regional cancer programs, RCC-hospitals hub and spoke model (caveat not all hospitals directly participate), fund all RCC oncology consults and radiation treatment, fund certain high end systemic treatments, fund extra cancer surgeries. Rest of care is directly funded by MOHLTC) Clinical Program respondents Prevention and Cancer Control (P&CC) respondents Legal, Finance and Audit portfolio respondents External user respondents Statistical work on responses is still in progress. The following gives a flavour of the findings based on scanning the narrative responses 20

21 A & I Respondents CCO developing enterprise wide policies and procedures to manage data assets, including stage, still in planning stage Cancer Analytics uses stage for indicator development, exploratory analyses for clinical programs using RCC stage depending on cancer type, question type, otherwise use CS. Want to understand staging better Stage is important for all sites, and we are currently missing relevant data on important cancers because RCCs never see all cancers of a type, because RCC stage high in missingness and unreviewed for quality We would use stage more if it was all the same type CCO Research office uses stage for internal/external research projects. About 80% of research projects requires stage. CS used if possible because of quality and completeness, otherwise RCC stage with caveats.etc. For my program, population level staging for all stageable cases is a necessity. This is a cancer agency that reports on cancer, how can we not have the appropriate data? How can we plan capacity and projects without knowing this information? (Cancer Surveillance is in A&I) 21

22 Regional Programs Respondents We use stage data to identify who survivors are -- we use it to exclude stage 4 patients from our cohort. Additionally, we use it for our best practice follow up guidelines for colorectal as they are only applicable for stages 2 and 3. Also helpful for stage 4 palliative care. Stage isn t used now but do need to use it in near future to prove MOHLTC initiated Quality Based Procedures. Want to use stage for outcomes, and appropriate testing (pathway concordance), and diagnostic testing protocol verification Staging data is extremely complicated and there is not a way to easily understand how it works, or to decide RCC vs. CS but RCC data is reportedly of poor quality Would use stage more if it were more complete for my cancers of interest, if it were higher quality and if I understood it better About 70% of my work requires stage. We are initiating a whole new body of measurement work in survivorship- for which stage is very important. Stage is going to be used for performance measurement but also for applied purposes, such as the breast screen expansion. 22

23 Clinical Program Respondents 70% of survivorship work requires stage.70% of Indicator development requires stage Prefer one system, either TNM or CS. New measurement work in survivorship. New performance measurement generally and for specific, such as the breast screening. Would also be important for palliative care. We have found it difficult to understand why stage has not been available for RCC cases. Population staging for all stageable cases is required to determine healthcare policy and confirm the necessity of screening programs. Assessment of quality is stage dependent. it would be better to have one system, all CS or all TNM. Do we know if we need population level staging for all cancers? Is it worth the cost? How can we determine this? Stage information is important for guideline adherence, data quality indicators, and patient population identification. We use RCC and CS. If the data is available in CS, we prefer it. Quality of RCC stage and case has worsened over time. We know nothing about the completeness or validity of stage data % of our work would benefit from stage, and we daily have work arounds to get around the fact that we don t have stage. 23

24 P&CC Respondents Quality of the CS data sounds decent but it is only available for four cancers. Our cancers of interest are not staged to population level, so RCC staging always insufficient. If it is incomplete, it is bad science. And only using a poor sample introduces bias that can result in false conclusions. I usually use CS first and if nothing else, use RCC. I don t go out of my way to use RCC. Regarding quality, CS is a 7, RCC TNM 2. The proportion of my work that requires stage is quite low, about 15%, but I wish it was higher. I have all of these survival analyses involving stage that are just too difficult w/o stage. I have had difficulty understanding why stage is not available for an RCC case. Screening programs only use CS stage and the SSFs, at least 20% of breast screening analysis involves stage. Stage at diagnosis is a big driver of cancer treatment cost, will help us better prepare and demonstrate value of interventions to improve survival. 24

25 Legal, Finance and Audit portfolio respondents Stage is not understood directly by Legal & Privacy Office or Finance. The Internal Auditor conducted value for money review of CS staging (to evaluate efficiency, effectiveness, etc. of programs). The value for money audit was in support of our corporate strategy, looking at stage s internal value. Peer reviews of the CS data appeared to be 9/10. A limitation is that RCC staging is unknown quality and not directly comparable. I do not believe the main problem is the understanding of stage, rather there are issues around the use of stage and the value of collecting the data. It is only useful if various programs are using it to support their program decisions and that is where the uncertainty is. I believe that it would only be useful if there was a program at CCO that supports the collection of the data. If a program is not using it, then its relevance/value will be put into question. Regarding population level staging for all stageable cases is nice to have. I think to achieve 100% is unrealistic and the issue is really, what level of data do you need to say that is it representative of what s actually happening? Conclusion of value for money audit foreshadowed the switch to all AJCC staging, suggested sharing staging with RCCs, as current segregation was wasteful. Clinical Programs read the audit, A&I largely did not, nor did Regional Programs, nor P&CC. 25

26 External User Respondents Staging pathologists When consulting RCC staging consider it excellent (note this is at source, not staging submitted to CCO) While CCO has fallen down in trying to sustain RCC staging. Not everyone is using the most current version among hospitals. As well, the resources and the expertise varies. Some hospitals are below the minimum standard of the hospital reporting we should be doing. Timeliness can also be an issue. Clinician staging champions I use stage for clinical patient care, and stage collection for the Canadian Partnership Against Cancer (CPAC). Specific uses of CS: Irrelevant because it is not going to be available after It is hard to understand why stage is not available for an RCC case, because without it cases cannot be re-staged upon relapse. 26

27 External User Respondents Clinical researchers Need to integrate stage into quality improvement research (external research and treatment protocol verification). They usetnm stage, and either stage abstract at source. In the past they were able to obtained better RCC staging from the OCR. 80% of research requires stage fairly common observation. I use stage data for health services research in cancer we did three province wide studies that involved abstracting stage ourselves too expensive. I always use TNM only. RCC TNM information is not complete and we would have to abstract it from the cancer centers ourselves. Population level staging is important for my work as a researcher. It is also important for all sites, as we need to understand if we are doing better or worse at a population level with improving outcomes for patients. If I do not use stage, it is because it is not available for some cases or specific years are not available. Completeness is the problem area in my opinion. I prefer to use CS, because RCC doesn t have quality checks in place. We must use it to demonstrate that this is a worthwhile investment for the ministry. 27

28 Appendix 2 Interview Questions Embedded Word Document 28

29 Usefulness of Scan Fit for which use? This study needs analytic support to be publication ready Regardless, for internal use, it is adequate now Having names and positions of all respondents, findings could be put before corporate CCO as a mirror of today CCO has complex choices to make these are not mutually exclusive Staging for provincial usefulness of immediate mandates Staging of more, selected cancers to population level The opportunity provided by reversion to AJCC sharing currently OCR CS staged cases with RCCs, giving OCR ability to stage other cancers, again sharing with RCC Moving outward to other tertiary care hospital (CCO systemic funding) Actively supporting its goal of many years of providing complete population staging 29

30 Conclusion Without meaning to, agencies can become internally siloed. They fail to think corporately. They do not collaborate on decisions that mutually effect various portfolios. At CCO, financial constraints in one area, resulting in defunding of an RCC staging analytic watch dog and engagement support, have resulted in RCC stage degrading just as the opportunity arises to merge OCR staging with it. CCO has to decide, consciously, as one co-dependent entity, and with consideration of their influential external stakeholders, to what extent the agency will support staging in the field. It seems sensible to start with the RCCs since they have a contractual agreement to supply stage. The advent of AJCC 8 th edition presents CCO with an opportunity to support RCCs to achieve their mandated data submission. Quality RCC staging fulfills many of the current provincial needs. However, it does not completely support research as RCC attendance at diagnosis is less than 50% (over a patient s lifetime it is 70%). Nor does this solution alone complete the surveillance picture either for Ontario or for Canada as a whole. 30

31 Acknowledgements CCO Research Office for providing funding for what is essentially a business case and communications project Ariane Carmona University of Toronto Master s Graduate, a seasoned interviewer, specializing in patient contact studies in the mental health field Bogdan Pylypenko OCR Senior Data Analyst, providing ongoing analytic and statistical support Mary Jane King, MPH, CTR, Manager, Ontario Cancer Registry ocrquestions@cancercare.on.ca 31

Role Description: Regional Colon Cancer Screening/GI Endoscopy Clinical Lead

South West Regional Cancer Program London Health Sciences Centre E4-100 Wing 800 Commissioners Road East London, Ontario N6A 5W9 Role Description: Regional Colon Cancer Screening/GI Endoscopy Clinical