Overview of WHO/UNICEF Immunization Coverage Estimates

Transcription

1 Overview of WHO/UNICEF Immunization Coverage Estimates Marta Gacic-Dobo, Anthony Burton, David Durrheim Meeting of the Strategic Advisory Group of Experts on Immunization (SAGE) 8-10 November 2011 CCV/CICG, Geneva

2 Data sources

3 Measuring immunization coverage Measles vaccination coverage=82% Administrative method Survey method Number of doses administered through routine services Number of population in target group Number of children in the sample vaccinated Number of children in the sample 3

4 Advantages and disadvantages of administrative and survey methods Administrative method l Advantages: Based on data necessary for service provision Timely management monitoring tool Provides data at local level l Disadvantage / Limitations : Denominator (target population may be projected based on old census data) Transcription or calculation errors Incomplete reporting May Include vaccination conducted outside the target group. May not include private sector 4 Survey method l Advantages: Estimate of immunization coverage can be obtained if the denominator is unknown. Provides additional information on social economical status of reached and unreached children Vaccinations given by the private sector reflected l Disadvantage / Limitations: Provides information on the previous birth year s cohort. Immunization card availability Reliance on recall in absence of card Interviewer interaction Length or complexity of the questionnaire may compromise accuracy Representativeness of sample

5 Background: WHO/UNICEF estimates of routine infant immunization coverage l Begun in 1999, methods were reviewed, approved, and first released in 2001, updated annually since l WHO & UNICEF joint endeavour. Produced as part of the WHO & UNICEF annual review of national immunization coverage. l Country-specific for 194 countries and territories. l Estimates are made for routine coverage: BCG, DTP1, DTP3, Polio3, measles, HepB3 DTP1, Hib3 (from 2005) PCV3, Rota last dose, Yellow Fever (from 2010) 5

6 Background: WHO/UNICEF estimates of routine infant immunization coverage l Reflects routine immunization system performance, does not include campaign or non-routine doses. l Annual revisions reflect updated estimates if new information is available. l Based on an evaluation of data. No statistical or mathematical model is used; estimate is based on a set of heuristics supplemented with judgment. l Does not "borrow" data from other countries. l Attempts to incorporate private sector vaccinations where possible. l Independent assessment: Estimates are sent to national authorities for review but national approval is not required. 6

7 7 External reviews of the method l July 2001: External Review of methods and findings of a review of national coverage data from l July 2009: Method published Burton A, et al WHO and UNICEF estimates of national infant immunization coverage: methods and processes. Bull World Health Organ 2009; 87: l October 2009: QUIVER Review of current method and ongoing activities to improve transparency and estimation methods Proposed approach found appropriate l 6 October 2011: QUIVER update Knowledge, Representation, Reasoning system method endorsed Call for validation (sero surveys, surveillance data) Endorsed use of quality description of estimates rather than quantitative ranges

8 Estimation method

9 Annual review of coverage data l National reports (JRF) Administrative coverage data Country official estimates l Published and grey literature DHS, MICS (UNICEF), other surveys l Additional information Stock-outs Data quality audits results l Local knowledge Rules Result: WHO/ UNICEF estimates of routine infant immunization coverage (WUENIC) 9

10 Estimation methods l Estimate = reported data if no other data OR other data do not challenge reported data l Reported data challenged if Inconsistent with quality survey results Unusual temporal changes Coverage for different vaccines inconsistent (DTP3 OPV3) l Decision: what is most consistent with the time series, what are the most likely biases (denominators etc)? l No averaging of results from multiple sources l 100% vaccination coverage not achievable l Include private sector (becoming increasingly important) 10

11 Formalization of the estimation methods: (Knowledge Representation and Reasoning) l Formal knowledge representation and reasoning (KRR) system to ensure estimates are documented, replicable, consistent, and transparent. l The KRR consists of: direct quantitative coverage data (national reports, survey results) working group decisions including local, context-specific information rules that allow estimates based on data and working group decisions l The KRR is described in natural language, computational logic and an executable programming language l Estimates are based solely on the data, decisions and rules documented by the KRR. l The objective of the KRR is to assists the working group in maintaining the replicability, consistency and transparency of the estimates not to replace the current methods. 11

12 WUENIC: processing levels l Data and information is represented in atomic sentence reported_data(country,vaccine,year,percent coverage). survey_results(country,vaccine,year,surveyid,description,percent coverage). wgd(country,vaccine,year,action,explanation). l Level I: survey and reported data "cleaning" accept, modify, ignore l Level II: make estimates at anchor points, where there are survey data for a country/vaccine/year. l Level III: make estimates between anchor points. l Level IV: complete the time series 12

13 Level 1: "cleaning" reported & survey data "cleaning" -- include, modify, ignore l Reported: ignore Coverage > 99% Sudden temporal changes Working group decision to ignore reported data l Survey: ignore Sample size < 300 Evidence is not based on card or on maternal recall Age cohort is not or months of age Working group decision to ignore survey results l Survey: adjust for recall bias 13

14 Level 1: Sudden temporal changes Reason to exclude reported: - inconsistent temporal change IF reported_data(t) reported_data(t-1) > threshold IF reported_data(t) reported_data(t-1) > threshold and reported_data(t) reported_data(t+1) > threshold and no wgd(accept reported data(t)) and reported_data(t) reported_data(t+1) > threshold and wgd(accept reported data(t)) THEN reason to exclude reported_data(t) THEN reason to accept reported_data(t) 14

15 Level 2: Make estimate at anchor point years IF abs(reported_data(t) survey_data(t)) > threshold and no wgd(assign_anchor(t)) THEN estimate(t) = survey_data(t) IF abs(reported_data(t) survey_data(t)) < threshold and no wgd(assign_anchor(t)) THEN estimate(t) = reported_data(t) 15

16 Level 3: Make estimates between anchor points l Decisions at the anchor points determine the estimates between anchor points: Surveys support reported data Surveys don't support reported data 16

17 Level 4: Complete the time series l Decisions at the anchor points determine the estimates for the beginning and end of the time series: Survey support beginning and end of the time series Survey supports beginning but not end of the time series 17

18 Result and uses

19 Availability and quality of data N of countries % of countries % total surviving infants Estimate = Reported data % 63% Multiple data source 75 39% 46% Single data source 76 39% 17% Estimate <> Reported 42 22% 37% Survey level reported trend 18 9% 12% Survey for level and trend 14 7% 22% Other adjustment 10 5% 4% Based on DTP3 coverage

20 Country profile

21 Global DTP3 reported and WHO/UNICEF estimated coverage, Global_reported Global estimated Source: WHO/UNICEF coverage estimates, 2010 revision as of July

22 22 WHO/UNICEF Estimates of National Immunization Coverage Uses l Official publications to report progress on immunization UNICEF State of the World's Children. WHO World Health Statistics / World Health Report. GAVI Annual Progress Report l Monitor international goals MDG 4: reduction of child mortality, indicator #15: Percent measles immunization coverage. GIVS goal: 90% national immunization coverage l Funding agencies Millennium Challenge Corporation / GAVI l Disease burden models: measles, NT, pertussis, Hib l Other uses? data are publicly available.

23 Cautions

24 Reporting completeness (% reports received per country, year, antigen) number of countries <50 50-<60 60-<70 70-<80 80-<90 90-< number of countries cumulative number of countries % of reports received 24

25 Population forecast accuracy l Size matters: Precision increases as population size increases. Population size does not appear to influence bias. l Magnitude and direction of change matters: Precision is greatest where population growth rates are positive but moderate. l Projection horizon matters: Precision declines as the length of the projection horizon increases. Projection horizon does not appear to introduce bias. l Projection method doesn't matter. l Averaging results matters: Averaging estimates from different methods improves precision. 25

26 80% of countries have had a census within the last 12 years Number and cummulative number of countries Myanmar 1984 DRC, Eriteria Afghanistan Lebanon, Angola Years since last census

27 Coverage surveys: number of years since most recent coverage survey, n= number of countries n of countries cumulative number of countries >10* * More than 10 years or no survey number of years since last survey 27

28 Alternatives to maintaining a time series l Estimates based on data available at the time and no subsequent changes. Data updates or new information not incorporated l Estimates lagged to allow complete reporting e.g., in 2010 make estimates up to 2006 l Update the entire time series as more complete or accurate data becomes available Previously made estimates may change 28

29 Update of historical data Estimates based on data up to 2008 Estimates based on data up to

30 Magnitude of change in 2009 country estimates - comparison of 2009 estimate and 2010 revision change in % coverage 30

31 44 countries had changes in their 2009 estimates in 2010 revision l Reported data was not available at the time of the estimates (4 countries - 9%) l Survey data was not available at the time of the estimates (9 countries - 20%) l Update received from country (21 countries - 48%) l Change in estimation methods (10 countries - 23%) 31

32 Further improving estimates

33 Next steps to improve coverage estimates l Improve data quality at sub-national and national level Reliable data from all countries (preferably quality administrative data periodically confirmed by a survey) l Improve country consultation process More direct contact with regions / countries to review coverage estimates Use EPI managers' meetings and other opportunities to use country expert opinion l Improve current estimation methods Improve current rule system 33 Address uncertainty

34 34 Describing estimate quality l The certainty of the WHO & UNICEF estimates vary between countries and, within countries, over time. l Alternatives to expressing different levels of confidence. Probability-based estimates of uncertainty: 1) precision and 2) confidence levels. e.g., the 95% confidence interval is 116, ,700. Alternative estimates based on different assumptions e.g., the UN Population division produces population forecast under a variety of assumptions regarding fertility. A linguistic grading of the estimate based on the scope, quality and source of information.

35 Levels of confidence l Fairly confident estimates are well supported, precision is high Multiple sources, consistent data no inconsistent data l Somewhat confident precision is lower, some conflicting information or lacking confirmatory data. Single consistent relevant data source Multiple sources, majority consistent l Not confident data inconsistent. Precision is low. Inconsistent single data source Multiple sources, inconsistent. 35

36 "Coverage levels measured by complete and accurate administrative data, validated regularly by high quality surveys would make the WHO & UNICEF estimates unnecessary; consolidation, analysis and dissemination would become the major contribution."

37 Questions to SAGE l What recommendations does SAGE have regarding the appropriate use of the estimates given the availability and quality of empirical data, estimation of percent coverage and revision of the time series given new data and improved methods? l What recommendations does SAGE have regarding further improvement for the estimation method? l What recommendations does SAGE have regarding improving the availability and quality of empirical data? 37

38 Additional slides

39 Forecast accuracy mean absolute percentage error (MAPE) Approximately 3 MAPE per 5 year interval. Siegel & Swanson. Methods and materials of demography. Census year + 5 years +10 years +15 years +20 years 39

40 Validation of the method l Compare estimates with data from countries with known high quality data l Validate with morbidity / mortality / supply statistics l Sero-surveys l Expert opinion 40

41 41 Validating coverage with sero-surveys l Conduct literature search for existing results / systematic abstraction. l Conducting sero-surveys Difficulty in distinguishing natural vs. artificial immunity is an issue. Cost/difficulty of interpretation makes sero-survey results problematic for only validating coverage. Should also provide important programmatic information (measles/polio/ pertussis) Biomarkers/interpretation for "new vaccines" S. pneumo, Hib, rotavirus. Tetanus appears to be the most robust marker. Difficulty of generalization

42 Using biomarkers to validate coverage: the Tajikistan serosurvey l Validating coverage is not the same as establishing a population susceptibility profile. Difficulty of translating biological response to immunization history (individual variation, cut-off values, assay choice). Most sero-markers cannot distinguish natural from artificially induced immunity (e.g., polio, measles) Ability to stratify by age is important to account for waning immunity and to establish immunization cohort. Ethical and logistics issues must be addressed. l Tetanus for children under five years of age, stratified by age, from a geographically representative sample is most likely to provide most direct information.

43 43 Experience from Madagascar "In the Madagascar DHS, women and kids were tested for tetanus and measles. Based on the results (see below), we found that the % vaccinated according to the interview was higher (overestimated??) than the % truly immunized (based on the test) among reported fully vaccinated 91-93% percent seropositive. We do not know the reasons of the differences: 1) overestimation of the vaccination based on the card/report (women reported an undetermined injection as a tetanus or measles vaccination), or 2) woman or child truly received the vaccination, but the vaccine did not work and the test was negative (i.e. there was no over-reporting). The survey was very complicated to implement and very costly and I do not think that we should repeated this type of test since we are not able to make any useful conclusion (overestimation, or problems with the vaccine)."

44 Coverage and diseases outbreak consistency Polio outbreaks in African region between countries reporting polio cases 50% estimates were lower than reported data 23 countries with no polio outbreaks 35% estimates were lower than reported data Reported < estimate Reported = estimate Reported > estimate Reported < estimate Reported = estimate Reported > estimate Estimate < 80% 0% 5% 36% Estimate < 80% 0% 4% 30% Estimate > 80 5% 41% 14% Estimate > 80 26% 35% 4% Total 5% 45% 50% Total 26% 39% 35% Coverage averaged between

45 Coverage and diseases outbreak consistency Polio outbreaks in African region between countries reporting polio cases 50% estimates were lower than reported data 23 countries with no polio outbreaks 35% estimates were lower than reported data % cooverage Chad DR Congo Angola Congo Guinea Niger Côte d Ivoire Benin Burkina Faso Kenya Mauritania Senegal CAF Liberia Uganda Sierra Leone Togo Ghana Mali Ethiopia Cameroon Coverage averaged between Burundi reported polio cases polio official estimate % cooverage Equatorial Guinea Gabon South Africa Guinea-Bissau Mozambique Madagascar Zimbabwe Comoros Zambia Lesotho Namibia Tanzania Rwanda Swaziland Malawi Algeria Botswana Gambia Sao Tome & Principe Cape Verde Eritrea Mauritius Seychelles reported polio cases polio official estimate

46 KRR data decisions and rules may be challenged l Data and information may be incorrect, out of date, or missing. l One may disagree with the decisions of the working group. l One may disagree with the relevance, appropriateness or application of the rules. l One may disagree with the final estimates based on rebutting information. 46

47 Changes in estimates, comparison of 2009 revision and 2010 revision, Nigeria 2009 revision 2010 revision 47

48 Changes in estimates, comparison of 2009 revision and 2010 revision, Afghanistan 2009 revision Revised official estimates 2010 revision 48