Overview of WHO/UNICEF Immunization Coverage Estimates

Overview of WHO/UNICEF Immunization Coverage Estimates Marta Gacic-Dobo, Anthony Burton, David Durrheim Meeting of the Strategic Advisory Group of Experts on Immunization (SAGE) 8-10 November 2011 CCV/CICG, Geneva

Data sources

Measuring immunization coverage Measles vaccination coverage=82% Administrative method Survey method Number of doses administered through routine services Number of population in target group Number of children in the sample vaccinated Number of children in the sample 3

Advantages and disadvantages of administrative and survey methods Administrative method l Advantages: Based on data necessary for service provision Timely management monitoring tool Provides data at local level l Disadvantage / Limitations : Denominator (target population may be projected based on old census data) Transcription or calculation errors Incomplete reporting May Include vaccination conducted outside the target group. May not include private sector 4 Survey method l Advantages: Estimate of immunization coverage can be obtained if the denominator is unknown. Provides additional information on social economical status of reached and unreached children Vaccinations given by the private sector reflected l Disadvantage / Limitations: Provides information on the previous birth year s cohort. Immunization card availability Reliance on recall in absence of card Interviewer interaction Length or complexity of the questionnaire may compromise accuracy Representativeness of sample

Background: WHO/UNICEF estimates of routine infant immunization coverage l Begun in 1999, methods were reviewed, approved, and first released in 2001, updated annually since 2001. l WHO & UNICEF joint endeavour. Produced as part of the WHO & UNICEF annual review of national immunization coverage. l Country-specific for 194 countries and territories. l Estimates are made for routine coverage: BCG, DTP1, DTP3, Polio3, measles, HepB3 DTP1, Hib3 (from 2005) PCV3, Rota last dose, Yellow Fever (from 2010) 5

Background: WHO/UNICEF estimates of routine infant immunization coverage l Reflects routine immunization system performance, does not include campaign or non-routine doses. l Annual revisions reflect updated estimates if new information is available. l Based on an evaluation of data. No statistical or mathematical model is used; estimate is based on a set of heuristics supplemented with judgment. l Does not "borrow" data from other countries. l Attempts to incorporate private sector vaccinations where possible. l Independent assessment: Estimates are sent to national authorities for review but national approval is not required. 6

7 External reviews of the method l 12-13 July 2001: External Review of methods and findings of a review of national coverage data from 1980-1999 l July 2009: Method published Burton A, et al WHO and UNICEF estimates of national infant immunization coverage: methods and processes. Bull World Health Organ 2009; 87:535-541 l 12-13 October 2009: QUIVER Review of current method and ongoing activities to improve transparency and estimation methods Proposed approach found appropriate l 6 October 2011: QUIVER update Knowledge, Representation, Reasoning system method endorsed Call for validation (sero surveys, surveillance data) Endorsed use of quality description of estimates rather than quantitative ranges

Estimation method

Annual review of coverage data l National reports (JRF) Administrative coverage data Country official estimates l Published and grey literature DHS, MICS (UNICEF), other surveys l Additional information Stock-outs Data quality audits results l Local knowledge Rules Result: WHO/ UNICEF estimates of routine infant immunization coverage (WUENIC) 9

Estimation methods l Estimate = reported data if no other data OR other data do not challenge reported data l Reported data challenged if Inconsistent with quality survey results Unusual temporal changes Coverage for different vaccines inconsistent (DTP3 OPV3) l Decision: what is most consistent with the time series, what are the most likely biases (denominators etc)? l No averaging of results from multiple sources l 100% vaccination coverage not achievable l Include private sector (becoming increasingly important) 10

Formalization of the estimation methods: (Knowledge Representation and Reasoning) l Formal knowledge representation and reasoning (KRR) system to ensure estimates are documented, replicable, consistent, and transparent. l The KRR consists of: direct quantitative coverage data (national reports, survey results) working group decisions including local, context-specific information rules that allow estimates based on data and working group decisions l The KRR is described in natural language, computational logic and an executable programming language l Estimates are based solely on the data, decisions and rules documented by the KRR. l The objective of the KRR is to assists the working group in maintaining the replicability, consistency and transparency of the estimates not to replace the current methods. 11

WUENIC: processing levels l Data and information is represented in atomic sentence reported_data(country,vaccine,year,percent coverage). survey_results(country,vaccine,year,surveyid,description,percent coverage). wgd(country,vaccine,year,action,explanation). l Level I: survey and reported data "cleaning" accept, modify, ignore l Level II: make estimates at anchor points, where there are survey data for a country/vaccine/year. l Level III: make estimates between anchor points. l Level IV: complete the time series 12

Level 1: "cleaning" reported & survey data "cleaning" -- include, modify, ignore l Reported: ignore Coverage > 99% Sudden temporal changes Working group decision to ignore reported data l Survey: ignore Sample size < 300 Evidence is not based on card or on maternal recall Age cohort is not 12-23 or 18-29 months of age Working group decision to ignore survey results l Survey: adjust for recall bias 13

Level 1: Sudden temporal changes Reason to exclude reported: - inconsistent temporal change IF reported_data(t) reported_data(t-1) > threshold IF reported_data(t) reported_data(t-1) > threshold and reported_data(t) reported_data(t+1) > threshold and no wgd(accept reported data(t)) and reported_data(t) reported_data(t+1) > threshold and wgd(accept reported data(t)) THEN reason to exclude reported_data(t) THEN reason to accept reported_data(t) 14

Level 2: Make estimate at anchor point years IF abs(reported_data(t) survey_data(t)) > threshold and no wgd(assign_anchor(t)) THEN estimate(t) = survey_data(t) IF abs(reported_data(t) survey_data(t)) < threshold and no wgd(assign_anchor(t)) THEN estimate(t) = reported_data(t) 15

Level 3: Make estimates between anchor points l Decisions at the anchor points determine the estimates between anchor points: Surveys support reported data Surveys don't support reported data 16

Level 4: Complete the time series l Decisions at the anchor points determine the estimates for the beginning and end of the time series: Survey support beginning and end of the time series Survey supports beginning but not end of the time series 17

Result and uses

Availability and quality of data N of countries % of countries % total surviving infants Estimate = Reported data 151 78% 63% Multiple data source 75 39% 46% Single data source 76 39% 17% Estimate <> Reported 42 22% 37% Survey level reported trend 18 9% 12% Survey for level and trend 14 7% 22% Other adjustment 10 5% 4% Based on DTP3 coverage 2010 19

Country profile www.who.int/immunization_monitoring/routine/immunization_coverage/en/index4.html http://www.childinfo.org/immunization_countryreports.html 20

Global DTP3 reported and WHO/UNICEF estimated coverage, 1990-2010 100 90 80 70 60 50 40 Global_reported Global estimated 30 20 10 0 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Source: WHO/UNICEF coverage estimates, 2010 revision as of July 2011 21

22 WHO/UNICEF Estimates of National Immunization Coverage Uses l Official publications to report progress on immunization UNICEF State of the World's Children. WHO World Health Statistics / World Health Report. GAVI Annual Progress Report l Monitor international goals MDG 4: reduction of child mortality, indicator #15: Percent measles immunization coverage. GIVS goal: 90% national immunization coverage l Funding agencies Millennium Challenge Corporation / GAVI l Disease burden models: measles, NT, pertussis, Hib l Other uses? data are publicly available.

Cautions

Reporting completeness (% reports received per country, year, antigen) 200 180 number of countries 160 140 120 100 80 60 40 20 0 74 65 29 16 4 2 4 <50 50-<60 60-<70 70-<80 80-<90 90-<100 100 number of countries cumulative number of countries % of reports received 24

Population forecast accuracy l Size matters: Precision increases as population size increases. Population size does not appear to influence bias. l Magnitude and direction of change matters: Precision is greatest where population growth rates are positive but moderate. l Projection horizon matters: Precision declines as the length of the projection horizon increases. Projection horizon does not appear to introduce bias. l Projection method doesn't matter. l Averaging results matters: Averaging estimates from different methods improves precision. 25

80% of countries have had a census within the last 12 years Number and cummulative number of countries 150 100 50 0 1 4 7 10 13 16 19 1983 Myanmar 1984 DRC, Eriteria 22 25 28 31 1979 Afghanistan 34 37 1970 Lebanon, Angola 40 26 Years since last census

Coverage surveys: number of years since most recent coverage survey, n=194 200 180 number of countries 160 140 120 100 80 60 40 20 0 0 2 7 21 19 31 21 6 3 1 3 79 n of countries cumulative number of countries 0 1 2 3 4 5 6 7 8 9 10 >10* * More than 10 years or no survey number of years since last survey 27

Alternatives to maintaining a time series l Estimates based on data available at the time and no subsequent changes. Data updates or new information not incorporated l Estimates lagged to allow complete reporting e.g., in 2010 make estimates up to 2006 l Update the entire time series as more complete or accurate data becomes available Previously made estimates may change 28

Update of historical data Estimates based on data up to 2008 Estimates based on data up to 2009 29

Magnitude of change in 2009 country estimates - comparison of 2009 estimate and 2010 revision -30-20 -10 0 10 20 30 change in % coverage 30

44 countries had changes in their 2009 estimates in 2010 revision l Reported data was not available at the time of the estimates (4 countries - 9%) l Survey data was not available at the time of the estimates (9 countries - 20%) l Update received from country (21 countries - 48%) l Change in estimation methods (10 countries - 23%) 31

Further improving estimates

Next steps to improve coverage estimates l Improve data quality at sub-national and national level Reliable data from all countries (preferably quality administrative data periodically confirmed by a survey) l Improve country consultation process More direct contact with regions / countries to review coverage estimates Use EPI managers' meetings and other opportunities to use country expert opinion l Improve current estimation methods Improve current rule system 33 Address uncertainty

34 Describing estimate quality l The certainty of the WHO & UNICEF estimates vary between countries and, within countries, over time. l Alternatives to expressing different levels of confidence. Probability-based estimates of uncertainty: 1) precision and 2) confidence levels. e.g., the 95% confidence interval is 116,400 366,700. Alternative estimates based on different assumptions e.g., the UN Population division produces population forecast under a variety of assumptions regarding fertility. A linguistic grading of the estimate based on the scope, quality and source of information.

Levels of confidence l Fairly confident estimates are well supported, precision is high Multiple sources, consistent data no inconsistent data l Somewhat confident precision is lower, some conflicting information or lacking confirmatory data. Single consistent relevant data source Multiple sources, majority consistent l Not confident data inconsistent. Precision is low. Inconsistent single data source Multiple sources, inconsistent. 35

"Coverage levels measured by complete and accurate administrative data, validated regularly by high quality surveys would make the WHO & UNICEF estimates unnecessary; consolidation, analysis and dissemination would become the major contribution."

Questions to SAGE l What recommendations does SAGE have regarding the appropriate use of the estimates given the availability and quality of empirical data, estimation of percent coverage and revision of the time series given new data and improved methods? l What recommendations does SAGE have regarding further improvement for the estimation method? l What recommendations does SAGE have regarding improving the availability and quality of empirical data? 37

Additional slides

Forecast accuracy mean absolute percentage error (MAPE) Approximately 3 MAPE per 5 year interval. Siegel & Swanson. Methods and materials of demography. Census year + 5 years +10 years +15 years +20 years 39

Validation of the method l Compare estimates with data from countries with known high quality data l Validate with morbidity / mortality / supply statistics l Sero-surveys l Expert opinion 40

41 Validating coverage with sero-surveys l Conduct literature search for existing results / systematic abstraction. l Conducting sero-surveys Difficulty in distinguishing natural vs. artificial immunity is an issue. Cost/difficulty of interpretation makes sero-survey results problematic for only validating coverage. Should also provide important programmatic information (measles/polio/ pertussis) Biomarkers/interpretation for "new vaccines" S. pneumo, Hib, rotavirus. Tetanus appears to be the most robust marker. Difficulty of generalization

Using biomarkers to validate coverage: the Tajikistan serosurvey l Validating coverage is not the same as establishing a population susceptibility profile. Difficulty of translating biological response to immunization history (individual variation, cut-off values, assay choice). Most sero-markers cannot distinguish natural from artificially induced immunity (e.g., polio, measles) Ability to stratify by age is important to account for waning immunity and to establish immunization cohort. Ethical and logistics issues must be addressed. l Tetanus for children under five years of age, stratified by age, from a geographically representative sample is most likely to provide most direct information.

43 Experience from Madagascar "In the 2003-04 Madagascar DHS, women and kids were tested for tetanus and measles. Based on the results (see below), we found that the % vaccinated according to the interview was higher (overestimated??) than the % truly immunized (based on the test) among reported fully vaccinated 91-93% percent seropositive. We do not know the reasons of the differences: 1) overestimation of the vaccination based on the card/report (women reported an undetermined injection as a tetanus or measles vaccination), or 2) woman or child truly received the vaccination, but the vaccine did not work and the test was negative (i.e. there was no over-reporting). The survey was very complicated to implement and very costly and I do not think that we should repeated this type of test since we are not able to make any useful conclusion (overestimation, or problems with the vaccine)."

Coverage and diseases outbreak consistency Polio outbreaks in African region between 2008-2010. 22 countries reporting polio cases 50% estimates were lower than reported data 23 countries with no polio outbreaks 35% estimates were lower than reported data % cooverage 100 90 80 70 60 50 40 30 20 10 0 Chad DR Congo Angola Congo Guinea Niger Côte d Ivoire Benin Burkina Faso Kenya Mauritania Senegal CAF Liberia Uganda Sierra Leone Togo Ghana Mali Ethiopia Cameroon Coverage averaged between 2008-2010 Burundi 140 120 100 80 60 40 20 0 reported polio cases polio official 08-10 estimate 08-10 % cooverage 100 90 80 70 60 50 40 30 20 10 0 Equatorial Guinea Gabon South Africa Guinea-Bissau Mozambique Madagascar Zimbabwe Comoros Zambia Lesotho Namibia Tanzania Rwanda Swaziland Malawi Algeria Botswana Gambia Sao Tome & Principe Cape Verde Eritrea Mauritius Seychelles 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 reported polio cases polio official 08-10 estimate 08-10 45

KRR data decisions and rules may be challenged l Data and information may be incorrect, out of date, or missing. l One may disagree with the decisions of the working group. l One may disagree with the relevance, appropriateness or application of the rules. l One may disagree with the final estimates based on rebutting information. 46

Changes in estimates, comparison of 2009 revision and 2010 revision, Nigeria 2009 revision 2010 revision 47

Changes in estimates, comparison of 2009 revision and 2010 revision, Afghanistan 2009 revision Revised official estimates 2010 revision 48