Monitoring, Evaluation, and Reporting (MER) Guidance (v.2.3): Orphans and Vulnerable Children (OVC)

Transcription

1 Monitoring, Evaluation, and Reporting (MER) Guidance (v.2.3): Orphans and Vulnerable Children (OVC) Presenters: Gretchen Bachman, Sr. Technical Advisor Caroline Cooney, OVC Policy Advisor Erin Schelar, Data Analysis Advisor Date: September 2018

2 Video Outline 1) Section 1: Overview of the technical area. 2) Section 2: Indicator changes in MER ) Section 3: Review of definitions and how to review this data. 4) Section 4: Overview of guiding narrative questions 5) Section 5: Additional Resources and Acknowledgments Please note: FY19 DATIM data entry screens are currently in development. Additional training materials on data entry requirements are forthcoming. 2

3 Section 1: Overview of the technical area and related indicators

4 OVC Programming Approach & Priorities Focus on socio-economic, and health promotion and access interventions that reduce vulnerability, contribute to prevention and risk prevention goals, and support access to and retention in treatment Provide family-centered, comprehensive care through case management and routine monitoring Partner with FBOs, local communities and government counterparts Prevention & protection for 9-14 year olds Finding children who are positive and in need of treatment (likely < age 10); keeping HIV+ children and adolescents on treatment Collaboration with DREAMS for AGYW in select SNUs 4

5 Overview of OVC Indicators Program Area Group Indicator Code Indicator Name Reporting Frequency Testing OVC_HIVSTAT Percentage of orphans and vulnerable children (<18 years old) with HIV status reported to implementing partner (including status not reported), disaggregated by HIV status Prevention OVC_SERV Number of active and graduated beneficiaries served by PEPFAR OVC programs for children and families affected by HIV Semi- Annually Semi- Annually Reporting Level Community & Facility Community & Facility 5

6 OVC Relationship with AGYW_PREV New AGYW_PREV indicator tracks layering of DREAMS services/interventions across budget codes and indicators Complementary to other MER indicators that track receipt of individual services such as OVC_SERV COP Funding Individual Services Received by AGYW DREAMS Layering HKID HVAB HVOP HTS Etc. OVC_SERV PP_PREV PREP_NEW PREP_CURR HTS_TST GEND_GBV PMTCT_STAT KP_PREV Etc. AGYW_PREV 6

7 DREAMS and OVC reporting overlap among year old females by program enrollment Program enrollment OVC services DREAMS services Count under OVC_SERV Count under AGYW_PREV OVC OVC and DREAMS Must meet OVC program requirements Must meet OVC program requirements N/A Must meet DREAMS program requirements; Layered interventions DREAMS only Qualifying eligible individual DREAMS service that is also an OVC service, e.g. education subsidy Must meet DREAMS program requirements; Layered interventions DREAMS only N/A Must meet DREAMS program requirements; Layered interventions 7

8 Section 2: OVC_SERV Indicator changes in MER 2.3

9 OVC_SERV: What s Changed? Change Minimum graduation benchmarks added Beneficiaries must have received at least one service in each of the preceding two quarters from a list of illustrative eligible services Active status requirements updated for <18 to include at least quarterly monitoring, an updated case plan, and receipt of at least one service in each of the preceding two quarters Age bands revised for additional disaggregations <10 and removed Active and graduated disaggregates will be collected by age/sex DREAMS disaggregates removed (OVC_SERV age/sex/service) Timing of data submitted for Q4 graduation, exit/transfer disaggs clarified Definitions and expectations for caregivers and households added Programmatic Rationale for Change Provide OVC programs, OVC caregivers, and vulnerable children definitions for success which the OVC program, caregivers and OVC can agree to work towards Assure that beneficiaries are reached promptly and regularly with specific services Assure that children and adolescents receive substantive, timely, and regular support based on a needs assessment after enrollment Alignment with PEPFAR standard disaggregates for <10; DREAMS beneficiaries will not be included in OVC_SERV so caregivers captured with 18+ Programs collect these data by age/sex and will improve ease of data entry with autocalculation New AGYW_PREV indicator added to MER to capture DREAMS layering Beneficiaries should only be counted once, with graduated and active status taking priority at Q4 To align with graduation benchmarks and right-size portfolio proportions of OVC <18 versus caregivers 9

10 Section 3: OVC_SERV review of numerator, denominator, and disaggregations

11 OVC_SERV Indicator Definition: Number of beneficiaries served by PEPFAR OVC programs for children and families affected by HIV Numerator: Denominator: Number of beneficiaries served by PEPFAR OVC programs for children and families affected by HIV (Active + Graduated) NONE Required Disaggregations: Program Participation Status by Age/Sex: Active: <1 F/M, 1-4 F/M, 5-9 F/M, F/M, F/M, 18+ F/M Graduated: <1 F/M, 1-4 F/M, 5-9 F/M, F/M, F/M, 18+ F/M Transfer/Exit (transferred out to a PEPFAR-supported partner, transferred out to a non-pepfar supported partner, exited without graduation) 11

12 OVC_SERV Program Implications OVC_SERV data is used for congressional reporting Program managers should monitor OVC_SERV data, including Panorama dashboard visuals To assess progress against targets To ensure regular receipt of eligible services based on needs To determine if there are many OVC exiting without graduation, and if so why and how to improve program retention To assess the distribution of beneficiaries by age/sex and program status To identify any open slots in their OVC program for new beneficiaries 12

13 OVC_SERV Panorama Visuals and Data Resources Panorama dashboard visuals for OVC_SERV results by 1) participation status and 2) results by age/sex ICPI OVC dashboard with OU and SNU comparisons 1) 2) 13

14 OVC_SERV beneficiary categories Beneficiary category WHO WHAT WHEN Active children children (ages 0-17) youth who are still in secondary school have received one or more eligible program services have a current case plan (updated within last year) are monitored at least quarterly In each of the past two quarters Or in past quarter if beneficiary was newly enrolled Active caregiver Up to two caregivers of beneficiary child Fulfill role of parent/guardian Active DREAMS only AGYW DREAMS participant who is not also actively enrolled in an OVC program Graduated children (ages 0-17) youth who are still in secondary school caregivers of beneficiary children have received one or more eligible program services have received one or more DREAMS services that is also an eligible OVC_SERV service Do not need an OVC case plan because DREAMS program has separate requirements (see AGYW_PREV) Entire household has achieved all relevant graduation benchmarks (may vary by age and HIV status of beneficiaries) Graduation of the household as a unit reflects the family-centered nature of OVC programming Same as above Same as above Graduated beneficiaries should be reported for previous 2 quarters at Q2 and for entire FY at Q4 14

15 8 Minimum Graduation Benchmarks Known HIV status (or test not required) Adherent/Virally suppressed Knowledgeable about HIV prevention Safe Healthy Not malnourished No violence reported in past 6 mos. Not in a child-headed household Stable Schooled Financially stable Children in school See MER v2.3 OVC_SERV indicator reference sheet and appendix for complete listing of minimum graduation benchmarks 15

16 How will OVC_SERV be calculated? MER follows WHO person-centered monitoring, so OVC_SERV follows individual beneficiary status At Q4 each individual should only be counted once Active beneficiary Graduated OVC_SERV total In DATIM at Q2 and Q4 report children and caregivers by age/sex who have received at least one eligible service in each of the past two quarters At Q2 in DATIM: Report children/caregivers by age/sex that graduated in the past two quarters At Q4 in DATIM: Report children/caregivers by age/sex that graduated in the past four quarters 16

17 OVC_SERV: Transfer/exit disaggregates Partners will continue to report in DATIM on transfers/exits to ensure work that may not have spanned to the end of the fiscal year is counted and understood These numbers are not included in OVC_SERV total Transferred to a PEPFARsupported partner Transferred to a non- PEPFAR supported partner Exited without graduation At Q2 in DATIM: Report children/caregivers that transferred out to a PEPFARsupported partner in the past two quarters At Q4 in DATIM: Report children/caregivers that transferred out to a PEPFARsupported partner in the past four quarters At Q2 in DATIM: Report children/caregivers that transferred out to a non-pepfar supported partner in the past two quarters At Q4 in DATIM: Report children/caregivers that transferred out to a non-pepfar supported partner in the past four quarters At Q2 in DATIM: Report children/caregivers that exited without graduation in the past two quarters At Q4 in DATIM: Report children/caregivers that exited without graduation in the past four quarters who did not return to active status 17

18 Timeline scenario 1: active then graduated Samuel, age 10, and his mother have been actively receiving OVC services since last year. Q4 Samuel receives an eligible service + monitoring + case plan upto-date Q1 Q2 Q3 Q4 Samuel receives an eligible service + monitoring + case plan upto-date Samuel receives an eligible service + monitoring + case plan upto-date Samuel and his mother meet all benchmarks to be counted as graduated Count as active at Q2 reporting Count as graduated at Q4 reporting 18

19 Timeline scenario 2: exit without graduation Rose, age 14, enrolls with her grandmother for the first time in Q1 but does not receive continuous services throughout the year. Q1 Q2 Q3 Q4 Rose and her grandmother assessed, enrolled, and case plan developed Not yet eligible to count under OVC_SERV Rose receives an eligible service + monitoring Rose does not receive a service Count as active at Q2 reporting Rose receives an eligible service + monitoring Count as exited without graduation at Q4 reporting 19

20 Timeline scenario 3: changing status Kabongo, age 16, has been enrolled and receiving OVC services for two years at the beginning of the fiscal year. Kabongo receives an eligible service + monitoring + case plan up-todate Q4 Kabongo receives an eligible service + monitoring + case plan up-todate Kabongo does not receive an eligible service Kabongo receives an eligible service + monitoring + case plan up-todate Kabongo receives an eligible service + monitoring + case plan up-todate Q1 Q2 Q3 Q4 Count as exited without graduation at Q2 reporting Count as active at Q4 reporting; for exited without graduation Q4 reporting DO NOT INCLUDE in DATIM but include in the narrative 20

21 What is an eligible OVC service? A service that the beneficiary: has received directly from the project was facilitated to obtain (provided transport or accompanied) or has a completed referral for PEPFAR provides a list of illustrative eligible interventions* by domain (healthy, safe, schooled, stable) by beneficiary population *for services that are not captured in the list, local USG funding agency approval must be received in order to count these services toward active OVC status 21

22 Reporting scenario for eligible OVC services Brenda lives with her grandmother, who is her primary caregiver. They are both enrolled in an OVC program. How should they be counted, using the list of eligible services, if they receive the following? Brenda receives an education subsidy at the beginning of Q1 to cover her full year of school. Her grandmother receives HIV adherence and disclosure support at least quarterly. What if Brenda does not receive an education subsidy but she and her grandmother receive family insurance coverage? Excerpt from PEPFAR list of illustrative services for active beneficiaries 22

23 Reporting scenario for eligible OVC services Brenda receives an education subsidy at the beginning of Q1 to cover her full year of school, so can be counted as ACTIVE at Q2 and Q4. Q1 Q2 Q3 Q4 Brenda receives an educational subsidy + monitoring + her case plan is updated for the fiscal year Brenda receives monitoring and attends school due to the bursary received in Q.1 Brenda s grandmother receives HIV adherence support in all four quarters, so can be counted as ACTIVE at Q2 and Q4. Q1 Q2 Q3 Q4 Grandmother receives disclosure support Grandmother receives disclosure and adherence support Brenda receives monitoring and attends school due to the bursary received in Q.1 Grandmother receives adherence support Brenda receives monitoring and attends school due to the bursary received in Q.1 Grandmother receives adherence support Q. What if Brenda does not receive an education subsidy but she and her grandmother receive family insurance coverage? 23

24 Section 4: OVC_HIVSTAT indicator changes in MER 2.3

25 OVC_HIVSTAT: What s Changed? Change Test not indicated disaggregate renamed to Test not required based on risk assessment No HIV status reported to the implementing partner (HIV status unknown) will no longer have subdisaggregates: test not required based on risk assessment will be its own disaggregate and other reasons will be removed Programmatic Rationale for Change Simplify language for clarity Disaggregate categories caused confusion; Review performance by creating a category of known status/risk (by combining those reported HIV-positive, HIV-negative, and those for whom a test is not required based on risk assessment) and compare this with OVC_SERV <18 HIV positive OVC for whom ART status is not documented will be reported under Reported HIV Positive -- Not currently receiving ART or ART status unknown OVC for whom HIV status is missing will be reported under No HIV status reported to the implementing partner (HIV status unknown) Children not on ART and for whom ART status is unknown require similar follow up for partner; provides clear instruction for data entry on where to include missing data Simplifies indicator to make it clear where missing data should be included, and narrative will collect additional information on reasons for HIV status unknown HIV positive category will be auto-calcuated from the disaggregates Reported HIV Positive -- currently receiving ART and Reported HIV Positive -- Not currently receiving ART or ART status unknown Clarifications made to highlight risk assessment Reduce data entry burden since data entry clerks only enter data in one place instead of in two places and ensure data quality and consistency to eliminate completeness issues Risk assessment programmatically important to find those who do not know their status and need to be tested 25

26 Section 5: OVC_HIVSTAT review of numerator, denominator, and disaggregations

27 OVC_HIVSTAT Indicator Definition: Percentage of orphans and vulnerable children (<18 years old) with HIV status reported to implementing partner Numerator: Number of orphans and vulnerable children (<18 years old) with HIV status reported, disaggregated by HIV status Denominator: Number of orphans and vulnerable children reported under OVC_SERV (<18 years old, total numerator including active and graduated) Required Disaggregations: Reported HIV positive Currently receiving ART Not currently receiving ART or ART status unknown Reported HIV negative Test not required based on risk assessment No status reported to implementing partner (HIV status unknown) 27

28 The collection of OVC_HIVSTAT data ensures that implementing partners are contributing to by: Identifying children who need HIV treatment Encouraging families to disclose to IP to enhance service delivery including adherence support (but never requiring disclosure as a criteria to receive services) Goal of monitoring OVC_HIVSTAT is to increase the proportion of children with a known HIV status or for whom an HIV test is not required 28

29 OVC and Testing Risk assessments ensure children who need treatment are found, and that there is targeted use of scarce HIV testing resources. An HIV risk assessment should always occur prior to HIV testing to determine if a test is required OVC_HIVSTAT does NOT provide positivity or yield because results are self-reported OVC IPs are not required to verify self-reports with clinical results 29

30 OVC, Risk Assessment, and Testing Implementing partners should conduct HIV risk assessment among OVC at intake, and then as often as needed, depending on child s age & circumstances How often should each of the following children be risk assessed? 16 year old girl, parents are HIV negative Two year old boy, HIV exposed at birth, tested negative through EID, no longer breastfeeding Eight year old girl, orphaned, parent s HIV status unknown, never tested 30

31 HIV Risk Assessment 1. Register OVC & elicit HIV status 2. Conduct HIV Risk Assessment 3. Refer atrisk children to testing 4. Elicit test result & document OVC may enter the project without a known HIV status Before being referred for HIV testing, each OVC should be assessed for HIV risk HIV risk assessment should occur as often as needed, depending on child s age & circumstances If they are determined to be at risk, they must be referred for HIV testing OVC programs must support the completion of the HIV referral, provide support for family centered disclosure, and document self reported test results Disclosure to IP is encouraged but not required 31

32 OVC_HIVSTAT risk assessment flow OVC who are not assessed Test not required based on needs assessment Never referred Referral not complete HIV status not disclosed or documented HIV egative* HIV positive* HIV Unknown Assessed for HIV risk At risk for HIV Referred for HIV testing HIV referral completed HIV status disclosure OVC are lost at every step of the flow Goal of monitoring OVC_HIVSTAT is to increase the proportion of children with a known HIV status or for whom an HIV test is not required Disclosure not required for services, but disclosure support and counseling critical *Not shown to scale 32

33 OVC_HIVSTAT disaggregate: Reported HIV positive Who: OVC beneficiaries <18 years old What: report to the IP that they are HIV positive based on an HIV test When: the HIV test may have been conducted during or prior to the reporting period ART status should be re-assessed at least once each reporting period to ensure updated adherence information Where: regardless of where the test occurred How: guardian or OVC self-reports HIV status (regardless of where test occurred) IP enters ART treatment status disaggregates into DATIM and reported HIV positive is auto-calculated 33

34 OVC_HIVSTAT disaggregate: Reported HIV positive, currently on ART Who: HIV positive beneficiaries <18 years old What: Report to the IP that are currently receiving ART When: The HIV treatment status must be updated in each reporting period How: Guardian or OVC self-reports that the child is currently on ART 34

35 OVC_HIVSTAT disaggregate: Reported HIV positive, not currently on ART or ART status unknown Who: HIV positive beneficiaries <age 18 What: Report to the IP that are not currently receiving ART or are not able to confirm ART treatment status The new definition of HIV positive not currently on ART includes all HIV positive OVC for whom ART status is missing When: The HIV treatment status must be updated in each reporting period How: Guardian or OVC self-reports that child is not currently on ART 35

36 OVC_HIVSTAT disaggregate: Reported HIV negative Who: OVC beneficiaries <age 18 What: report to the IP that they are HIV negative based on an HIV test When: a negative HIV test result is valid unless the implementing partner suspects that child s risk has changed Where: regardless of where the test occurred How: Guardian or OVC self-reports HIV status (regardless of where test occurred) 36

37 OVC_HIVSTAT: Test not required based on risk assessment Formerly known as test not indicated and a disaggregate of unknown status Who: OVC beneficiaries <age 18 What: based on an HIV risk assessment by IP, OVC does not require an HIV test during reporting period When: a status of test not required is valid unless the implementing partner suspects that child s risk has changed Where: if possible, at OVC s household or another place where there is visual and auditory privacy How: by IP, using a standardized data collection tool to guide the risk assessment 37

38 OVC_HIVSTAT disaggregate: No status reported to implementing partner (HIV status unknown) No longer includes sub-disaggregates Other reasons and Test not indicated Who: OVC beneficiaries <age 18 What: OVC or guardian does not know OVC HIV status or HIV status is missing When: during reporting period How: self report 38

39 OVC_HIVSTAT disaggregate: No status reported to implementing partner (HIV status unknown) Formerly the disaggregate known as No status other reasons IP may choose to collect and report on these possible reasons for HIV status unknown in the narratives: 1. HIV unknown: OVC enrolled but not yet assessed for HIV risk 2. Refuse HIV assessment: OVC have been approached, but did not agree to answer the risk assessment questions 3. At risk for HIV: OVC have been assessed and are at risk for HIV, but have not yet accepted referral for HIV testing 4. HIV referral: OVC have accepted HIV testing referral, but have not yet completed the test 5. HIV referral completed: OVC have completed the HIV test although test result is not available. 6. Refuse report: OVC have been approached by community volunteer but have not yet agreed to share test result 7. Missing: no available data 39

40 Current Panorama visual Test not indicated and other reasons disaggregates broken out separately Challenges with reporting completeness and missing data 40

41 Current ICPI dashboard visual Currently several different views are necessary to consider completeness of reporting/missing data, and the subdisaggregates for HIV positive and no status 41

42 How to Count OVC_HIVSTAT in FY19 OVC_SERV 18+ Missing HIV Data Known status proxy, 92% HIV status unknown (includes missing) Test not required based on risk assessment Reported positive OVC_SERV OVC_SERV <18 * OVC_HIVSTAT Reported negative Not on ART/status unknown ART coverage FY2018 Q2 OVC_SERV <18+18> OVC <18 HIV Status Reported HIV Status Types Reported Updated visuals anticipated to reflect modifications to disaggregates and highlight important of known status proxy including test not required based on risk assessment How to calculate annual total: use Q4 results On ART 42

43 OVC_HIVSTAT performance metric: Known status proxy HIV positive + HIV negative + HIV test not required based on risk assessment Number of orphans and vulnerable children reported under OVC_SERV (<18 years old) 90% Captures OVC <18 who either have a known status or are deemed not to need a test through the risk assessment Expect to see this metric increase over time and be at least 90% 43

44 Section 6: Overview of guiding narrative questions

45 Context Matters! Narrative questions provide an important opportunity to share additional qualitative information on results and performance IMs and OU teams should answer the narrative questions in detail in order to provide further insight into performance Explain how the indicator changes may have affected expected outcomes or achievement 45

46 OVC_SERV: Guiding Narrative Questions 1. Please explain reasons and context for highest/lowest performing partners performance (i.e. results/target) for OVC_SERV total numerator and OVC_SERV <18, including any programmatic shifts or monitoring updates that were made as a result of the change in indicator guidance for MER v

47 OVC_SERV: Guiding Narrative Questions 2. Please explain results by Program Participation Status: a. For active beneficiaries, were there any interventions that were provided and approved by local USG funding agency that were not included in the illustrative examples (Figures 1 and 2)? b. For graduation, were any of the benchmarks especially challenging to achieve or monitor? If so, which? c. Of those who are reported to be active, what percentage are newly enrolled in the reporting period? Any re-enrollments of those LTFU (i.e. exited without graduation)? If yes, how many? Are any partners especially good at finding and re-enrolling those LTFU? Are these pieces of information tracked by MIS systems? How many active beneficiaries <18 were counted as active because they had received all eligible services but they or other household members did not meet the criteria for graduation? 47

48 OVC_SERV: Guiding Narrative Questions 3. Please explain results by exited/transferred: a. How many beneficiaries exited without graduation? Please explain the reasons for exiting without graduation and try to quantify with percentages if possible. Are there certain partners with higher rates of exiting without graduation? How are you managing this with the partner(s)? b. How many beneficiaries were transferred? To whom (e.g., other NGOs, government support, etc.). Where were beneficiaries transferred? Please provide disaggregates for beneficiaries transferred to specific sources of support. 48

49 OVC_HIVSTAT: Guiding Narrative Questions 1. If the sum of reported HIV negative + reported HIV positive + Test not required based on risk assessment is less than 90% of OVC_SERV <18, please explain why such a high proportion are being reported in the category of HIV Status Unknown (i.e. the performance metric described in the how to use section). Are there certain partners that are struggling with reporting or understanding the disaggregates? How is the Mission responding? 2. Please explain the breakdown of those reported under HIV Status Unknown. What percentage of caregivers refused to disclose a child s HIV status? What percentage represents those who have been referred for testing but do not yet have results? What percentage represents missing data where an implementing partner failed to document the child s HIV status? 49

50 OVC_HIVSTAT: Guiding Narrative Questions 3. For children reported as Reported HIV Positive - not currently on ART or ART Status Unknown, what efforts are being undertaken in response? Are there certain partners with low ART coverage, why? Is this an issue related to community case management? Or are partners having a hard time collecting timely confirmation of treatment status (i.e. missing)? 50

51 Section 7: Data quality considerations for reporting and analysis

52 OVC_SERV: Data quality checks and FAQs Ensure no double counting of the same person by the same partner in the reporting period Disaggregates are mutually exclusive, with graduated and active status highest priority Program Participation Status should not show high deviations from program targets Review IP and site results for deviations from one period to the next which may indicate rapid exit and entry of beneficiaries or high sudden graduation rate in one, versus another period Evolution of OVC_SERV in past few years, reference MER guidance FAQ for additional reporting scenarios and changes since MER v2.2 with questions 52

53 OVC_HIVSTAT: Data quality check #1 HIV positive + HIV negative + HIV status unknown + HIV test not required based on risk assessment Number of orphans and vulnerable children reported under OVC_SERV (<18 years old) 100% Every beneficiary included in OVC_SERV (<18 years old) must be reported under OVC_HIVSTAT in one of the four HIV status categories. No OVC 18+ should be included in OVC_HIVSTAT The new definition of HIV status unknown includes all OVC for whom HIV status is missing for any reason. 53

54 OVC_HIVSTAT: Indicator misconceptions and frequently asked questions OVC_HIVSTAT cannot be used to calculate a yield OVC_HIVSTAT is a measure of self-report of HIV status and is not an indicator of HIV tests performed or receipt of testing results Status disclosure to the IP is NOT a prerequisite for enrollment or continuation in an OVC program Programs must only report HIV status for active + graduated beneficiaries under OVC_SERV 54

55 Section 8: Additional Resources and Acknowledgments

56 Additional Resources 2012 PEPFAR Guidance for OVC Programming OVC Graduation Benchmark Guidance, forthcoming AGYW_PREV MER training video 56

57 Acknowledgments Thank you to OVC focal points, the OVC taskforce and implementing partners, MEASURE Evaluation, and interagency short-term task team members including Amy Aberra, Eniko Akom, Gretchen, Bachman, Kristine Clark, Viva Combs Thorsen, Caroline Cooney, Eboni Galloway, Salewa Oyelaran, and Erin Schelar. 57

58 Thank you