Developing an active and adaptive case-finding procedure for use in coverage assessments of therapeutic feeding programs

Transcription

1 Developing an active and adaptive case-finding procedure for use in coverage assessments of therapeutic feeding programs Mark Myatt and Sophie Woodhead

2 Developing an active and adaptive case-finding procedure This document provides guidance on developing case-finding procedures for use in SQUEAC small-area surveys, SQUEAC likelihood surveys, SLEAC surveys, and CSAS / S3M surveys. The within-community case-finding procedure most often used in SQUEAC small-area surveys, SQUEAC likelihood surveys, SLEAC surveys, and CSAS / S3M surveys is known as active and adaptive case-finding (AACF). The procedure should be: Active : The procedure actively searches for cases rather than just expecting cases to be found in a sample. Adaptive : The procedure uses information found during case-finding to inform and improve the search for cases. Active and adaptive case-finding is sometimes called snowball sampling, optimally biased sampling, chainreferral sampling, or respondent-driven sampling. These types of sampling procedures are used when subjects are hard to sample because they are rare in the population (e.g. cases of severe acute malnutrition) or because of stigma or legal considerations associated with the condition of interest (e.g. injecting drug users). A good active and adaptive case-finding procedure is one which finds all (or nearly all) cases in a sampled community. Designing a good active and adaptive case-finding procedure for cases of severe acute malnutrition (SAM) involves undertaking a semi-quantitative investigation gathering information regarding: Local beliefs about the aetiology (causation) of wasting, severe wasting, and oedema. Treatment seeking behaviours for wasting, severe wasting, and oedema. Local terms for wasting, severe wasting, and oedema. Local persons living in communities that have knowledge about children with wasting, severe wasting, and oedema. You should employ the SQUEAC principals of triangulation by source and method and sampling to redundancy to guide the investigation. Typical sources and appropriate methods are: Table AACF.1 : Sources and methods / sampling plan Source of information Carers of children attending the program Clinical staff (CMAM program and others) Community-based volunteers Village leaders Religious leaders (and their spouses) Traditional healers Traditional birth attendants Family matriarchs Method(s) In-depth interview Group discussions Semi-structured interviews Semi-structured interviews Group discussions Semi-structured interviews Group discussions Semi-structured interviews In-depth interview In-depth interview In-depth interview Group discussions SQUEAC data-collection, data-organisation, and data-analysis techniques such as the BBQ-like tools, mind maps, concept maps, and small studies may prove useful during the investigation.

3 Developing an active and adaptive case-finding procedure : Data gathering Before you start collecting the data needed to develop an active and adaptive case-finding procedure, you should make a sampling plan that lists your sources and methods. Table AACF.1 shows a typical sampling plan. You should develop an interview guide for each source and method in your sampling plan. Box AACF.1, for example, shows an interview guide that has proved useful when collecting data from carers of children attending the program. Note how this interview guide addresses each of our information needs: Local beliefs about the aetiology (causation) of wasting, severe wasting, and oedema. This is addressed by questions (1) and (2). It is also addressed by question (3) because names of diseases are often informed by beliefs about causality. For example, kwashiorkor means supplanted one in the Ga language of West Africa and is often translated as the sickness that the older child gets when the next baby is born. Treatment seeking behaviours for wasting, severe wasting, and oedema. This by addressed in questions (1) and (2). Local terms for wasting, severe wasting, and oedema. This is addressed by question (3). Local persons living in communities that have knowledge about children with wasting, severe wasting, and oedema. This is addressed by question (4). When you make your own interview guides you should check that the interview guides addresses each of these information. If you are carrying out a SQUEAC investigation then the data needed to develop an active and adaptive casefinding procedure can be collected in the first (qualitative) phase of SQUEAC data collection. If you are carrying out a SLEAC or CSAS / S3M survey then you will need a day or two to collect the information needed to develop an active and adaptive case-finding procedure and to test the active amd adaptive case-finding procedure. When collecting and analysing data you should employ: The SQUEAC principals of triangulation by source and method and sampling to redundancy to guide the investigation. SQUEAC data-collection, data-organisation, and data-analysis techniques such as the BBQ-like tools, mind maps, concept maps, and small studies. After you have collected data from a number of sources using a number of methods you should find that there is a lot of agreement between the different sources and the different methods. When this happens you can change your interview guides to focus on clarifying areas of non-agreement. Remember to keep the work focussed on the information needs.

4 Box AACF.1 : Example interview guide for interviews with carers of children in a program (1) How did this child get to be in this program? The intention of this question is to: Elicit a history. Explore local SAM aetiologies. Explore treatment seeking behaviour / pathways to care. The carer may start by describing events around case-finding and referral. Keep this as a reference point during the interview and probe: What happened after that? What happened before that? Encourage narratives / histories. Reflect back responses to elicit further information. (2) Do you know of any children in your village that are like your child but are not attending this program? When asking and following up on this question, refer to / ask about: The index child s specific history (from above). Common SAM aetiologies (e.g. not recovered well after an illness). Specific signs (e.g. thin arms, swollen feet, kwashiorkor signs). Treatment seeking behaviour / pathways to care. Encourage narratives / histories. Reflect back responses to elicit further information. (3) If I wanted to find children like your child and the children we have spoken about, how would I best describe them to other people? The intention of this question is to discover local terms and aetiologies for SAM. Probe for definitions of local terms. Some terms will be descriptive. Other terms may reflect local / folk aetiologies. Give examples of specific signs and ask for local terms. Probe : What does [TERM] mean? Probe : Any other names for this? Probe : Will most people understand if I ask about [TERM]?. (4) If I wanted to find children like your child and the children we have spoken about, who would best be able to help me to find them?

5 Probe : Anyone else?. Make sure you ask directly about midwives, traditional birth attendants, traditional healers, the people mentioned in histories when exploring treatment seeking behaviour / pathways to care, and the people used by the program for case-finding and referral. Probe : Why? and Why not?. Confirm : You are saying that I should ask [PERSON] to take me to see children with [TERMS]. Is that right?

6 Developing an active and adaptive case-finding procedure : Data gathering It is important to realise that interview guides are not survey questionnaires. You are free to follow leads as they occur. You should also adapt interview guides to account for new information. If, for example, you find five common terms for wasting and an interviewee gives you fewer terms then you should ask: I have also heard that some people use the word [TERM]. Have you heard that term? If YES : Is that a good term to use? What is the difference between this term and the terms you have given me? If, for example, you hear a new term, you could ask something like: What does [TERM] mean? Does it mean something different from [KNOWN TERM]? Will many / most people know this term? It is important that you identify which terms are stigmatised terms as you may need to avoid using these terms with some informants during case-finding: Is [TERM] something that people are embarrassed or ashamed about? Is [TERM] something that I can ask about without appearing rude or causing embarrassment? Is there a good way of asking about [TERM]? Remember to employ: The SQUEAC principals of triangulation by source and method and sampling to redundancy to guide the investigation. SQUEAC data-collection, data-organisation, and data-analysis techniques such as the BBQ-like tools, mind maps, concept maps, and small studies.

7 Developing an active and adaptive case-finding procedure : Data gathering and testing After you have collected data using your sampling plan you should be in a position to formulate your casefinding procedure. This has two components: The informant : This is the person who will help you find cases. This will usually be a community health worker, traditional birth attendant, or traditional healer. Care needs to be exercised in the choice of informant. Community leaders are a useful point of entry, but seldom make useful informants. They are most useful in helping you find and recruit useful informants. You should avoid relying solely on community health workers or volunteers that are attached to the program since they may be unable or reluctant to take you to see children that are not in the program and this may result in an upward bias in coverage estimates. The question : This is usually a simple question (addressed to the informant) such as: Can you take me to see children that are sick, thin, have swollen legs or feet, or have recently been sick and have not recovered fully, or are attending a feeding program? using the local terms that you have found earlier. Markers of risk (e.g. orphans, twins, single parents, neglected or abused children, households without land or livestock) may also be included in the casefinding question if indicated. It is important to avoid stigmatised terms (e.g. terms associated with poverty, child abuse or neglect, sexual libertinage, alcoholism, substance abuse) because community members may be reluctant to slander their neighbours to help you find SAM cases. It is important to ask about children attending a feeding program (or specific feeding programs). Failure to do this may result in in a downward bias in coverage estimates. Once you have formulated your case-finding procedure you can test it. The first stage of testing need only be a simple reality check using easy and cheap to access sources such as carers of children attending the program, community-based volunteers, and clinical staff. Explain your case-finding procedure to them and ask if they think it will work and what they think could be done to improve it. If you are carrying out a SQUEAC investigation then this first stage of testing can be performed in the first (qualitative) phase of SQUEAC data collection. If you are carrying out a SLEAC or CSAS / S3M survey then you will need to dedicate time for testing. The second stage of testing is to test the case-finding procedure in a few communities. If you are carrying out a SQUEAC investigation then this first stage of testing can be performed in the second (semi-quantitative / hypothesis testing) phase of SQUEAC data collection. If you are carrying out a SLEAC or CSAS / S3M survey then you will need to dedicate time for testing. With a good case-finding procedure it should be obvious that the population understands what you are asking and you should start to find cases very quickly. If this does not happen then further work is required. This is often just a matter of changing your informant (e.g. from a village leader to a traditional birth attendant). When you find cases, point out key signs and confirm the terms that you are using in your question. Ask for new terms and ask for help to improve your case-finding procedure: This is the sort of child I am looking for... How could I have better described what I was looking for? Who would be the best person to help me find children like this child? A key part of an active and adaptive case-finding procedure is snowball or chain-referral sampling. When you find SAM cases you can ask the carer and neighbours of these cases to help you find more cases using the existing cases as exemplars. Point out key signs, confirm terms, and ask: Are there more children like this living around here? Can you give me their names or tell me how I can find them?

8 Developing an active and adaptive case-finding procedure : Data gathering and testing In some settings you may find that people are reluctant to identify cases. This may be a particular problem if you are using stigmatised terms. It is always worth asking: I am a stranger in this village. I do not mean to be rude or to embarrass anyone. Please be honest have I been rude or asked embarrassing questions? If YES : I am sorry. What should I have said or done differently? At the end of a test in a particular community you should: 1. Check that you have found cases (if any exist) that you already knew about because they are attending the program. If there are cases that you did not find then ask to see them and ask the informant why they think that the case was not found. 2. Ask the informants for suggestions on how to improve the case-finding procedure. You should develop the method as you test the method. After each community test you should reflect on what you have learned and change the case-finding procedure accordingly. Apply the SQUEAC principal of sampling to redundancy. This means that you should keep testing your case-finding method until you learn nothing new and feel that thecxase-finding procedure cannot be further improved. It is important to realise that active and adaptive case-finding procedures will fail in some settings. The method has been found not to work well in some refugee and IDP camp settings, in urban locations where there is a high population turnover (e.g. around railway and bus stations and in newly established or growing peri-urban shanties ), and in displaced and displacing populations. These settings are typified by a lack or loss of strong extra-familial relationships, extended familial relationships, strong local kinship ties, collective loyalty, and simple (traditional) social structures. In these settings it may be very difficult to find useful key informants or local guides, and snowball sampling will not work well for finding SAM cases when people do not know their neighbours well. In these settings it is also sensible to search for cases by moving house-tohouse and door-to-door, making sure that you measure all children by taking a verbal household census before asking to measure children. This avoids sick or sleeping children being hidden to avoid them being disturbed by the survey team. At the end of the process described here you will usually have confidence that your case-finding procedure will find all (or nearly all) cases of SAM in communities. It is usually not necessary to formally test the performance of your case-finding procedure. If you have concerns about the performance of your casefinding procedure then you can undertake a capture-recapture study to formally test it.

9 Using capture-recapture studies to investigate the performance of case-finding procedures Mark Myatt, Johannah Wegerdt, and Monica Zanchettin

10 Investigating the performance of a case-finding procedure Any case-finding procedure that you develop must be practicable. A case-finding procedure that is more expensive and time-consuming than, for example, a house-to-house / door-to-door census is probably of little value. The primary motivation for developing an active and adaptive case-finding procedure is to save time and money finding cases in coverage assessments. A case-finding procedure must also be sensitive. The sensitivity (or exhaustivity) of a case-finding procedure is a simple statistical measure of how well a case-finding procedure performs at finding cases: Sensitivity(%) = Numberofcasesfound Totalnumberofcases 100 If there are twelve cases in a community and your case-finding procedure finds eleven of these cases then the sensitivity of your case-finding procedure is: Sensitivity(%) = Numberofcasesfound Totalnumberofcases 100 = = = 91.7% The sensitivity of a case-finding procedure is the proportion of all cases that are found by the case-finding procedure and runs from zero percent (no cases found) to one hundred percent (all cases found). When estimating or classifying coverage it is important that your case-finding procedure has a high casefinding sensitivity. If your case-finding procedure has a poor case-finding sensitivity then your coverage survey may overestimate coverage because the results will be based on the easiest to find cases who may be more likely to be covered by the program being assessed. There is also an ethical consideration. Cases of severe acute malnutrition (SAM) are at high risk of death if they remain untreated. Your coverage survey should, therefore, make every effort to find and refer as many SAM cases as possible from the sampled villages. Calculating the sensitivity of your case finding procedure: Sensitivity(%) = Numberofcasesfound Totalnumberofcases 100 requires that you know the total number of cases in your population. This is almost never known but can be estimated. A good way of estimating the total number of cases in a community is to use a method borrowed from ecology called the capture-recapture study. Note : Capture-recapture studies can be difficult, expensive, and time consuming to do properly. Before undertaking a capture-recapture study, you should do some informal testing of your case-finding procedure in a few communities. With a good case-finding procedure it should be obvious that the population understands what you are asking and you should start to find cases very quickly. If this does not happen then further work is required. This is often just a matter of changing your key informant or local guide (e.g. from a village leader to a traditional birth attendant). In some cases you may have to do more work or even abandon active and adaptive case-finding procedures in favour of using a house-to-house / door-to-door census making sure that you measure all children by taking a verbal household census before asking to measure children. This avoids sick or sleeping children being hidden to avoid them being disturbed by the survey team.

11 Capture-recapture studies described Capture-recapture studies are used to estimate the size of a population when a census may be infeasible or impossible to conduct. The basic idea of capture-recapture studies is to sample and identify individuals or cases from a population and then resample the same population to see what fraction of individuals or cases in the second sample were also identified in the first sample (i.e. the fraction of individuals or cases that were found in both samples). In this document we will describe the basic capture-recapture study method and explore the use of capturerecapture studies to estimate the sensitivity of a case-finding procedure. This document focuses on severe acute malnutrition (SAM) but the methods outlined here may also be applied to moderate acute malnutrition (MAM) as well as non-nutritional conditions. Figure CR.1 shows a capture-recapture study in diagram form. Figure CR.1 : A capture recapture study presented as a Venn diagram In this diagram: N = Total number of cases in the study population. M = Number of cases found in the first sample. C = Number of cases found in the second sample. R = Number of cases found in both samples. The letters used to label each set of case reflect the ecological origins of the capture-recapture method and stand for Number, Marked, Captured, and Recaptured. Note that there is no requirement that either sample (i.e. either cases-finding procedure) find 100% of the cases present in the population in order to estimate the total number of cases in the study population.

12 Capture-recapture studies described For this application, one sample will usually be collected using a census (or census-like) case-finding procedure such as central-location screening or house-to-house / door-to-door screening and the other sample will be collected using a rapid case-finding procedure (this will be the method under test). Note that we do not require that either case-finding procedure is 100% sensitive in order to estimate the total number of cases in the study population. Figure CR.2 presents a capture-recapture study with four sets of cases labelled a, b, c, and x as a Venn diagram (as in Figure CR.1) and as a two-by-two table. Figure CR.2 : A capture recapture study presented as a Venn diagram and as a two-by-two table In this diagram (Figure 2): M = a + b C = a + c R = a N = a + b + c + x Where x, the number of cases not found by either case-finding procedure, is unknown. Our task is to find x. Once we have a value for x, we can calculate N since we will then know all of the terms in: N = a + b + c + x Once we have a value for N (i.e. the total number of cases in the study population) we can calculate the sensitivity of either or both case-finding procedures. If we assume: 1. The population is closed (i.e. there is no change in the population during the investigation) 2. Cases sampled on both occasions can be identified and matched (perfect matching) 3. Each case has an equal chance of being included in each sample (equal catchability) 4. The presence of a case in the second sample is not influenced by the presence of the same case in the first sample (independence) then we can calculate a value for the unknown x cell in the table.

13 Capture-recapture studies described Under these assumptions (and particularly under the assumption of independence) the probability (P) of a case being present in the second sample if it is present in the first sample is: a a + b and the probability (P) of a case being present in the second sample if it is not present in the first sample is: c c + x are the same. That is: a a + b = c c + x This formula can be re-arranged to find x: a a + b = c c + x c(a + b) = a(c + x) ac + bc = ac + ax bc = ax x = bc a Knowing x allows us to estimate the total number of cases in the study population: N = a + b + c + bc a N = (a + b)(a + c) a This can be expressed as: N = M C R This formula can overestimate the total number of cases in the study population when the population is small as is the the case with a rare condition such as SAM and can lead to a division by zero when there are no recaptured cases. The following formula: N = (M + 1) (C + 1) 1 R + 1 should be used in preference to the simpler formula. Since it is impossible to have part of a case, the estimate of N should be rounded to the nearest whole number. Capture-recapture studies described

14 The standard error for the estimate of N is approximately: (M + 1) (C + 1) (M R) (C R) SE(N) (R + 1) 2 (R + 2) The approximate 95% confidence interval for the estimate of N is: (M + 1) (C + 1) (M R) (C R) 95%CI N ± 1.96 (R + 1) 2 (R + 2) These formulae are commonly implemented in software (e.g. such as the EpiTable module in EpiInfo and in the Tools for SQUEAC and SLEAC software from the Coverage Monitoring Network) that can work with data from simple capture-recapture studies (see Figure CR.3 and Figure CR.4). Figure CR.3 : The EpiTable module in EpiInfo (running under DosBox) DosBox allows EpiInfo and other MS-DOS programs to run on Windows, OS-X, Linux, UNIX, and Android devices

15 Estimating case-finding sensitivity from capture-recapture study data Once we have made an estimate of the total number of cases (N) in the study population we can estimate the sensitivity of the case-finding procedures used to collect each sample: Sensitivityofmethodone(%) = Sensitivityofmethodtwo(%) = a + b N 100 = M N 100 a + c N 100 = C N % confidence limits can be calculated for these estimates by substituting the upper and lower 95% confidence limits for N in place of N in these formulae. These calculations are usually performed using purpose designed software such as Tools for SQUEAC and SLEAC software from the Coverage Monitoring Network: Figure CR.4 : Capture-recapture calculations in Tools for SQUEAC and SLEAC software This software is available for Windows, OS-X, and Linux We are usually only interested in the sensitivity of our case-finding procedure. More specifically, we are interested in confirming that the sensitivity of our case-finding procedure is high enough to be useful. This usually means having a case-fining sensitivity above 75%.

16 Underlying assumptions of capture-recapture studies It is important that the assumptions behind the estimate of the total number of cases (N) in the study population are not violated. If the closed population assumption is violated then there may be some cases found in one sample that cannot be found in the other sample. This reduces the probability of recapture and will lead to an overestimation of N and an underestimation of case-finding sensitivity. This assumption can usually be met provided that only a short time is allowed between the collection of the two samples so that cases are not lost from the study population due to mortality, recovery, migration, or displacement and cases do not enter the study population either as new cases or as existing cases arriving from outside of the study area. It is also important to ensure that neither sample is taken on market days, feast days, holidays, distribution days, or days / times when many people may be absent from their home community. In one capture-recapture study we found that people from neighbouring villages were attracted by central-location screening activities. This meant that the population for central-location screening was larger than the population for rapid case-finding. This was likely to lead to an overestimation of N and an underestimation of the sensitivity of the rapid casefinding procedure. In another study, a study village was located close to a major road and passers-by attempted to join the central-location screening. Again, this meant that the population for central-location screening was larger than the population for rapid case-finding. This was likely to lead to an overestimation of N and an underestimation of the sensitivity of the rapid case-finding procedure. In such situations it is important to identify and censor excess members of the study population by asking the carers of children where they live and confirming this with a key informant or local guide such as a village health worker, traditional birth attendant, or traditional healer. It is important, for ethical reasons, that outsiders are screened and cases referred to the program. Such problems may also be avoided by careful selection of study sites. In many studies central-location screening has acted as an attractor and the use of alternative methods such as house-to-house / door-to-door screening should be contemplated. Ensuring a closed population can be very difficult in camps for refugees and other displaced persons during the early stages of on an emergency. This is because the camp population may change on a daily basis as refugees arrive or are dispersed to other camps or to live within the host-population. The problem still exists once a camp population has stabilised as the population that can be feasibly sampled may change from dayto-day as carers attend activities such as education / health-promotion / growth monitoring / vaccination programs and food or clothing distributions. In these situations it is important to work closely with camp authorities to identify if a violation of the closed population assumption is likely, and to become familiar with activity timetables to ensure that you take both samples when carers are likely to be at home with their children. Camps for internally displaced persons (IDPs) are more challenging than refugee camps since IDP camps tend to be less organised than refugee camps and it can be very difficult to track population changes and IDPs may leave the camp for work or foraging activities on an irregular basis or to access health care outside of the camp. In such situations you will probably be restricted to identifying whether or not a violation is likely to have occurred and use this information when interpreting sensitivity estimates. Care should be taken when sampling to ensure that the physical boundaries of areas to be sampled are well defined. Failure to do so is likely to result in different areas being used for the two samples. Since mistakes will tend to be made at the boundaries of sampling areas this can have a large effect on the size of the population sampled on each occasion. This may be a particular problem in urban areas. In such situations it is a good idea to use existing boundaries (e.g. roads) and to have the study team produce a rough map of the area used for the first sample which can be used when collecting the second sample. In some situations you may find direct evidence of a violation of the closed population assumption and be able to correct for this during data-collection. In one study (during the recapture phase) the study team were directed to a household and informed by the grandmother that the mother had taken the child to another village. The grandmother was able to provide matching information and the child could be matched with a case found in the capture phase of the study. In this case the child was treated as a recaptured case. In another study, the study team were directed to a household in which the identified child had died between the capture and recapture phases of the study. In this case the child was matched and treated as a recaptured case.

17 Underlying assumptions of capture-recapture studies Cases found in both samples must be reliably identified and matched. If true matches are missed then the number of recaptured cases will be falsely reduced, leading to an overestimation of N and an underestimation of case-finding sensitivity. If false matches are created then the number of recaptured cases will be falsely increased leading to an underestimation of N and an overestimation of case-finding sensitivity. Matching should not prove to be a problem provided sufficient identifying information is collected. The minimum identifying information required is full names (i.e. first, middle, and last names) of carer and the child and the sex and age of the child. The equal catchability assumption does not require that the probability of being found is the same for both methods. It requires that cases have an equal chance of being found for each method. If some cases have a low probability of being found by either method then N will be underestimated and case-finding sensitivity overestimated. Carers of very sick children (i.e. children likely to be cases) may tend to be reluctant to attend central-location screening, particularly when simultaneous screening of several communities at a central point (leading to children being screened outside of their home communities) is used. The use of alternatives to central-location screening (such as house-to-house / door-to-door screening) should be contemplated if you have any doubts about the ability of the rapid case-finding procedure to find very sick children. You should be aware, however, that sick children may also be hidden in house-to-house / door-to-door screening. It is vital to have a good understanding of how the local population defines sickness and malnutrition in order to develop a sensitive rapid case-finding method. In one study there was a marked reluctance in the population to attend for central-location screening and to co-operate with screening activities using a rapid case-finding procedure. This was due to screening fatigue caused by the villages being repeatedly screened without any obvious and direct advantage to the community or to the screened individuals. This is likely to be a problem when villages are selected because of their proximity to feeding centres and main roads. Such problems may be avoided by careful selection of study sites. Screening fatigue may also be introduced by the capture-recapture study itself and this may negatively affect case-finding sensitivity during a subsequent coverage survey. For this reason it is recommended that villages used for capture-recapture studies are excluded from the subsequent coverage assessment. Referring all cases found by the capture-recapture study to the CMAM program helps to counteract screening fatigue. The assumption of independence (i.e. that the presence of a case in the second sample is not influenced by the presence of the same case in the first sample) underlies the derivation of the estimator for N. If there is positive dependence (i.e. if cases found in the first sample are more likely than cases not found in the first sample to be found in the second sample) then N will be underestimated and case-finding sensitivity overestimated. If there is negative dependence (i.e. if cases found in the first sample are less likely than cases not found in the first sample to be found in the second sample) then N will be overestimated and case-finding sensitivity underestimated. The assumption of independence is violated in almost all capture-recapture studies but efforts should be made to minimise the bias that this may introduce. For testing case-finding procedures for use in coverage surveys you should, as far as is possible, use separate teams assisted by different informants / guides to take the two samples (i.e. the same team should not take both samples from a given village) so as to avoid introducing positive dependence. Referral slips can be issued (to all cases found in both samples) only after the second sample has been taken so as to avoid introducing negative dependence. From the above it should be clear that capture-recapture studies should be undertaken with considerable care so as to avoid gross violations of the assumptions behind the method. These assumptions and measures to avoid their gross violation should inform the design, planning, and execution of capture-recapture studies (see Table CR.1). The conduct of a capture-recapture study should always be evaluated to check if any of the assumptions underlying the capture-recapture method are likely to have been violated. Most capturerecapture studies will violate one or more of the underlying assumptions to some degree but can still provide useful results because we can usually identify the violation and predict its likely effect on the estimate of case-finding sensitivity (see Table CR.1). It is usually possible to identify the direction but not the magnitude of a bias that may have been introduced by violations of the underlying assumptions. Any violations and their likely effect on the estimate of case-finding sensitivity should be documented in the capture-recapture study report or the coverage assessment report. Estimates of case-finding sensitivity may be qualified by phrases such as at most x% or at least x%. This has been done in the example shown in Box CR.1.

18 Table CR.1 : Assumptions, possible violations, effects, and preventative measures Effect on... Violated assumption N Sensitivity estimate Preventive measures Closed population Short time between samples. Avoid sampling on certain days (e.g. market days, holidays, and CMAM clinic days). Careful selection of study sites. Appropriate use of central-location screening. Testing for residency. Clearly define the physical boundary of each study site: Existing boundaries. Map study area of first sample. Ensure that local guides understand the scope of the study population and do not introduce bias by systematic exclusion of some individuals. Use community leaders and key informants to identify and exclude outsiders from inclusion in the study. Perfect matching True matches missed False matches created Collect sufficient identifying data. Take care when matching. Equal catchability Use a method sensitive to finding very sick children. This may require qualitative work to understand how the local population defines sickness and malnutrition. Use house-to-house / door-to-door screening for one sample. Careful selection of study sites. Independence Positive dependence Use separate teams to collect each sample. Use different informants / guides for each sample. Negative dependence Refer cases to service after second sample.

19 Effect on... Violated assumption N Sensitivity estimate Preventive measures Inadequate sample size (i.e. very few cases found) Increase size of study population (if possible) by sampling from more communities. Box CR.1 : Assessing case-finding sensitivity and accounting for a violation of underlying assumptions A capture-recapture study found: Cases found by central-location screening (M) : 30 Cases found by active case-finding (C) : 43 Cases found by both methods (R) : 22 The total number of cases in the study population is estimated to be: (M + 1) (C + 1) N = 1 R = 1 23 = 58 Since it is impossible to have part of a case, the estimate of N has been rounded to the nearest whole number. The sensitivity of the active case-finding procedure is: Sensitivity(%) = C N 100 = = 74.1% During the capture-recapture study there was some concern that the closed population assumption had been violated due to people from neighbouring villages being attracted by central-location screening activities. This is likely to have led to to an overestimation of N and an underestimation of case-finding sensitivity. The true case-finding sensitivity was, therefore, likely to be higher than the calculated estimate. This could be qualified in the study report or coverage survey as at least 74%.

20 Sample sizes for capture-recapture studies Sample size calculations for capture-recapture studies are informed by the fact that the formula: N = (M + 1) (C + 1) 1 R + 1 is an unbiased estimator provided: (M + C) > N and the estimate is very nearly unbiased if: R > 7 The estimate of the total number of cases in the study population is unbiased if both of these conditions are met and nearly unbiased if only one or the other of these conditions is met. A capture-recapture recapture study can still provide useful results if these conditions are not met but the estimate of N will tend to be biased upwards and the estimate of sensitivity biased downwards. Calculating the sample size for a capture-recapture study requires estimates of the case-finding sensitivity of both case-finding procedures and an estimate of the total number of cases in the study population. These estimates need only be rough guesses. Suitable guesses for case-finding sensitivity are: Central location screening : 50% 70% House-to-house screening : 80% 100% Active case-finding : 75% 100% The estimate of the total number of cases in a population should be informed by the prevalence of the condition (e.g. from a recent nutritional anthropometry survey) and the size of the study population adjusted to take into account program eligibility / admission criteria: N = Population Proportion Aged6 59months Prevalence Census data should be used when it is available but it is usually safe to assume that about 20% of the total population is aged between 6 and 59 months. Box CR.2 shows how to apply the sample size conditions when planning a capture-recapture study. If the sample size conditions are not met then the size of the study population should be increased. In some situations this may not be possible (e.g. due to cost or time constraints in very low prevalence situations). It is also possible to apply sample size constraints retrospectively to check whether they were met. This has been done in Box CR.3. Just as with the assumptions underlying the the capture-recapture method, results from studies that do not meet the sample size conditions can still be useful. Failure to meet the sample size constraints will tend to lead to an overestimation of N and an underestimation of sensitivity.

21 Box CR.2 : Applying sample size conditions during study planning A capture-recapture study is planned to take place in eight communities with an estimated total population of 4680 persons. The prevalence of severe acute malnutrition (SAM) is estimated to be around 2%. Census data indicates that 19% of the total population are aged between 6 and 59 months. The total number of cases in the study population will be about: N = = 18 If we estimate that the sensitivity of the first case finding method (central-location screening) will be 55% then the number of cases found by that method (M) would be about: M = = 10 If we estimate that the sensitivity of the active case-finding method will be 80% then the number of cases found by that method (C) would be about: C = = 14 In this example M + C would be: R would be: M + C = = 24 M C R = N = 18 = 8 The sample size conditions: and: (M + C) > N R > 7 are likely to be met in the planned study.

22 Box CR.3 : Applying sample size conditions retrospectively A capture-recapture study found: Cases found by central-location screening (M) : 6 Cases found by active case-finding (C) : 8 Cases found by both methods (R) : 5 The total number of cases in the study population is estimated to be: N = (M + 1) (C + 1) 1 R + 1 = = 10 This study did meet the condition: since: (M + C) > N = 14 which is larger than the estimate for N. This study did not meet the condition: since: R > 7 R = 5 The estimate of sensitivity of the active case-finding procedure in this study is: Sensitivity(%) = C N 100 = = 80% This is likely to be a slight underestimate of the true case-finding sensitivity. We might say that the case-finding sensitivity was likely to be at least 80%.

23 Case-finding sensitivity and coverage estimates Coverage assessment methods such as SQUEAC, SLEAC, CSAS, and S3M rely upon using a case-finding procedure with high sensitivity. Ideally this should be 100%. With careful design and testing it is often possible to achieve a case-finding sensitivity of 100% (or near 100%) particularly for severe acute malnutrition (SAM) cases. An estimated case-finding sensitivity of below 100% found by a capture-recapture study could be due to violations of the assumptions behind the capture-recapture method or failure to meet the sample size conditions. It may also be due to a poorly designed or applied case-finding procedure, or to the fact that you are working with a difficult population (an example of a difficult population is one in which people do not know their neighbours, such as the population of a recently established IDP camp). In such situations you should evaluate whether the capture-recapture assumptions were violated and whether the sample size constraints were met and take this into account when assessing case-finding sensitivity. You should also seriously consider whether your case-finding procedure is well designed and was properly applied during the capture-recapture study. If you have any doubts regarding your case-finding procedure then you should either redesign the procedure and test it again using a new capture-recapture study or consider whether to adopt a house-to-house / door-to-door census sampling approach which will usually have close to 100% sensitivity if done well. You should certainly consider doing this if the case-finding sensitivity is below about 75% after taking into account the likely effects of violations of the capture-recapture assumptions or failures to meet the sample size conditions. It is possible, but not recommended, to proceed with a coverage survey using a low sensitivity case-finding method. Doing so will produce a biased survey result. The nature of the bias will depend upon the nature of the underlying faults in the case-finding procedure but the most likely effect will be to exclude cases that are difficult to find. Surveys using a low sensitivity case-finding procedure will be restricted to assessing coverage in easy to find cases. Since such cases are likely to have been found and recruited by the program this may cause the survey to overestimate program coverage. This likely bias should be noted in survey reports. Using a case-finding procedure with a sensitivity of between 75% and 100% is reasonable because such a method is likely to exclude only the least severe borderline SAM cases.

24 Case-finding sensitivity and coverage estimates for two or more programs It is important to avoid confusion regarding case-definitions when developing case-finding procedures and estimating their sensitivities for use in coverage surveys that estimate the coverage of two or more related programs with differing case-definitions. For example, a coverage survey may need to estimate the coverage of an outpatient therapeutic care program (OTP) and an associated therapeutic supplementary feeding program (TSFP). These programs will have different program entry-criteria (case-definitions). For example: Example entry-criteria for OTP and TSFP programs Program OTP TSFP Case-definition / program entry-criteria MUAC < 115 mm or bilateral pitting oedema MUAC between 115 mm and 124 mm (inclusive) or recent discharge from OTP In these situation you should estimate sensitivity separately for each case-definition using a capturerecapture study with a sample size calculated for the less common (i.e. more severe) condition. When you do this you will probably find that the case-finding method is 100% sensitive for the more severe condition but is considerably less than 100% sensitive for the less severe condition (typically, it will be between 60% and 80%). It is reasonable to proceed with a coverage survey with a case-finding sensitivity that is between 60% and 80% for the less severe condition because case-finding sensitivity is likely to be satisfactory for the more severe cases. For example: Example case-finding sensitivities and severity of wasting Program MUAC Sensitivity OTP < 115 mm 100% TSFP Between 115 mm and 119 mm (inclusive) 75% Between 120 mm and 124 mm (inclusive) 50% It is sometimes better to use a mixture of case-finding procedures: Case-finding procedures for OTP and TSFP programs Program OTP TSFP Case-finding procedure Active and adaptive case-finding House-to-house / door-to-door screening If you use this approach you may reduce the number of communities in which house-to-house / door-to-door screening is used as MAM prevalence is usually much higher than SAM prevalence.