ORIGINAL ARTICLE. Hemant D. Shewade & Arun K. Aggarwal & Bhavneet Bharti

Transcription

1 DOI /s ORIGINAL ARTICLE Integrated Management of Neonatal and Childhood Illness (IMNCI): Skill Assessment of Health and Integrated Child Development Scheme (ICDS) Workers to Classify Sick Under-five Children Hemant D. Shewade & Arun K. Aggarwal & Bhavneet Bharti Received: 22 March 2012 /Accepted: 15 June 2012 # Dr. K C Chaudhuri Foundation 2012 Abstract Objective To assess the skills (diagnostic/counseling) of Integrated Management of Neonatal and Childhood Illness (IMNCI) trained workers; and to assess the degree of agreement between the physician and the IMNCI trained workers of Raipurrani block, district Panchkula, India, to classify sick under-five children in field. Methods The cross-sectional study was conducted in Raipurrani in the outpatient departments of the community health centre and one primary health centre in Workers from health department and Integrated Child Development Scheme (ICDS) were assessed in this study. They received IMNCI training in 2006, with 1 day refresher training in Investigator noted his observations using a skill assessment checklist. Under-five child observations were the unit of study. Results Sixteen IMNCI trained workers made 128 child observations. Considering color-coded categorization under IMNCI, agreement with investigator (Kappa) was intermediate; red and yellow categorizations had poor agreement. Morbidity-wise agreement (Kappa) was poor for possible serious bacterial infection, feeding problem, respiratory problem and anemia. Considering final diagnosis, investigator and H. D. Shewade : A. K. Aggarwal School of Public Health, Postgraduate Institute of Medical Education and Research, Chandigarh, India B. Bharti Department of Pediatrics, Advanced Pediatric Centre, Postgraduate Institute of Medical Education and Research, Chandigarh, India H. D. Shewade (*) Department of Community Medicine, Indira Gandhi Medical College and Research Institute (IGMCRI), Puducherry, India hemantjipmer@gmail.com IMNCI trained worker completely agreed in 45 % child observations. All symptoms were asked only in 15 %. Skills were poor overall for young infants. For children between 2 mo to 5 y, danger signs, neck stiffness, edema, wasting and pallor were checked in <40 % observations. Immunization card was asked for in 20 % observations. IMNCI trained workers performed well in all aspects of counseling, except follow up. Conclusions Training without effective implementation plans will not result in long term skill retention. Keywords IMNCI. Skill assessment. Color-coded categorization. Under-five children. Counseling Introduction During the mid-1990s, the World Health Organization (WHO), in collaboration with UNICEF, developed a strategy known as the Integrated Management of Childhood Illness (IMCI). This strategy was expanded in India to include all neonates and was renamed as Integrated Management of Neonatal and Childhood Illness (IMNCI). IMNCI training is designed primarily for skill improvement pertaining to neonatal and child health issues as well as appropriate and prompt management. IMNCI training in District Panchkula, India started in In 2006, School of Public Health, Post Graduate Institute of Medical Education and Research (PGIMER), Chandigarh trained a set of workers from District Panchkula [1], that included health workers called Auxiliary Nurse Midwives (ANMs) and Integrated Child Development Scheme (ICDS) workers called Anganwadi Workers (AWWs). Refresher training in classroom setting for the same was conducted in September 2009 [2]. The present study aimed to assess the skills of the same set of workers at field level. In this context, the present

2 study was conducted in Raipurrani block of District Panchkula in the year 2010 with the following objectives: (i) To ascertain overall degree of agreement between the physician and the IMNCI trained workers (ANMs/AWWs) in color-coded categorization (pink/yellow/green) of underfive sick children at the field level based on IMNCI guidelines. (ii) To assess the specific diagnostic skills of these IMNCI trained workers. (iii) To assess the counseling skills of these IMNCI trained workers Material and Methods Panchkula is a district in north Haryana. It has two administrative blocks, each with one Community Health Centre (CHC). CHC Raipurrani caters to a population of 96,717. It has three Primary Health Centres (PHCs) and 19 Health Sub-Centres (HSCs). Under-five child observations were the units of study. Study period was from 5 July to 19 November Cross-sectional study design was used for sample size calculation. It was assumed that health workers would correctly classify 61 % of the child observations [3]. At precision of 10 % and alpha of 0.025, a sample size of 120 child observations was calculated. Alpha of was assumed to allow for one subgroup analysis (say AWW vs. ANM). Two lists were prepared out of the 46 IMNCI workers trained by authors in block Raipurrani: one for ANMs and another for AWWs. Simple random sampling (random number table using a pre-planned methodology) was used to select 8 IMNCI trained workers in each list. Out Patient Departments (OPDs) of CHC Raipurrani and PHC Barwala were chosen for IMNCI skill assessment for the high patient load in these OPDs. In case a selected ANM/AWW from the list were not available for the study, unbiased replacement was done. At the end of the study period, 128 child observations were done (8 child observations per worker per wk). The study was initiated after the investigator attained a level of agreement (Kappa) of >/0 0.9 with an IMNCI master trainer. Under-five children attending the OPD for symptoms of any illness were included in the study, on first come first serve basis. Under-five children attending for follow up, immunization, or well baby visits and very sick children requiring resuscitative measures were excluded from the study. The child, after being examined by the investigator was presented to the ANM/AWW. This being a field assessment (not classroom assessment), the investigator did not insist on the use of IMNCI case assessment form. Observations were entered by investigator in case record form, checklist of diagnostic skills and checklist of counseling skills. These are standard checklists recommended by WHO/UNICEF and Govt. of India for use in the national programme. However, a pre-testing was conducted to standardize the operational aspects of the skill assessment. The child was then sent to the medical officer for routine OPD care. The diagnostic checklist had items/subgroups/morbidities which represented a diagnosis. Each item had component(s) under them. Final color-coded categorization for each under-five child observation was based upon the most severe color-coded categorization for individual morbidities. At the end of completion of eight child observations, supportive supervision was done based on the observations made by the investigator. Data analysis was done using SPSS version 17 computer software. Unweighted kappa statistics was calculated using an online software [4]. Kappa (K) < 0.4 was taken as poor agreement; (including extremes) as intermediate, > 0.75 as good; and > 0.9 as excellent agreement [5]. The same quality cut-offs were used for other indicators. For ease of analysis of color-coded categorization, under-five children with no diagnosis/normal children were put into green category. Approval of the Institute Ethics Committee, PGIMER and Haryana State Health Services was taken before conducting the study. Results Of the 128 child observations, 26 (20.3 %) were young infants (<2 mo) and 102 (79.7 %) were of 2 mo to 5 y of age. There was no significant difference in the number of children examined by each category of worker based on distribution of age group and color-coded categorization. The investigator categorized 15 % of child observations as red, 48 % as yellow and 37 % as green. The IMNCI trained workers correctly categorized (colorcoded) 65 % of the under-five child observations. Overall agreement of color-coded categorization for child observations (Unweighted Kappa, K) was intermediate (Table 1). Agreement was intermediate for green categorization; poor for yellow and red categorization. Kappa value for colorcoded categorization was also analyzed for individual morbidities (Table 2). Sensitivity was highest for green categorization (excellent, 91 %) and least for red categorization (poor, 32 %). Positive predictive value was least for red categorization (intermediate, 40 %) and maximum for yellow categorization (intermediate, 74 %). Specificity (maximum for red) and negative predictive value (maximum for green) was greater than or equal to 66 % for all color-codes. Complete agreement in final diagnosis (all subgroup/morbidities) was 45 %. (Table 3). When sensitivity in correct overall diagnosis was analyzed morbidity wise, it was found

3 Table 1 Degree of agreement in categorization of under-five child observations against investigator categorization by type of IMNCI trained worker, age group wise IMNCI trained worker categorization (n0eligible child observations) Investigator categorization Kappa 0.95 CI Red Yellow Green Overall (n0128) Red , 0.56 Yellow Green AWW (n064) Red , 0.56 Yellow Green ANM (n064) Red , 0.68 Yellow Green Young Infants (n026) Red , 0.62 Yellow Green mo - 5 y child (n0102) Red , 0.59 Yellow Green that correct diagnosis was highest for respiratory problems (76 %), followed by diarrhea (67 %), fever (50 %) and feeding problem (40 %). It was least for possible serious bacterial infection (20 %), followed by anemia (31 %) and malnutrition (38 %). In young infant group, the common causes of disagreement were missing increased respiratory rate (5 instances) and wrong assessment of feeding problem (5 instances); and in 2 mo to 5 y group, failure to detect pallor (29 instances) and failure to plot/wrong interpretation of weight for age chart (21 instances) were noted. There were instances of categorization without giving a justification and of using wrong age group classification. Very few health workers used IMNCI case assessment forms. It was found that all items in diagnostic skill checklist pertaining to age group were asked or checked in 15 % (Table 4). Counseling skills revealed that the workers scored more than 68 % in all the items but for explaining follow up visit, in which they scored 28 % (Table 5). There was no significant difference in performance between ANMs and AWWs in diagnosis/categorization. Skill wise, AWWs performed better in assessment of malnutrition and ANMs were better in assessment of young infant, anemia and immunization. Discussion It appears that most of the intermediate agreement (Kappa) in color-coded categorization was contributed by green category. The sensitivity for correct color-coded categorization was 65 %. Poor sensitivity in detecting severe illness (32 %) results in not getting appropriate treatment or referral in time. It was 46.8 % during evaluation of IMCI assessments in South Africa [6]. Low positive predictive value would result in unnecessary referrals as in case of red categorization (40 %). When final diagnosis was considered, in 71 % child observations, there was agreement in final diagnosis in at least one diagnosis; with complete in 44 % and partial in 27 %. In Purulia [7], there was agreement in at least one sub group in two-thirds of the cases: with complete in 30.6 % and no agreement at all in 33.9 %. Complete agreement was around 30 % in South Africa [6] and Uganda [8]; around 20 % in Bangladesh [9] and Kenya [10]; and around 60 % in Tanzania [11] and Brazil [12]. In the young infant group, the agreement beyond chance (Kappa) and sensitivity for detecting both possible serious bacterial infection and feeding problem was poor. Though the sample size was inadequate, it does give some indication. All components under possible serious bacterial infection (PSBI) and feeding problem were checked only in 12 % and 27 % respectively. The above findings were also corroborated by the following poor diagnostic skills: asking seizures (34 %) and looking for attachment and suckling (35 %). In Purulia [7], worker performance was better in assessment of all components of PSBI (32 %), feeding problem and immunization. Workers in the present study

4 Table 2 Degree of agreement in categorization of 2 mo to 5 y child observations, by morbidities assessed IMNCI Trained Worker Categorization (n0eligible child observations) Investigator Categorization Kappa 0.95 CI Red Yellow Green a In two child observations health worker did not grade malnutrition in final diagnosis b Sample size inadequate Overall (n0128) Red , 0.56 Yellow Green Respiratory (n0102) Red , 0.80 Yellow Green Diarrhea (n0102) Red , 0.93 Yellow Green Fever (n0102) Red , 0.74 Yellow Green Malnutrition (n0100) a Red , 0.83 Yellow Green Anemia (n0102) Red , 0.52 Yellow Green Possible serious bacterial infection b (n026) Yes No , 0.68 Yes 2 0 No 8 16 Feeding roblem b (n026) Yes , 0.86 No 3 19 fared better in assessing temperature and skin pustules; worse in assessing seizures and respiratory rate. Similarly, in the 2 mo to 5 y age group, agreement (Unweighted Kappa) was poor for anemia and respiratory problem. Poor agreement for anemia can be explained by poor skill in assessment of palmar pallor. Also, the most common cause of disagreement in final diagnosis was failure to check pallor. This was not the case in Purulia [7], where palmar pallor was checked in 76 % of cases with correct identification of pallor in 53 %. Both respiratory rate and chest in-drawing were checked in 44 % of eligible children, whereas the figure was higher in Purulia (60 %). This could be the reason for poor agreement for respiratory problem. Intermediate agreement (kappa) for diarrhea, fever and malnutrition should be interpreted with caution, Table 3 Agreement in overall diagnosis, by type of IMNCI trained worker, age-group wise All child observations [% (0.95 CI)] AWW [% (0.95 CI)] ANM [% (0.95 CI)] < 2 mo [% (0.95 CI)] 2 mo 5 y [% (0.95 CI)] Eligible child observations Complete agreement 58 [45 (37,54)] 28 [44 (32,56)] 30 [47 (35,59)] 15 [58 (39,77)] 43 [42 (33,52)] in overall diagnosis Partial agreement (at 33 [26 (18,33)] 13 [20 (10,30)] 20 [31 (20,43)] 1 a [3.8] 32 [31 (22,40)] least one subgroup) No agreement at all 37 [29 (21,36)] 23 [36 (24,48)] 14 [22 (12,32)] 10 [38 (20,57)] 27 [27 (18,35)] a Sample size inadequate

5 Table 4 Diagnostic skills of IMNCI trained workers, age group wise Table 4 (continued) Ask/Check the Following No. of Eligible Children Yes % (0.95 CI) Ask/Check the Following No. of Eligible Children Yes % (0.95 CI) All items checked/asked (9,21) A. Young infants (< 2 mo) All items asked/checked ANM (16,53) Possible serious bacterial infection Ask seizures (13,49) Look at watch (43,80) Count RR for 60 s (39,77) Expose chest (35.73) Look for chest in-drawing (51,87) Check for fever (76,101) Look for skin pustules (61,93) Check for inactivity (35,73) Checked all components aanm ( 1, 24) Checked at least one component a (89,104) Ask for diarrhea (43,80) Assessed skin pinch Look for sunken eyes Ask/check for feeding problem Ask frequency of breast feeds (43,80) Other feeds (43,80) Look for attachment b (16,53) Look for suckling b (16,53) Checked all components (10,44) Checked at least one component (71,98) Ask for immunization status (39,77) Immunization card ANM (13,49) B. Children 2 mo to 5 y All items asked/checked (4,16) Look for danger signs (25,44) Ask for respiratory symptoms c 82 (75,90) Look at watch (45,71) Expose chest (57,81) Count RR for 60 s (34,60) Look for chest in-drawing (63,86) Checked all components (27,53) Checked at least one component AWW (67,89) Ask for diarrhea d 67 (58,76) Look for skin pinch (51,85) Look for sunken eyes (43,79) Checked both components (43,79) Checked at least one component (51,85) Ask/Check for fever e 84 (77,91) Fever duration (73,95) Look for neck stiffness (13,39) Checked both components (25,54) Checked at least one component (29,59) Check malnutrition Weigh the child (83,95) Plot weight in weight for age chart AWW (46,66) Look for edema AWW (11,26) Look for wasting AWW (6,18) Checked all components AWW (5,17) Checked at least one component f 90 (84,96) Check anemia ANM g 46 (36,56) Look for palmar pallor ANM (30,49) Check immunization status ANM (41,61) Ask immunization card (9,24) a Sample size inadequate b 4 young infants were not accompanied by their mothers Presenting complaint- c (32), d (22), e (33), f (6), g (4) AWW Performance of AWW was significantly better than ANM ANM Performance of ANM was significantly better than AWW as most of the agreement was contributed by green category (Table 2). Similarly, relatively higher sensitivity in correctly diagnosing respiratory problem, diarrhea and fever was because most of the children had no complications. There was discrepancy in sensitivity for correct color coding for malnutrition (88 %) and sensitivity in correct diagnosis of malnutrition (38 %). It appears that yellow color code camouflaged the false grading (grade II, III, IV) of malnutrition by the health worker. Overall, cough, diarrhea and fever was asked in 55 % of cases; it was 72 % in Purulia [7], 15 % in Bangladesh [9]and 95 % in Tanzania [11]. All components pertaining to diarrhea were checked in 61 % cases; it was 45 % in Purulia [7]. Component wise, performance was poor in looking for neck stiffness; it was 44 % in Purulia [7]. All components of malnutrition were checked in 11 % of child observations; edema and wasting being checked in 19 % and 12 %. In the present study, weight was plotted in weight for age chart in 56 % cases; worse than Tanzania (77 %) [11] and better than Purulia (52 %) [7]. In Purulia, slightly lower figure could be because their figure also included cases with correct interpretation in addition to proper performance of the skill. In Bangladesh [9], weight wasn t plotted even in a single case. Overall, Immunization status was asked in 52 %, with card being asked only in 20 % of child observations. Health care providers in Purulia [7] and Brazil [12], assessed immunization status in about three-fifths of children. Health workers fared better in Tanzania [11]. The authors found that in only 15 % of child observations, IMNCI trained workers asked/assessed the entire items/

6 Table 5 Counseling skills of IMNCI trained workers in assessing under-five children (n0child observations), by type of health worker * Chi square test ** Statistically significant difference a Fischer exact test Item Overall (%) (n0128) AWW (%) (n064) ANM (%) (n064) p value * Greet the mother Sit at level with mother Play with the child Ask problems a Listen carefully to child s problems a Use local verbatim a Give time to ask question Give time to answer Explains child s condition Explain treament Explain follow up Ask immunization status ** Ask immunization card ** diagnosis/symptoms specific to the age group; it was 18 % in South Africa [6]; <10 % in Kenya [10]. The most common cause for poor comprehensive assessment was poor performance in asking for danger signs (34 %); similar poor performance was found in South Africa [6] and Bangladesh [9]. In Purulia [7], the performance was relatively better with all danger signs being correctly assessed in 52 % of cases. Health workers in the present study counseled the children adequately though follow up visit was less emphasized (28 %). It could be because the place of study was different from the actual field area of the health worker. Follow-up visit was informed to parents in around 20 % of cases in Purulia[7], 23.7 % in Uganda [8], 42.2 % in Brazil [12], 66 % in Tanzania [11], but only 0.6 % in Bangladesh [9]. IMNCI evaluation requires the worker to use the specific IMNCI assessment forms. Training programmes strongly emphasize that workers shall not cram these. However, it was observed that health system did not provide them with these forms at their work places. This could be the reason for lack of practice post training; resulting in failure of comprehensive assessment. In the same set of workers in 2009, during a follow up evaluation in classroom setting, there was a significant decline in skills, 3 years post training [2]. The mean skill score during the follow up evaluation for respiratory problem, diarrhea and counseling was 75 %, 49 % and 70 % respectively. These findings are similar to the results in the present study, except for diarrhea. It appears that even the good skill levels of few items checked by Venkatachalam et al. are not getting transformed into correct diagnosis/classification [2]. Poor kappa score for color coding of respiratory problem and poor sensitivity in correct diagnosis of malnutrition corroborate this finding. It has been showed that health worker skills can be retained by early follow up visit post training and supportive supervision later on [13]. In addition, timely refresher courses can help in retaining skills. Pariyo et al. [8] have also drawn similar conclusions in Uganda that training alone is not sufficient. IMNCI trained workers in the present study had not received supportive supervision for on the job fine-tuning of skills. Comparing the respective performances (skill-wise) of ANMs and AWWs, one is tempted to presume that, if not for their routine work, their performance in the items in which they have fared better (AWW- malnutrition; ANMimmunization, anemia, young infant), might have been poor. Though, item/component wise poor performance has been detected, the authors would not recommend a reinforcement of those items in the primary training schedule; rather the weak areas may be looked into and reinforced upon during supportive supervision and refresher trainings. This analysis of IMNCI performance presents some tantalizing data describing health worker performance in detecting Possible Serious Bacterial Infection in young infants. This suggests that there is a serious discrepancy in IMNCI worker ability to detect illness in this very young age group compared with older children. But the sample size for this important subgroup analysis is rather too small to draw important conclusions. As under-five children were recruited in the study based on first come first serve, adequate number of young infants could not be studied. Bias cannot be ruled out as the investigator himself was the gold standard for health worker categorization; but the likelihood of bias is less as the investigator was not involved in the training process. Inter-observer variances cannot be ruled out. Conclusions Poor agreement for yellow/red categorizations, low skill levels in comprehensive assessment of all components

7 under an item/subgroup/morbidity suggests that there is a serious deficiency in IMNCI worker ability to detect illness. For effective skill retention, there is a need to strengthen implementation of IMNCI with specific attention to supportive supervision, timely refresher courses, maintenance of constant drug supply, strong referral mechanism and general system strengthening. Acknowledgements The authors would like to thank Project Director, Reproductive and Child Health II (RCH II), district Panchkula, Haryana, India, for providing support for this work; Also,Dr. Dinesh Kumar and Dr. Venkatachalam for providing timely suggestions during the design and conduct of the study. Contributions HDS: Conception and design of the study; planning and conducting the study; analysis and interpretation of data; and drafting the paper; AKA: Conception and design of the study; providing guidance; and revising the draft critically for substantial intellectual content; BB: Conception and design of the study; and revising the draft critically for substantial intellectual content. Conflict of Interest None. Role of Funding Source Project Director, Reproductive and Child Health-II (RCH-II), District Panchkula, Haryana, provided support for the travel of IMNCI trained health workers to respective assessment sites. References 1. Kumar D, Aggarwal AK, Kumar R. The effect of interrupted 5-day training on Integrated Management of Neonatal and Childhood Illness on the knowledge and skills of primary health care workers. Health Policy Plan. 2009;24: Venkatachalam J, Kumar D, Gupta M, Aggarwal AK. Knowledge and skills of primary health care workers trained on integrated management of neonatal and childhood illness: follow-up assessment 3 y after the training. Indian J Public Health. 2011;55: Bandyopadhyay S, Kumar R, Singhi S, Aggarwal AK. Are primary health care workers skilled enough to assess the severity of illness among young infants? Indian Pediatr. 2003;40: Kappa calculation online software [Internet]. Available from faculty.vassar.edu/lowry/kappa.html. Accessed 2011 Feb Gordis L. Epidemiology. In: Gordis L, ed. Assessing validity and reliability of diagnostic and screening tests. 4th ed. Philadelphia: Saunders Elsevier; pp Horwood C, Vermaak K, Rollins N, Haskins L, Nkosi P, Qazi S. An evaluation of the quality of IMCI assessments among IMCI trained health workers in South Africa. PLoS One. 2009;4:e Biswas AB, Mukhopadhyay DK, Mandal NK, Panja TK, Sinha N, Mitra K. Skill of frontline workers implementing integrated management of neonatal and childhood illness: experience from a district of West Bengal, India. J Trop Pediatr. 2011;57: Pariyo GW, Gouws E, Bryce J, Burnham G, Uganda IMCI Impact Study Team. Improving facility-based care for sick children in Uganda: training is not enough. Health Policy Plan. 2005;20: i Arifeen SE, Bryce J, Gouws E, et al. Quality of care for under-fives in first-level health facilities in one district of Bangladesh. Bull World Health Organ. 2005;83: Lin Y, Tavrow P. Assessing health worker performance of IMCI in Kenya. Quality assurance project case study. Bethesda, Maryland, USA: Published for the US Agency for International Development (USAID) by the Quality Assurance Project (QAP); Schellenberg JA, Bryce J, de Savagny D, Tanzania IMCI Multi- Country Evaluation Health Facility Survey Study Group, et al; The effect of Integrated Management of Childhood Illness on observed quality care of under five in rural Tanzania. Health Policy Plan. 2004;19: Amaral J, Gouws E, Bryce J, Leite AJ, Cunha AL, Victora CG. Effect of Integrated Management of Childhood Illness (IMCI) on health worker performance in Northest-Brazil. Cad Saude Publica. 2004;20:S Chaudhary N, Mohanthy PN, Sharma M. Integrated management of childhood illness (IMCI) follow up of basic health workers. Indian J Pediatr. 2005;72:735 9.