Participation in everyday life activities requires executive functioning skills to

Transcription

1 Development of a Performance Assessment of Executive Function: The Children s Kitchen Task Assessment Kristy Rocke, Paige Hays, Dorothy Edwards, Christine Berg KEY WORDS cognition executive function pediatrics performance assessment OBJECTIVE. This study developed and explored the validity of the Children s Kitchen Task Assessment (CKTA), a performance assessment of executive function. METHOD. The development of the CKTA is described. Children were given the CKTA and neuropsychological assessments of executive functioning. Parents completed the Parent Behavior Rating Inventory of Executive Function (Parent BRIEF). RESULTS. Forty-nine typically developing children ages 8 to 12 years participated in this study. Interrater reliability and internal consistency were established. Preliminary evidence of discriminant validity was reflected in significant differences on neuropsychological tests and the Parent BRIEF between high- and low-scoring CKTA groups. Age-related differences in CKTA performance further supported discriminant validity. Support for concurrent validity was observed in moderate correlations with established neuropsychological tests. CONCLUSION. Preliminary results suggest the CKTA is a valid performance assessment that evaluates the level of cognitive assistance children require to complete a challenging functional activity. Rocke, K., Hays, P., Edwards, D., & Berg, C. (2008). Development of a performance assessment of executive function: The Children s Kitchen Task Assessment. American Journal of Occupational Therapy, 62, Kristy Rocke, OTD, OTR/L, is a graduate of the Washington University School of Medicine Program in Occupational Therapy, St. Louis, MO. Paige Hays, MS, OTR/L, is a graduate of the Washington University School of Medicine Program in Occupational Therapy, St. Louis, MO. Dorothy Edwards, PhD, is Associate Professor, Kinesiology Occupational Therapy, Neurology, and Medicine, University of Wisconsin Madison, Madison, WI. Christine Berg, PhD, OTR/L, is Instructor, Program in Occupational Therapy and Neurology, Washington University School of Medicine, 4444 Forest Park Avenue, St. Louis, MO 63108; cberg@wustl.edu Participation in everyday life activities requires executive functioning skills to formulate goals, to plan strategies to achieve those goals, and to self-evaluate during these activities (Lezak, 1982). Lezak (1982) described executive functioning as essential for social and constructive performance that is required for productive life. Because executive function is vital for the performance and maintenance of occupations across the lifespan, occupational therapists must evaluate and address these performance issues (Katz & Hartman-Maeir, 2005). Children and adolescents with executive function deficits often experience difficulties with participation in everyday, meaningful activities (Biederman et al., 2004). Developmental psychologists have written extensively about the assessment and treatment of dysexecutive syndromes in children with various diagnoses (e.g., attention deficit/hyperactivity disorder, autism, epilepsy, traumatic brain injury, and sickle cell disease; Geurts, Verte, Oosterlaan, Roeyers, & Sergeant, 2004; Gioia, Isquith, Kenworthy, & Barton, 2002; Levin & Hanten, 2005; Parrish et al., 2007; Wang et al., 2001). Traditionally, neuropsychological assessment of children is based on evaluation of the subcomponents of cognition and executive function such as memory, attention, inhibition, and verbal or spatial skills. Neuropsychological assessments provide valuable information about the level of cognitive functioning in a specific area; however, the ecological validity, or the degree to which an assessment can evaluate or be related to performance in everyday activities, has been questioned (Burgess, Alderman, Evans, Emslie, & Wilson, 1998; Chaytor & Schmitter-Edgecombe, 2003; Manchester, Priestly, & Jackson, 2004). Direct assessment of functional activities is thought to demonstrate better ecological validity and 528 September/October 2008, Volume 62, Number 5

2 thus provide a more accurate indicator of performance in a wide range of everyday activities and contexts (Alderman, Burgess, Knight, & Henman, 2003; Baum & Edwards, 1993; Burgess, 2000; Burgess et al., 1998; Channon & Crawford, 1999; Goel, Grafman, Tajik, Gana, & Danto, 1997; Shallice & Burgess, 1991). Direct observation of performance in context allows an occupational therapist to view the child s strengths and weaknesses and to identify factors that facilitate or inhibit performance. Currently, few reliable and valid standardized performance assessments measure executive function, particularly for children and adolescents; thus, development of additional performance assessments for the use of pediatric occupational therapists is needed. The most common performance assessments are the Kitchen Task Assessment (KTA; Baum & Edwards, 1993), the Executive Functioning Performance Test (EFPT; Baum et al., 2008), the Assessment of Motor and Processing Skills (AMPS; Fisher, 2003), and the School AMPS (Fingerhut, Madill, Darrah, Hodge, & Warren, 2002). The KTA and EFPT were designed to objectively measure the level of assistance adults require to perform a specific instrumental activity of daily living (IADL; Baum & Edwards, 1993; Baum, Edwards, Hahn, & Morrison, 2003). The KTA objectively measures the level of cognitive assistance a person needs during a cooking task by assessing the cognitive or executive function elements of task performance. These elements include initiation, organization, sequencing and planning, judgment and safety, and completion (Baum & Edwards, 1993, 2003). The EFPT uses the same format and scoring system as the KTA with an expanded group of IADLs, including the KTA cooking task, bill paying, medication management, and making a telephone call. Both the KTA and the EFPT provide detailed information for the development of intervention strategies for clients with performance deficits (Baum & Edwards, 2003; Baum et al., 2008). The AMPS and School AMPS evaluate motor and processing skills during a variety of daily living tasks in home and school environments, but unlike the KTA and EFPT, the AMPS and School AMPS do not evaluate specific executive function skills or determine the level of assistance required to complete the task (Fingerhut et al., 2002; Fisher, 2003). Determining cognitive capacity and the level of assistance needed for successful task completion are essential for the promotion of performance and development in children with cognitive impairments (Vygotsky, 1978). Therefore, a pediatric assessment tool documenting the level of cognitive assistance needed to support successful performance would be beneficial. This information can be used to educate parents or teachers about the level and types of assistance needed for a child to be successful in everyday activities. The goal of the current study was to modify the KTA for use with children. This pediatric version of the KTA is named the Children s Kitchen Task Assessment (CKTA). This article describes the development of the CKTA and examines concurrent and discriminant validity of the CKTA in a sample of typically developing African-American children. Development of the CKTA The CKTA is an adaptation of the KTA, a valid and wellestablished performance assessment of adult performance during a simple cooking task of making stovetop oatmeal (Baum & Edwards, 1993). The CKTA uses the same approach used to develop and validate the KTA and EFPT. The CKTA uses a safe and age-appropriate activity for children (making play dough) and an expanded version of the scoring and cueing system validated for the KTA and EFPT. The testing and scoring procedures are described in detail in Table 1 and the Appendix. The first step in the development of the CKTA was to identify a child-friendly and motivating functional activity that requires executive functioning skills to complete. The essential elements of executive function are similar across the lifespan (Zelazo, Craik, & Booth, 2004); therefore, those elements were considered when selecting an appropriate activity. Several different activities were considered as part of the development of the CKTA, including making microwave popcorn, macaroni and cheese, and play dough. Making play dough was chosen because it is a novel, ageappropriate, and goal-directed activity for most children (see Table 1). The novelty and uniqueness of the play dough recipe reduced the likelihood of procedural memory or prior experience aiding in the completion of the task. Table 1. Components of the Children s Kitchen Task Assessment INITIATION: Taking recipe out of the box and turning to appropriate page ORGANIZATION: Gathering appropriate materials PLANNING/SEQUENCING: Following the recipe 1. Measure 1 cup of flour. Put in bowl. 2. Measure ¼ cup of salt. Put in bowl. 3. Measure 1½ teaspoons of cream of tartar. Put in bowl. 4. Measure 1½ teaspoon of oil. Put in bowl. 5. Mix with a spoon. Set bowl aside 6. Measure ½ cup of water. STOP! Do not add to bowl! 7. Add 3 drops of food coloring into the water. Do not add to bowl! 8. Heat the water in the microwave for 1 min. 9. Add the water to the bowl. 10. Mix for 1 min. 11. Let play dough sit for 1 min. 12. Mix the play dough with your hands. Make the play dough into a ball. JUDGMENT/SAFETY: Ensuring the play dough is safe to touch. Using safe judgment while carrying water and throughout other steps. COMPLETION: Putting the play dough in a bag. Recognizing task is completed. The American Journal of Occupational Therapy 529

3 After selecting an appropriate activity, a play dough recipe was created. The recipe needed to require cooking to test the element of judgment and safety, but stovetop cooking was not thought to be accessible or safe for this age range. Heating water in a microwave was determined to be an appropriate cooking element to assess judgment and safety. To minimize the language and reading demands, the directions were broken down into simple words, and pictures were added to accompany the written recipe. Steps of the recipe were determined to be at a reading level below the first grade, as determined by the Flesch-Kincaid Grade Level (Flesch, 1948). Before administration of the CKTA to the current study sample, the recipe was pilot tested with 6 children ages 6 to 10 years who had various levels of reading skills to determine the ease of following the written and picture recipe. The recipe and materials were revised on the basis of the performance and feedback from these children. After completing the recipe, the CKTA scoring system was adapted to create a uniform means of evaluating the level of assistance required to complete the activity. As in the KTA, standard cueing guidelines were created for each step of the recipe to ensure consistent cueing across testers. A consistent and reliable cueing system is vital because the amount and level of cues determine the children s total score and level of assistance required to complete the activity. Like the KTA, the participant is scored on the basis of the number and type of cues needed to successfully complete each task component. Assistance starts with nonspecific verbal guidance and progresses through a standard series of increasingly more specific and direct cues until the participant has completed the step. The examiner waits 10 s before cueing the child to determine whether the child self-corrects. The type of cue given is scored from 0 to 5, beginning with no cues (0), progressing to general verbal guidance (1), gesture guidance (2), direct verbal assistance (3), physical assistance (4), and, finally, doing for the participant (5) (see the Appendix; Baum et al., 2003). The CKTA uses the components of executive function defined in the KTA, which include initiation, organization, planning and sequencing, judgment and safety, and completion (see Table 1; Baum & Edwards, 1993; Baum et al., 2003). Initiation was defined as the ability to start the activity; this component includes retrieving the recipe book and opening it to the appropriate page. Organization was determined by the ability to find the necessary materials throughout the task and manage the workspace. Planning and sequencing was the largest component of the assessment and consisted of following and completing the steps of the recipe. Children were given a score for each step of the recipe and the sum of the steps created a total planning and sequencing score. Judgment and safety was defined as the ability to demonstrate safe behaviors and appropriate judgment throughout the assessment. Specific tasks within the play dough recipe were created that required judgment and safety (e.g., safely heating water in the microwave and pouring it into the bowl). Finally, completion was scored when the child finished the task by putting the play dough in a bag and acknowledging completion. The CKTA score form reflects a detailed scoring system that captures the level of assistance (cues) required to complete each step of the recipe. Scoring is based on the number and level of cues required to complete each step and section (initiation, organization, planning and sequencing, judgment and safety, and completion) within the task. The total weighted score reflects the number and level of cues needed to successfully complete the task. Total weighted score on the CKTA can range from 0 to 400. A higher score indicates a higher level of assistance required to complete the task and therefore indicates lower executive function skills. Built into this task is the learning effect of corrective cueing. Within the sequenced steps of the recipe, three of the steps are immediately repeated to allow for learning: measuring 1 cup of flour followed by measuring ¼ cup of salt; measuring 1½ teaspoons of cream of tartar followed by 1½ teaspoons of oil; using the timer to stir for 1 min followed by allowing the play dough to cool for 1 min. In addition, the tester can make note of self-talk and the number of times a child needs to reread an instruction, reflecting the child s strategies for memory. After the scoring system was finalized, pre- and postassessment questionnaires were developed to obtain additional information from the children. The preassessment questionnaire addressed the children s previous cooking experience, microwave use, past experience making play dough, and a prediction of the amount of help they would require to make play dough. During this questionnaire, children were asked to read a recipe card ( Measure 1 cup of flour ) and explain how to follow this directive. This activity was to ensure that the children understood the words and the concept of following the recipe directions. The testers explained any words or concepts that the child did not understand. The postassessment questionnaire evaluated self-awareness regarding performance while making the play dough. This was determined by asking participants to reflect on their task performance, the amount of help required, and anything they would do differently next time. The administration of the entire CKTA takes approximately 15 to 20 min. Analyzing occupations within the context of everyday life contributes to our understanding of the role of executive function ability and performance of complex everyday life tasks (Katz & Hartman-Maeir, 2005). Occupational therapists have greatly contributed to this area of practice with the 530 September/October 2008, Volume 62, Number 5

4 development of performance assessments for adults, such as the KTA and the EFPT, that are designed to measure the level of cognitive assistance required to complete a complex activity (Baum & Edwards, 1993; Goverover et al., 2005; Katz, Tadmor, Felzen, & Hartman-Maeir, 2007). These performance assessments of complex real-world tasks complement the information obtained from neuropsychological tests (Manchester et al., 2004). Such tests are needed because neuropsychological assessments may lack ecological validity, the ability to accurately predict function in real-world environments (Alderman et al., 2003). Performance tests provide valuable information for treatment planning, as well as a perspective on normal developmental changes on task performance. Method Participants Participants in this study were recruited from a local elementary school, community acquaintances, and siblings from a larger ongoing study evaluating children with sickle cell disease compared with their age-, gender-, and race-matched peers. Because sickle cell disease primarily affects the African American population, the age-, gender-, and race-matched control group from which children were recruited includes typically developing African American children. Inclusion criteria for this sample thus consisted of typically developing African American children between ages 8 and 12 years. Exclusion criteria were diagnosis of a learning disorder, mental retardation, physical impairment, communication disorder, or receipt of special education services. Validation Measures Behavior Rating Inventory of Executive Function (Parent Form). The Behavior Rating Inventory of Executive Function (BRIEF; Gioia, Isquith, Guy, & Kenworthy, 2000) is a standardized parent questionnaire designed to assess realworld behaviors in children related to executive functioning in the home. The assessment consists of eight subscales: inhibit, shift, emotional control, initiate, working memory, plan and organize, organization of materials, and monitor. The first three subscales combine to form the Behavioral Regulation Index (BRI), and the last five subscales combine to form the Metacognition Index (MI). The BRI and MI combine into one Global Executive Composite score. The normative sample of the Parent BRIEF includes 11.9% African Americans (Gioia et al., 2000). Delis Kaplan Executive Function System Color-Word Interference Test. The Delis Kaplan Executive Function System (D KEFS) Color-Word Interference Test (Delis, Kaplan, & Kramer, 2001) is a standardized neuropsychological assessment that measures a person s ability to inhibit overlearned responses. This assessment is divided into four conditions: naming color patches, reading words printed in black ink, naming the ink color of printed words, and switching between naming the ink color and reading the words. Participants are scored on the basis of the time taken to complete each condition and the number of errors made during each condition. D KEFS Sorting Test. The D KEFS Sorting Test (Delis et al., 2001) is a standardized neuropsychological assessment that measures concept formation and problem-solving skills. The test has two conditions: free sorting and sort recognition. During the free sorting test, the participants are asked to sort six cards into two categories in as many ways as possible and to describe the concepts used to make the sort. During the sort recognition test, the tester sorts the same six cards and asks the participant to identify and describe the sorted groups. Age-adjusted scaled scores are available for people ages 8 to 89 years. Normative data for the D KFES battery was consistent with the 2000 U.S. data in terms of race/ethnicity (Delis et al., 2001; Homack, Lee, & Riccio, 2005). Wechsler Intelligence Scale for Children IV, Digit Span. The Wechsler Intelligence Scale for Children IV (WISC IV) Digit Span (Wechsler, 2003) is a standardized measure of working memory in children. The children are read a series of numbers and asked to repeat the numbers. As the participant correctly recalls the numbers, the amount of numbers given increases until the participant is unable to correctly recite the numbers. Next, the participants are read numbers and asked to repeat the numbers in reverse order. Once again, as the participant correctly recalls the numbers backward, the amount of numbers given increases until the participant is unable to correctly recall the numbers. The participants are scored on the basis of the number of correctly recalled series of numbers. Normative data provide age-adjusted norms and scaled scores for children ages 6 to 16 (Wechsler, 2003). Additional Measures Parents and guardians completed a demographic questionnaire to identify whether their children had any known medical conditions or developmental disabilities. Additional information collected included participants history of grade retention or special education services and any additional diagnoses, such as attention disorders, learning disabilities, or autism. The Hollingshead Four Factor Index of Social Status score (Hollingshead, 1975) is derived from information provided by parents on the demographic questionnaire. Parental occupations and education levels are scaled and The American Journal of Occupational Therapy 531

5 combined. The number of adults contributing to the child s daily care is acknowledged, resulting in a classification of social status between 1 and 5, with 1 being the lowest and 5 being the highest (Hollingshead, 1975). Procedures The study was approved by the Washington University School of Medicine institutional review board, and all parents, guardians, and participants provided informed consent or assent before beginning testing. Occupational therapy graduate students and faculty members were trained in the entire assessment battery and administered all assessments. All testers were trained to criterion and were required to achieve 90% or greater interrater reliability on CKTA total scores before being allowed to administer the test to research participants. No specific order of testing was used. Administration of both the CKTA and the neuropsychological test battery took less than 1 hr. Most participants were recruited from a local elementary school and were tested at their school. Other participants were recruited from the community using similar methods and were tested at the St. Louis Children s Hospital General Clinical Research Center or at the Washington University Program in Occupational Therapy. All children assented to participate before data collection. The children completed the CKTA and the neuropsychological assessments. All participants were given a $5 certificate to a local restaurant for participating in the study. Parents completed the demographic questionnaire in an interview format. Children tested at the elementary school were sent home with the Parent BRIEF. Parents were called after testing to complete the demographic questionnaire and to notify them that the Parent BRIEF had been sent home for them to fill out and return to the school nurse. Parents of children tested elsewhere completed the BRIEF during the testing session. Data Analysis All statistical analyses were completed with SPSS for Windows version 13.0 (SPSS, Inc., Chicago). Descriptive statistics were computed for all variables. Student s t tests were used to test for differences between groups for normally distributed continuous variables, and chi-square analyses were used for dichotomous variables. Interrater reliability was computed using intraclass correlation coefficients (ICCs). Internal consistency was evaluated with Cronbach s alpha statistic. Discriminant validity was examined in two ways. First, differences by age were tested with a one-way analysis of variance (ANOVA; Age Group Total Score). Second, two groups were created on the basis of total CKTA scores and compared for differences between the groups on the neuropsychological tests and Parent BRIEF. A median split (MacCallum, Zhang, Preacher, & Rucker, 2002) was used to create a dichotomous measure of performance on the CKTA. On the basis of the median total score, scores of 10 were considered a good outcome, whereas scores of 11 were indicative of poor performance in this sample. Spearman rho correlation coefficients were used to examine associations among study measures. A priori alpha levels of.05 were required for significance. Results All participants were African American. The children ranged in age from 8 years, 10 months to 12 years, 6 months (M = 10.4 ± 1.12 years) and were enrolled in second to seventh grade. Most of the participants were female (69%), and almost half (49%) lived in single-parent homes. The mean socioeconomic status score was 2.71 (SD = 1.25), indicative of lower middle class, although scores varied from 1 (lowest) to 5 (highest) within the sample. The demographic characteristics of the sample are presented in Table 2. Forty-two of the 49 participants came from a local elementary school. Interrater reliability of the CKTA had been established as part of an ongoing larger study. Twenty-two participants Table 2. Demographic Characteristics of the Sample Participants (N = 49) Variable M SD Age (years) Socioeconomic status Gender n % Male Female Living situation Single parent Two parent Other relative 2 4 Grade 2nd 1 2 3rd th th th 3 6 7th September/October 2008, Volume 62, Number 5

6 were videotaped and scored independently by two blinded raters. The ICC for the total score of the test was.98. Internal consistency of the CKTA was found to be moderate with a Cronbach s alpha of.68. All participants completed the CKTA. The scores are presented in Table 3. Several sets of scores have been computed. First, a total score and total number of cues were created from the combination of the scores of each section of the test (initiation, organization, planning and sequencing, judgment and safety, and completion). The total cues for each section of the assessment were recorded. Section scores reflect the number and level of cues needed to successfully complete each section of the test. The children tested required <1 cue for each of the following: initiation, organization, judgment and safety, and completion. The mean score for planning and sequencing was also found to be <1 cue per step averaged over the 12 steps. The mean total score of all participants was ± 8.80, and the number of cues for the test as a whole averaged 9.02 ± Discriminant validity was examined by comparing performance across age groups using a one-way ANOVA to determine whether the test scores increased with age. The mean scores are shown by 1-year age increments in Figure 1. Performance significantly improved as age increased (F[4,45] = 3.83, p <.008), suggesting that the CKTA is sensitive to the expected age-related differences in executive ability observed in typically developing children. Performance on the neuropsychological tests and Parent BRIEF scores were used to examine the discriminant validity of the CKTA. Thirty-seven Parent BRIEF forms were returned and were included in this analysis. The BRIEF provides eight subscale scores. Parent scores on the eight subscales of the BRIEF were all within the average range of the standardized scores for each subtest. The Color-Word Interference and Sorting subtests of the D KEFS battery and Table 3. Total and Component Scores on the Children s Kitchen Task Assessment Participants (N = 49) Variable M SD Range Total score Total number of cues Component scores a Initiation Organization Planning/sequencing b Judgment/safety Completion a Component scores from total number of cues. b Indicates mean number of cues for all steps. Figure 1. Children s Kitchen Task Assessment total scores and total cues by age. WISC IV Digit Span subtest were used to assess executive function. Performance on the Color-Word Interference and Sorting subtests was within the average range based on comparison to the normative values for these tests. Children in this sample demonstrated poorer performance on the Digit Span backwards, a test of working memory, when compared with the normative values. To further explore the discriminant validity of the CKTA, we used a median split of the total score to create high- and low-scoring groups. We then compared the performance of high- and low-scoring CKTA groups on the Parent BRIEF and three neuropsychological tests. Participants with lower scores on the CKTA required less assistance to complete the task, and children with higher scores on the CKTA required more assistance to complete the task. Chisquare analyses were used to compare groups for CKTA differences by gender and socioeconomic status. No significant differences were found between the high- and low-scoring CKTA groups for gender (χ 2 = 2.08, p >.05, [1]) or socioeconomic status (χ 2 = 3.42, p >.05, [4]). Group comparisons were computed using t tests corrected for multiple comparisons to determine differences between high- and low-scoring CKTA groups performance on the BRIEF and the neuropsychological assessments (Tables 4 and 5). The group with higher CKTA scores demonstrated higher scores on the Parent BRIEF, indicating that children who required more assistance to complete the CKTA were also reported by parents as having demonstrated more characteristics of executive dysfunction. Significant differences between the groups were found for the BRIEF Inhibition subtest score (p <.003) and BRI score (p <.01). Two scores on the Parent BRIEF approached significance: Initiation subtest (p <.06) and the Global Executive Composite (p <.08). Children with better scores on the CKTA performed better on all three neuropsychological tests. Significant differences were found between the two The American Journal of Occupational Therapy 533

7 Table 4. Comparison Between Children s Kitchen Task Assessment High- and Low-Scoring Groups on Parent Behavior Rating Inventory of Executive Function groups on the D KEFS Confirmed Correct Card Sorts (p <.03) and WISC IV Digit Span backwards (p <.04). As part of the preliminary examination of the concurrent validity of the CKTA, we examined the relationships between the results of the CKTA and the results of the Parent BRIEF and neuropsychological tests using Spearman correlation coefficients. All of the coefficients were in the low to moderate range. For example, low to moderate correlations were found between the total score of the CKTA and scores on the WISC IV Digit Span backward (rs =.31) and with the Inhibition subscale of the BRIEF (rs =.34). The organization section of the CKTA correlated significantly with the Plan and Organize subscale of the BRIEF (rs =.38). The judgment and safety section of the CKTA correlated significantly with the Monitor subscale of the BRIEF (rs =.35). The CKTA planning/sequencing score correlated significantly with the scores on the D KEFS Confirmed Correct Card Sorts (rs =.33) and with the Total Description Score (rs =.30). Discussion High-Score Group (n = 17) Low-Score Group (n = 20) Variable M SD M SD p Subtest or index a Inhibition Plan/organize Organization of materials Monitor performance Shifting attention Initiation Working memory Behavioral regulation index Metacognition index Global executive composite a Normative scaled mean equals 50 (SD = 10). We developed the CKTA as a modification of the KTA, an assessment used to measure executive function skills in adults and determine the level of assistance required for successful completion of a simple cooking task (Baum & Edwards, 1993). The CKTA was developed to meet the need for a reliable and valid performance assessment of executive function capabilities in children. The purpose of this study was to examine the reliability and validity of the CKTA in a sample of typically developing African American children between ages 8 and 12. Table 5. Comparison Between Children s Kitchen Task Assessment High- and Low-Scoring Groups on Neuropsychological Tests D KEFS Color-Word Interference Test a High-Score Group (n = 24) Low-Score Group (n = 25) M SD M SD p Condition Condition Total errors Condition Total errors Condition Composite scaled score D KEFS Card Sorting Test a Confirmed correct sorts Description Sort recognition WISC IV Digit Span a Forward Backward Total scaled Note. D KEFS = Delis Kaplan Executive Function System; WISC IV = Wechsler Intelligence Scale for Children IV. a Normative scaled mean equals 10 (SD = 2.5). Children are given the task of making play dough from a recipe; however, the children must plan an approach to the task and carry out their plan. The implicit rule would be that when one step is completed, the child moves to the next step in a linear fashion, managing the materials and workspace as he or she proceeds. The role of the tester is to prompt, after waiting a set time for the child to self-correct, when the child is struggling with a step. Although the CKTA is divided into components of executive functioning, the integration of numerous executive function skills must be used to complete the task; therefore, detection of specific deficits is not the purpose of the CKTA. Our study provides preliminary evidence of the reliability and validity of the CKTA and establishes the CKTA as a clinically useful tool to evaluate children s performance during a challenging real-life activity. We found high levels of interrater reliability and moderate internal consistency when the test is administered and scored by trained testers. In addition, the discriminant validity of the CKTA is supported by the incremental improvement in performance seen as age increases, as well as by the ability to discriminate between high- and lowscoring participants. Developmental theory suggests that executive function skills develop continuously as children mature throughout childhood (Anderson, 2002; Zelazo et al., 2004). The role of social interaction in the development of cognitive skills in children is supported by Vygotsky s (1978) theory and the use of scaffolding. Problem solving is learned within a social context when children receive support, or scaffolding, from adults who, knowing a child s current cognitive abilities, cue 534 September/October 2008, Volume 62, Number 5

8 the child to a higher potential level of performance (Vygotsky, 1978; Wertsch, 1985). Typically, as children s cognitive skills mature through this guidance, the amount of support required to complete a task lessens until the child is able to successfully problem solve a cognitively challenging task independently. In this study, this developmental process is reflected in the greater need for cues and therefore higher total scores on the CKTA in the younger participants. Further support of discriminant validity is seen in the significant differences in the test scores between high- and low-scoring CKTA groups. Children in the high-scoring CKTA group had poorer performance on the D KEFS Sorting Test, indicating that they had more difficulty with abstract reasoning, cognitive flexibility, and initiation of problem-solving strategies. This group also did less well on the WISC IV Digit Span backwards, indicating more difficulty with working memory. Significant differences on the Parent BRIEF support the validity of the CKTA. Although most of the children in this study scored within the normal range on the BRIEF, the parents of children in the high- scoring CKTA group noted more problems with initiation, behavioral regulation, and inhibition. The construct validity of the CKTA is supported by the correlations with the Parent BRIEF, parts of the D KEFS battery, and the WISC IV Digit Span. The correlations of the CKTA with the neuropsychological measures observed in this sample are consistent with the findings of previous studies examining the relationship between performance assessments of executive function and traditional neuropsychological measures in adults (Burgess et al., 2006). The moderate correlations between performance measures and neuropsychological tests have been observed in a variety of adult populations (Burgess et al., 1998; Chaytor & Schmitter-Edgecombe, 2003; Shallice & Burgess, 1991). The authors of these studies have argued for the construct validity of performance measures, providing the evidence that there are higher correlation coefficients observed between the performance assessments and the executive measures and lower coefficients observed between performance assessments and measures of other cognitive functions such as memory. We observed a similar pattern. For example, the CKTA scores were more highly correlated with the WISC IV Digit Span backwards, which is thought to measure working memory, than with the WISC IV Digit Span forwards, which measures short-term memory. Godefroy (2003) suggested that differences in test content account for the observed low to moderate correlation coefficients among the measures. The neuropsychological tests are designed to assess isolated cognitive functions, whereas the performance measures require simultaneous integration and synthesis of several cognitive operations (Burgess et al., 2006). A multitask directive from a parent, such as go clean your room, or from the teacher, such as put everything away and get ready for recess, requires the integration of many executive function skills from the child. Teachers and parents are familiar with the child who has difficulty initiating the task independently, staying focused on the task, sequencing the steps to complete the task, using judgment and safety awareness, or organizing materials. When therapists are able to reliably identify the amount and type of cueing required to enhance task performance, they can instruct parents and teachers on how to support a child s performance similar to what Baum and Edwards (2003) have done for older adults with dementia and their caregivers. By objectively identifying the level of support children require for successful participation in meaningful activities through the CKTA, parents, teachers, and therapists will be able to create a supportive learning environment that enhances a child s capacity for occupational success. Strengths, Limitations, and Future Directions The African American sample is a strength of this preliminary study and a valuable contribution to the literature for this understudied population. However, the generalizability of this study s findings may be limited until testing of a more representative sample is completed. Future research should focus on assessing a variety of races and ethnicities of children with the CKTA to establish cross-cultural validity of the assessment. Moreover, the utility of the CKTA should be addressed by studying the performance of children with a variety of disabilities. Throughout the ongoing development of this assessment, modifications have been made to strengthen the CKTA for future research and clinical practice. Conclusion The development of the CKTA created a motivating and childfriendly assessment of executive functioning that demonstrated validity as a pediatric performance assessment tool. Use of the CKTA allows pediatric occupation-based practitioners to determine the level of cognitive assistance children require to successfully complete a functional activity, and assessment results can immediately guide treatment and inform parents and teachers of cognitive strategies to improve children s performance and participation in everyday activities. s Acknowledgments We thank the participants and their families; Adams Elementary School in St. Louis, MO; Surilla Shaw, RN; and The American Journal of Occupational Therapy 535

9 the graduate students in the Berg Research Lab. We acknowledge the Saint Louis Children s Hospital General Clinical Research Center (EMMA1 RR0036) for their support of this project. References Alderman, N., Burgess, P. W., Knight, C., & Henman, C. (2003). Ecological validity of a simplified version of the multiple errands shopping test. Journal of the International Neuropsychological Society, 9, Anderson, P. (2002). Assessment and development of executive function (EF) during childhood. Child Neuropsychology, 8, Baum, C. M., Connor, L. T., Morrison, T., Hahn, M., Dromerick, A. W., & Edwards, D. F. (2008). Reliability, validity, and clinical utility of the Executive Function Performance Test: A measure of executive function in a sample of people with stroke. American Journal of Occupational Therapy, 62, Baum, C., & Edwards, D. F. (1993). Cognitive performance in senile dementia of the Alzheimer s type: The Kitchen Task Assessment. American Journal of Occupational Therapy, 47, Baum, C. M., & Edwards, D. F. (2003). What persons with Alzheimer s disease can do: A tool for communication about daily activities. Alzheimer s Care Quarterly, 4, Baum, C. M., Edwards, D. F., Hahn, M., & Morrison, T. (2003). Revised Kitchen Task Assessment (KTA). Unpublished manuscript, Washington University School of Medicine. Biederman, J., Monuteaux, M. C., Doyle, A. E., Seidman, L. J., Wilens, T. E., Ferrero, F., et al. (2004). Impact of executive function deficits and attention deficit/hyperactivity disorder on academic outcomes in children. Journal of Consulting and Clinical Psychology, 72, Burgess, P. W. (2000). Strategy application disorder: The role of the frontal lobes in human multitasking. Psychological Research, 63, Burgess, P. W., Alderman, N., Evans, J., Emslie, H., & Wilson, B. A. (1998). The ecological validity of tests of executive function. Journal of the International Neuropsychological Society, 4, Burgess, P. W., Alderman, N., Forbes, C., Costello, A., Coates, L. M., Dawson, D., et al. (2006). The case for the development and use of ecologically valid measures of executive function in experimental and clinical neuropsychology. Journal of the International Neuropsychological Society, 12, Channon, S., & Crawford, S. (1999). Problem-solving in real-lifetype situations: The effects of anterior and posterior lesions of performance. Neuropsychologia, 37, Chaytor, N., & Schmitter-Edgecombe, M. (2003). The ecological validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychology Review, 13, Delis, D. C., Kaplan, E., & Kramer, J. H. (2001). D KEFS executive function system: Examiner s manual. San Antonio, TX: Psychological Corporation. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32, Fingerhut, P., Madill, H., Darrah, J., Hodge, M., & Warren, S. (2002). Brief report: Classroom-based assessment: Validation for the School AMPS. American Journal of Occupational Therapy, 56, Fisher, A. G. (2003). Assessment of Motor and Process Skills. Vol. 1: Development, standardization, and administration manual (5th ed.). Fort Collins, CO: Three Star Press. Geurts, H. M., Verte, S., Oosterlaan, J., Roeyers, H., & Sergeant, J. A. (2004). How specific are executive functioning deficits in attention deficit hyperactivity disorder and autism? Journal of Child Psychology and Psychiatry, 45, Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2000). Behavior Rating Inventory of Executive Function. Odessa, FL: Psychological Assessment Resources. Gioia, G. A., Isquith, P. K., Kenworthy, L., & Barton, R. I. (2002). Profiles of everyday executive function in acquired and developmental disorders. Child Neuropsychology, 8, Godefroy, O. (2003). Frontal syndromes and disorders of executive functions. Journal of Neurology, 250, 1 6. Goel, V., Grafman, J., Tajik, J., Gana, S., & Danto, D. (1997). A study of the performance of patients with frontal lobe lesions in a financial planning task. Brain, 120, Goverover, Y., Kalmar, J., Gaudino-Goering, E., Shawaryn, M., Moore, N. B., Halper, J., et al. (2005). The relation between subjective and objective measures of everyday life activities in persons with multiple sclerosis. Archives of Physical Medicine and Rehabilitation, 86, Hollingshead, A. B. (1975). Four factor index of social status. Unpublished working paper, Yale University. Homack, S., Lee, D., & Riccio, C. A. (2005). Test review: Delis Kaplan Executive Function System. Journal of Clinical and Experimental Neuropsychology, 27, Katz, N., & Hartman-Maeir, A. H. (2005). Higher-level cognitive functions: Awareness and executive functions enabling engagement in occupation. In N. Katz (Ed.), Cognition and occupation across the life span (2nd ed., pp. 3 25). Bethesda, MD: AOTA Press. Katz, N., Tadmor, I., Felzen, B., & Hartman-Maeir, A. (2007). Validity of the executive function performance test in individuals with schizophrenia. Occupational Therapy Journal of Research, 27, 1 8. Levin, H., & Hanten, G. (2005). Executive functions after traumatic brain injury in children. Pediatric Neurology, 33(2), Lezak, M. D. (1982). The problem of assessing executive functions. International Journal of Psychology, 17, MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, Manchester, D., Priestly, N., & Jackson, H. (2004). The assessment of executive functions coming out of the office. Brain Injury, 18, Parrish, J., Geary, E., Jones, J., Seth, R., Hermann, B., & Seidenberg, M. (2007). Executive functioning in childhood epilepsy: Parent-report and cognitive assessment. Developmental Medicine and Child Neurology, 49, September/October 2008, Volume 62, Number 5

10 Shallice, T., & Burgess, P. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Wang, W., Enos, L., Gallagher, D., Thompson, R., Guarini, L., Vichinsky, E., et al. (2001). Neuropsychological performance in school-aged children with sickle cell disease: A report from the Cooperative Study of Sickle Cell Disease. Journal of Pediatrics, 139, Wechsler, D. (2003). Wechsler Intelligence Scale for Children (4th ed.). San Antonio, TX: Psychological Corporation. Wertsch, J. V. (1985). Vygotsky and the social formation of the mind. Cambridge, MA: Harvard University Press. Zelazo, P. D., Craik, F. M., & Booth, L. (2004). Executive function across the life span. Acta Psychologica, 115, Appendix. Levels of Cueing for the Children s Kitchen Task Assessment No cues required (0): The participant requires no help or reassurance, does not ask questions for clarification, and goes directly to task. Self-talk is acceptable. The participant independently completes the step. Verbal guidance (1): The participant requires prompting with an open-ended question or an affirmation that will help him or her move on. Verbal guidance is in the form of a question, not a direct instruction; for example, What should you do now? What is the next step? What else do you need? What does the recipe say? Gesture guidance (2): The participant requires gesture prompting. The tester is not physically involved with the task. Tester makes a gesture that mimics the action that is necessary to complete the subtask or makes a movement that guides the participant back on task; for example, the tester may move his or her hands in a stirring motion, points to where the participant may find the item, or points to the recipe picture. Direct verbal assistance (3): The participant requires a direct phrase or command. The tester is telling the participant what to do. The tester delivers a one-step command, so that the tester is cueing the participant to take the action; for example, saying, You need 1 cup of flour, Put the salt into the bowl, or Use the timer. Physical assistance (4): Tester is physically assisting the participant with the step but not doing the step for the participant. The tester may hold the bowl while the participant adds the ingredients, retrieve a necessary item for the participant, and so forth. Do for participant (5): The tester is required to do the step for the participant. Participant is unable to complete the task. The tester only completes the specific step the participant is demonstrating difficulty with. Participant continues with the next step. The American Journal of Occupational Therapy 537