Performance-Based Measure of Executive Function: Comparison of Community and At-Risk Youth

Joan Toglia; Christine Berg

doi:10.5014/ajot.2013.008482

Abstract

OBJECTIVE. We compared abilities and strategy use of at-risk youth aged 16–21 yr with those of a community sample of high school students using a performance measure of executive function, the Weekly Calendar Planning Activity (WCPA).

METHOD. We recruited students from an alternative school for at-risk youth (n = 113) and from community high schools in the same region (n = 49). We collected demographic information from and administered the WCPA to both groups.

RESULTS. The at-risk group made more errors, used fewer strategies, and broke more rules than the community group; however, the groups were similar in average time for planning and task completion. Moderate relationships were found between WCPA and academic performance in the at-risk group.

CONCLUSION. Comparison of at-risk and community youth provides support for discriminant validity of the WCPA and indicates that the WCPA is useful in identifying adolescents who are at risk for occupational performance deficits.

The literature suggests that executive cognitive impairments are a major contributor to the underlying learning, behavioral, and psychosocial deficits observed in at-risk youth (Best, Miller, & Naglieri, 2011; Pihet, Combremont, Suter, & Stephan, 2012; Riccio, Hewitt, & Blake, 2011). These impairments compromise the ability of at-risk youth to effectively use strategies needed to cope with challenges in daily life occupations such as school and work. Executive cognitive abilities and strategy use are crucial to occupational performance and fall within the domain of occupational therapy practice. Comparison of similarities and differences between typical and at-risk youth on multistep activities requiring integrated use of strategies and executive cognitive abilities can provide a better understanding of the occupational performance deficits of at-risk youth. This population could benefit greatly from occupational therapy services; however, additional knowledge and evidence-based tools are needed as a foundation on which to develop interventions.

Background

During adolescence, youth experience increased demands to juggle multiple school assignments, integrate higher level academic material, and negotiate social peer groups. Developmentally, they also experience a steady increase in executive function (EF) and efficiency of cognitive control, mirrored by structural changes in the brain (Blakemore & Choudhury, 2006; Cowan, Morey, AuBuchon, Zwilling, & Gilchrist, 2010). The capacity to hold in mind multiple concepts, suppress inappropriate thoughts and responses, delay gratification, self-monitor and reflect on performance, think more strategically, and sustain goal-directed actions increases (Best & Miller, 2010; Crone, 2009; Zelazo & Carlson, 2012). With increases in effectiveness and self-regulation of strategies, adolescents are better able to cope with challenging activities and unfamiliar situations (Blakemore & Choudhury, 2006; Magar, Phillips, & Hosie, 2010).

Strategy Use and Development

During development, the mixture of strategies children use during problem solving and multistep or cognitively challenging tasks changes. As children grow older, they use more efficient strategies that require less working memory and time. For example, Winsler and Naglieri (2003) found that verbal strategies moved from overt (i.e., talk aloud) to covert with increasing age. Although common in younger children (aged 5–8 yr), talking aloud was the least common strategy by age 17. Talking aloud in older children, they found, was not related to performance and indicated the persistence of an ineffective, immature strategy when other more efficient strategies were available. Thus, whereas in younger children the overt verbalization strategy appeared to increase effectiveness of performance, the adolescents who used it received no benefit (Winsler & Naglieri, 2003).

Researchers have also suggested that different types of verbalization may influence performance. Children’s rote repetition of what they read or just did appears to be generally ineffective in improving performance, whereas stating aloud their metacognitive reflections (e.g., commenting on why they did something) may enhance their performance (Bunce, Flens, & Neiles, 2010; Tarricone, 2011). Other strategies such as self-checking, planning ahead, rereading, and selecting and summarizing key points have been reported to differentiate between low- and high-performing students (Carr, 2010). These findings suggest that strategies that encourage self-monitoring, elaboration, or deeper processing of relevant information, regardless of modality, are most effective in enhancing learning (Glogger, Schwonke, Holzäpfel, Nückles, & Renkl, 2012).

Because strategies involve the process of dealing with a challenging activity and can affect performance across a wide range of activities, it is important to understand the characteristics of efficient and inefficient performance. Other than Winsler and Naglieri’s (2003) study, little information is available on the strategies adolescents use outside the context of specific academic areas. The few studies that have explored this area have focused on verbal strategies in reading, arithmetic, or contrived problem-solving tasks. Exploration of spontaneous strategy use within challenging everyday activities is required.

Strategy Use and Executive Function

Effective strategy use has been described as “good information processing” (Pressley & Harris, 2006); the most effective learners have a large repertoire of strategies they can combine and adjust to meet different task demands and thus are able to use multiple strategies spontaneously during challenging activities (Harris, Alexander, & Graham, 2008). Children and adolescents who struggle academically often fail to implement effective learning and organizational strategies spontaneously (Reid & Lienemann, 2006).

Both quantity and quality of cognitive strategies have been found to predict learning outcomes (Conley, 2008; Glogger et al., 2012). Glogger and colleagues (2012) examined the number of strategies youth in high school used (quantity) and the extent to which those strategies fulfilled the purpose of the task and promoted successful performance (quality) in mathematics and biology. They demonstrated that higher quality strategies encouraged deeper processing of information. For example, they rated highlighting text as a high-quality strategy if the youth highlighted the most important points and as a low-quality strategy if the youth highlighted the majority of text or missed key points. These findings are consistent with others indicting that low- and high-performing students use different types of strategies for math (Wu et al., 2008) and that students with poor reading skills use fewer strategic planning strategies for reading comprehension (Locascio, Mahone, Eason, & Cutting, 2010).

Restricted use of effective strategies has been linked to EF limitations (Bouazzaoui et al., 2010; Lemaire, 2010). Fewer cognitive resources or decreased working memory may make it harder to encode all aspects of a task and select the most efficient strategy. Reduced inhibition or flexibility in thinking may decrease an adolescent’s ability to let go of a previously used strategy or to adjust and switch strategies for different task aspects (Lemaire, 2010).

Executive Function Impairments and At-Risk Youth

A growing body of evidence indicates that EF impairments, particularly during adolescence, can contribute to risky behaviors (Romer, 2010), early substance abuse (Hester, Lubman, & Yücel, 2010; Pharo, Sim, Graham, Gross, & Hayne, 2011), physical aggression (Harris et al., 2010; Riccio et al., 2011), developmental psychopathology (Pihet et al., 2012), behavioral dyscontrol (Iselin & DeCoster, 2012), development of antisocial behavior (DeLisi & Vaughn, 2011; Ogilvie, Stewart, Chan, & Shum, 2011), and learning difficulties or lack of school success (Best et al., 2011). At-risk youth experience multiple factors as they grow up that may contribute to poorer EF development, including impoverished environments (Sarsour et al., 2011), inadequate nutrition (Lukowski et al., 2010), prenatal drug exposure (Fisher et al., 2011), and depression (Han et al., 2012). In addition, heightened cumulative life stress and adverse life experiences such as deprivation and abuse have been associated with enduring abnormalities in brain structure and organization that can affect learning and cognitive performance during development and into adulthood (Anda et al., 2006; Hackman & Farah, 2009; Raizada & Kishiyama, 2010; Shonkoff et al., 2012).

Studies suggesting that EF impairment is a major challenge in at-risk youth are supported by perceptions of staff and teachers in alternative schools (Dirette & Kolak, 2004). Dirette and Kolak (2004) found that 73.7% of staff in alternative schools perceived that 60%–100% of students had problems with higher level thinking skills or executive cognitive functioning. For example, more than 80% of the staff reported that only 0%–20% of students were successful in multitasking and time management skills. Thus, the majority of at-risk students struggled with coordinating task components and attending to multiple instructions.

Summary

Although adolescence is a critical period during development for the fine tuning and refinement of effective strategy use and executive cognitive skills, most of the literature on EF focuses on younger children. Similarly, despite evidence suggesting that at-risk youth are more vulnerable to EF impairments, few studies have compared EF and strategy use of at-risk youth with those of youth who are not at risk. The handful of studies that do exist focused on individual components of EF using abstract tasks and did not compare performance on complex activities that involve integrated use of EF components or multiple strategies. Because effective strategy use is an inherent part of occupational performance, it is important for occupational therapy practitioners to have a better understanding of the strategies youth use as a foundation for interventions to optimize learning, skill acquisition, and occupational performance in this population (Toglia, Rodger, & Polatajko, 2012).

The purpose of our study was to compare similarities and differences among at-risk and community youth on a recently developed, ecologically valid assessment of everyday EF, the Weekly Calendar Planning Activity (WCPA). We sought to provide a better understanding of typical adolescent performance and of the factors that may contribute to difficulties in occupational performance for at-risk youth. The WCPA has been shown to have practicality, interrater reliability, and face validity with at-risk adolescents transitioning to adulthood (Weiner, Toglia, & Berg, 2012).

We hypothesized that all of the WCPA scores for the at-risk youth would be lower than those of the community youth. We also hypothesized that we would find significant relationships between WCPA scores and academic performance in the at-risk group. Last, we hypothesized that strategy use would differ between the two groups, with the at-risk group using fewer strategies than the community youth.

Method

Research Design

This study used a quasi-experimental, two-group comparison, cross-sectional research design. The institutional review board of the university required verbal consent from the youth to participate and to allow researchers access to their academic records. For the control group, we obtained written parental and participant consent according to university guidelines.

Participants

Participants included 162 adolescents aged 16–21 yr, 113 of whom were at risk and recruited from an alternative Midwestern city high school. This high school enrolls youth who have dropped out of high school or who are at risk of dropping out, and it draws from the entire city, not just from a restricted catchment area. At-risk participants had the following risk factors: 30% were pregnant or a parent, 99% had previously dropped out of high school, 35% were homeless, 6% were in foster care, 45% had mental health issues, 35% had substance abuse issues, and 18% had legal problems. The control group was a convenience sample of 49 adolescents from the same region who had no known diagnoses and whom we recruited through fliers placed around the city. We sought a representative sample of gender, race, age, and educational backgrounds through a stratified sampling method.

Measures

Weekly Calendar Planning Activity.

We administered the WCPA to 113 at-risk youth on their entry to the alternative high school; those scores were reported in Weiner et al. (2012) . The WCPA is a performance measure of EF (Toglia, 2010). An examiner presents participants with 18 appointments in a randomly ordered list. Participants are required to enter the appointments into a 1-wk schedule while recognizing and managing conflicts and adhering to five written rules: (1) Leave Wednesday free, (2) do not cross out appointments once they are entered, (3) inform the examiner when it is a specified time, (4) do not respond to distracting questions from the examiner, and (5) inform the examiner when finished. The examiner observes and records the strategies participants used during the task on a list of 16 pre-identified strategies. Scores include total accuracy of appointment placement on the calendar, errors made in appointment placement, planning time and total task time, number of rules followed, and types of strategy used (Weiner et al., 2012). When the task is completed, participants are asked six after-task questions and score their performance on a 6-point Likert scale.

Before assessing participants for this study, the examiners, including the second author (Berg), received 5 hr of training in administering the WCPA. Interrater reliability was established with 38 participants; the intraclass correlation for total accuracy score was .986, indicating a high level of interrater reliability for total accuracy score (Portney & Watkins, 2008). Further validation of this instrument is in progress at this time.

Northwest Evaluation Association’s Measures of Academic Progress.

The assistant principal tested the academic performance of the at-risk group when they entered the alternative high school program using the Northwest Evaluation Association’s Measures of Academic Progress, a computerized test of reading, math, and language (writing) abilities (Northwest Evaluation Association, 2009). Both Rasch unit (RIT) and grade level are reported. RITs measure academic growth on an equal-interval scale. Scores range from 140 to 300; scores of 240 to 300 indicate typical high school achievement (Northwest Evaluation Association, 2009). Each at-risk student’s RIT score was converted to a grade using the 50th percentile score equivalent. For example, a student’s reading RIT of 214 would be at the 50th percentile for the sixth-grade reading level (213–216).

Procedures

Participants were individually tested by one examiner, including the second author, in a private testing room following the WCPA protocol. Each at-risk participant received a battery of tests, but in this article we report only the WCPA assessment results.

Analysis

We managed data using REDCap, an electronic data storage system (Harris et al., 2009). Data from REDCap were exported to PASW statistical analysis software, Version 19 (IBM Corporation, Armonk, NY). Descriptive statistics, Pearson χ² with Fisher’s exact test significance, and t-test comparisons are reported.

Results

Characteristics of both samples are presented in Table 1. Nine of the 113 at-risk youth stopped the testing, saying they could not complete the WCPA; no academic scores were available for them because they dropped out of the alternative high school before academic testing could occur. The groups did not differ by average age; the at-risk group had a higher percentage of males, Pearson χ² (1, N = 153) = 4.98, p = .03, and of African American or multiracial youth.

Table 1.

Demographic Characteristics of At-Risk and Community Participants

Characteristic	At-Risk Group (n = 104)	Community Group (n = 49)
Age, yr; M (SD)	18.4 (1.3)	18.9 (1.7)
Gender, n (%)
Male	54 (52)	16 (33)
Race, n (%)
African American	88 (85)	4 (8)
Multiracial	9 (9)	2 (4)
Asian	0	3 (6)
White	0	36 (73)
Hispanic	0	3 (6)
Native American	0	1 (2)

Note. M = mean; SD = standard deviation.

View Large

Table 1.

Demographic Characteristics of At-Risk and Community Participants

Characteristic	At-Risk Group (n = 104)	Community Group (n = 49)
Age, yr; M (SD)	18.4 (1.3)	18.9 (1.7)
Gender, n (%)
Male	54 (52)	16 (33)
Race, n (%)
African American	88 (85)	4 (8)
Multiracial	9 (9)	2 (4)
Asian	0	3 (6)
White	0	36 (73)
Hispanic	0	3 (6)
Native American	0	1 (2)

Note. M = mean; SD = standard deviation.

For the at-risk youth, the average reading grade (n = 63) was 6.0 (standard deviation [SD] = 3.4), the average math grade (n = 61) was 4.7 (SD = 2.4), and the average language grade (n = 54) was 5.5 (SD = 2.8). Reading, math, and language scores had significant moderate relationships with WCPA total accuracy score and WCPA number of rules followed: The higher the participant’s academic scores, the higher were his or her WCPA total accuracy and number of rules followed (Table 2).

Table 2.

Correlations Among Academic Scores, WCPA Accuracy, and Rules Followed

Measure	Total Accurate	Rules Followed
Reading grade (n = 63)	.45**	.43**
Reading RIT (n = 63)	.40**	.35*
Language grade (n = 54)	.48**	.50**
Language RIT (n = 54)	.41*	.46**
Math grade (n = 61)	.40**	.33*
Math RIT (n = 61)	.42**	.34*
Number of strategies (n = 153)	.37**	.10

Note. RIT = Rasch unit; WCPA = Weekly Calendar Planning Activity.

*p ≤ .01. **p ≤ .001.

View Large

Table 2.

Correlations Among Academic Scores, WCPA Accuracy, and Rules Followed

Measure	Total Accurate	Rules Followed
Reading grade (n = 63)	.45**	.43**
Reading RIT (n = 63)	.40**	.35*
Language grade (n = 54)	.48**	.50**
Language RIT (n = 54)	.41*	.46**
Math grade (n = 61)	.40**	.33*
Math RIT (n = 61)	.42**	.34*
Number of strategies (n = 153)	.37**	.10

Note. RIT = Rasch unit; WCPA = Weekly Calendar Planning Activity.

*p ≤ .01. **p ≤ .001.

The at-risk students were screened for depression and anxiety at the initial evaluation (Turner-Stokes, Kalmus, Hirani, & Clegg, 2005). On a Likert scale ranging from none (1) to severe (6), 10% self-reported moderate to severe anxiety (scores of 5 or 6) and 3% reported moderate to severe depression; no participant reported moderate to severe levels of both anxiety and depression.

The distribution of planning time (n = 152; 1 participant’s planning time was not recorded) was positively skewed (3.46; standard error [SE] = 0.20) and kurtotic (12.53; SE = 0.39), with 77% of the scores falling within 1 min of planning time. Planning time data were monotonically transformed into four quartiles to reduce the effects of skewness and kurtosis on the statistics. Additionally, the distribution of entered appointments (n = 153) was skewed negatively (−3.34; SE = 0.20) and kurtotic (14.48; SE = 0.39), with 89% of the scores ranging from 15 to 18. Entered appointments data were monotonically transformed by arcsine to reduce the effects of skewness and kurtosis on the statistics. The t-test comparison between the two groups on WCPA scores, with Bonferroni correction of p = .004 (.05/12) to avoid a Type I error, revealed significant differences, with the exception of total time (p = .91), planning time (p = .97), and rules followed (p = .03; Portney & Watkins, 2008; Table 3). The two groups took the same amount of time to plan and to complete the task and had no difference in number of rules followed.

Table 3.

Comparison of Average WCPA Scores Between At-Risk and Community Groups

Score	At-Risk Group,^a Mean (SD)	Community Group,^b Mean (SD)	Independent t Test (p)
Total time, min	15.9 (5.4)	15.96 (6.1)	−0.108 (.91)
Entered appointments transformed	1.13 (0.3)	1.40 (0.2)	−5.83 (.000)*
Total accurate	7.8 (3.9)	14.20 (2.6)	−12.17 (.000)*
Total errors	7.9 (3.6)	3.27 (2.4)	9.46 (.000)*
Location errors	3.5 (2.6)	1.39 (1.4)	6.58 (.000)*
Self-report errors	0.83 (1.2)	0.22 (0.6)	4.21 (.000)*
Repetition errors	0.59 (1.3)	0.04 (0.3)	4.08 (.000)*
Inaccuracy errors	0.33 (0.8)	0.29 (0.6)	0.31 (.76)
Time errors	3.4 (2.2)	1.53 (1.4)	6.22 (.000)*
Rules followed	3.89 (1.1)	4.29 (0.9)	−2.24 (.03)
Number strategies	3.1 (1.9)	4.31 (2.0)	−3.44 (.001)*
Planning time transformed	2.54 (1.2)	2.55 (1.2)	−0.035 (.97)

Note. M = mean; SD = standard deviation; WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004.

View Large

Table 3.

Comparison of Average WCPA Scores Between At-Risk and Community Groups

Score	At-Risk Group,^a Mean (SD)	Community Group,^b Mean (SD)	Independent t Test (p)
Total time, min	15.9 (5.4)	15.96 (6.1)	−0.108 (.91)
Entered appointments transformed	1.13 (0.3)	1.40 (0.2)	−5.83 (.000)*
Total accurate	7.8 (3.9)	14.20 (2.6)	−12.17 (.000)*
Total errors	7.9 (3.6)	3.27 (2.4)	9.46 (.000)*
Location errors	3.5 (2.6)	1.39 (1.4)	6.58 (.000)*
Self-report errors	0.83 (1.2)	0.22 (0.6)	4.21 (.000)*
Repetition errors	0.59 (1.3)	0.04 (0.3)	4.08 (.000)*
Inaccuracy errors	0.33 (0.8)	0.29 (0.6)	0.31 (.76)
Time errors	3.4 (2.2)	1.53 (1.4)	6.22 (.000)*
Rules followed	3.89 (1.1)	4.29 (0.9)	−2.24 (.03)
Number strategies	3.1 (1.9)	4.31 (2.0)	−3.44 (.001)*
Planning time transformed	2.54 (1.2)	2.55 (1.2)	−0.035 (.97)

Note. M = mean; SD = standard deviation; WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004.

Comparison between the groups for strategies used revealed significant differences (see Table 4). The community group consistently more often used the strategies of “fixed appointments first” (e.g., dentist, 3 p.m., Thursday), “uses finger” (e.g., used finger to keep track of place on list of appointments), “crossed off/checked off” (e.g., crossed off appointments after writing them on the calendar), and “notes to self” than did the at-risk youth, supporting our hypothesis. One exception was noted for “talks out loud”; the at-risk group (36%) used this strategy more frequently than the community group (18%), although this difference did not reach significance, Pearson χ² (1, N = 153) = 4.69, p = .03.

Table 4.

Comparison of WCPA Strategies Used by At-Risk and Community Youth

	At-Risk Group,^a	Community Group,^b
Strategy	n (%)	n (%)	Pearson χ² (p)
Reread	23 (22)	8 (16)	0.69 (.27)
Fixed appointments first	10 (10)	20 (41)	20.28 (.000)*
Talks aloud	37 (36)	9 (18)	4.69 (.02)
Self-checks	34 (33)	18 (37)	0.24 (.38)
Uses finger	64 (61)	41 (84)	7.58 (.004)*
Covers lines	8 (8)	3 (6)	0.10 (.52)
Key words	5 (5)	7 (14)	4.06 (.05)
Highlighted	7 (7)	3 (6)	0.02 (.60)
Color code	1 (1)	1 (2)	0.29 (.54)
Cross off/check off	33 (32)	31 (63)	13.61 (.000)*
List or outline	5 (5)	9 (18)	7.37 (.01)
Notes to self	3 (3)	9 (18)	11.05 (.002)*
Cross off Wednesday	4 (4)	6 (12)	3.77 (.06)

Note. WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004; 1-sided Fisher’s exact test.

View Large

Table 4.

Comparison of WCPA Strategies Used by At-Risk and Community Youth

	At-Risk Group,^a	Community Group,^b
Strategy	n (%)	n (%)	Pearson χ² (p)
Reread	23 (22)	8 (16)	0.69 (.27)
Fixed appointments first	10 (10)	20 (41)	20.28 (.000)*
Talks aloud	37 (36)	9 (18)	4.69 (.02)
Self-checks	34 (33)	18 (37)	0.24 (.38)
Uses finger	64 (61)	41 (84)	7.58 (.004)*
Covers lines	8 (8)	3 (6)	0.10 (.52)
Key words	5 (5)	7 (14)	4.06 (.05)
Highlighted	7 (7)	3 (6)	0.02 (.60)
Color code	1 (1)	1 (2)	0.29 (.54)
Cross off/check off	33 (32)	31 (63)	13.61 (.000)*
List or outline	5 (5)	9 (18)	7.37 (.01)
Notes to self	3 (3)	9 (18)	11.05 (.002)*
Cross off Wednesday	4 (4)	6 (12)	3.77 (.06)

Note. WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004; 1-sided Fisher’s exact test.

Discussion

The findings of the current study provide support for the discriminant validity of the WCPA; the scores distinguished between at-risk high school youth and community youth. The community group was more likely to follow the rules, enter appointments accurately, and self-report errors and used a greater number of strategies than the at-risk group. These findings are consistent with literature indicating that at-risk youth are more likely to have lower levels of EF and suggest that occupational therapists should consider using the WCPA with adolescents and young adults as a performance-based assessment of EF.

The moderate relationships found between the WCPA and academic performance in the at-risk group suggest that the skills required to complete the WCPA (i.e., planning, organization, adhering to rules, keeping track, implementing strategies) are important aspects of reading, math, and language academic performance. This finding is similar to that of Best and colleagues (2011), who reported moderate correlations between EF and both reading and math performance in children aged 5–17 yr. These results suggest that EF contributes to general abilities that extend across different academic domains rather than to specific areas. Interventions directed at EF, therefore, could have broad effects on performance because they could potentially benefit more than one academic area (Best et al., 2011).

Contrary to expectations, the at-risk group and community group were similar in both average planning time and total time for task completion. Both groups demonstrated a tendency to jump into the activity, dedicating little time to planning in advance. Although the average time to plan and complete the WCPA was similar in both groups, a significant difference was observed in how the groups spent their time doing the task. The community youth spontaneously selected and implemented a greater number of strategies than the at-risk group. Increased strategy use was associated with better accuracy, suggesting that interventions to promote strategy use may be important in enhancing task performance. This finding indicates that the WCPA provides important information on the process that participants use in completing an everyday task, including analysis of error patterns and strategy use.

Our findings are consistent with evidence indicating differences in strategy use among lower and higher achieving students (Best et al., 2011). In this sample, it was surprising that the majority of at-risk youth did not use relatively simple external strategies such as checking off or crossing off items on a list. Checking or crossing off items as they are completed reduces working memory load so that cognitive resources can be used more efficiently, and the majority of the community youth used this strategy. The lack of implementation of these strategies by at-risk youth places greater demands on mental tracking, further increases task difficulty, and makes accurate completion of the task more challenging.

Prerequisites to effective strategy use include strategy knowledge, repertoire, motivation, and beliefs about strategies, as well as EF (Toglia et al., 2012). A combination of any these components may have contributed to the at-risk youths’ decreased spontaneous initiation of strategies in completing the WCPA and should be further explored. In addition, lack of experience and exposure to efficient task strategies may have contributed to the restricted range of strategies observed. The extent to which knowledge or experience contributed to decreased strategy use needs to be further explored in future studies.

In addition to the number of strategies used, we found differences in the types of strategy implemented. For example, one-third of the at-risk group used “talk aloud” as a strategy; however, this strategy was uncommon among the community group. As noted earlier in this article, overt verbalization in this age group may suggest persistence of an immature and ineffective strategy (Winsler & Naglieri, 2003). Likewise, 41% of the community group entered fixed appointments first, whereas only 10% of the at-risk group used this strategy. Entering fixed appointments first is a more sophisticated strategy because it requires taking into consideration all the variables, mentally identifying similarities and differences among appointments, and using a classification strategy.

Limitations and Future Research

This study was limited to a single Midwestern city. At-risk youth were recruited from one high school, and the control group was a convenience sample. The groups were not equivalent in size, ethnicity, or gender representation. These factors limit the generalizability of findings. Testing of a wider representative sample with a matched control group is needed in future research. In addition, cultural and ethnic differences between the examiners and at-risk youth may have increased anxiety and introduced error.

Limitations in knowledge of strategies, experience with testing situations or calendars, or motivation to do a calendar task may have influenced the participants’ performance and require careful consideration in future studies. Participants appeared interested and engaged in the calendar task, but future studies should include formal ratings of interest level and perceived relevance of the activity.

The WCPA categorizes the type and number of strategies; efficiency or quality of strategy use and impact on performance could be explored in future studies. Further psychometric development of the WCPA should include comparison of WCPA scores with standardized neuropsychological and behavioral measures of EF and examination of the extent to which the WCPA relates to or predicts performance on other challenging everyday tasks. An alternative version of the task can easily be developed to examine test–retest reliability and provide a means of assessing change in strategy use and performance with intervention.

Implications for Occupational Therapy Practice

The results of this study have the following implications for occupational therapy practice:

Executive cognitive difficulties, including low levels of self-monitoring and ineffective strategy use, may contribute to occupational performance deficits in at-risk youth and therefore deserve the attention of occupational therapy practitioners.
Our preliminary data on typical adolescent performance on the WCPA in community high school students provide a baseline for screening those who may be at risk for occupational performance deficits.
Discriminant validity findings provide additional psychometric support for use of the WCPA as an appropriate and valid tool to assess EF and strategy use in at-risk youth.
Strategy-based interventions that focus on effective methods for coping or managing activity challenges may have potential for increasing performance across broad areas of academics and function in at-risk youth.

Conclusion

Comparison of at-risk and typical youth on a recently developed ecologically valid assessment of everyday EF, the WCPA, provided insight into factors that may contribute to difficulties in occupational performance. Our findings support the discriminant validity of the WCPA and provide preliminary normative data for adolescents. This study helps fill the wide gap in the literature on EF and strategy use in youth aged 16–21 yr; most studies in this area have focused on younger children. The need remains to further investigate how adolescents use cognitive strategies within the context of everyday complex tasks and to identify characteristics of effective strategy use as a foundation for promoting competence and success in occupational performance.

This study describes an assessment tool that allows observation and analysis of strategy use, thus yielding valuable information that can help occupational therapy practitioners target interventions. Methods that promote effective strategy use within the context of relevant life activities may have the potential to enhance outcomes for at-risk youth across broad areas of function. The ability to generate alternative strategies to obstacles or challenges and think flexibly is important in building adaptive capacities necessary for resilience and coping with adversity. At-risk youth have diverse needs that require a holistic perspective and an understanding of the connections between behavioral, psychosocial, and cognitive skills. The skill set of occupational therapy practitioners is well suited to assist in promoting successful outcomes in this population, and we encourage practitioners to expand their roles beyond traditional services to meet the needs of this population.

Acknowledgments

We thank the students, staff, and graduate students who participated in this study and assisted with the data collection and management.

References

Anda, R. F., Felitti, V. J., Bremner, J. D., Walker, J. D., Whitfield, C., Perry, B. D., … Giles, W. H. (2006). The enduring effects of abuse and related adverse experiences in childhood: A convergence of evidence from neurobiology and epidemiology. European Archives of Psychiatry and Clinical Neuroscience, 256, 174–186. http://dx.doi.org/10.1007/s00406-005-0624-4 [Article] [PubMed]

Best, J. R., & Miller, P. H. (2010). A developmental perspective on executive function. Child Development, 81, 1641–1660. http://dx.doi.org/10.1111/j.1467-8624.2010.01499.x [Article] [PubMed]

Best, J. R., Miller, P. H., & Naglieri, J. A. (2011). Relations between executive function and academic achievement from ages 5 to 17 in a large, representative national sample. Learning and Individual Differences, 21, 327–336. http://dx.doi.org/10.1016/j.lindif.2011.01.007 [Article] [PubMed]

Blakemore, S. J., & Choudhury, S. (2006). Development of the adolescent brain: Implications for executive function and social cognition. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 47, 296–312. http://dx.doi.org/10.1111/j.1469-7610.2006.01611.x [Article] [PubMed]

Bouazzaoui, B., Isingrini, M., Fay, S., Angel, L., Vanneste, S., Clarys, D., & Taconnat, L. (2010). Aging and self-reported internal and external memory strategy uses: The role of executive functioning. Acta Psychologica, 135, 59–66. http://dx.doi.org/10.1016/j.actpsy.2010.05.007 [Article] [PubMed]

Bunce, D. M., Flens, E. A., & Neiles, K. Y. (2010). How long can students pay attention in class? A study of student attention decline using clickers. Journal of Chemical Education, 87, 1438–1443. http://dx.doi.org/10.1021/ed100409p [Article]

Carr, M. (2010). The importance of metacognition for conceptual change and strategy use in mathematics. In H. S.Waters & W.Schneider (Eds.), Metacognition, strategy use, and instruction (pp. 176–197). New York: Guilford Press.

Conley, M. W. (2008). Cognitive strategy instruction for adolescents: What we know about the promise, what we don’t know about the potential. Harvard Educational Review, 78, 84–106.

Cowan, N., Morey, C. C., AuBuchon, A. M., Zwilling, C. E., & Gilchrist, A. L. (2010). Seven-year-olds allocate attention like adults unless working memory is overloaded. Developmental Science, 13, 120–133. http://dx.doi.org/10.1111/j.1467-7687.2009.00864.x [Article] [PubMed]

Crone, E. A. (2009). Executive functions in adolescence: Inferences from brain and behavior. Developmental Science, 12, 825–830. http://dx.doi.org/10.1111/j.1467-7687.2009.00918.x [Article] [PubMed]

DeLisi, M., & Vaughn, M. G. (2011). The importance of neuropsychological deficits relating to self-control and temperament to the prevention of serious antisocial behavior. International Journal of Child, Youth and Family Studies, 2, 12–35.

Dirette, D., & Kolak, L. (2004). Occupational performance needs of adolescents in alternative education programs. American Journal of Occupational Therapy, 58, 337–341. http://dx.doi.org/10.5014/ajot.58.3.337 [Article] [PubMed]

Fisher, P. A., Lester, B. M., DeGarmo, D. S., Lagasse, L. L., Lin, H., Shankaran, S., … Higgins, R. (2011). The combined effects of prenatal drug exposure and early adversity on neurobehavioral disinhibition in childhood and adolescence. Development and Psychopathology, 23, 777–788. http://dx.doi.org/10.1017/S0954579411000290 [Article] [PubMed]

Glogger, I., Schwonke, R., Holzäpfel, L., Nückles, M., & Renkl, A. (2012). Learning strategies assessed by journal writing: Prediction of learning outcomes by quantity, quality, and combinations of learning strategies. Journal of Educational Psychology, 104, 452–468. http://dx.doi.org/10.1037/a0026683 [Article]

Hackman, D. A., & Farah, M. J. (2009). Socioeconomic status and the developing brain. Trends in Cognitive Sciences, 13, 65–73. http://dx.doi.org/10.1016/j.tics.2008.11.003 [Article] [PubMed]

Han, G., Klimes-Dougan, B., Jepsen, S., Ballard, K., Nelson, M., Houri, A., … Cullen, K. (2012). Selective neurocognitive impairments in adolescents with major depressive disorder. Journal of Adolescence, 35, 11–20. http://dx.doi.org/10.1016/j.adolescence.2011.06.009 [Article] [PubMed]

Harris, A. W., Large, M. M., Redoblado-Hodge, A., Nielssen, O., Anderson, J., & Brennan, J. (2010). Clinical and cognitive associations with aggression in the first episode of psychosis. Australian and New Zealand Journal of Psychiatry, 44, 85–93. http://dx.doi.org/10.3109/00048670903270423 [Article] [PubMed]

Harris, K. R., Alexander, P., & Graham, S. (2008). Michael Pressley’s contributions to the history and future of strategies research. Educational Psychologist, 43, 86–96. http://dx.doi.org/10.1080/00461520801942300 [Article]

Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42, 377–381. http://dx.doi.org/10.1016/j.jbi.2008.08.010 [Article] [PubMed]

Hester, R., Lubman, D. I., & Yücel, M. (2010). The role of executive control in human drug addiction. Current Topics in Behavioral Neurosciences, 3, 301–318. http://dx.doi.org/10.1007/7854_2009_28 [PubMed]

Iselin, A. M. R., & DeCoster, J. (2012). Unique relations of age and delinquency with cognitive control. Journal of Adolescence, 35, 367–379. http://dx.doi.org/10.1016/j.adolescence.2011.08.004 [Article] [PubMed]

Lemaire, P. (2010). Cognitive strategy variations during aging. Current Directions in Psychological Science, 19, 363–369. http://dx.doi.org/10.1177/0963721410390354 [Article]

Locascio, G., Mahone, E. M., Eason, S. H., & Cutting, L. E. (2010). Executive dysfunction among children with reading comprehension deficits. Journal of Learning Disabilities, 43, 441–454. http://dx.doi.org/10.1177/0022219409355476 [Article] [PubMed]

Lukowski, A. F., Koss, M., Burden, M. J., Jonides, J., Nelson, C. A., Kaciroti, N., … Lozoff, B. (2010). Iron deficiency in infancy and neurocognitive functioning at 19 years: Evidence of long-term deficits in executive function and recognition memory. Nutritional Neuroscience, 13, 54–70. http://dx.doi.org/10.1179/147683010X12611460763689 [Article] [PubMed]

Magar, E. C. E., Phillips, L. H., & Hosie, J. A. (2010). Brief report: Cognitive regulation across the adolescent years. Journal of Adolescence, 33, 779–781. http://dx.doi.org/10.1016/j.adolescence.2009.10.002 [Article] [PubMed]

Northwest Evaluation Association. (2009). Technical manual for Measures of Academic Progress and Measures of Academic Progress for Primary Grades. Portland, OR: Author.

Ogilvie, J. M., Stewart, A. L., Chan, R. C. K., & Shum, D. H. K. (2011). Neuropsychological measures of executive function and antisocial behavior: A meta-analysis. Criminology, 49, 1063–1107. http://dx.doi.org/10.1111/j.1745-9125.2011.00252.x [Article]

Pharo, H., Sim, C., Graham, M., Gross, J., & Hayne, H. (2011). Risky business: Executive function, personality, and reckless behavior during adolescence and emerging adulthood. Behavioral Neuroscience, 125, 970–978. http://dx.doi.org/10.1037/a0025768 [Article] [PubMed]

Pihet, S., Combremont, M., Suter, M., & Stephan, P. (2012). Cognitive and emotional deficits associated with minor and serious delinquency in high-risk adolescents. Psychiatry, Psychology and Law, 19, 427–438. http://dx.doi.org/10.1080/13218719.2011.598634 [Article]

Portney, L. G., & Watkins, M. P. (2008). Foundations of clinical research: Applications to practice (3rd ed.). Upper Saddle River, NJ: Prentice Hall.

Pressley, M., & Harris, K. R. (2006). Cognitive strategies instruction: From basic research to classroom instruction. In E.Anderman, P. H.Winne, P. A.Alexander, & L.Corno (Eds.), Handbook of educational psychology (2nd ed., pp. 265–286). Mahwah, NJ: Erlbaum.

Raizada, R. D. S., & Kishiyama, M. M. (2010). Effects of socioeconomic status on brain development, and how cognitive neuroscience may contribute to leveling the playing field. Frontiers in Human Neuroscience, 4, 1–11. http://dx.doi.org/10.3389/neuro.09.003.2010 [PubMed]

Reid, R., & Lienemann, T. O. (2006). Strategy instruction for students with learning disabilities. New York: Guilford Press.

Riccio, C. A., Hewitt, L. L., & Blake, J. J. (2011). Relation of measures of executive function to aggressive behavior in children. Applied Neuropsychology, 18, 1–10. http://dx.doi.org/10.1080/09084282.2010.525143 [Article] [PubMed]

Romer, D. (2010). Adolescent risk taking, impulsivity, and brain development: Implications for prevention. Developmental Psychobiology, 52, 263–276. http://dx.doi.org/10.1002/dev.20442 [PubMed]

Sarsour, K., Sheridan, M., Jutte, D., Nuru-Jeter, A., Hinshaw, S., & Boyce, W. T. (2011). Family socioeconomic status and child executive functions: The roles of language, home environment, and single parenthood. Journal of the International Neuropsychological Society, 17, 120–132. http://dx.doi.org/10.1017/S1355617710001335 [Article] [PubMed]

Shonkoff, J. P.; & Garner, A. S.; Committee on Psychosocial Aspects of Child and Family Health; Committee on Early Childhood, Adoption, and Dependent Care; & Section on Developmental and Behavioral Pediatrics., (2012). The lifelong effects of early childhood adversity and toxic stress. American Academy of Pediatrics Policy, 129, e232–e246. http://dx.doi.org/10.1542/peds.2011-2663

Tarricone, P. (2011). The taxonomy of metacognition. New York: Psychology Press.

Toglia, J. (2010). Weekly Calendar Planning Activity. Dobbs Ferry, NY: Mercy College.

Toglia, J., Rodger, S., & Polatajko, H. (2012). Anatomy of strategy use: A therapist’s primer for enabling occupational performance. Canadian Journal of Occupational Therapy, 79, 225–236. http://dx.doi.org/10.2182/cjot.2012.79.4.4 [Article]

Turner-Stokes, L., Kalmus, M., Hirani, D., & Clegg, F. (2005). The Depression Intensity Scale Circles (DISCs): A first evaluation of a simple assessment tool for depression in the context of brain injury. Journal of Neurology, Neurosurgery, and Psychiatry, 76, 1273–1278. http://dx.doi.org/10.1136/jnnp.2004.050096 [Article] [PubMed]

Weiner, N. W., Toglia, J., & Berg, C. (2012). Weekly Calendar Planning Activity (WCPA): A performance-based assessment of executive function piloted with at-risk adolescents. American Journal of Occupational Therapy, 66, 699–708. http://dx.doi.org/10.5014/ajot.2012.004754 [Article] [PubMed]

Winsler, A., & Naglieri, J. (2003). Overt and covert verbal problem-solving strategies: Developmental trends in use, awareness, and relations with task performance in children aged 5 to 17. Child Development, 74, 659–678. http://dx.doi.org/10.1111/1467-8624.00561 [Article] [PubMed]

Wu, S. S., Meyer, M. L., Maeda, U., Salimpoor, V., Tomiyama, S., Geary, D. C., & Menon, V. (2008). Standardized assessment of strategy use and working memory in early mental arithmetic performance. Developmental Neuropsychology, 33, 365–393. http://dx.doi.org/10.1080/87565640801982445 [Article] [PubMed]

Zelazo, P. D., & Carlson, S. M. (2012). Hot and cool executive function in childhood and adolescence: Development and plasticity. Child Development Perspectives, 6, 354–360. http://dx.doi.org/10.1111/j.1750-8606.2012.00241.x [Article]

Table 1.

Demographic Characteristics of At-Risk and Community Participants

Characteristic	At-Risk Group (n = 104)	Community Group (n = 49)
Age, yr; M (SD)	18.4 (1.3)	18.9 (1.7)
Gender, n (%)
Male	54 (52)	16 (33)
Race, n (%)
African American	88 (85)	4 (8)
Multiracial	9 (9)	2 (4)
Asian	0	3 (6)
White	0	36 (73)
Hispanic	0	3 (6)
Native American	0	1 (2)

Note. M = mean; SD = standard deviation.

View Large

Table 1.

Demographic Characteristics of At-Risk and Community Participants

Characteristic	At-Risk Group (n = 104)	Community Group (n = 49)
Age, yr; M (SD)	18.4 (1.3)	18.9 (1.7)
Gender, n (%)
Male	54 (52)	16 (33)
Race, n (%)
African American	88 (85)	4 (8)
Multiracial	9 (9)	2 (4)
Asian	0	3 (6)
White	0	36 (73)
Hispanic	0	3 (6)
Native American	0	1 (2)

Note. M = mean; SD = standard deviation.

Table 2.

Correlations Among Academic Scores, WCPA Accuracy, and Rules Followed

Measure	Total Accurate	Rules Followed
Reading grade (n = 63)	.45**	.43**
Reading RIT (n = 63)	.40**	.35*
Language grade (n = 54)	.48**	.50**
Language RIT (n = 54)	.41*	.46**
Math grade (n = 61)	.40**	.33*
Math RIT (n = 61)	.42**	.34*
Number of strategies (n = 153)	.37**	.10

Note. RIT = Rasch unit; WCPA = Weekly Calendar Planning Activity.

*p ≤ .01. **p ≤ .001.

View Large

Table 2.

Correlations Among Academic Scores, WCPA Accuracy, and Rules Followed

Measure	Total Accurate	Rules Followed
Reading grade (n = 63)	.45**	.43**
Reading RIT (n = 63)	.40**	.35*
Language grade (n = 54)	.48**	.50**
Language RIT (n = 54)	.41*	.46**
Math grade (n = 61)	.40**	.33*
Math RIT (n = 61)	.42**	.34*
Number of strategies (n = 153)	.37**	.10

Note. RIT = Rasch unit; WCPA = Weekly Calendar Planning Activity.

*p ≤ .01. **p ≤ .001.

Table 3.

Comparison of Average WCPA Scores Between At-Risk and Community Groups

Score	At-Risk Group,^a Mean (SD)	Community Group,^b Mean (SD)	Independent t Test (p)
Total time, min	15.9 (5.4)	15.96 (6.1)	−0.108 (.91)
Entered appointments transformed	1.13 (0.3)	1.40 (0.2)	−5.83 (.000)*
Total accurate	7.8 (3.9)	14.20 (2.6)	−12.17 (.000)*
Total errors	7.9 (3.6)	3.27 (2.4)	9.46 (.000)*
Location errors	3.5 (2.6)	1.39 (1.4)	6.58 (.000)*
Self-report errors	0.83 (1.2)	0.22 (0.6)	4.21 (.000)*
Repetition errors	0.59 (1.3)	0.04 (0.3)	4.08 (.000)*
Inaccuracy errors	0.33 (0.8)	0.29 (0.6)	0.31 (.76)
Time errors	3.4 (2.2)	1.53 (1.4)	6.22 (.000)*
Rules followed	3.89 (1.1)	4.29 (0.9)	−2.24 (.03)
Number strategies	3.1 (1.9)	4.31 (2.0)	−3.44 (.001)*
Planning time transformed	2.54 (1.2)	2.55 (1.2)	−0.035 (.97)

Note. M = mean; SD = standard deviation; WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004.

View Large

Table 3.

Comparison of Average WCPA Scores Between At-Risk and Community Groups

Score	At-Risk Group,^a Mean (SD)	Community Group,^b Mean (SD)	Independent t Test (p)
Total time, min	15.9 (5.4)	15.96 (6.1)	−0.108 (.91)
Entered appointments transformed	1.13 (0.3)	1.40 (0.2)	−5.83 (.000)*
Total accurate	7.8 (3.9)	14.20 (2.6)	−12.17 (.000)*
Total errors	7.9 (3.6)	3.27 (2.4)	9.46 (.000)*
Location errors	3.5 (2.6)	1.39 (1.4)	6.58 (.000)*
Self-report errors	0.83 (1.2)	0.22 (0.6)	4.21 (.000)*
Repetition errors	0.59 (1.3)	0.04 (0.3)	4.08 (.000)*
Inaccuracy errors	0.33 (0.8)	0.29 (0.6)	0.31 (.76)
Time errors	3.4 (2.2)	1.53 (1.4)	6.22 (.000)*
Rules followed	3.89 (1.1)	4.29 (0.9)	−2.24 (.03)
Number strategies	3.1 (1.9)	4.31 (2.0)	−3.44 (.001)*
Planning time transformed	2.54 (1.2)	2.55 (1.2)	−0.035 (.97)

Note. M = mean; SD = standard deviation; WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004.

Table 4.

Comparison of WCPA Strategies Used by At-Risk and Community Youth

	At-Risk Group,^a	Community Group,^b
Strategy	n (%)	n (%)	Pearson χ² (p)
Reread	23 (22)	8 (16)	0.69 (.27)
Fixed appointments first	10 (10)	20 (41)	20.28 (.000)*
Talks aloud	37 (36)	9 (18)	4.69 (.02)
Self-checks	34 (33)	18 (37)	0.24 (.38)
Uses finger	64 (61)	41 (84)	7.58 (.004)*
Covers lines	8 (8)	3 (6)	0.10 (.52)
Key words	5 (5)	7 (14)	4.06 (.05)
Highlighted	7 (7)	3 (6)	0.02 (.60)
Color code	1 (1)	1 (2)	0.29 (.54)
Cross off/check off	33 (32)	31 (63)	13.61 (.000)*
List or outline	5 (5)	9 (18)	7.37 (.01)
Notes to self	3 (3)	9 (18)	11.05 (.002)*
Cross off Wednesday	4 (4)	6 (12)	3.77 (.06)

Note. WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004; 1-sided Fisher’s exact test.

View Large

Table 4.

Comparison of WCPA Strategies Used by At-Risk and Community Youth

	At-Risk Group,^a	Community Group,^b
Strategy	n (%)	n (%)	Pearson χ² (p)
Reread	23 (22)	8 (16)	0.69 (.27)
Fixed appointments first	10 (10)	20 (41)	20.28 (.000)*
Talks aloud	37 (36)	9 (18)	4.69 (.02)
Self-checks	34 (33)	18 (37)	0.24 (.38)
Uses finger	64 (61)	41 (84)	7.58 (.004)*
Covers lines	8 (8)	3 (6)	0.10 (.52)
Key words	5 (5)	7 (14)	4.06 (.05)
Highlighted	7 (7)	3 (6)	0.02 (.60)
Color code	1 (1)	1 (2)	0.29 (.54)
Cross off/check off	33 (32)	31 (63)	13.61 (.000)*
List or outline	5 (5)	9 (18)	7.37 (.01)
Notes to self	3 (3)	9 (18)	11.05 (.002)*
Cross off Wednesday	4 (4)	6 (12)	3.77 (.06)

Note. WCPA = Weekly Calendar Planning Activity.

an = 104. ^bn = 49.

*Bonferroni correction, p ≤ .004; 1-sided Fisher’s exact test.