Weekly Calendar Planning Activity (WCPA): A Performance-Based Assessment of Executive Function Piloted With At-Risk Adolescents

Nikki Williamson Weiner; Joan Toglia; Christine Berg

doi:10.5014/ajot.2012.004754

Free

Research Article | November 2012

Weekly Calendar Planning Activity (WCPA): A Performance-Based Assessment of Executive Function Piloted With At-Risk Adolescents

Nikki Williamson Weiner; Joan Toglia; Christine Berg

Author Affiliations

Article Information

American Journal of Occupational Therapy, November/December 2012, Vol. 66, 699-708. doi:10.5014/ajot.2012.004754

Abstract

OBJECTIVE. We piloted the Weekly Calendar Planning Activity (WCPA), a performance-based measure of executive function (EF), to establish a baseline for at-risk adolescents.

METHOD. Participants were 113 youths ages 16–21 who were enrolled at a charter school for youth returning to high school after dropping out. We administered the WCPA and collected demographic information.

RESULTS. On average, participants spent 15.9 min on the WCPA, made 7.9 errors, and followed 4.0 of 5 possible rules. No ceiling effect was observed in overall accuracy. Participants used a mean of 3.1 strategies (standard deviation = 1.9) while completing the WCPA. Participants who used more strategies spent more time planning and completing the task and were more accurate.

CONCLUSION. The WCPA may be useful to occupational therapists as a performance measure of EF. This assessment allows evaluation of complex task performance, strategy use, self-evaluation of performance, and error patterns, which can be used in developing intervention strategies.

Executive function (EF), associated with the frontal lobes of the brain, allows a person to adapt to novel situations and perform goal-oriented behavior (Burgess & Simons, 2005). EF, which manifests in childhood and continues to develop into early adulthood, “involve[s] maintenance of multiple goals and subgoals, with priorities that change over time requiring self-initiative, self-monitoring, and self regulation” (Lamberts, Evans, & Spikman, 2010, p. 57). EF represents complex and dynamic control processes that are often context dependent (Gioia, Kenworthy, & Isquith, 2010), so the measurement of its impact on everyday life is not straightforward.

Cognition and Adolescence

As environmental and task demands evolve between childhood and adolescence, the brain, particularly the prefrontal cortex, mirrors these changes, as evidenced by magnetic resonance imaging studies (Blakemore & Choudhury, 2006; Paus, 2005). Developmentally, adolescence is a period of dramatic cognitive maturation and change. Forming an identity, choosing peer groups, and shifting opportunities for action (Gestsdottir & Lerner, 2008) are but a few examples of complex behaviors that the brain must execute as adulthood emerges. With EF development, including inhibitory control over impulses, increased attention and concentration, more self-control, and less distractibility or hyperactivity (Fuster, 2002), the typical adolescent is able to better navigate increasing task demands. Additionally, adolescents typically improve at decision making and multitasking while demonstrating increased processing speed (Blakemore & Choudhury, 2006).

At-risk adolescents, according to Davis, Kerr, and Robinson Kurpius (2003), are youth aged 13–19 who are at a disadvantage in completing developmental tasks such as graduating from high school. These youth are more likely to live in poverty, witness aberrant behavior, perform poorly at school, exhibit risk-taking behavior, and be of minority status (Davis et al., 2003). Several studies have suggested that poverty or low socioeconomic status affect neural systems that mediate development of language and EF (Kishiyama, Boyce, Jimenez, Perry, & Knight, 2009; Sarsour et al., 2011). EF impairments have been associated with behavioral dyscontrol, propensity to drug addiction (Hester, Lubman, & Yücel, 2010), and risky behaviors (Romer et al., 2011).

The challenges inherent in growing up in an impoverished environment make EF a survival factor for adolescents (Gestsdottir & Lerner, 2008). For example, exposure to chronic stress within the environment may disrupt persistence in goal-directed behaviors and therefore disrupt the ability to self-regulate behavior (Eisenberg et al., 1997). Goal-directed behaviors associated with EF are documented predictors of overall successful life management and the avoidance of maladaptive or deviant behaviors (Freund & Baltes, 1998; Gardner, Dishion, & Connell, 2008). EF may be particularly important for at-risk youth transitioning to adulthood because the cognitive demands of daily living in the adult world necessitate many aspects of EF; for example, being successful at work, college, and independent life requires organization, planning, sustained attention, timeliness, management of distraction, multitasking, and strategy use. The use of metacognitive strategies has been determined to improve learning outcomes in school-age youth (Conley, 2008; Harris & Pressley, 1991).

In summary, the literature strongly suggests that at-risk youth may have underlying EF difficulties that may contribute to difficulties in learning, behavior, and social and emotional functioning. Because EF is important for coping and management of daily life challenges, it is important to examine EF within the context of challenging activities to obtain a complete picture of the person’s functioning. EF may contribute to the survival skills of at-risk youth that promote success despite environmental disadvantage; however, scant attention has been paid to the measurement and exploration of EF in high-risk transition-age young adults.

Assessment of Executive Function

EF is crucial for daily function, yet few assessments capture this construct in the context of the real world. Cognitive testing is often conducted with standardized neurocognitive instruments in structured environments, free of distraction, with a directing clinician. This environment prohibits the multitasking and goal-oriented behaviors that make up complex, everyday tasks (Burgess, Alderman, Evans, Emslie, & Wilson, 1998; Burgess et al., 2006; Gioia & Isquith, 2004; Manchester, Priestley, & Jackson, 2004). Such assessments are not representative of actual performance and not predictive of real-world challenges (Alderman, Burgess, Knight, & Henman, 2003; Kvavilashvili & Ellis, 2004). This reduced ecological validity yields only weak to moderate correlations between neurocognitive test performance and performance in everyday life (Manchester et al., 2004).

An ecological approach to the assessment of EF using less traditional means, such as performance-based assessment, has been recommended (Worthington, 2005). Performance-based assessment is thought to have the potential to detect even the most subtle of EF difficulties because the tasks capture the complexity of real-life performance and cannot be feigned (Alderman et al., 2003; Baum et al., 2008; Lamberts et al., 2010; Toglia, Johnston, Goverover, & Dain, 2010). Performance-based assessment allows clinicians not only to evaluate the outcomes of testing, but also to analyze the process, including the general approach the client takes, strategy use, and observable problem-solving behaviors. A current limitation to performance-based testing, however, is that these tests are often time consuming, are not suitable for retesting because of the loss of novelty (Lamberts et al., 2010), and may require substantial environment setup.

Performance-Based Assessments of Executive Function

One of the challenges that occupational therapists face in working with at-risk youth is locating assessment tools that can provide an understanding of the factors that might be contributing to performance. We searched the literature for a performance-based measure of EF that was age appropriate, practical, and informative.

The Behavioral Assessment of the Dysexecutive Syndrome (BADS; Wilson, Alderman, Burgess, Emslie, & Evans, 1996), for example, is a performance-based assessment of EF that measures cognitive flexibility, novel problem solving, planning, judgment and estimation, and behavioral regulation with six subtests. The BADS is well validated in people with brain injury (Wilson, Evans, Emslie, Alderman, & Burgess, 1998), but this assessment, which takes approximately 1.5 hr to administer and score in its entirety, may not be as sensitive to the subtle executive dysfunction seen in more neurotypical populations (Norris & Tate, 2000).

Several performance-based cognitive assessments have been developed for school-age youth, including the Children’s Kitchen Task Assessment (Rocke, Hays, Edwards, & Berg, 2008) for ages 8–12, the Children’s Cooking Task (Chevignard, Catroppa, Galvin, & Anderson, 2010) for ages 8–20, and the school version of the Assessment of Motor and Process Skills (Fisher, Bryze, Hume, & Griswold, 2007) for ages 3–13. These assessments, however, either did not match the age of our youth or required kitchen facilities, limiting utility. Additionally, we wanted a task that was novel and could show improvement after 1 year of engagement in an intensive academic and internship program. One performance-based assessment, the Party Planning Assessment (Todd, 1996), is specifically for use in adolescents, but this task focuses heavily on planning without the possibility of distraction management or multitasking. Only one study has contributed to the validation of this measure (Pentland, Todd, & Anderson, 1998).

Objective

The Weekly Calendar Planning Activity (WCPA), developed by Toglia (2009), is a performance-based assessment of EF with promising ecological validity and clinical utility. Until now, no research studies have reported on the use of this assessment. The objective of our study was to describe the baseline EF profile of at-risk youth who were participating in a 2-year return-to-school education program. We sought to explore relationships between accuracy, time, strategy use, error patterns, and self-evaluation of performance.

Method

The institutional review board at Washington University in St. Louis approved this study and waived written consent. We placed individual written reports on the assessment results in each student’s academic file for teacher access.

Participants

Participants were 113 youth aged 16–21 enrolled at a midwestern city charter school for youth who have dropped out of high school or are at risk of dropping out. All students who enrolled in the school from August 2010 to December 2010 participated in this study. Participants lived in urban, low-income areas and, according to the high school social worker, were likely to have the following risk factors: pregnant or parenting teen (30%), previous dropout from high school (99%), homelessness (35%), substance abuse (35%), mental health issues (45%), or criminal history (18%).

Measures

We administered the WCPA to each student and gathered demographic information in interviews and from school records (age, birth date, gender, and race). The purpose of testing was to track program outcomes and enhance curriculum approaches.

The WCPA was developed to examine how subtle EF difficulties influence a person’s ability to perform multistep activities in daily life (Toglia, 2009). The WCPA has three levels of difficulty, from Level 1, the easiest, in which appointments are preordered, to Level 3, the most difficult, in which appointments are randomly ordered in paragraph form. We chose to use Level 2 for this study after considering the developmental stage and expected performance level of our study population.

In Level 2 of the WCPA, the participant is presented with a list of appointments in random order that he or she must schedule during 1 wk by filling in a paper calendar. Some appointments conflict with each other, so the participant must strategically place the appointments on the calendar to minimize errors and satisfy constraints. The participant is asked to follow five rules, which we made available for reference throughout the task:

Do not schedule appointments on Wednesday.
Do not answer questions from the examiner.
Tell the examiner when it is [7 min after the test begins].
Do not cross out entered appointments on the calendar.
Tell the examiner when you are finished.

The five rules aim to facilitate assessment of aspects of distraction management, prospective memory, and problem-solving ability.

The examiner observes the participant during the assessment and notes observable strategy use. Examples of strategies include entering fixed appointments before flexible appointments, rereading appointments before beginning, or self-talk. After the participant is finished with the task, the examiner administers a brief after-task interview and self-rating scale to determine the participant’s self-evaluation of his or her performance on the task. In the after-task interview, the examiner asks the participant about both observed and unobserved strategy use and the participant’s general approach to the task. The participant responds to the following six statements using a rating scale ranging from 1 = completely disagree to 6 = completely agree:

This task was easy for me.
I used an efficient approach to this task.
I had no difficulty doing this task.
I kept track of everything I needed to do.
I do tasks like this—use a planner or keep a schedule.
I would do this task the same way if I did it again.

The examiner tracks five possible error types participants make in entering appointments into the calendar: location errors, repeat errors, incomplete errors, time errors, and self-recognition errors. A location error occurs when the person places an appointment in the wrong location. A repeat error occurs when the person enters an appointment more than once on the calendar. An incomplete error occurs when the person writes an incomplete appointment name. A time error occurs when the person fails to allot the correct amount of time for an appointment. When a person recognizes that he or she has made an error, the examiner marks a self-recognition error in addition to one of the other four error types. Self-recognition errors are recorded if the person attempts to correct an error during testing, verbalizes acknowledgment of an error during testing, or reports the error during the after-task interview. For example, a participant may acknowledge that he or she made an error in the after-task interview but chose not to correct it because of the rule “Do not cross out entered appointments on the calendar.”

We used a modified version of the WCPA for this study that included activities more suited to the daily lives of our participants. For example, pick up dry cleaning was changed to pay phone bill at store. We also rewrote the exit questions to eliminate double negatives, which could be confusing to participants. Participant results consist of a profile of scores, rather than a single composite score. Preliminary normative data are available for people aged 18–90 (Toglia, 2009).

Procedures

Participants were individually tested in a private testing room using a standardized approach. The examiner presented the participant with the list of appointments, a blank weekly calendar, a list of directions and rules, scrap paper, pens and highlighters, and a visible clock in the room to monitor time. The examiner read a script outlining instructions and rules aloud to the participant. As the participant performed the task, the examiner kept track of time, observable strategy use, and adherence to rules and provided three distractions at 2, 5, and 10 min following initiation of the task. The distractions were in the form of questions: “When is your birthday?” “Do you know what the weather is supposed to be like tomorrow?” “Do you have any pets?” As noted previously, participants were instructed not to answer questions by the examiner. After the participant completed the task, the examiner administered the after-task interview and rating scale by verbally reading the questions and recording the participant’s answers on a structured answer form.

Analysis

We managed the study data using REDCap, an electronic data storage and management system hosted by a consortium of research and educational institutions. REDCap provides an interface for validated data entry, audit trails for tracking data manipulation and export, automated export for data downloads to common statistical packages, and procedures for importing data from external sources (Harris et al., 2009). We exported our data from REDcap to PASW Statistics Version 18 (IBM Corporation, Somers, NY). Descriptive statistics and Pearson correlations are reported.

Results

The characteristics of the study sample (N = 113) are presented in Table 1. The sample included 53 young women (47%) and 60 young men (53%). The mean age of the participants was 18.4 yr (range = 16–21 yr; 1 participant did not report age). The participants were predominantly African-American (91%; 8 participants did not report race). We attempted to recover missing demographic information, but this information was not held in school records, and the students were no longer enrolled.

Table 1.

Participant Characteristics

Characteristic	n	%
Gender, N = 113
Female	53	47
Male	60	53
Race, N = 105
African American	96	91
Multiracial, other	9	9
Age, yr, N = 112
16–17	28	25
18	36	32
19	22	20
20–21	26	23

View Large

Table 1.

Participant Characteristics

Characteristic	n	%
Gender, N = 113
Female	53	47
Male	60	53
Race, N = 105
African American	96	91
Multiracial, other	9	9
Age, yr, N = 112
16–17	28	25
18	36	32
19	22	20
20–21	26	23

Interrater Reliability

We determined interrater reliability for two trained scorers who independently rated 38 assessments. Each examiner recorded the scores, and a statistician calculated interrater reliability. Intraclass correlation for accuracy scores was .99, indicating a high level of interrater reliability for total accuracy scores (Portney & Watkins, 2008).

WCPA Scores

Nine students attempted but did not complete the WCPA, resulting in a sample size of 104. Participants took a mean of 15.9 (standard deviation [SD] = 5.4) min to complete the calendar task. This calculation excludes 2 participants whose time was not collected at the time of testing. Mean planning time (time from start of task to entry of first appointment on calendar) was 51.7 s (SD = 74.0); one outlier was excluded whose planning time was 1,528 s. The mean number of entered appointments was 15.7 (SD = 2.8) of 18 possible. The mean number of errors was 7.9 (SD = 3.6). No ceiling effect was observed in overall accuracy; the highest score for accurate appointments was 17. The mean number of rules followed was 4.0 (SD = 1.0) of 5 possible. Participants used a mean of 3.1 strategies (SD = 1.9) in completing the WCPA. Location errors (mean = 3.5, SD = 2.6) and time errors (mean = 3.4, SD = 2.2) were the most common error types. Mean scores are reported in Table 2.

Table 2.

Means and Standard Deviations of Scores on the Weekly Calendar Planning Activity (N = 104)

	Mean	Standard Deviation	Range
Entered appointments	15.7	2.8	1–18
Missing appointments	2.3	2.8	0–17
Total accurate	7.8	3.9	0–17
Total errors	7.9	3.6	1–17
Self-recognition errors	0.8	1.2	0–8
Location errors	3.5	2.6	0–15
Repeat errors	0.6	1.3	0–10
Incomplete errors	0.3	0.8	0–6
Time errors	3.4	2.2	0–9
Planning time, s	51.7	74.0	0–420
Total time, min	15.9	5.4	6–40
Rules followed	4.0	1.0	1–5
Strategies	3.1	1.9	0–9

View Large

Table 2.

Means and Standard Deviations of Scores on the Weekly Calendar Planning Activity (N = 104)

	Mean	Standard Deviation	Range
Entered appointments	15.7	2.8	1–18
Missing appointments	2.3	2.8	0–17
Total accurate	7.8	3.9	0–17
Total errors	7.9	3.6	1–17
Self-recognition errors	0.8	1.2	0–8
Location errors	3.5	2.6	0–15
Repeat errors	0.6	1.3	0–10
Incomplete errors	0.3	0.8	0–6
Time errors	3.4	2.2	0–9
Planning time, s	51.7	74.0	0–420
Total time, min	15.9	5.4	6–40
Rules followed	4.0	1.0	1–5
Strategies	3.1	1.9	0–9

Rules Followed

The most commonly broken rule was Tell the examiner when it is [7 min after the test begins]; 34% of participants broke this rule. The next most common rules broken were Do not answer questions from the examiner (30%) and Do not schedule appointments on Wednesday (27%). The least common rules broken were Tell the examiner when you are finished (13%) and Do not cross out entered appointments on the calendar (9%).

Strategy Use

Figure 1 presents the strategies participants used most commonly in completing the task. The most commonly used strategy was use of the index finger to help track the participant’s place on the appointment list or calendar; 62% of the sample used this strategy. Fifty-eight percent of participants engaged in management of distractions beyond the three scripted distractions, including noise outside the testing room or unplanned interruptions during testing. The next most commonly used strategies were talking aloud (36%), self-checking (33%), checking off or crossing out appointments after entering them (32%), or other strategies (26%) such as asking the examiner questions or taking breaks. The least-used strategies were writing notes to self (3%) and using color coding (1%).

Figure 1.

Strategies participants used in completing the Weekly Calendar Planning Activity (N = 104).

Full Size | Slide (.ppt)

After-Task Interview

Participants rated each after-task interview question from 1 = completely disagree to 6 = completely agree. The mode answer was 4 = slightly agree for This task was easy for me and I had no difficulty doing this task; 5 = mostly agree for I used an efficient approach to this task and I kept track of everything I needed to do; 1 = completely disagree for I do tasks like this—use a planner or keep a schedule; and 6 = completely agree for I would do this task the same way if I did it again. Pearson correlations between total errors and two interview statements—I kept track of everything I needed to do (r = .03, p = .72) and I would do this task the same way if I did it again (r = −.08, p = .45)—were not significant.

Within-WCPA Correlations

Pearson correlations for the WCPA results are presented in Table 3. Total time correlated significantly with planning time, as expected (p < .05); total time includes planning time. The number of entered appointments correlated significantly with total time (p < .01), indicating that participants who spent more time on the calendar task also entered more appointments. Total number of accurate appointments correlated significantly with total time (p < .05), indicating that participants who spent more time on the WCPA were more accurate. Accuracy also significantly correlated with total entered appointments, as expected (p < .01); appointments have to be entered to be accurate. The number of rules followed correlated significantly with total accuracy scores (p < .05), indicating that participants who were more accurate also tended to follow more rules. The number of strategies used correlated significantly with planning time (p < .05) and with total time and total accuracy (p < .01), meaning that participants who used more strategies also spent more time planning and completing the task and were more accurate.

Table 3.

Pearson Correlations Among Weekly Calendar Planning Activity Results (N = 104)

Variable	Planning Time	Planning Time Transformed	1	2	3	4	5
1. Total time	.22*	−.02
2. Entered appointments	.09	.07	.28**
3. Entered appointments transformed	.10	.04	.28**	—
4. Total accuracy	.09	−.07	.24*	.49**	.51**
5. Rules followed	.03	−.03	−.16	−.02	−.03	.24*
6. Strategies	.25*	−.09	.42**	.18	.22*	.29**	−.07

Note. — = correlation was not calculated.

*p < .05. **p < .01.

View Large

Table 3.

Pearson Correlations Among Weekly Calendar Planning Activity Results (N = 104)

Variable	Planning Time	Planning Time Transformed	1	2	3	4	5
1. Total time	.22*	−.02
2. Entered appointments	.09	.07	.28**
3. Entered appointments transformed	.10	.04	.28**	—
4. Total accuracy	.09	−.07	.24*	.49**	.51**
5. Rules followed	.03	−.03	−.16	−.02	−.03	.24*
6. Strategies	.25*	−.09	.42**	.18	.22*	.29**	−.07

Note. — = correlation was not calculated.

*p < .05. **p < .01.

The distribution of planning time was positively skewed (3.1, standard error [SE] = 0.24) and kurtotic (11.2, SE = 0.47): 80% of participants scored <1 min of planning time. Planning time data were monotonically transformed into four quartiles to reduce the effects of skewness and kurtosis on the correlation statistics. Pearson correlations between the transformed planning time variable and other WPCA variables remained unchanged, except prior significant relationships with total time and number of strategies were no longer significant (see Table 3).

Additionally, the distribution of entered appointments was skewed negatively (−2.97, SE = 0.24) and kurtotic (10.8, SE = 0.47), with 85% of scores ranging from 14 to 18. Entered appointments were monotonically transformed by arcsine to reduce the effects of skewness and kurtosis on the correlation statistics (see Table 3). Pearson correlations between the transformed entered appointment variable and other WCPA variables remained unchanged except for a weak and significant relationship to number of strategies.

Discussion

The results of this study indicate that the WCPA offers unique information for capturing the everyday EF in transition-age at-risk adolescents. First, the complexity of the task and the broad range of scores suggest that the WCPA may be useful in assessing even subtle EF difficulties that affect performance in daily life. Second, the WCPA may facilitate structured activity analysis and assessment of elements of complex task performance, including errors, self-evaluation, and strategy use, which may guide intervention. Finally, our findings contribute to evidence regarding the clinical utility of the WCPA, including its interrater reliability, face validity, and practicality.

Complexity of the WCPA

It is noteworthy that participants who struggled with the task struggled with multiple components of the WCPA, supporting EF as a dynamic and interwoven control process (Stuss & Alexander, 2000; Suchy, 2009). Participants who were less accurate tended to follow fewer rules and use less time and fewer strategies. More complex tasks may recruit more frontal lobe resources, challenging the validity of single-construct neurocognitive tests (Stuss & Alexander, 2000). The WCPA, with its complexity, is more analogous to real-world tasks (e.g., driving) that elicit the use of multiple EFs at one time.

Spooner and Pachana (2006) emphasized that newly developed measures of EF should be sensitive enough to provide a broad range of scores that are of use with neurotypical populations. The broad range of scores and lack of a ceiling effect that we observed suggest that the WCPA may reveal information about even the subtlest of EF difficulties; however, this preliminary finding warrants further investigation.

Error Patterns, Self-Evaluation, and Strategy Use

In past studies with neurocognitive tests, errors have generally been shown to decrease with age from early adolescence to late adolescence (Somsen, 2007). Adolescence, however, is a period of neurodiversity, during which development of EF may follow variable developmental trajectories related to vulnerabilities such as stress, risk taking, and the social context (Crone, 2009; Steinberg, 2005). Adolescents whose developmental course is challenged by complications such as poverty and lack of family support may have a wider range of variability in their neurocognitive development, as reflected in the range of scores on the WCPA of the study participants. The finding that correlations were not significant between total errors and two statements from the after-task interview—I kept track of everything I needed to do and I would do this task the same way if I did it again—implies that these youth may have lacked awareness about the quality of their own performance after task completion. This possibility requires further study.

The WCPA may reveal aspects of EF that cannot be uncovered through self-report alone; Manchester and colleagues (2004) called for an observation-based approach to cognitive evaluation because of weak to moderate relationships between standardized tests and everyday behavior. Although rating scales for EF that include items related to everyday performance, such as the Behavior Rating of Executive Function, have promising ecological validity (Gioia et al., 2010), use of such tools in isolation may not have provided a complete picture for our sample of at-risk youth. Because proxy reports from parents or teachers were unavailable for our transient population, our performance results may suggest that these youth do not reliably report problems with EF. It also may suggest that, before addressing EF strategies, students need assistance in learning to monitor and self-evaluate their ongoing performance.

In addition to shedding light on participants’ awareness of their performance, the WCPA may be a valuable tool for observing strategy use and patterns of task errors through a task analysis method. Participants who performed better on the task were more deliberate and accurate overall—they took more time, used more strategies, and followed more rules. Pacing emerged as a strategy for performance of this task, but participants did not take full advantage of planning time. They may have considered taking time to decide on strategies as a cost in their cost–benefit analysis, in part guided by former experiences. Without performance feedback or memories to apply to this novel situation, the participants may have weighed time use against accuracy, influencing their approach to the task; the WPCA offers a window into this aspect of EF (Marewski & Schooler, 2011). Occupational therapy practitioners can develop interventions that help adolescents consider the use of time as a strategy and practice the process of strategy selection in multiple-step activities to enhance performance, an approach that is supported in educational research (Harris & Pressley, 1991). Training in cognitive strategies (i.e., the planning and self-regulation of goal-directed behavior needed to complete a task) is a promising yet underused educational method for improving learning outcomes in adolescents (Conley, 2008) and may be especially important for populations at risk of school failure. As Conley (2008) emphasized, success in the transition from adolescence to adulthood in the areas of work or higher education requires that students master the use of cognitive strategies, including complex problem solving. Existing research in the area of cognitive strategy training, however, mostly involves young children performing simple cognitive or motor tasks, so research in this area is needed addressing at-risk youth.

Moreover, it is interesting that examiners reported greater use of external strategies and little use of internal strategies. For example, the strategy of finger pointing (an external strategy) is much less sophisticated than entering all fixed appointments first (an internal strategy). The examination and between-group comparison of quality and type of strategies used may be important, and further investigation is warranted.

Utility of the WCPA

The teachers of our participants found the information generated by the WCPA to be useful in establishing teaching approaches for each student. Additionally, this study contributed to evidence regarding the utility of this measure. According to Matheson (1994), the utility of an instrument is determined by a combination of safety, reliability, validity, and practicality. We found the task to be safe, with no obvious risk or harm (beyond frustration) from completing the task. Reliability was supported through the establishment of interrater reliability for accuracy scores.

Validity is defined as the ability of results to be predictive of everyday performance in a given target area (Matheson, 1994). We found the task itself—the management of a calendar—to have face validity for our population because this is a feasible task for transition-age young adults entering the workforce or college. Moreover, the appointment list can be modified easily, suggesting that it can be customized to increase face validity for different populations.

Last, we found the WCPA to be a practical measure. A disadvantage of many other performance-based tests of EF is that they may require substantial time or environmental setup. For example, the Multiple Errands Shopping Test (Alderman et al., 2003; Shallice & Burgess, 1991) is useful for examining subtle EF difficulties, but it requires that the participant be assessed in a shopping center. The Executive Secretarial Task (EST; Lamberts et al., 2010), like the WCPA, requires the person to organize and prioritize multiple tasks and deal with multiple distractions, but the EST takes 3 hr to complete. On average, our participants completed the WCPA in 16 min. In addition, the WCPA requires little environmental setup, cost, or materials, making it practical for use in a variety of contexts, clinical and research.

Implications for Occupational Therapy Practice

The results of this study have the following implications for occupational practice:

Occupational therapy has a practical performance-based tool for identifying EF skills in high school students struggling academically.
We have established some psychometric properties of the WPCA, including its interrater reliability and face validity with at-risk youth ages 16 to 21.
The WCPA has potential to lead directly to interventions such as the development of strategies connected to school success.
Occupational therapy has an opportunity to expand its role to better prepare at-risk youth with the skills and habits necessary for success in life.

Limitations and Future Directions

At-risk transition-age adolescents are an understudied population that deserves consideration by researchers and service providers. Our results cannot be generalized to other populations, however, until testing occurs with a more representative sample of 16–21-yr-olds. Evaluation of EF may be particularly important for high-risk populations, and further studies should compare EF in high-risk populations versus typical age-matched controls.

The socioeconomic and race divide between our examiners and our sample was a limitation, in addition to the fact that low levels of literacy may have been present in some participants, particularly the 9 who did not complete the WCPA. The students might have had no interest in this task, affecting performance, and it is possible that the task may not be valid for this age group if they rely on electronic calendar systems.

Researchers should continue to explore the potential of this new instrument. Future studies should focus on assessing a variety of populations, both neurotypical and with diagnosed cognitive disabilities, from a wide range of geographic and ethnic groups.

Conclusion

The objective of this study was to describe use of the WPCA to evaluate the baseline EF profile of at-risk youth who were participating in a 2-year return-to-school education program. The WCPA is a new evaluation tool that can aid occupational therapy practitioners in identifying the cognitive strategies youth use to complete a complex task. This information may support interventions with youth at risk of failing high school, including training in cognitive strategies such as time use and self-awareness of performance.

Although occupational therapists have traditionally worked in school settings to fulfill legal mandates, an unmet opportunity exists to expand their role to better prepare high school youth with the skills and habits necessary for success in life (Gangl, Neufeld, & Berg, 2011). Occupational therapy practitioners can be classroom consultants, making recommendations to improve classroom performance. As we confirmed through the partnership formed with this high school, “expanding our roles beyond the traditional service to persons with specific disabilities provides opportunities for occupational therapy knowledge to support a more successful and satisfying experience for everyone” (Law, Baum, & Dunn, 2005, p. 377).

Acknowledgments

We thank all students and staff who participated in this study. We also thank Lisa Connor, Erin Foster, and Debora Davidson for their valuable input to this article.

References

Alderman, N., Burgess, P. W., Knight, C., & Henman, C. (2003). Ecological validity of a simplified version of the Multiple Errands Shopping Test. Journal of the International Neuropsychological Society, 9, 31–44. http://dx.doi.org/10.1017/S1355617703910046 [Article] [PubMed]

Baum, C. M., Connor, L. T., Morrison, T., Hahn, M., Dromerick, A. W., & Edwards, D. F. (2008). Reliability, validity, and clinical utility of the Executive Function Performance Test: A measure of executive function in a sample of people with stroke. American Journal of Occupational Therapy, 62, 446–455. http://dx.doi.org/10.5014/ajot.62.4.446 [Article] [PubMed]

Blakemore, S. J., & Choudhury, S. (2006). Development of the adolescent brain: Implications for executive function and social cognition. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 47, 296–312. http://dx.doi.org/10.1111/j.1469-7610.2006.01611.x [Article] [PubMed]

Burgess, P. W., Alderman, N., Evans, J., Emslie, H., & Wilson, B. A. (1998). The ecological validity of tests of executive function. Journal of the International Neuropsychological Society, 4, 547–558. http://dx.doi.org/10.1017/S1355617798466037 [Article] [PubMed]

Burgess, P. W., Alderman, N., Forbes, C., Costello, A., Coates, L. M., Dawson, D. R.et al (2006). The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology. Journal of the International Neuropsychological Society, 12, 194–209. http://dx.doi.org/10.1017/S1355617706060310 [Article] [PubMed]

Burgess, P. W., & Simons, J. S. (2005). Theories of frontal lobe executive function: Clinical applications. In Halligan, P. W., & Wade, D. T. (Eds.), The effectiveness of rehabilitation for cognitive deficits (pp. 211–231). Oxford, England: Oxford University Press.

Chevignard, M. P., Catroppa, C., Galvin, J., & Anderson, V. (2010). Development and evaluation of an ecological task to assess executive functioning post childhood TBI: The Children’s Cooking Task. Brain Impairment, 11, 125–143. http://dx.doi.org/10.1375/brim.11.2.125 [Article]

Conley, M. W. (2008). Cognitive strategy instruction for adolescents: What we know about the promise, what we don’t know about the potential. Harvard Educational Review, 78, 84–106.

Crone, E. A. (2009). Executive functions in adolescence: Inferences from brain and behavior. Developmental Science, 12, 825–830. http://dx.doi.org/10.1111/j.1467-7687.2009.00918.x [Article] [PubMed]

Davis, T. L., Kerr, B. A., & Robinson Kurpius, S. E. (2003). Meaning, purpose, and religiosity in at-risk youth: The relationship between anxiety and spirituality. Journal of Psychology and Theology, 31, 356–365.

Eisenberg, N., Fabes, R. A., Shepard, S. A., Murphy, B. C., Guthrie, I. K., Jones, S.et al (1997). Contemporaneous and longitudinal prediction of children’s social functioning from regulation and emotionality. Child Development, 68, 642–664. http://dx.doi.org/10.2307/1132116 [Article] [PubMed]

Fisher, A. G., Bryze, K., Hume, V., & Griswold, L. A. (2007). School AMPS: School version of the Assessment of Motor and Process Skills (2nd ed.). Fort Collins, CO: Three Star Press.

Freund, A. M., & Baltes, P. B. (1998). Selection, optimization, and compensation as strategies of life management: Correlations with subjective indicators of successful aging. Psychology and Aging, 13, 531–543. http://dx.doi.org/10.1037/0882-7974.13.4.531 [Article] [PubMed]

Fuster, J. M. (2002). Frontal lobe and cognitive development. Journal of Neurocytology, 31, 373–385. http://dx.doi.org/10.1023/A:1024190429920 [Article] [PubMed]

Gangl, C., Neufeld, P. S., & Berg, C. (2011). A qualitative study of occupational therapy’s role in adolescent transition in a midwestern coalition of many school districts. Journal of Occupational Therapy, Schools, and Early Intervention, 4, 154–167. http://dx.doi.org/10.1080/19411243.2011.595313 [Article]

Gardner, T. W., Dishion, T. J., & Connell, A. M. (2008). Adolescent self-regulation as resilience: Resistance to antisocial behavior within the deviant peer context. Journal of Abnormal Child Psychology, 36, 273–284. http://dx.doi.org/10.1007/s10802-007-9176-6 [Article] [PubMed]

Gestsdottir, S., & Lerner, R. M. (2008). Positive development in adolescence: The development and role of intentional self-regulation. Human Development, 51(3), 202–224. [Article]

Gioia, G. A., & Isquith, P. K. (2004). Ecological assessment of executive function in traumatic brain injury. Developmental Neuropsychology, 25, 135–158. [Article] [PubMed]

Gioia, G. A., Kenworthy, L., & Isquith, P. K. (2010). Executive function in the real world: BRIEF lessons from Mark Ylvisaker. Journal of Head Trauma Rehabilitation, 25, 433–439. http://dx.doi.org/10.1097/HTR.0b013e3181fbc272 [Article] [PubMed]

Harris, K. R., & Pressley, M. (1991). The nature of cognitive strategy instruction: Interactive strategy construction. Exceptional Children, 57, 392–404.

Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42, 377–381. http://dx.doi.org/10.1016/j.jbi.2008.08.010 [Article] [PubMed]

Hester, R., Lubman, D. I., & Yücel, M. (2010). The role of executive control in human drug addiction. Current Topics in Behavioral Neurosciences, 3, 301–318. http://dx.doi.org/10.1007/7854_2009_28 [PubMed]

Kishiyama, M. M., Boyce, W. T., Jimenez, A. M., Perry, L. M., & Knight, R. T. (2009). Socioeconomic disparities affect prefrontal function in children. Journal of Cognitive Neuroscience, 21, 1106–1115. http://dx.doi.org/10.1162/jocn.2009.21101 [Article] [PubMed]

Kvavilashvili, L., & Ellis, J. (2004). Ecological validity and the real life/laboratory controversy in memory research: A critical and historical review. History and Philosophy of Psychology, 6, 59–80.

Lamberts, K. F., Evans, J. J., & Spikman, J. M. (2010). A real-life, ecologically valid test of executive functioning: The executive secretarial task. Journal of Clinical and Experimental Neuropsychology, 32, 56–65. [Article] [PubMed]

Law, M., Baum, C., & Dunn, W. (2005). Challenges and strategies in applying an occupational performance measurement approach. In Law, M., Baum, C., & Dunn, W. (Eds.), Measuring occupational performance: Supporting best practice in occupational therapy (pp. 375–381). Thorofare, NJ: Slack.

Manchester, D., Priestley, N., & Jackson, H. (2004). The assessment of executive functions: Coming out of the office. Brain Injury, 18, 1067–1081. http://dx.doi.org/10.1080/02699050410001672387 [Article] [PubMed]

Marewski, J. N., & Schooler, L. J. (2011). Cognitive niches: An ecological model of strategy selection. Psychological Review, 118, 393–437. http://dx.doi.org/10.1037/a0024143 [Article] [PubMed]

Matheson, L. N. (1994). Basic requirements for utility in the assessment of physical disability. American Pain Society Journal, 3, 193–199.

Norris, G., & Tate, R. L. (2000). The Behavioural Assessment of the Dysexecutive Syndrome (BADS): Ecological, concurrent, and construct validity. Neuropsychological Rehabilitation, 10, 33–45. http://dx.doi.org/10.1080/096020100389282 [Article]

Paus, T. (2005). Mapping brain maturation and cognitive development during adolescence. Trends in Cognitive Sciences, 9, 60–68. http://dx.doi.org/10.1016/j.tics.2004.12.008 [Article] [PubMed]

Pentland, L., Todd, J. A., & Anderson, V. (1998). The impact of head injury severity on planning ability in adolescence: A functional analysis. Neuropsychological Rehabilitation, 8, 301–317. http://dx.doi.org/10.1080/713755572 [Article]

Portney, L. G., & Watkins, M. P. (2008). Foundations of clinical research: Applications to practice (3rd ed.). Upper Saddle River, NJ: Prentice Hall.

Rocke, K., Hays, P., Edwards, D., & Berg, C. (2008). Development of a performance assessment of executive function: The Children’s Kitchen Task Assessment. American Journal of Occupational Therapy, 62, 528–537. http://dx.doi.org/10.5014/ajot.62.5.528 [Article] [PubMed]

Romer, D., Betancourt, L. M., Brodsky, N. L., Giannetta, J. M., Yang, W., & Hurt, H. (2011). Does adolescent risk taking imply weak executive function? A prospective study of relations between working memory performance, impulsivity, and risk taking in early adolescence. Developmental Science, 14, 1119–1133. http://dx.doi.org/10.1111/j.1467-7687.2011.01061.x [Article] [PubMed]

Sarsour, K., Sheridan, M., Jutte, D., Nuru-Jeter, A., Hinshaw, S., & Boyce, W. T. (2011). Family socioeconomic status and child executive functions: The roles of language, home environment, and single parenthood. Journal of the International Neuropsychological Society, 17, 120–132. http://dx.doi.org/10.1017/S1355617710001335 [Article] [PubMed]

Shallice, T., & Burgess, P. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727–741. http://dx.doi.org/10.1093/brain/114.2.727 [Article] [PubMed]

Somsen, R. J. (2007). The development of attention regulation in the Wisconsin Card Sorting Task. Developmental Science, 10, 664–680. [Article] [PubMed]

Spooner, D. M., & Pachana, N. A. (2006). Ecological validity in neuropsychological assessment: A case for greater consideration in research with neurologically intact populations. Archives of Clinical Neuropsychology, 21, 327–337. http://dx.doi.org/10.1016/j.acn.2006.04.004 [Article] [PubMed]

Steinberg, L. (2005). Cognitive and affective development in adolescence. Trends in Cognitive Sciences, 9, 69–74. http://dx.doi.org/10.1016/j.tics.2004.12.005 [Article] [PubMed]

Stuss, D. T., & Alexander, M. P. (2000). Executive functions and the frontal lobes: A conceptual view. Psychological Research, 63, 289–298. http://dx.doi.org/10.1007/s004269900007 [Article] [PubMed]

Suchy, Y. (2009). Executive functioning: Overview, assessment, and research issues for non-neuropsychologists. Annals of Behavioral Medicine, 37, 106–116. http://dx.doi.org/10.1007/s12160-009-9097-4 [Article] [PubMed]

Todd, J. A. (1996). Planning skills in head-injured adolescents and their peers. Neuropsychological Rehabilitation, 6, 81–100. http://dx.doi.org/10.1080/713755500 [Article]

Toglia, J. (2009). Weekly Calendar Planning Activity. Dobbs Ferry, NY: Mercy College.

Toglia, J., Johnston, M. V., Goverover, Y., & Dain, B. (2010). A multicontext approach to promoting transfer of strategy use and self regulation after brain injury: An exploratory study. Brain Injury, 24, 664–677. http://dx.doi.org/10.3109/02699051003610474 [Article] [PubMed]

Wilson, B. A., Alderman, N., Burgess, P. W., Emslie, H., & Evans, J. J. (1996). Behavioral Assessment of the Dysexecutive Syndrome. Bury St. Edmunds, England: Thames Valley Test Company.

Wilson, B. A., Evans, J. J., Emslie, H., Alderman, N., & Burgess, P. W. (1998). The development and validation of an ecologically valid test for assessing patients with a dysexecutive syndrome. Neuropsychological Rehabilitation, 8, 213–228. http://dx.doi.org/10.1080/713755570 [Article]

Worthington, A. (2005). Rehabilitation of executive deficits. In Halligan, P. W., & Wade, D. T. (Eds.), The effectiveness of rehabilitation for cognitive deficits (pp. 257–267). Oxford, England: Oxford University Press.

Table 1.

Participant Characteristics

Characteristic	n	%
Gender, N = 113
Female	53	47
Male	60	53
Race, N = 105
African American	96	91
Multiracial, other	9	9
Age, yr, N = 112
16–17	28	25
18	36	32
19	22	20
20–21	26	23

View Large

Table 1.

Participant Characteristics

Characteristic	n	%
Gender, N = 113
Female	53	47
Male	60	53
Race, N = 105
African American	96	91
Multiracial, other	9	9
Age, yr, N = 112
16–17	28	25
18	36	32
19	22	20
20–21	26	23

Table 2.

Means and Standard Deviations of Scores on the Weekly Calendar Planning Activity (N = 104)

	Mean	Standard Deviation	Range
Entered appointments	15.7	2.8	1–18
Missing appointments	2.3	2.8	0–17
Total accurate	7.8	3.9	0–17
Total errors	7.9	3.6	1–17
Self-recognition errors	0.8	1.2	0–8
Location errors	3.5	2.6	0–15
Repeat errors	0.6	1.3	0–10
Incomplete errors	0.3	0.8	0–6
Time errors	3.4	2.2	0–9
Planning time, s	51.7	74.0	0–420
Total time, min	15.9	5.4	6–40
Rules followed	4.0	1.0	1–5
Strategies	3.1	1.9	0–9

View Large

Table 2.

Means and Standard Deviations of Scores on the Weekly Calendar Planning Activity (N = 104)

	Mean	Standard Deviation	Range
Entered appointments	15.7	2.8	1–18
Missing appointments	2.3	2.8	0–17
Total accurate	7.8	3.9	0–17
Total errors	7.9	3.6	1–17
Self-recognition errors	0.8	1.2	0–8
Location errors	3.5	2.6	0–15
Repeat errors	0.6	1.3	0–10
Incomplete errors	0.3	0.8	0–6
Time errors	3.4	2.2	0–9
Planning time, s	51.7	74.0	0–420
Total time, min	15.9	5.4	6–40
Rules followed	4.0	1.0	1–5
Strategies	3.1	1.9	0–9

Table 3.

Pearson Correlations Among Weekly Calendar Planning Activity Results (N = 104)

Variable	Planning Time	Planning Time Transformed	1	2	3	4	5
1. Total time	.22*	−.02
2. Entered appointments	.09	.07	.28**
3. Entered appointments transformed	.10	.04	.28**	—
4. Total accuracy	.09	−.07	.24*	.49**	.51**
5. Rules followed	.03	−.03	−.16	−.02	−.03	.24*
6. Strategies	.25*	−.09	.42**	.18	.22*	.29**	−.07

Note. — = correlation was not calculated.

*p < .05. **p < .01.

View Large

Table 3.

Pearson Correlations Among Weekly Calendar Planning Activity Results (N = 104)

Variable	Planning Time	Planning Time Transformed	1	2	3	4	5
1. Total time	.22*	−.02
2. Entered appointments	.09	.07	.28**
3. Entered appointments transformed	.10	.04	.28**	—
4. Total accuracy	.09	−.07	.24*	.49**	.51**
5. Rules followed	.03	−.03	−.16	−.02	−.03	.24*
6. Strategies	.25*	−.09	.42**	.18	.22*	.29**	−.07

Note. — = correlation was not calculated.

*p < .05. **p < .01.