To address the objective of positioning occupational therapy as an evidence-based and science-driven profession in accordance with the American Occupational Therapy Association’s (AOTA’s; 2007) Centennial Vision, we critically appraised articles addressing intervention effectiveness for children and youth published in 2012 in the American Journal of Occupational Therapy (AJOT). We ascertained how well this group of studies facilitated the generation of evidence sufficient to support practice that fosters innovation in occupational therapy practice, embraces excellence, and sharpens occupational therapy’s competitive edge (Clark, 2011). This overall effort was designed to examine AJOT’s performance in three key areas: (1) guiding practitioners to make evidence-based decisions by disseminating well-designed clinical trials; (2) raising the awareness of practitioners, clients, and third-party payers of science-driven and evidence-based innovations that have the potential to improve participation and quality of life; and (3) ultimately influencing health care decisions for current and future clients.

The World Health Organization (WHO; 2007) has identified the primary purpose of occupational therapy intervention for children and youth as fostering quality of life and ensuring societal involvement in everyday activities and real-world situations or occupations. The area of children and youth has been identified by AJOT as one of the six main practice areas of the occupational therapy profession. Occupational therapy research addressing children and youth is an important focus area for evaluating progress toward the Centennial Vision because a substantial proportion of occupational therapy practitioners work with children and youth in schools and in early intervention (26.9% of occupational therapists, 23.2% of occupational therapy assistants; AOTA, 2010). AOTA’s (2006)  Children and Youth Ad Hoc Committee identified the following key areas of research to inform practice in the area of children and youth (p. 8):

• Basic and applied scientific studies related to skills, processes, and foundations for childhood and adolescent occupations

• Factors that contribute to the success or failure of a specific frame of reference

• Both qualitative and quantitative methodologies to address multiple facets of these factors

• Efficacy studies that examine interventions (efficacy, effectiveness, outcomes development)

• Theory development and development of conceptual models that promote integration of theory and practice

• Empirical studies conducted in context

• Translational research providing information on applications to practice, policy development, systems change, and program development

• Roles and participation of parents, siblings, and other family members within family-centered services

• Longitudinal studies of the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood

• Studies that examine factors central to children, youth, and their families such as finding a friend, participating in community life, and procuring and maintaining jobs

• Studies that examine the emotional and social costs of occupational deprivation and occupational injustice to children, youth, and their family, such as depression, alcohol and substance abuse, and suicide in disenfranchised youth, as well as to society.

In addition to the Ad Hoc Committee’s recommendations, Gutman (2008, p. 499) further charged the profession to fulfill the mission of the Centennial Vision by meeting five specific research priorities:

1. Provide additional evidence for the efficacy of clinical practice

2. Test the reliability and validity of our assessment instruments

3. Examine how engagement in occupation can promote developmental milestones, health and wellness throughout the lifespan, and productive aging

4. Provide fundamental or basic research information regarding how specific disability experiences affect community and social participation—with the intent to ultimately use this information to develop clinical guidelines that can be tested for efficacy

5. Explore topical questions (i.e., current issues) whose answers will provide direction for the profession’s continued growth and evolution.

As the profession strives to meet the ambitious charge to position occupational therapy within higher evidentiary strata, it is important to define and distinguish efficacy of practice from effectiveness trials. Evidence for practice falls along a continuum, or into a hierarchy, with research designs with higher levels of rigor being less vulnerable to bias and error, better able to be generalized, and more likely to have outcomes attributed to the intervention being studied. Multiple methods are available for classifying evidence, in part because of discipline-specific methods that have distinct purposes, orientations, theories, and research methods uniquely developed to answer discipline-specific questions (Reichow, Volkmar, & Cicchetti, 2008). Efficacy refers to the ability of an intervention to produce the desired beneficial effects when implemented by expert clinicians and in ideal circumstances and essentially encompasses four key concepts, each typically assessed in a separate research study (Sussman, Valente, Rohrbach, Skara, & Pentz, 2006). Efficacy studies ask questions about the effectiveness of clinical interventions, their safety, their cost–benefit ratio in regard to time and cost, and their acceptability to clients.

Similarly, effectiveness intervention studies compare two independent variables such as intervention, medication, surgery, or therapeutic practice and evaluate the extent to which the intervention produces the intended change in the outcome of interest. Comparative effectiveness research compares existing health care interventions to determine which works best, for whom, and under what circumstances. Researchers have suggested that occupational therapy practitioners need to look at all levels of evidence (I–V) and not fixate on a rigid standard or they may miss effective and emerging treatments.

Pragmatic intervention studies look at questions related to the overall effectiveness of an intervention as it is delivered in treatment-as-usual methods, thus referring to clinical research that evaluates a therapy holistically, as it is used in normal practice. When everyday clinical practice is modeled, researchers are allowed to individualize protocols, adapt treatment, and ultimately test the overall package of care, but pragmatic intervention studies are not designed to empirically evaluate the contributions of an intervention’s components in isolation (McPherson, Smith-Lovin, & Brashears, 2006). The strength of pragmatic research is its ability to test the effectiveness of an intervention as it is meant to be used in practice. This ability is balanced against the benefit of controlling variables in a true experimental model. Many occupational therapy effectiveness trials may be better regarded as pragmatic interventions. What is important is to help clinicians learn to evaluate and analyze research to assess appropriate and reasonable interventions (Reichow et al., 2008).

To analyze the 11 intervention effectiveness studies related to children and youth published in AJOT during 2012, we first identified the research design used in each study, determined whether it had the rigor to provide evidence for practice, and discussed implications of use of this evidence by practitioners, clients, and third-party payers. We then used the level-of-evidence hierarchy system developed by the AOTA Evidence-Based Literature Review Project (Lieberman & Scheer, 2002) to classify the reviewed studies. Level I is the highest level of research, encompassing systematic reviews, meta-analyses, and randomized controlled trials. Level II study designs consist of two-group pretest–posttest designs in which a control is present and randomization is not (e.g., cohort designs, case–control studies). Level III study designs include neither control nor randomization but instead use a one-group pretest–posttest design. Level IV includes studies involving single-subject designs, descriptive studies, and case series. Level V involves case studies or expert opinion studies that are not based on systematic review. Consistent with the review of 2011 articles on children and youth (Hilton & Smith, 2012), we used a pediatric adaptation of Baum’s (2011)  expanded International Classification of Functioning, Disability and Health categories (WHO, 2007) to examine levels of mechanisms of rehabilitation science. Also, following Gutman’s (2008)  charge, we identified alignment with research priorities for the Centennial Vision. Finally, to optimize clinicians’ ability to apply research to informed practice, we assessed the 11 articles against criteria of efficacy, effectiveness, comparative effectiveness, and pragmatic methodology (see Table 1).

Intervention effectiveness studies in the area of children and youth have shown an increasing trend in numbers and evidence level of studies over the past 4 yr (Table 2). Ages of children examined in the 2012 studies ranged from <4 to 21 yr. All 11 studies involved quantitative data analysis, although 4 studies (27%) also incorporated qualitative measures (Case-Smith, Holland, Lane, & White 2012; de Brito Brandão, Gordon, & Mancini, 2012; Kinnealey et al., 2012; Schaaf, Hunt, & Benevides, 2012). Only 1 study enrolled children younger than age 3 (Silva, Schalock, Garberg, & Smith, 2012). Four of the studies enrolled children younger than age 6 (Case-Smith, DeLuca, Stevenson, & Ramey, 2012; Dunn, Cox, Foster, Mische-Lawson, & Tanquary, 2012; Schaaf et al., 2012; Silva et al., 2012), 6 enrolled children between ages 6 and 12 (Case-Smith, Holland, et al., 2012; de Brito Brandão et al., 2012; Dunn et al., 2012; Koenig, Buckley-Reen, & Garg, 2012; Palsbo & Hood-Szivek, 2012; Wells, Chasnoff, Schmidt, Telford, & Schwartz, 2012), and 2 enrolled adolescents (Gutman, Raphael-Greenfield, & Rao, 2012; Kinnealey et al., 2012). Four (36%) of the studies were Level I (Case-Smith, DeLuca, et al., 2012; de Brito Brandão et al., 2012; Silva et al., 2012; Wells et al., 2012), 2 (18%) were Level II (Gutman et al., 2012; Koenig et al., 2012), 3 (27%) were Level III (Case-Smith, Holland, et al., 2012; Dunn et al., 2012; Palsbo & Hood-Szivek, 2012), and 2 (18%) were Level IV (Kinnealey et al., 2012; Schaaf et al., 2012).

Researchers addressed five of the seven areas of the framework mechanisms. Specific mechanisms examined included body function and body structure (Kinnealey et al., 2012; Schaaf et al., 2012; Wells et al., 2012), functional limitations (Case-Smith, DeLuca, et al., 2012; Gutman et al., 2012; Koenig et al., 2012; Silva et al., 2012; Wells et al., 2012), activity (Case-Smith, Holland, et al., 2012; de Brito Brandão et al., 2012; Gutman et al., 2012), participation (Dunn et al., 2012), and environment (Kinnealey et al., 2012). See Table 1 for framework categories.

More than half (63%) of the studies used pragmatic trial methodology, assessing treatment in context as it is meant to be delivered (Case-Smith, DeLuca, et al., 2012; Case-Smith, Holland, et al., 2012; de Brito Brandão et al., 2012; Dunn et al., 2012; Gutman et al., 2012; Koenig et al., 2012; Wells et al., 2012;Table 3). One study addressed comparative effectiveness (de Brito Brandão et al., 2012). Additional criteria specific to the profession, including reliability and validity of test instruments, were examined in 2 of the studies (Palsbo & Hood-Szivek, 2012; Silva et al., 2012). Engagement in occupation and its ability to promote developmental milestones, health, and wellness were examined in 3 studies (Gutman et al., 2012; Kinnealey et al., 2012; Koenig et al., 2012). Three studies included examination of how specific disability experiences affect community and social participation (Dunn et al., 2012; Gutman et al., 2012; Schaaf et al., 2012), and 6 included questions that guide professional evolution (Case-Smith, Holland, et al., 2012; Dunn et al., 2012; Gutman et al., 2012; Kinnealey et al., 2012; Palsbo & Hood-Szivek, 2012; Silva et al., 2012). Acceptability of treatment was appraised in one study (Case-Smith, DeLuca, et al., 2012), although client satisfaction was considered more often (Case-Smith, Holland, et al., 2012; Dunn et al., 2012; Kinnealey et al., 2012). Schaaf et al. (2012)  specifically explored factors that contribute to the success or failure of a specific frame of reference, namely the Ayres Sensory Integration approach.

Several of the studies included measures to increase scientific rigor. Use of a manualized intervention or reference to use of a specific treatment protocol was described in 7 (64%) studies (Case-Smith, DeLuca, et al., 2012; Case-Smith, Holland, et al., 2012; de Brito Brandão et al., 2012; Gutman et al., 2012; Koenig et al., 2012; Silva et al., 2012; Wells et al., 2012), with one study (Dunn et al., 2012) measuring the usability of a manualized instrument to address parenting stress in preparation of future efficacy trials. Palsbo and Hood-Szivek (2012)  used a robot-assisted intervention with consistent procedures but no manual or specific protocol describing the verbal instructions or lesson plans. Assessments of adherence to intervention fidelity were included in 7 (64%) studies (Case-Smith, DeLuca, et al., 2012, Case-Smith, Holland, et al., 2012; Dunn et al., 2012; Gutman et al., 2012; Kinnealey et al., 2012; Koenig et al., 2012; Schaaf et al., 2012). In 1 study (Dunn et al., 2012), coaching logs were used to attain intervention fidelity for the use of reflective questioning strategies.

Evaluators were blinded to the participants’ condition in 2 (18%) studies (Case-Smith, DeLuca, et al., 2012; Kinnealey et al., 2012) and were partially blinded in another study (Case-Smith, Holland, et al., 2012). Two (18%) studies (Koenig et al., 2012, on economically disadvantaged children; Wells et al., 2012, on fetal alcohol syndrome) examined the emotional and social costs of occupational deprivation and occupational injustice to children and youth and to family as well as society.

Intervention was administered in a variety of contexts in 1 study (Wells et al., 2012) and with participants from a variety of environments supporting the potential for generalizability of the outcomes in another (Silva et al., 2012). Multisite administration of the intervention was an aspect included in 1 study (Case-Smith, DeLuca, et al., 2012).

Maintenance of improvement was examined in 2 (18%) studies (Gutman et al., 2012; Silva et al., 2012). Gutman and associates (2012)  found carryover of skills during a no-intervention period between Phases 1 and 2 of the intervention. Three (27%) studies included an ecologically relevant measure of parenting stress (Dunn et al., 2012), parent report of treatment acceptability (Case-Smith, DeLuca, et al., 2012), or caregiver burden (de Brito Brandão et al., 2012). Two others examined teacher reflection on efficacy (Case-Smith, Holland, et al., 2012) and student reflection (Kinnealey, et al., 2012). Silva and associates (2012)  used parent intervention a priori to establish relevant outcome goals.

Contributions to occupational performance and participation were seen in 3 (27%) studies (Dunn et al., 2012; Gutman et al., 2012; Schaaf et al., 2012), and engagement was seen as a variable of interest in 3 (27%) studies (Gutman et al., 2012; Kinnealey et al., 2012; Koenig et al., 2012). Although acceptability of the intervention was addressed by only one research team (Case-Smith, Holland, et al., 2012), client satisfaction was examined in 6 (55%) studies (Case-Smith, DeLuca, et al., 2012; de Brito Brandão et al., 2012; Dunn et al., 2012; Koenig et al., 2012; Schaaf et al., 2012; Silva et al., 2012). One study (Kinnealey et al., 2012) uniquely examined student satisfaction (adolescents). Understudied populations were examined in 3 (27%) studies (Gutman et al., 2012; Koenig et al., 2012; Silva et al., 2012).

Having a very small research cohort (n < 10) was a limitation in 3 (27%) of the studies (Gutman et al., 2012; Kinnealey et al., 2012; Schaaf et al., 2012). Most studies (64%) examined 10–50 participants (Case-Smith, DeLuca, et al., 2012; Case-Smith, Holland, et al., 2012; Dunn et al., 2012; de Brito Brandão et al., 2012; Koenig et al., 2012; Palsbo & Hood-Szivek, 2012; Silva et al., 2012), with 1 study examining >50 (Wells et al., 2012). Of the studies examining intervention effectiveness in educational contexts (Case-Smith, Holland, et al., 2012; Gutman et al., 2012; Kinnealey et al., 2012; Koenig et al., 2012; Wells et al., 2012), none addressed alignment with core curriculum standards in its literature review. Except for 1 study (Case-Smith, DeLuca, et al., 2012), methodologies addressing client satisfaction did not clearly demonstrate that acceptability of intervention was also measured (de Brito Brandão et al., 2012; Dunn et al., 2012; Kinnealey et al., 2012; Koenig et al., 2012; Schaaf et al., 2012; Silva et al., 2012). One study addressed the cost–benefit of the intervention (Case-Smith, DeLuca, et al., 2012), but only peripherally. Siblings and other family members were not examined in the studies addressed by this review, nor did any of the articles reviewed describe longitudinal studies.

Multiple studies examined in this review were aimed at creating conceptual models, translating theory into practice, and informing program development as it relates to best practices for treatment, policy development, and program development. Studies that examined factors central to children, youth, and their families, such as finding a friend and participating in community life, are reflected in the studies examined for this article, but factors such as barriers to and supports for procuring and maintaining jobs are not.

Intervention effectiveness studies in the area of children and youth have shown signs of moving in a positive direction toward guiding practitioners to make evidence-based decisions affecting skills, processes, and foundations for childhood and adolescent occupations in comparison with studies published in 2011 (Hilton & Smith, 2012). One study examined children younger than age 3, and the majority were between ages 3 and 12, but 2 of the studies did address adolescents, an often overlooked population, which shows important progress toward meeting the objectives of the Centennial Vision charging researchers to examine the participation of children with special needs in their daily lives as they transition through childhood and adolescence into adulthood. Studies also included qualitative methodology, helping to better explain the roles and participation of parents as they experience disabilities in the context of family-centered services. Researchers must examine the emotional and social costs of occupational deprivation and occupational injustice, such as depression, alcohol and substance abuse, and suicide in disenfranchised youth, to children, youth, and their families, as well as to society.

Investigators are highly involved in examining interventions in context and across sites. Studies addressed a wide range of mechanisms from the framework areas: body function and body structure, functional limitations, activity, participation, and the environment, continuing to build evidence for the broad scope of practice within the field of occupational therapy. Discipline-specific methods to uniquely address the charge for science-driven practice have evolved, and pragmatic, comparative effectiveness, and mixed methods have all been used to inform practice, guide clinical decision making, and allow future researchers to attribute outcomes to interventions used in practice. For example, the majority of studies included manualized interventions, demonstrated adherence to intervention fidelity, and used established treatment protocols. Also, the inclusion of parental stress, teacher reflection, and student reflection demonstrates the profession’s commitment to understand intervention effectiveness framed within an ecological system.

Inclusion of measures of intervention fidelity has been identified as commonly absent in intervention effectiveness studies in AJOT (Murphy & Gutman, 2012). Intervention fidelity examines the extent to which the intervention is delivered as it was intended (Gearing et al., 2011). Describing the specific intervention protocol, which might be in the form of a manual, is the first step toward being able to achieve fidelity in adherence to the protocol. Use of intervention fidelity measures in several of the studies published during 2012 is promising, but it is an area for greater adherence in future studies. Inclusion of measures of client acceptability and cost–benefit analysis of services is lacking, and future research must address these issues to further establish the profession’s contribution to the practice area of children and youth. Discussing the major changes in health care delivery in the next decades will certainly include cost–benefit analysis of services within the larger conversation of improved quality of health care within a contracting budget.

Perhaps it is tacit knowledge and the commonness of the concern obscures its importance, but only 1 study addressed the acceptability of intervention. An even greater omission was that not one of the studies examining effectiveness of school-based intervention used alignment with the core curriculum standards as a variable of interest. Generalization and maintenance of improvement continues to be an area of weakness, and future investigators are encouraged to address this gap in the evidence.

Future reviews may want to evaluate trends in the number of efficacy, effectiveness, comparative effectiveness, and pragmatic trials because these trends are pertinent to fostering evidence that informs an evidence-based and science-driven practice. An even more important outcome is that such reviews will allow the profession to maintain a practice-informed evidentiary dialogue. Continued appraisal of identified research priorities (efficacy, reliability of test instruments, engagement, impact of disability on community and social participation, and the generation of questions that guide the future growth and evolution of occupational therapy) is essential and should become part of our self-assessment in the upcoming years.

In general, occupational therapy research is interested in understanding the pragmatic relevance of intervention and the extent to which intervention promotes participation, a core edict of the Centennial Vision. Future studies will need to include acceptability of the intervention and alignment with outcomes relevant to the context, such as the core curriculum standards in school-based interventions as well as the impact on families and children with disabilities, especially siblings and other family members. Cost–benefit analysis of intervention and comparative effectiveness trials, areas of concern neglected in the existing research, will likely become increasingly urgent in light of proposed health care reform laws and policies that will affect health care delivery in the next decade, with particular attention paid to the emotional and social costs to children, youth, and families—and to society—of occupational deprivation and occupational injustice.

Compared with studies published in previous years, the studies examined in this review showed improvement in their ability to guide practitioners to make evidence-based decisions by increasing the understanding of the pragmatic relevance of intervention and the extent to which intervention promotes participation in childhood and adolescent occupations. Studies’ level of evidence increased, as did efforts to increase scientific rigor. Intervention fidelity was, however, not consistently included in the studies. Several studies included qualitative methodology, helping to better explain the roles and participation of parents as they experience disabilities within family-centered services. Other strengths identified in this review include those aimed at creating conceptual models, translating theory into practice, and informing program development as it relates to best practices for treatment, policy development, and program development.

Although examination of factors central to children, youth, and their families was reflected in the studies examined for this article, barriers to and supports for procuring and maintaining jobs were not. Inclusion of longitudinal studies, measures of client acceptability, cost–benefit analysis, and alignment with educational standards (how coccupational therapy intervention supports academic success) also need more attention in future studies. In addition, occupational therapy research needs to do more to examine the emotional and social costs of occupational deprivation and occupational injustice for children and youth. Generalization and maintenance of improvement continue to be areas of weakness. Future studies should include acceptability of the intervention and alignment with outcomes relevant to the context and examine siblings and other family members.

*Indicates studies that were systematically reviewed for this article.