Fidelity of intervention is a critical aspect of effectiveness research and therefore important for therapists to consider when they examine research to guide their practice decisions. In the context of evidence-based practice, fidelity refers to the extent to which the intervention delivered in a study is true to the underlying therapeutic principles on which it is based (Teague, Bond, & Drake, 1998; Waltz, Addis, Koerner, & Jacobson, 1993). When conducting outcomes research, investigators must systematically manualize the intervention (i.e., describe its philosophy, therapeutic principles, and procedures in a manual for training interveners) and then monitor its delivery during the study to ensure that it is provided in a manner that accurately represents its philosophy and guiding principles. Ideally, a fidelity instrument guides this systematic analysis of the intervention. The use of a fidelity instrument not only allows the researcher to verify that the therapeutic strategies used in the study represent the defined intervention but also makes the study replicable (DePoy & Gitlin, 2005; Nelson & Mathiowetz, 2004). Outcomes research that uses a carefully thought out fidelity instrument allows practitioners to appraise the relevance of a study’s intervention procedures to their own practice challenges.

Intervention guided by sensory integration theory (Ayres, 1972) is commonly used by occupational therapists who work with children (Case-Smith & Miller, 1999; National Board for Certification in Occupational Therapy, 2004; Roley, Blanche, & Schaaf, 2001). However, specific intervention methods called sensory integration vary widely across geographic locations and practice settings. Such differences in practice may be the result of variations in entry-level training curricula (Jacobs, Koomar, Mailloux, & Roley, 1999).

Disparities in intervention delivery methods are apparent in effectiveness research. The description of an intervention in one study may be quite different from that in another study, even though both claim to evaluate “sensory integration.” More than 70 articles examining the efficacy of sensory integration based on Ayres’ work (1972, 1989, 2005) have been published, but intervention fidelity is a major concern affecting the validity of this research because researchers do not usually report whether they designed the intervention to represent Ayres’ original therapeutic principles or whether they monitored intervention delivery during the study to ensure that it maintained a high degree of fidelity (May-Benson & Koomar, 2010; Miller, 2003; Parham et al., 2007).

The project reported in this article is a product of the Sensory Integration Research Collaborative (SIRC), a work group of occupational therapy practitioners, educators, and researchers collaborating to improve the state of outcomes research on sensory integration intervention. The SIRC emerged from a research project that received funding from the National Institutes of Health to form a collaborative research group to address sensory integration intervention outcomes (Roley et al., 2005). One of SIRC’s first goals was to develop the fidelity measure reported in this article. This sensory integration fidelity measure provides a tool for ensuring that intervention called sensory integration is replicable and consistently adheres to the principles of Ayres’ sensory integration frame of reference, now trademarked as Ayres Sensory Integration (ASI; Roley, Mailloux, Miller-Kuhaneck, & Glennon 2007). It may also facilitate the ease with which practitioners appraise the validity of research claiming to evaluate the effectiveness of ASI intervention. Ultimately, we anticipate that a reliable and valid ASI fidelity instrument will improve the quality and value of future studies on the effectiveness of occupational therapy using ASI intervention.

The purpose of the Ayres Sensory Integration Fidelity Measure developed by SIRC is to provide a tool that will enable (1) documentation of whether intervention is carried out in accordance with the essential procedural aspects of ASI intervention, (2) monitoring of replicable ASI intervention delivery in research such as randomized clinical trials, and (3) differentiation between ASI and other types of intervention. In this article, we describe the development, reliability, and validity of the Ayres Sensory Integration Fidelity Measure with particular attention to the section of the instrument that addresses the dynamic process of intervention sessions. Specifically, we address these questions:

• Does the process section of the fidelity measure show acceptable interrater reliability?

• Does the process section of the fidelity measure have acceptable internal consistency?

• Does the process section of the fidelity measure demonstrate adequate validity in differentiating ASI from other intervention approaches in occupational therapy?

• Does the entire fidelity measure demonstrate content validity in addressing key elements of ASI intervention?

The Ayres Sensory Integration Fidelity Measure addresses the key structural and process elements of ASI intervention identified by SIRC (Parham et al., 2007). Parts 1–4 of the instrument measure the structural elements (Table 1), which reflect commonly documented features such as therapist credentials, including postprofessional training and mentorship; record review, including detailed assessment results; physical space and equipment; and evidence of parent–therapist collaboration on goal setting. Part 5 of the instrument measures therapist adherence to 10 process elements (Table 2) that reflect the key therapeutic strategies involved in delivery of ASI intervention. All structural and process elements are considered essential to adequate provision of ASI intervention. Because scoring of the process section involves a high degree of subjectivity in rating therapist strategies, this section was developed across a series of reliability and validity studies to ensure that the final instrument would be psychometrically sound.

In the initial stages of instrument development, raters scored the process section for frequencies of particular therapist behaviors during videotaped segments of ASI therapy sessions. However, raters were unable to agree on scores based on behavioral frequencies. Moreover, frequency scores did not accurately represent a therapist’s adherence to a particular ASI construct because ASI strategies are meant to be contingent on or synchronous with child behavior.Therefore, a 4-point rating method was eventually developed whereby raters judged whether they thought an observed therapist was intentionally using each therapeutic strategy. This method generated an acceptable degree of rater agreement and was used throughout the instrument’s development.

In the process section, each item represents 1 of the 10 ASI process elements, framed as a therapeutic strategy. Each therapeutic strategy being measured is defined by key concepts; additional descriptors of commonly observed therapist behaviors that exemplify the strategy are provided. To score each item, the rater assigns a rating of 1 to 4 on the basis of his or her judgment as to whether the intervener is intentionally using the strategy as a key element of intervention (1 = no, the therapist does not use this strategy; 2 = doubtful that the therapist uses this strategy; 3 = probably the therapist uses this strategy; 4 = certainly the therapist uses this strategy). Note that the rater bases the score on the intervener’s faithfulness to the key therapeutic strategies of ASI intervention, not on the child’s performance or whether the intervention session appears to be productive or successful. Because the rater must infer whether the intervener is using a therapeutic strategy from observation of the intervener’s behavior during intervention, the rater should be postprofessionally trained and experienced in ASI intervention.

To compute the Total Fidelity score, item scores are weighted so that the maximum score for each item is 10 and the minimum is 0; higher scores indicate greater adherence to ASI strategies. The maximum possible Total Fidelity score of 100 represents a perfect match to ASI intervention strategies, whereas the minimum possible Total Fidelity score of 0 represents a perfect mismatch. A Total Fidelity score of 80 was designated as the tentative cutpoint for determining whether an observed intervention session adhered to ASI therapeutic principles.

Because scoring the process section requires the rater to infer the intervener’s intent and might be influenced by rater expertise with ASI, the instrument was refined across a series of four studies that examined the reliability and validity of scores obtained from different sets of raters and trainers in diverse geographical areas and with varying degrees of ASI experience. A summary of these studies follows.

A preliminary instrument was developed, and the process section was tested with five ASI experts at one site in Southern California. Raters and the trainer (Parham) were occupational therapists with master’s or doctoral degrees and were instructors for the University of Southern California/Western Psychological Services Comprehensive Program in Sensory Integration. Each rater had been mentored by A. J. Ayres or by a mentee of Ayres and had 10–40 yr ASI experience. Results led to major changes in item content and scoring, including elimination of an original item (“sets up room to engage child”) because of weak reliability and creation of a new item (“challenges postural, ocular, and bilateral development”) to represent a critical aspect of ASI that we felt had not emerged in the original nominal group process used to identify key ASI elements (Parham et al., 2007). A subgroup of SIRC members systematically reviewed key literature on ASI principles to ensure that the new item addressed a critical ingredient of ASI. Additionally, independent researchers from a different U.S. geographical area used the preliminary fidelity instrument in research on ASI’s effectiveness (Watling & Dietz, 2007) and provided feedback that influenced revisions to the instrument.

Revisions from the preliminary study were tested in a subsequent pilot training program in which participants were 9 national experts on ASI, including SIRC members and their associates residing in diverse areas across the United States. The trainer was the same as for the preliminary study. Raters all had master’s or doctoral degrees, had been mentored by A. J. Ayres or a mentee of Ayres, and had 15–35 yr ASI experience. Results showed a marked increase in reliability and validity over that obtained with the preliminary instrument. Item refinements were made to create a revised instrument for the next study.

The revised instrument was examined in a train-the-trainers project using data from 16 raters (12 from diverse areas of the United States, and 4 from other countries), all of whom were occupational therapists with ASI experience ranging from 5 to >20 yr. All were certified to administer and interpret the Sensory Integration and Praxis Tests (SIPT; Ayres, 1989). Trainers were the original trainer and one other SIRC member (Roley) Findings led to further refinement of item guidelines for scoring to produce the final instrument.

The next step was to conduct a second-generation study to evaluate the reliability and validity of the final version of the process section. In this project, 3 participants from the previous study provided fidelity instrument training to therapists who had widely variable levels of experience with ASI. This step was important because all of the previous analyses had involved a particular trainer and raters who were nationally and internationally recognized experts in ASI. Whereas all the previous trainings had taken place in California, the training on the final version took place in the northeastern United States.

Fourteen occupational therapists who were experienced in ASI intervention were invited to serve as raters in the second-generation training. Three raters had a bachelor’s degree, 10 had a master’s degree, and 1 had a doctorate. All raters had >5 yr experience practicing as an occupational therapist. One therapist had ≥20 years experience practicing specifically in ASI, 7 had >10 yr, 5 had >5 yr, and 1 had only 1 yr of experience with ASI. All participants were certified to administer and interpret the SIPT. For 10 participants, it had been <4 yr since their certification.

Training was conducted by two SIRC members who had not previously served as trainers (May-Benson and Koomar) and an additional expert master’s-level therapist. Approximately 6 hr of training covered the instrument’s purpose and history, instructions for scoring, and individual practice with group discussion on scoring video-recorded clips of ASI and contrasting non-ASI interventions. After this training, the raters silently viewed and independently scored five anonymous cases for reliability and validity analysis. As in all of the preceding reliability–validity studies, all video-recorded cases used for training and for reliability–validity analysis were contributed by several SIRC sites across the United States; signed informed consent for video recording had been previously obtained.

The five reliability–validity cases consisted of adult–child dyads interacting in real intervention settings. For each case, raters did not have any information about the background of the child, the adult, or the situation that was video recorded. Only one case was known by the trainers to represent ASI intervention. Two cases were not occupational therapy sessions: In one, a babysitter accompanied a child in free play in a setting with SI equipment; in the other, an adult interacted with a child in a tabletop play activity that the child had selected. The other two cases were non-ASI sessions delivered by an occupational therapist: One was a sensory–motor session characterized by therapist-directed manual techniques and use of SI equipment, and the other was a therapist-directed perceptual–motor type of intervention focusing on practice of motor coordination and balance activities in a therapy room with SI equipment.

In the studies of the pilot, revised, and final versions of the process section, we analyzed ratings of individual items and Total Fidelity scores to evaluate internal consistency and interrater reliability using Cronbach’s α and intraclass correlation coefficients (ICCs; Shrout & Fleiss, 1979), respectively. We conducted analyses of variance (ANOVAs) to examine the validity of the Total Fidelity score in differentiating ASI from other interventions. If ANOVA results were significant, we planned post hoc t tests to test the hypothesis that the ASI case received significantly higher Total Fidelity scores than the other cases. Detailed results for the final version are presented in this article.

Table 3 depicts results of reliability analyses. As with earlier versions of the instrument, the final version demonstrated excellent internal consistency for the Total Fidelity score, with a Cronbach’s α of .99 across the 14 raters and five cases. Alphas for individual items ranged from .96 to .98. Because α coefficients >.70 are considered acceptable (Nunnaly, 1978), these findings indicate that the final version of the process section has strong internal consistency and that individual items contribute approximately equally to the reliability of the Total Fidelity score.

Interrater reliability coefficients improved substantially between the second and third versions of the instrument and remained high in the final version. Table 3 shows that for the final version, the ICC for the Total Fidelity score across all raters was quite high (.99), indicating that the instrument’s overall reliability is strong when rater scores are pooled. Inspection of the raw data, however, suggested a high degree of individual rater variability for individual items. Because it is desirable for research purposes that any one rater be as reliable as possible, the Total Fidelity score and the individual item scores were further analyzed for reliability of any single rater rather than limiting the reliability analysis to the rater pool as a group. When reliability of the Total Fidelity score was examined for any individual rater, the ICC dropped but was still acceptable at .85. Reliability of individual item scores across pooled raters was high (ICCs = .95–.98). As expected, reliability coefficients were lower for individual items as scored by any individual rater, with ICCs ranging from somewhat low (.58) to strong (.81). Three items had ICCs for any individual rater that fell below a minimum acceptable level of .70: “Establishes therapeutic alliance” (.58), “Presents sensory opportunities” (.66), and “Challenges praxis and organization of behavior” (.69).

Table 4 presents the means and standard deviations of the Total Fidelity scores assigned by the raters for the five cases. Total Fidelity scores on the final version sustained the high discriminant validity that had been demonstrated on all the instrument’s previous versions. Results of the ANOVA of Total Fidelity scores and post hoc t tests with α set at .05 indicated that the mean Total Fidelity score for the ASI case was significantly higher than those of all four of the non-ASI cases (F[4, 70] = 80.34, p < .001).

Twenty occupational therapists with expertise in SI theory and practice assessed the content validity of the structural and process sections of the final version of the fidelity measure to determine whether item contents were consistent with ASI. For Parts 1–4, they answered the question, “Do you perceive these structural elements as being essential for the provision of ASI intervention?” For Part 5, they answered the question, “Do you perceive these process elements as being the core components of ASI intervention?” They rated each item on a scale ranging from 1 (strongly disagree) to 5 (strongly agree).

Questionnaires were returned by 19 experts in sensory integration from six different countries who had not participated in the development of the Fidelity Measure. The 20 people recruited represented six separate associations related to ASI in the United States and Europe, including those offering training in sensory integration and the SIPT (1 from Spain, 3 from the United Kingdom or Ireland, 1 from Greece, 1 from Portugal, 1 from Austria, and 11 from the United States). Two responders were not affiliated with associations specifically related to ASI but were considered experts in the field, each with >30 yr of experience. Respondents were not identifiable by name on the returned questionnaires.

Results of the questionnaire indicated high content validity for the Fidelity Measure’s structural and process components, as shown in Tables 1 and 2. Mean ratings for all items and item components of the structural elements ranged from 4.11 to 4.95, indicating that on average, the respondents agreed with the statement, “Do you perceive these structural elements as being essential for the provision of ASI Intervention?” Of the respondents, 100% agreed or strongly agreed on 20 of 28 structural items rated. The lowest percentage of agree–strongly agree responses on the remaining items was 73.7% for the therapist qualification–supervision item. Written feedback on the questionnaire indicated that the respondents who did not agree with this item recommended a more intense level of supervision or mentorship than that indicated on the fidelity measure. The mean ratings for all items on the process elements ranged from 4.89 to 5.00, indicating high agreement with the statement, “Do you perceive these process elements as being the core components of ASI intervention?” Of the 19 respondents, 18 indicated that they strongly agreed with all 10 process components. For one item, “Presents sensory opportunities,” 100% of respondents assigned a strongly agree rating. For all other items, 1 respondent assigned a rating of agree.

Results provided affirmative answers to all four research questions that guided this project. Content validity data indicated a high level of agreement among experts that the structural and process sections of the instrument accurately represent essential features of the ASI intervention. Analyses of the final process section of the Ayres Sensory Integration Fidelity Measure suggested that its validity as a measure of ASI intervention is strong. Internal consistency is excellent. Item interrater reliability is generally acceptable, and interrater reliability of the Total Fidelity score appears to be excellent.

The Total Fidelity score significantly differentiated ASI from alternative interventions, such as perceptual–motor training, that are sometimes used as comparison interventions in ASI effectiveness research. At times in past research, SI was distinguished from an alternative intervention on the basis of the type of equipment or specific activities used rather than the dynamic therapist–child interactions that characterize ASI (Parham et al., 2007). A strength of the fidelity measure is that it distinguishes ASI when the alternatives are delivered using the same kinds of equipment in the same kinds of settings as in ASI intervention. The instrument is sensitive to the dynamic process of therapy that differentiates ASI from other interventions.

Interrater reliability coefficients for individual items of the instrument’s process section were generally weaker for individual raters compared with the high reliability coefficients for total score and for pooled raters. This finding was not surprising, because pooled raters usually generate higher reliability coefficients than individual raters, and total scores usually demonstrate stronger reliability than individual items of an instrument.

Three items did not meet the ICC criterion of .70 for individual rater reliability. The item with the lowest ICC (.58), “Fosters therapeutic alliance,” may be affected by the extent to which raters must rely on subtle nonverbal cues to make judgments about the relationship between therapist and child as well as by the absence of information regarding the history of the child and of the dyad. The item “Presents sensory opportunities” also generated a relatively low ICC (.66), perhaps because it is often difficult to judge whether the therapist is supporting or offering an activity with the intent to provide augmented sensory experiences versus motor opportunities or practice of a particular motor skill. The item “Challenges praxis and organization of behavior” nearly reached criterion (ICC = .69). This coefficient was much lower than the ICCs obtained for this item in the earlier studies, in which raters were all recognized experts in ASI. Perhaps this item’s weak reliability in the final instrument occurred because less experienced therapists have greater difficulty discerning when praxis or organization of behavior is being challenged or because this item is not defined clearly enough in the instrument for less experienced therapists to rate it with optimal reliability.

The reliability and validity studies of the process section involved raters who were all occupational therapists with postgraduate training, mentorship, and practice experience in ASI intervention. Rater qualifications undoubtedly contributed to the generally strong evidence of reliability and validity that we obtained. The instrument’s process section may not be reliable or valid when scored by an observer without those qualifications.

The interveners who provided ASI intervention in the videorecordings we used for fidelity training and data collection met the professional qualifications that the content validity raters indicated are essential for ASI intervention. Those requirements include ≥50 hr of education in ASI after completion of basic professional education in occupational therapy, plus additional supervision or structured mentorship. The ASI interveners’ competency level contributed to the trustworthiness of our reliability and validity data.

Reliability has been examined only for the Fidelity Measure’s process section. We do not yet know whether the section measuring structural elements is adequately reliable.

Several limitations may have affected our results. In the process section, scoring involves assignment of a numerical value to raters’ subjective impressions. The subjectivity of the scoring may make it difficult to increase some items’ reliability. Despite the subjectivity involved, we were able to obtain adequate agreement across experienced raters. It may be that the use of subjective judgment by sophisticated observers is the most efficient and accurate way to measure dynamic therapist–child interactions.

Another limitation is that we revised the wording of items with each version of the instrument’s process section with the intent to strengthen its reliability and validity. However, the raters’ level of expertise also changed with the final version, making it impossible to distinguish whether changes in reliability seen in the final version were the result of differences in rater expertise or changes in item description or order. Fortunately, reliability and validity of the Total Fidelity score remained strong in the final version.

Our findings indicate that the Ayres Sensory Integration Fidelity Measure provides a valid measure of ASI intervention for use in effectiveness studies. Adherence to the Fidelity Measure’s structural and process elements will increase the likelihood that interventions called SI and provided by qualified therapists are faithful to ASI principles not only in research but also in education and practice.

For the purposes of research, the Fidelity Measure provides an international standard by which to determine whether an intervention represents ASI. In the Fidelity Measure’s process section, the Total Fidelity score should be used as a marker of adherence to ASI intervention principles. Our data supported a total process score of ≥80 as indicative of an ASI intervention session. Therefore, the Fidelity Measure can be used in effectiveness studies to identify intervention sessions that do not adhere to ASI principles, as well as those that do. In well-designed studies, fidelity raters are blinded to the study design and to assignment of participants to interventions. Our data show that the Total Fidelity score of the process section is reliable and valid when used by trained raters who observe and score video-recorded intervention sessions with no prior information about the therapist, child, or environment (i.e., school, hospital, clinic, or community site).

Content validity data indicated that occupational therapists who have extensive education and supervision in ASI are appropriate interveners in an effectiveness study. Specifically, we found that international experts in ASI strongly agreed that therapists providing ASI intervention should have ≥50 hr of education in this approach, after basic professional education. Moreover, these experts agreed that supervised experience or structured mentorship in ASI was essential to ensure competence in delivering this intervention.

To ensure reliability and validity of the Fidelity Measure’s process section, raters should be occupational therapists with a high level of previous postprofessional training and experience with ASI who have completed a formal training program on scoring the measure. Training and mentored practice experience with ASI are likely to be important qualifications of raters on this section of the instrument because scoring requires subjective appraisals of therapist intent to use specific ASI therapeutic strategies. We doubt that ratings on the process section would be valid if scored by raters without prior extensive background in ASI intervention or without specific training on the instrument. It is not yet known whether the structural section of the Fidelity Measure will demonstrate adequate reliability. Highly trained raters with expertise in ASI may not be required in order to produce reliable scores on the structural section.

Well-designed intervention studies use an intervention manual that describes the philosophy, principles, and strategies of the intervention being studied to train interveners who will deliver the intervention in the study. Manualization maximizes consistency in the delivery of intervention, which is monitored and documented with a fidelity instrument. Currently, an ASI intervention manual that is complementary to the Fidelity Measure is in development by SIRC members. This manual is intended for training occupational therapists who meet the postgraduate educational and mentorship requirements in ASI (as discussed earlier) to serve as interveners in research on ASI intervention.

Although the Ayres Sensory Integration Fidelity Measure was developed to meet critical needs of researchers wanting to study ASI’s effectiveness, we became aware of its usefulness as a teaching tool early in its development. The instrument’s structural section is valuable in delineating the professional and environmental qualifications required for ASI intervention. The instrument’s process section is useful in coaching occupational therapists who are developing new knowledge and skills in ASI intervention, particularly in regard to honing clinical reasoning and on-the-spot intervention skills using ASI knowledge. Perhaps formal applications of the Fidelity Measure to postprofessional education and mentorship in ASI will prove fruitful in the future.

Content validity raters strongly agreed that detailed evaluations and communications through written assessment reports and intervention plans, as well as direct interchanges with parents and teachers, are essential in the delivery of ASI intervention. These communications allow the therapist to address not only the child’s sensory integration capacities but also the ways in which they relate to the child’s health and participation in daily life. This communication process is critical in the collaborative formation of intervention goals and identification of salient outcomes.

Occupational therapists work in a diverse array of settings. Expert ratings of content validity indicated strong agreement on the physical features of the environment required for delivery of this intervention; for example, a space large enough for equipment suspended from several overhead hooks to be used safely, flexibility in arrangement of mats and equipment to ensure safety, and an array of equipment well suited for ASI intervention are mandatory. These environments can be created regardless of whether the intervention is provided in hospitals, clinics, community-based settings, or schools.

Our data show that the Ayres Sensory Integration Fidelity Measure is a reliable and valid instrument for determining whether an observed intervention session represents occupational therapy using ASI. Given that sensory integration is one of many methods used by occupational therapists, future research may address effectiveness of occupational therapy using ASI in conjunction with other methods, such as motor skills training or Lifestyle Redesign®. Future researchers may also examine fidelity when ASI intervention is delivered by physical therapists or speech–language pathologists who have postprofessional training and mentorship equivalent to that of the occupational therapist qualifications defined in this study.

Support for the initial work on this project was provided by an R21 National Institutes of Health One-Year Collaborative Planning grant (1 R21 HD41614–01) to Lucy J. Miller (Principal Investigator; University of Colorado, Denver, at the time of the award), with Collaborating Principal Investigators Ellen S. Cohn (Boston University) and L. Diane Parham (University of Southern California at the time of the award). We acknowledge the following key organizations and people who contributed to this project: SPIRAL Foundation–Watertown, MA; Occupational Therapy Associates–Watertown, MA; Pediatric Therapy Network, Torrance, CA; Alison Teasdale; Stefanie Bodison; Melanie Salort; therapists enrolled in the University of Southern California graduate course in sensory integration (OT610) at Pediatric Therapy Network; community therapists who provided feedback on training; the therapists and clients who contributed videotaped cases; and the raters in each of the studies.