Interrater Reliability and Discriminative Validity of the Structural Elements of the Ayres Sensory Integration® Fidelity Measure©

Teresa A. May-Benson; Susanne Smith Roley; Zoe Mailloux; L. Diane Parham; Jane Koomar; Roseann C. Schaaf; Annamarie Van Jaarsveld; Ellen Cohn

doi:10.5014/ajot.2014.010652

Free

Research Article | September 2014

Interrater Reliability and Discriminative Validity of the Structural Elements of the Ayres Sensory Integration^® Fidelity Measure^©

Teresa A. May-Benson; Susanne Smith Roley; Zoe Mailloux; L. Diane Parham; Jane Koomar; Roseann C. Schaaf; Annamarie Van Jaarsveld; Ellen Cohn

Author Affiliations

Teresa A. May-Benson, ScD, OTR/L, FAOTA, is Executive Director, SPIRAL Foundation, 74 Bridge Street, Newton, MA 02458; tmaybenson@thespiralfoundation.org
Susanne Smith Roley, OTD, OTR/L, FAOTA, is Project Director, Division of Occupational Science and Occupational Therapy, University of Southern California, Los Angeles
Zoe Mailloux, OTD, OTR/L, FAOTA, is Program and Professional Development Consultant, Redondo Beach, CA
L. Diane Parham, PhD, OTR/L, FAOTA, is Professor, Occupational Therapy Graduate Program, School of Medicine, University of New Mexico, Albuquerque
Jane Koomar, PhD, OTR/L, FAOTA, was Executive Director, OTA The Koomar Center, and President, SPIRAL Foundation, Newton, MA, at the time of the study
Roseann C. Schaaf, PhD, OTR/L, FAOTA, is Professor and Chair, Department of Occupational Therapy, Jefferson School of Health Professions, Thomas Jefferson University, Philadelphia, PA
Annamarie Van Jaarsveld, MOT (UFS), is Head of Department: Occupational Therapy, Faculty of Health Sciences, University of the Free State, Bloemfontein, Republic of South Africa
Ellen Cohn, ScD, OTR/L, FAOTA, is Clinical Professor and MSOT Program Director, Occupational Therapy, Department of Occupational Therapy, College of Health and Rehabilitation Sciences, Boston University, Boston, MA

Article Information

American Journal of Occupational Therapy, September/October 2014, Vol. 68, 506-513. doi:10.5014/ajot.2014.010652

Abstract

This study examined the reliability and validity of the structural section of the Ayres Sensory Integration^® Fidelity Measure© (ASIFM), which provides a method for monitoring the extent to which an intervention was implemented as conceptualized in studies of occupational therapy using sensory integration intervention methods (OT–SI). We examined the structural elements of the measure, including content of assessment reports, availability of specific equipment and adequate space, safety monitoring, and integration of communication with parents and other team members, such as collaborative goal setting with parents or family and teacher education, into the intervention program. Analysis of self-report ratings by 259 occupational therapists from 185 different facilities indicated that the structural section of the ASIFM has acceptable interrater reliability (r ≥ .82) and significantly differentiates between settings in which therapists reportedly do and do not practice OT–SI (p < .001).

Evidence of fidelity to intervention principles is an essential feature of rigorous intervention studies in health-related fields, including occupational therapy (Blanche, Fogelberg, Diaz, Carlson, & Clark, 2011; Bond, Becker, & Drake, 2011; Chandler, 2011; Gutman, 2010; Kaderavek & Justice, 2010). Measurement of fidelity ensures that interventions are delivered in a systematic manner congruent with the principles of the intervention. The structural elements of an intervention are generally the framework for service delivery, and the process elements focus on the direct application of the intervention (Mowbray, Holter, Teague, & Bybee, 2003; O’Donnell, 2008).

The Ayres Sensory Integration^® Fidelity Measure^© (ASIFM) evaluates the structural and process elements that are central to the delivery of occupational therapy using Ayres Sensory Integration intervention principles (OT–SI; Parham et al., 2011). The process elements address the dynamics of therapist–child interaction during occupational therapy sessions. The structural elements address the intervention setting, including therapist qualifications; the physical features of the environment in which intervention is delivered; and the systems for assessment, goal setting, and communication with parents. Therefore, the structural elements of the ASIFM examine whether OT–SI intervention involves qualified occupational therapists (or in some cases, qualified physical therapists or speech–language pathologists); whether intervention is provided in an adequate environment; and whether formal procedures for assessing, setting goals, and communicating with team members are implemented.

Research on the ASIFM has indicated that scores for the process section are reliable and valid. Interrater reliability of the total process score is high (intraclass correlation coefficient = .99), and the total process score discriminates between OT-SI and non-OT–SI videotaped intervention sessions (Parham et al., 2011). The current effort evaluated the structural section of this measure; thus, the purposes of this article are to describe the development and content of the structural section of the ASIFM and to present evidence for the reliability and validity of ASIFM items that measure structural elements.

Development of Structural Elements of the Ayres Sensory Integration Fidelity Measure

In the early development of the ASIFM, eight structural elements of OT–SI intervention were identified by 10 experts in sensory integration from across the United States after a review of 11 key publications in the sensory integration literature. A systematic review of 34 published studies that claimed to evaluate outcomes of OT–SI found that only two of the eight structural elements were described in the majority of studies: professional background of interveners (almost always reported as occupational therapists) and identification of therapeutic equipment used in the study (Parham et al., 2007). This finding underscored the critical need to identify and measure the full range of structural elements necessary in the provision of OT–SI.

The next step in the development of the structural section of the ASIFM involved experienced practitioners of OT–SI from a variety of practice settings who generated and reviewed items to clarify, revise, and expand the initial set of structural elements described by Parham et al. (2007) . Across a 5-yr period, experienced practitioners who were attending continuing education courses on OT–SI across the United States were invited to review and submit written comments on the contents of the structural section and to participate in focus groups that discussed how the essential structural elements of OT–SI should be represented in the ASIFM. After each wave of practitioner review, new items were added and existing items were revised or deleted to strengthen the clarity and feasibility of the structural section. The revised structural section was then presented for review by the next group of practitioners. This iterative process of review and refinement was performed to ensure that the structural items developed for the ASIFM would adequately operationalize the structural requirements for delivery of OT–SI and would be consistent with and feasible for contemporary practice.

The initial version of the ASIFM structural section was generated during the iterative review process just described. It consisted of 28 items categorized under five headings: (1) therapist qualifications, including postprofessional training in sensory integration and clinical experience or supervision in OT–SI; (2) components of the occupational therapy assessment report, including history, occupational profile, and evaluation of sensory integration functions; (3) physical environment and equipment, including space for vigorous as well as quiet activity; (4) provisions for safety of therapeutic equipment; and (5) communication with parents and teachers, including collaborative goal setting and family or teacher education. As part of the development of the ASIFM, a content validity study was conducted on these 28 structural items (Parham et al., 2011). Ratings by international experts in OT–SI strongly agreed that each structural item was essential for the provision of OT–SI. Subsequently, additional refinements were made to create the final version of the structural section after several more rounds of practitioner reviews. The final version retained all items from the content validity study; however, some items were reworded slightly or separated into individual items (e.g., each piece of therapeutic equipment and specific assessment report component became a separate item) for a total of 55 items.

In the final version of the ASIFM’s structural section, which we analyzed in this study, therapist qualifications are treated as a “go–no-go” item, that is, the therapist delivering OT–SI must have formal postprofessional training in OT–SI and mentorship from a therapist who is trained and experienced with this intervention. If the therapist does not meet these criteria, then fidelity is compromised, and no need exists to measure fidelity any further. However, if these criteria are met, then additional structural elements are evaluated by scoring items in the four remaining components of the structural section: Safe Environment (Part I), Assessment Reports (Part II), Physical Space and Equipment (Part III), and Communication With Parents and Teachers (Part IV). Parts I–IV are summed to obtain a total structural section score. Similar to the process elements section, a score of 85 (reflecting a 75% pass rate) was tentatively set as the minimum cutpoint for determining whether a given therapy setting meets standards for the provision of high-fidelity OT–SI intervention.

Research Questions

The following questions guided this research:

Does the structural section of the ASIFM show acceptable interrater reliability?
Does the structural section of the ASIFM demonstrate adequate discriminative validity in differentiating settings that provide OT–SI from those that do not?
Does the tentative total structural section cutoff score of 85 serve as a valid differential between settings that provide OT–SI and those that do not?
Do different types of practice settings that provide OT–SI yield structural section scores that differ from each other?
Do structural section scores differ between facilities that provide OT–SI in the United States and those that provide OT–SI in South Africa?

Method

Instrument

The final version of the structural section of the ASIFM (Parham et al., 2011) was converted into a survey format consisting of background and demographic questions (Table 1) and the four structural section subcomponents: Part I (Safe Environment), Part II (Assessment Reports), Part III (Physical Space and Equipment), and Part IV (Communication With Parents and Teachers) (Table 2). All parts were scored dichotomously (yes = 2, no = 0). Parts I–IV were summed to obtain a total structural section score. We used the following structural fidelity scores for analyses: total scores for Parts I, II, III, and IV; two subsection scores for Part III; individual scores for all items; and a total structural fidelity score (sum of Parts I–IV). Participants were asked to rate their facility on the basis of their knowledge of the standard facility practices and expected procedures.

Table 1.

Therapist and Facility Characteristics by Country

Survey Items	United States	South Africa	Other	Total
OT–SI is routinely provided, n (%)
Yes	118 (69)	63 (86)	9 (69)	190 (74)
No	54 (31)	10 (14)	4 (31)	68 (26)
Highest educational degree, n (%)
Bachelor’s	49 (29)	64 (93)	5 (42)	118 (47)
Master’s	110 (65)	5 (7)	6 (50)	121 (48)
Doctoral	11 (6)	0	1 (8)	12 (5)
Therapist is SIPT certified, n (%)
Yes	89 (52)	56 (77)	9 (69)	154 (60)
No	83 (48)	17 (23)	4 (31)	104 (40)
Primary work facility, n (%)
Private clinic	70 (41)	55 (75)	7 (54)	132 (52)
School system	80 (47)	16 (22)	4 (31)	100 (39)
Hospital	16 (10)	2 (3)	0	18 (7)
Other	3 (2)	0	2 (15)	5 (2)
Years of practice experience in occupational therapy, M (SD)	15.9 (10.78)^a	13.3 (8.21)^b	13.9 (9.27)^c	15.1 (10.07)^d
Years of practice experience in OT–SI, M (SD)	12.5 (10.32)^e	8.6 (6.33)^b	9.3 (7.20)^c	11.2 (9.34)^f

Note. N = 258. M = mean; OT–SI = occupational therapy using sensory integration intervention; SD = standard deviation; SIPT = Sensory Integration and Praxis Tests. ns vary as a result of nonresponse for some items.

^an = 170. ^bn = 73. ^cn = 13. ^dn = 257. ^en = 166. ^fn = 252.

View Large

Table 1.

Therapist and Facility Characteristics by Country

Survey Items	United States	South Africa	Other	Total
OT–SI is routinely provided, n (%)
Yes	118 (69)	63 (86)	9 (69)	190 (74)
No	54 (31)	10 (14)	4 (31)	68 (26)
Highest educational degree, n (%)
Bachelor’s	49 (29)	64 (93)	5 (42)	118 (47)
Master’s	110 (65)	5 (7)	6 (50)	121 (48)
Doctoral	11 (6)	0	1 (8)	12 (5)
Therapist is SIPT certified, n (%)
Yes	89 (52)	56 (77)	9 (69)	154 (60)
No	83 (48)	17 (23)	4 (31)	104 (40)
Primary work facility, n (%)
Private clinic	70 (41)	55 (75)	7 (54)	132 (52)
School system	80 (47)	16 (22)	4 (31)	100 (39)
Hospital	16 (10)	2 (3)	0	18 (7)
Other	3 (2)	0	2 (15)	5 (2)
Years of practice experience in occupational therapy, M (SD)	15.9 (10.78)^a	13.3 (8.21)^b	13.9 (9.27)^c	15.1 (10.07)^d
Years of practice experience in OT–SI, M (SD)	12.5 (10.32)^e	8.6 (6.33)^b	9.3 (7.20)^c	11.2 (9.34)^f

^an = 170. ^bn = 73. ^cn = 13. ^dn = 257. ^en = 166. ^fn = 252.

Table 2.

ASIFM Structural Section Contents and Median Interrater Joint Probability of Agreement

Section (No. of Items)	Description	Sample Abbreviated Items	Interrater Agreement
Part I: Safe Environment (5)	Examines provisions for ensuring physical safety in the therapy environment	Use of mats, cushions, or pillows Adjustable equipment Equipment monitored for safe use Unused equipment stored safely Routine monitoring and repair for safety	1.00^a
Part II: Assessment Reports (19)	Checks for standard assessment processes and reports that address evaluation results, goal setting, and intervention planning	Assessment reports contain • History and occupational profile • Reason for referral • Evaluation of sensory processing (modulation and discrimination) • Evaluation of praxis, postural–ocular control, visual perception, fine and gross motor skills • Evaluation of organizational skills • Interpretation of results in relation to child’s life • As appropriate, intervention plan and collaborative goals related to participation.	0.95^b
Part III: Physical Space and Equipment (28)	Examines physical environmental affordances for sensory–motor exploration and movement through space	Space allows for flow of physical activity, designated quiet space, and flexible arrangement of equipment.	0.96^a
IIIA: Physical Space (6)		Ceiling hooks, rotational devices, and bungee cords for suspended equipment	1.00^c
IIIB: Equipment (22)		Wide variety of suspended and nonsuspended equipment and sensory materials available	0.95^a
Part IV: Communication With Parents and Teachers (3)	Checks for ongoing formal and informal discussions with parents and teachers	Therapists routinely meet with parents and teachers regarding course of intervention as related to child’s occupational performance.	1.00^a
Total structural section score (55)			0.95^d

Note. ASIFM = Ayres Sensory Integration Fidelity Measure.

^aFacilities with joint probability of agreement ≥ .80 on subsection scores = 80. ^bFacilities with joint probability of agreement ≥ .80 on subsection scores = 96. ^cFacilities with joint probability of agreement ≥ .80 on subsection scores = 92. ^dFacilities with joint probability of agreement ≥ .80 on subsection scores = 100.

View Large

Table 2.

ASIFM Structural Section Contents and Median Interrater Joint Probability of Agreement

Section (No. of Items)	Description	Sample Abbreviated Items	Interrater Agreement
Part I: Safe Environment (5)	Examines provisions for ensuring physical safety in the therapy environment	Use of mats, cushions, or pillows Adjustable equipment Equipment monitored for safe use Unused equipment stored safely Routine monitoring and repair for safety	1.00^a
Part II: Assessment Reports (19)	Checks for standard assessment processes and reports that address evaluation results, goal setting, and intervention planning	Assessment reports contain • History and occupational profile • Reason for referral • Evaluation of sensory processing (modulation and discrimination) • Evaluation of praxis, postural–ocular control, visual perception, fine and gross motor skills • Evaluation of organizational skills • Interpretation of results in relation to child’s life • As appropriate, intervention plan and collaborative goals related to participation.	0.95^b
Part III: Physical Space and Equipment (28)	Examines physical environmental affordances for sensory–motor exploration and movement through space	Space allows for flow of physical activity, designated quiet space, and flexible arrangement of equipment.	0.96^a
IIIA: Physical Space (6)		Ceiling hooks, rotational devices, and bungee cords for suspended equipment	1.00^c
IIIB: Equipment (22)		Wide variety of suspended and nonsuspended equipment and sensory materials available	0.95^a
Part IV: Communication With Parents and Teachers (3)	Checks for ongoing formal and informal discussions with parents and teachers	Therapists routinely meet with parents and teachers regarding course of intervention as related to child’s occupational performance.	1.00^a
Total structural section score (55)			0.95^d

Note. ASIFM = Ayres Sensory Integration Fidelity Measure.

Procedures

The survey was initially distributed to a sample of occupational therapists attending several continuing education courses on sensory integration in the United States and South Africa. The goal was to obtain a cross-section of therapists practicing in a variety of settings that serve children and who may or may not use OT–SI. Preliminary demographic analysis of data collected from these initial surveys suggested limited diversity in type of settings; therefore, to obtain better representation of different types of settings, email invitations to participate in an online version of the survey were sent to additional occupational therapists across the United States to increase sample size and diversity of facilities represented. Therapists receiving invitations were encouraged to forward the invitation to colleagues in diverse pediatric practice settings. Participation was entirely voluntary, and surveys were submitted anonymously in all cases.

Participants

Two hundred eighty-six occupational therapists representing 185 facilities completed the structural section survey. Of these, 28 surveys were missing critical data and were omitted from the analysis. For the 258 surveys included in the analysis, 75% stated that OT–SI was provided at the facility. Respondents included 172 participants from the United States, 73 from South Africa, and 13 from other countries (Canada, China, Greece, Ireland, Portugal, Singapore, and the United Kingdom). Of the 258 respondents, 132 (51%) primarily worked in a private practice, 100 (39%) in a school system, 18 (7%) in a hospital, and 4 (2%) in other settings (e.g., early intervention), and 3 (1%) did not respond to the question regarding type of setting. Of the total 185 facilities, 24 had >1 respondent; therefore, we used data from these facilities to analyze interrater agreement on the structural elements of the ASIFM. Table 1 presents demographic data on therapist qualifications and facilities in various practice settings and by country.

Data Analysis

We summarized responses to each item using descriptive statistics before evaluating reliability and validity. Several of the useable forms were missing data for one or more questions. Missing values were replaced by their section mean if the number of missing items was limited, specifically, no more than one item from Part I (Safe Environment), two items from Part II (Assessment Reports), one item from Part IIIA (Physical Space subsection), and two items from Part IIIB (Equipment subsection). Forms with additional missing values were used only in those analyses for which the required data were present; therefore, sample sizes varied by analysis.

Interrater reliability was assessed using joint probability of agreement to calculate agreement among raters who worked at the same facility (Miller & Vanni, 2005), because we expected that respondents in facilities in which OT–SI was provided would answer affirmatively to the majority of questions, thus violating the assumption of randomness made by intraclass correlation procedures. When multiple raters existed at a site, we determined joint probability of agreement by computing the mean of the pairwise percentages of agreement, and we used this figure to represent the reliability estimate for that site. This procedure calculates the proportion of times that ratings coincide.

With regard to validity analyses, we used nonparametric procedures for all analyses of aggregated data, because aggregate scores had nonnormal distributions. We used the Mann–Whitney U test (Sheskin, 2011) to analyze differences between two groups, for example, between facilities that reportedly do and do not provide OT–SI. We used the χ² test for post hoc analyses of two-group tests and the Kruskal–Wallis test (Sheskin, 2011) to detect multiple group differences, that is, across practice settings or countries. All data analyses were conducted using SPSS Version 15.0 for Windows (SPSS Inc., Chicago).

Results

Interrater Reliability

Interrater reliability was determined for 24 facilities for which 2 or more respondents submitted data for each part of the structural section of the ASIFM. These facilities consisted of private clinics or school systems. Seventeen facilities were located in the United States, 6 in South Africa, and 1 in Portugal. The number of respondents from private clinics varied from 2 to 6 per facility, and the number of raters per school district ranged from 2 to 25. Data from a total of 93 respondents were analyzed, of whom 83 indicated they provided OT–SI. The 10 who did not provide OT–SI worked in four school districts. Interrater reliability was determined primarily from U.S. private practices; data from school systems and South African therapists were limited because, in many cases, respondents from these facilities were the sole therapist working in that particular school district or private practice.

For the total structural score, each of the 24 facilities analyzed had a joint probability of agreement ≥.82, with a median agreement of .95. For each of the 24 sites, median agreement for each subscore on Parts I–IV was ≥.95 (see Table 2).

Facilities That Did and Did Not Provide OT–SI

Discriminative Validity of Scores.

To evaluate the discriminative validity of the structural section total and subsection scores, we used the Mann–Whitney U test to examine whether scores differed between settings that did and did not provide OT–SI. Surveys were included in this analysis if respondents had submitted adequate responses to calculate a total structural score. We compared ratings from 178 respondents who stated that their facilities provided OT–SI with ratings from 55 respondents who indicated that their facilities did not provide OT–SI. As shown in Table 3, we found significant differences between the two groups on the total structural section score and all subsection scores, with facilities providing OT–SI showing a greater likelihood of higher scores (p < .001). Although some medians and ranges shown in Table 3 were identical for both groups, the groups significantly differed because of a preponderance of lower scores for the group that did not provide OT–SI compared with the group that did.

Table 3.

Structural Subsection and Total Section Scores of Facilities That Did and Did Not Provide OT–SI

Section	OT–SI			Non-OT–SI
Section	n	Median	Range	n	Median	Range
Part I, Safe Environment	189	10	6–10	63	10	0–10
Part II, Assessment Reports	187	38	24–38	64	32	12–38
Part III, Physical Space and Equipment	181	52	8–56	60	37	0–56
A. Physical Space	187	12	0–12	61	6	0–12
B. Equipment	181	42	8–44	62	29	0–44
Part IV, Communication With Parents and Teachers	184	6	0–6	61	6	0–6
Total structural section score	178	104	58–110	55	84	36–110

Note. Mann–Whitney U tests for all score differences between facilities that did and did not provide OT–SI were significant (p < .001). OT–SI = occupational therapy using sensory integration intervention.

View Large

Table 3.

Structural Subsection and Total Section Scores of Facilities That Did and Did Not Provide OT–SI

Section	OT–SI			Non-OT–SI
Section	n	Median	Range	n	Median	Range
Part I, Safe Environment	189	10	6–10	63	10	0–10
Part II, Assessment Reports	187	38	24–38	64	32	12–38
Part III, Physical Space and Equipment	181	52	8–56	60	37	0–56
A. Physical Space	187	12	0–12	61	6	0–12
B. Equipment	181	42	8–44	62	29	0–44
Part IV, Communication With Parents and Teachers	184	6	0–6	61	6	0–6
Total structural section score	178	104	58–110	55	84	36–110

We then used χ² analysis to examine whether a total structural section score of >85 discriminated between facilities in which OT–SI was and was not reportedly provided. We detected a significant difference between the two types of facilities, χ²(1) = 77.1, p < .001. Among respondents who indicated that OT–SI was provided at their facilities, 96% scored >85 on the total structural section score. In contrast, 45% of therapists who stated that OT–SI was not provided at their facilities scored >85.

Discriminative Validity of Items.

We evaluated discriminative validity of the individual items in the structural section of the ASIFM by comparing item ratings of respondents in facilities in which OT–SI was and was not reportedly offered. The Mann–Whitney U test revealed statistically significant differences (p < .001) between the OT–SI and non-OT–SI facilities on Part II (Assessment Reports), with the OT–SI facilities more often including content in their assessment reports that addressed reason for referral, sensory modulation and discrimination, postural control, motor coordination, praxis, interpretation of the relationship of sensory integration problems to performance, and reason for referral. Of 33 equipment and safety items identified in Part I (Safe Environment) and Part III (Physical Space and Equipment), especially those involving suspended equipment, 29 were less frequently found in facilities in which OT–SI was not provided (p < .001). We found no significant differences between facilities on the other items in these sections, including safety documentation and presence of therapy balls, weighted objects, and materials to practice everyday living skills. Last, facilities in which OT–SI was provided reported significantly more communication with parents and discussion of the influence of sensory integration problems on performance and participation (p < .001).

Differences Across Types of Facilities Providing OT–SI

We compared the total structural section scores of respondents in private practice, school, and hospital settings that provide OT–SI to examine whether they differed in the extent to which structural elements were implemented. Of the 190 therapists who reported that they worked in a facility that provides OT–SI, 120 (63%) indicated their primary workplace was a private practice; 55 (29%), a school system; 12 (6%), a hospital; and 3 (2%), other settings (e.g., early intervention). Therapists working in other settings were not included in this analysis because of the very small sample size. A Kruskal–Wallis test found no significant difference in total structural section scores across private practice, school, and hospital settings that provide OT–SI, χ²(2) = 5.4, p = .067. However, a significant difference across settings was indicated in the overall percentage of facilities that scored at or above the designated cut point of 85, χ²(2) = 6.5, p = .039. Private clinics had the highest percentage of facilities reporting scores >85 (98% of 116 facilities), followed by hospitals (92% of 12 hospitals), and then school systems (89% of 47 schools).

Differences Between U.S. and South African Facilities Providing OT–SI

We also compared the total structural section scores of respondents in U.S. and South African facilities providing OT–SI to detect any differences that might exist, and a significant difference was found (U = 2,435.5, p = .003). This difference in total structural section score was accounted for by differences in two subsections: Part I (Safe Environment; U = 2,578.5, p < .001) and Part III (Physical Space and Equipment; U = 2,237, p < .001). In Part I, the U.S. group scored higher than the South African group, primarily because facilities in South Africa did not routinely document monitoring of equipment safety (U = 2,452.5, p < .001). In Part III, the U.S. facilities reportedly had more equipment available, specifically more rotational devices (U = 2,854.5, p < .001), platform swings (U = 2,138, p < .001), and tire swings (U = 2,033.5, p < .001).

Discussion

This study provides evidence that the structural section of the ASIFM has acceptable reliability and discriminative validity for use in research on OT–SI intervention. Interrater reliability estimates are strong for the total structural section score as well as subsection scores. Total and subsection scores significantly differed between therapists whose facilities provided OT–SI and those whose facilities did not. The items contributing most strongly to discriminative validity were those addressing assessment reports and communication with parents and other team members. These items examine whether assessments and communication address specific sensory integration problems and how they relate to individual child performance and participation. Items that address safety monitoring and presence of specific therapeutic equipment also differentiated between settings that did and did not implement OT–SI. A significant difference in pass rate (total structural section score >85) between facilities that did and did not provide OT–SI supports the establishment of this cut point for passing. These findings, along with previous research that reported strong content validity for the structural section as well as strong reliability and validity of the ASIFM process section (Parham et al., 2011), indicate that the entire ASIFM has adequate psychometric properties for use in research. Therefore, the ASIFM structural section should be used together with the process section to evaluate fidelity of OT–SI intervention in research.

Findings suggest that OT–SI settings in private clinics are somewhat more likely to meet the passing score for structural fidelity than those in school systems and hospitals. Perhaps this difference in pass rates for the structural section is because therapists in private clinics have greater autonomy in establishing procedures for assessment and communication with parents, as well as greater control of space usage and equipment purchases, than those working in educational or medical settings. Nevertheless, no difference in mean total structural section scores was found across these settings. This finding indicates that occupational therapy settings in private clinics, hospitals, and schools all offer opportunities to practice OT–SI with strong fidelity to structural elements.

Although we found the expected difference in pass rate between respondents in facilities that did and did not provide OT–SI, it is noteworthy that nearly half of the therapists in facilities that did not provide OT–SI submitted scores that met the passing total score criterion of >85. This finding may reflect the permeation of OT–SI concepts and therapeutic equipment into general pediatric occupational therapy practice, even when other intervention approaches are used. Alternatively, it may reflect overlap between the structural elements of OT–SI and other specific occupational therapy interventions.

Therapists in South African facilities that provided OT–SI reportedly did less safety monitoring and had fewer types of therapeutic equipment than their colleagues in the United States. These differences may reflect a lack of accountability for safety or lack of awareness of the importance of systematic safety monitoring in South Africa, as well as limited availability of a wide range of affordable equipment. Both of these shortcomings could be remedied at individual facilities by establishing safety monitoring procedures and expanding the types of equipment available. In some settings, it may be desirable to design and construct affordable equipment that can be made locally.

Limitations and Future Research

A limitation of this study is that the respondents were samples of convenience. Participants were recruited from attendees of sensory integration conferences and training programs, as well as through email invitations distributed to therapists in a variety of practice settings, and therefore may not be representative of all occupational therapists who use or do not use OT–SI. Respondents may have been biased by social desirability when assigning item ratings to their facilities. A more rigorous way to evaluate structural fidelity would be through direct observation of the setting to evaluate it rather than relying on therapist report. Moreover, we relied on therapist reports of whether OT–SI was provided as routine practice in the respondents’ facilities on the basis of the therapists’ knowledge of facility-expected procedures and observations of facility staff. The research design would have been stronger had we also collected fidelity scores on the process section of the ASIFM to ensure that self-reports of OT–SI provision were valid, but doing so was not feasible because of the time and expense required. Although we made an attempt to represent a range of practice settings, a broader survey of practice settings with larger sample sizes may be helpful in future research.

Implications for Occupational Therapy Practice

The results of this study suggest the following implications for occupational therapy practice:

Researchers planning to study the outcomes of OT–SI should use both the structural and the process sections of the ASIFM to ensure that the OT–SI intervention delivered in the study demonstrates strong fidelity.
Researchers and occupational therapy practitioners who seek to evaluate the quality of evidence supporting OT–SI should examine whether structural and process aspects of the intervention have been systematically monitored in studies of OT–SI using instruments such as the ASIFM.
Occupational therapists who want to engage in best practice of OT–SI might use the ASIFM structural and process sections to evaluate and improve their own practice setting. For example, the structural section could be used to set criteria for therapist qualifications, to establish procedures for safety monitoring, to design or update OT–SI treatment spaces, or to thoughtfully select an array of equipment for use in intervention.
Consumers seeking OT–SI may be informed by the structural elements of the ASIFM to identify appropriately qualified therapists, to differentiate OT–SI from other services, and to better understand the significance of assessment methods, collaborative planning, safety, space, and equipment that are essential to the provision of OT–SI.

Conclusion

OT–SI intervention is one of many theoretical perspectives that guide occupational therapy intervention (Schaaf et al., 2009). Since A. Jean Ayres originally developed sensory integration theory, assessment, and intervention (Ayres, 1972), its use has become commonplace in occupational therapy (Lane, Smith Roley, & Champagne, 2013; Watling, Koenig, Davies, & Schaaf, 2011), yet research on the effectiveness of OT–SI has been severely limited by poor intervention fidelity (Parham et al., 2007). Results of this study show that the structural section of the ASIFM, along with the process section (Parham et al., 2011), is reliable and valid for use in effectiveness studies to ensure that interventions claiming to provide OT–SI are congruent with the underlying principles of this intervention. Moreover, the structural elements identified in the ASIFM may provide a tool to guide therapists who intend to provide OT–SI so that they can acquire the necessary professional qualifications, provide comprehensive evaluations, engage in active collaboration with families and other professionals, and deliver the intervention in a safe and adequately equipped therapeutic space.

Acknowledgments

We acknowledge Alison Teasdale, Senior Research Assistant, SPIRAL Foundation, Newton, MA, for her statistical analysis and contributions to this article. We thank the therapists who participated in this study by rating their facility. Support for the initial work on this project was provided by an R21 National Institutes of Health One-Year Collaborative Planning grant (1 R21 HD41614-01) to Lucy J. Miller (Principal Investigator; University of Colorado, Denver, at the time of the award), with Collaborating Principal Investigators Ellen Cohn (Boston University) and Diane Parham (University of Southern California at the time of the award).

References

Ayres, A. J. (1972). Sensory integration and learning disorders. Los Angeles, CA: Western Psychological Services.

Blanche, E. I., Fogelberg, D., Diaz, J., Carlson, M., & Clark, F. A. (2011). Manualization of occupational therapy interventions: Illustrations from the pressure ulcer prevention research program. American Journal of Occupational Therapy, 65, 711–719. http://dx.doi.org/10.5014/ajot.2011.001172 [Article] [PubMed]

Bond, G. R., Becker, D. R., & Drake, R. E. (2011). Measurement of fidelity of implementation of evidence-based practices: Case example of the IPS fidelity scale. Clinical Psychology: Science and Practice, 18, 126–141. http://dx.doi.org/10.1111/j.1468-2850.2011.01244.x [Article]

Chandler, D. W. (2011). Fidelity and outcomes in six integrated dual disorders treatment programs. Community Mental Health Journal, 47, 82–89. http://dx.doi.org/10.1007/s10597-009-9245-0 [Article] [PubMed]

Gutman, S. A. (2010). Reporting standards for intervention effectiveness studies. American Journal of Occupational Therapy, 64, 523–527. http://dx.doi.org/10.5014/ajot.2010.09644 [Article] [PubMed]

Kaderavek, J. N., & Justice, L. M. (2010). Fidelity: An essential component of evidence-based practice in speech-language pathology. American Journal of Speech-Language Pathology, 19, 369–379. http://dx.doi.org/10.1044/1058-0360(2010/09-0097) [Article] [PubMed]

Lane, S. J., Smith Roley, S., & Champagne, T. (2013). Sensory integration and processing: Theory and applications to occupational performance. In B. A. Boyt Schell, G. Gillen, & M. E. Scaffa (Eds.), Willard and Spackman’s occupational therapy(12th ed., pp. 816–868). Baltimore: Lippincott Williams & Wilkins.

Miller, K. J., & Vanni, M. (2005, September). Inter-rater agreement measures and the refinement of metrics in the PLATO MT evaluation paradigm. Paper presented at the 10th Machine Translation Summit, Phuket, Thailand.

Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340 [Article]

O’Donnell, C. L. (2008). Defining, conceptualizing and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention. Review of Educational Research, 78, 33–84. Retrieved from http://www.jstor.org/stable/40071121. http://dx.doi.org/10.3102/0034654307313793 [Article]

Parham, L. D., Cohn, E. S., Spitzer, S., Koomar, J. A., Miller, L. J., Burke, J. P., … Summers, C. A. (2007). Fidelity in sensory integration intervention research. American Journal of Occupational Therapy, 61, 216–227. http://dx.doi.org/10.5014/ajot.61.2.216 [Article] [PubMed]

Parham, L. D., Roley, S. S., May-Benson, T. A., Koomar, J., Brett-Green, B., Burke, J. P., … Schaaf, R. C. (2011). Development of a fidelity measure for research on the effectiveness of the Ayres Sensory Integration^® intervention. American Journal of Occupational Therapy, 65, 133–142. http://dx.doi.org/10.5014/ajot.2011.000745 [Article] [PubMed]

Schaaf, R. C., Schoen, S., Smith Roley, S., Lane, S., Koomar, J., & May-Benson, T. (2009). A frame of reference for sensory integration. In P. Kramer & J. Hinojosa (Eds.), Frames of reference for pediatric occupational therapy(3rd ed., pp.99–186). Baltimore: Lippincott Williams & Wilkins.

Sheskin, D. J. (2011). Handbook of parametric and nonparametric statistical procedures(5th ed.). Boca Raton, FL: Chapman & Hall/CRC.

Watling, R. L., Koenig, K. P., Davies, P. L., & Schaaf, R. C. (2011). Occupational therapy practice guidelines for children and adolescents with challenges in sensory processing and sensory integration. Bethesda, MD: AOTA Press.

Table 1.

Therapist and Facility Characteristics by Country

Survey Items	United States	South Africa	Other	Total
OT–SI is routinely provided, n (%)
Yes	118 (69)	63 (86)	9 (69)	190 (74)
No	54 (31)	10 (14)	4 (31)	68 (26)
Highest educational degree, n (%)
Bachelor’s	49 (29)	64 (93)	5 (42)	118 (47)
Master’s	110 (65)	5 (7)	6 (50)	121 (48)
Doctoral	11 (6)	0	1 (8)	12 (5)
Therapist is SIPT certified, n (%)
Yes	89 (52)	56 (77)	9 (69)	154 (60)
No	83 (48)	17 (23)	4 (31)	104 (40)
Primary work facility, n (%)
Private clinic	70 (41)	55 (75)	7 (54)	132 (52)
School system	80 (47)	16 (22)	4 (31)	100 (39)
Hospital	16 (10)	2 (3)	0	18 (7)
Other	3 (2)	0	2 (15)	5 (2)
Years of practice experience in occupational therapy, M (SD)	15.9 (10.78)^a	13.3 (8.21)^b	13.9 (9.27)^c	15.1 (10.07)^d
Years of practice experience in OT–SI, M (SD)	12.5 (10.32)^e	8.6 (6.33)^b	9.3 (7.20)^c	11.2 (9.34)^f

^an = 170. ^bn = 73. ^cn = 13. ^dn = 257. ^en = 166. ^fn = 252.

View Large

Table 1.

Therapist and Facility Characteristics by Country

Survey Items	United States	South Africa	Other	Total
OT–SI is routinely provided, n (%)
Yes	118 (69)	63 (86)	9 (69)	190 (74)
No	54 (31)	10 (14)	4 (31)	68 (26)
Highest educational degree, n (%)
Bachelor’s	49 (29)	64 (93)	5 (42)	118 (47)
Master’s	110 (65)	5 (7)	6 (50)	121 (48)
Doctoral	11 (6)	0	1 (8)	12 (5)
Therapist is SIPT certified, n (%)
Yes	89 (52)	56 (77)	9 (69)	154 (60)
No	83 (48)	17 (23)	4 (31)	104 (40)
Primary work facility, n (%)
Private clinic	70 (41)	55 (75)	7 (54)	132 (52)
School system	80 (47)	16 (22)	4 (31)	100 (39)
Hospital	16 (10)	2 (3)	0	18 (7)
Other	3 (2)	0	2 (15)	5 (2)
Years of practice experience in occupational therapy, M (SD)	15.9 (10.78)^a	13.3 (8.21)^b	13.9 (9.27)^c	15.1 (10.07)^d
Years of practice experience in OT–SI, M (SD)	12.5 (10.32)^e	8.6 (6.33)^b	9.3 (7.20)^c	11.2 (9.34)^f

^an = 170. ^bn = 73. ^cn = 13. ^dn = 257. ^en = 166. ^fn = 252.

Table 2.

ASIFM Structural Section Contents and Median Interrater Joint Probability of Agreement

Section (No. of Items)	Description	Sample Abbreviated Items	Interrater Agreement
Part I: Safe Environment (5)	Examines provisions for ensuring physical safety in the therapy environment	Use of mats, cushions, or pillows Adjustable equipment Equipment monitored for safe use Unused equipment stored safely Routine monitoring and repair for safety	1.00^a
Part II: Assessment Reports (19)	Checks for standard assessment processes and reports that address evaluation results, goal setting, and intervention planning	Assessment reports contain • History and occupational profile • Reason for referral • Evaluation of sensory processing (modulation and discrimination) • Evaluation of praxis, postural–ocular control, visual perception, fine and gross motor skills • Evaluation of organizational skills • Interpretation of results in relation to child’s life • As appropriate, intervention plan and collaborative goals related to participation.	0.95^b
Part III: Physical Space and Equipment (28)	Examines physical environmental affordances for sensory–motor exploration and movement through space	Space allows for flow of physical activity, designated quiet space, and flexible arrangement of equipment.	0.96^a
IIIA: Physical Space (6)		Ceiling hooks, rotational devices, and bungee cords for suspended equipment	1.00^c
IIIB: Equipment (22)		Wide variety of suspended and nonsuspended equipment and sensory materials available	0.95^a
Part IV: Communication With Parents and Teachers (3)	Checks for ongoing formal and informal discussions with parents and teachers	Therapists routinely meet with parents and teachers regarding course of intervention as related to child’s occupational performance.	1.00^a
Total structural section score (55)			0.95^d

Note. ASIFM = Ayres Sensory Integration Fidelity Measure.

View Large

Table 2.

ASIFM Structural Section Contents and Median Interrater Joint Probability of Agreement

Section (No. of Items)	Description	Sample Abbreviated Items	Interrater Agreement
Part I: Safe Environment (5)	Examines provisions for ensuring physical safety in the therapy environment	Use of mats, cushions, or pillows Adjustable equipment Equipment monitored for safe use Unused equipment stored safely Routine monitoring and repair for safety	1.00^a
Part II: Assessment Reports (19)	Checks for standard assessment processes and reports that address evaluation results, goal setting, and intervention planning	Assessment reports contain • History and occupational profile • Reason for referral • Evaluation of sensory processing (modulation and discrimination) • Evaluation of praxis, postural–ocular control, visual perception, fine and gross motor skills • Evaluation of organizational skills • Interpretation of results in relation to child’s life • As appropriate, intervention plan and collaborative goals related to participation.	0.95^b
Part III: Physical Space and Equipment (28)	Examines physical environmental affordances for sensory–motor exploration and movement through space	Space allows for flow of physical activity, designated quiet space, and flexible arrangement of equipment.	0.96^a
IIIA: Physical Space (6)		Ceiling hooks, rotational devices, and bungee cords for suspended equipment	1.00^c
IIIB: Equipment (22)		Wide variety of suspended and nonsuspended equipment and sensory materials available	0.95^a
Part IV: Communication With Parents and Teachers (3)	Checks for ongoing formal and informal discussions with parents and teachers	Therapists routinely meet with parents and teachers regarding course of intervention as related to child’s occupational performance.	1.00^a
Total structural section score (55)			0.95^d

Note. ASIFM = Ayres Sensory Integration Fidelity Measure.

Table 3.

Structural Subsection and Total Section Scores of Facilities That Did and Did Not Provide OT–SI

Section	OT–SI			Non-OT–SI
Section	n	Median	Range	n	Median	Range
Part I, Safe Environment	189	10	6–10	63	10	0–10
Part II, Assessment Reports	187	38	24–38	64	32	12–38
Part III, Physical Space and Equipment	181	52	8–56	60	37	0–56
A. Physical Space	187	12	0–12	61	6	0–12
B. Equipment	181	42	8–44	62	29	0–44
Part IV, Communication With Parents and Teachers	184	6	0–6	61	6	0–6
Total structural section score	178	104	58–110	55	84	36–110

View Large

Table 3.

Structural Subsection and Total Section Scores of Facilities That Did and Did Not Provide OT–SI

Section	OT–SI			Non-OT–SI
Section	n	Median	Range	n	Median	Range
Part I, Safe Environment	189	10	6–10	63	10	0–10
Part II, Assessment Reports	187	38	24–38	64	32	12–38
Part III, Physical Space and Equipment	181	52	8–56	60	37	0–56
A. Physical Space	187	12	0–12	61	6	0–12
B. Equipment	181	42	8–44	62	29	0–44
Part IV, Communication With Parents and Teachers	184	6	0–6	61	6	0–6
Total structural section score	178	104	58–110	55	84	36–110