Examining Content Validity and Reliability of the Assessment of Children’s Hand Skills (ACHS): A Preliminary Study

Chi-Wen Chien; Ted Brown; Rachael McDonald

doi:10.5014/ajot.2010.08158

Free

Research Article | September 2010

Examining Content Validity and Reliability of the Assessment of Children’s Hand Skills (ACHS): A Preliminary Study

Chi-Wen Chien; Ted Brown; Rachael McDonald

Author Affiliations

Article Information

American Journal of Occupational Therapy, September/October 2010, Vol. 64, 756-767. doi:10.5014/ajot.2010.08158

Abstract

OBJECTIVE. We developed the Assessment of Children’s Hand Skills (ACHS) to evaluate hand skills using naturalistic observation and examined the assessment's interrater and test–retest reliability.

METHOD. We developed the hand skill framework, performed expert review, and pilot tested the ACHS. The ACHS's reliability was examined by recruiting 54 children (30 typically developing children and 24 children with disabilities).

RESULTS. The test–retest reliability for the ACHS was satisfactory at the individual item level (0.42 ≤ κ ≤ 0.79) and the total scale level (Spearman’s ρ = 0.78, p < .01). Moderate interrater agreement of the total scale scores was demonstrated (ρ = 0.63, p < .01), but individual items exhibited varied interrater agreement.

CONCLUSION. The ACHS demonstrated adequate content validity and preliminary reliability evidence and could be used to quantify children’s hand skill use. Construct validity should be established in a clinical setting.

Using hand skills for manipulating objects or attaining functional needs plays a vital role in children’s development and successful participation in daily occupations (Case-Smith et al., 1998; Exner, 1993; Henderson, 2006). Studies have reported that children spend more than one-third of a typical school day on hand skill–related activities (Marr, Cermak, Cohn, & Henderson, 2003; McHale & Cermak, 1992). Children participating in out-of-school contexts also use their hands to complete activities such as play and self-care (Hofferth & Sandberg, 2001; Robinson & Bianchi, 1997). Consequently, effective use of hand skills can support completion of daily occupational performance activities, participation in education-related tasks, and the development of emotional well-being and social skills (Case-Smith et al., 1998; Henderson & Pehoski, 2006; Rogers & Case-Smith, 2002).

Children with disabilities may have difficulties or problems with hand skills that impede their daily life participation. Limited daily life participation often results in referral for occupational therapy services (Brown, Rodger, Brown, & Roever, 2005; Case-Smith, 1996). Consequently, occupational therapists have advocated the use of occupation-centered assessments that support a top-down approach to therapy evaluation and intervention (American Occupational Therapy Association [AOTA], 2008; Coster, 1998; Hocking, 2001). The top-down approach emphasizes a global perspective of considering children’s participation and roles to determine what is important to the child and the parents rather than discrete components of certain skills or abilities. The occupation-centered assessment with top-down approach allows therapists to focus the evaluation of children’s occupational performance on their meaningful occupations in relevant environments (Law et al., 1996) and, accordingly, assists therapists in identifying critical occupational issues potentially caused by hand skill difficulties.

Clinicians do not have an occupation-centered assessment tool that specifically evaluates the hand skills of children with different health conditions. On the basis of a recent literature search by Randall, Imms, and Carey (2008), 13 measures of hand skills in children were reported. Five of the tests evaluate a child’s best hand skill capacity as demonstrated during optimal conditions (e.g., what he or she can do in a standardized environment). These tests include the Peabody Developmental Motor Scales (Folio & Fewell, 2000) and the Bruininks–Oseretsky Test of Motor Proficiency (Bruininks & Bruininks, 2005). Children’s completion of test items does not necessarily equate to their actual performance on daily hand skill tasks in everyday life contexts. Moreover, instruments usually focus on certain underlying hand skill components (e.g., manual dexterity, visual–motor integration skills, or grip strength) and contribute to a top-down, occupation-centered assessment in a limited way.

The remaining eight assessment tools are likely to be able to assist occupational therapists in understanding children's occupational performance and participation affected by hand skill difficulties. However, their applicability remains limited. The In-Hand Manipulation Test (Breslin & Exner, 1999) assesses only in-hand manipulation skills of preschool-age children. The Melbourne Assessment of Unilateral Upper Limb Function (Randall, Johnson, & Reddihough, 1999) and the Quality of Upper Extremity Skills Tests (QUEST; DeMatteo et al., 1992) both cover a wider age range (e.g., 5–15 yr and 1.5–8 yr, respectively) and measure hand skill movement quality, but they do not assess the bimanual skills essential for performing the majority of daily life activities. By contrast, the Assisting Hand Assessment (Krumlinde-Sundholm & Eliasson, 2003) and the ABILHAND–Kids (Arnould, Penta, Renders, & Thonnard, 2004) assess bimanual skills; however, the two tests can be used to examine only children with cerebral palsy or unilateral dysfunction. The Assessment of Motor and Process Skills (AMPS; Fisher, 1997), the Pediatric Evaluation of Disability Inventory (Haley, Coster, Ludlow, Haltiwanger, & Andrellos, 1992), and the Activity Scale for Kids (Young, Williams, Yoshida, & Wright, 2000) are commonly used instruments suitable for use with a range of children with disabilities, but they are not designed specifically for the assessment of hand skills.

Hence, a need exists for a valid and reliable hand skill assessment tool that measures a comprehensive range of children’s hand skills and can be applied with different populations of children over a wide age range. Such an instrument must focus on children’s actual hand skill performance on relevant occupations within their naturalistic contexts to fit with the philosophy of occupation-centered assessment.

We recently proposed the Children’s Hands Skills Framework (CHSF) as a model to guide development of a pediatric hand skill assessment (Chien, Brown, & McDonald, 2009). The framework was developed after extensive review of published hand skill models, classifications, and instruments and was validated by international experts’ reviews. The CHSF divides the functional repertoire of children’s hand skill use into six distinct categories: manual gesture, body-contact hand skills, adaptive skilled hand use, arm–hand use, bimanual use, and general quality. Manual gesture and body-contact hand skills are the categories in which the hands do not contact any object. Object-related hand skills—in which the hands are used to handle various objects—include adaptive skilled hand use, arm–hand use, and bimanual use. The final category in the CHSF is related to general quality of children’s hand skills. The six hand skill categories are further divided into 22 subcategories that assist in detailed analysis and establishment of a complete hand skill profile in children.

This study aimed to describe the development process of the Assessment of Children’s Hand Skills (ACHS), which operationalizes the constructs presented in the CHSF into measurable items for monitoring progress of children’s hand skills. The ACHS intends, by means of naturalistic observation, to evaluate how effectively children ages 2–12 use their hands when engaged in meaningful occupations. In the concept of the ACHS, the six CHSF hand skill categories with comprehensive subcategories will be used as observational components to analyze and rate children’s actual hand skill performance in their relevant environments rather than best capacity in standardized settings. The ACHS will also include a range of difficulty levels of hand skill activities within three childhood occupations of leisure and play, school-related work, and activities of daily living. A corresponding parent-report questionnaire will be developed to obtain parents’ perceptions about which activities present appropriate challenges for the children being observed. The children’s functional hand skill assessment can be attained by observing their engagement in a selection of specific challenging activities. After this conceptualization and establishment of the ACHS's content validity, we assessed its interrater and test–retest reliability.

Method

Development of the ACHS

Four steps were completed in the ACHS development process: (1) generation of working definitions and descriptions for observable hand skill items from the CHSF, (2) selection of a pool of hand skill activities that were representative of childhood occupations and could elicit children’s most appropriate hand skill performance across different age levels, (3) completion of performance analyses that define how the hand skills may be performed in the selected activities, and (4) construction of rating scales for each hand skill item (reflecting the effectiveness of its performance) and scoring of a sample of children to generate a preliminary ACHS research version.

Generation of Hand Skill Items.

Twenty hand skill items were generated in the ACHS that reflected the conceptualization of the six CHSF categories with 22 corresponding subcategories. The manual gesture and body-contact hand skills were the first two ACHS skill items. Although the manual gesture category in the CHSF included four subcategories, generating four individual items that measure each subtype of gesturing was considered to be excessively detailed. Thus, they were integrated into a single manual gesturing item.

The remaining 18 hand skill items were consistent with the proposed CHSF categories and subcategories and can be grouped as follows: adaptive skilled hand use, 5 items (grasping, holding, in-hand manipulating, releasing, and isolated finger movement); arm–hand use, 7 items (reaching, turning, carrying, throwing, catching, moving, and stabilizing); bimanual use, 3 items (transferring, using both hands simultaneously, and using both hands cooperatively); and general quality of hand skills, 3 items (accuracy, pace, and movement quality). Each hand skill item was operationally defined by providing explanatory descriptions and examples to help prospective users understand the assessment of the skill items. The 20 hand skill items made up the core of the ACHS to describe and evaluate how children use their hand skills when performing certain activities.

Selection of Hand Skill Activities.

A pool of child-relevant hand skill activities was developed after establishing individual hand skill items. The intent was to have a range of difficulty levels of hand skill activities in the ACHS that could be used to elicit appropriate hand skill performances in children ages 2–12 yr. Four selection criteria were developed to assist with the selection of the hand skill activities: (1) to be representative of common childhood occupations that require hand use; (2) to present specific difficulty (or age-appropriate difficulty) to children ages 2–12 yr; (3) to be easily observed naturalistically while placing minimal demands on language, cognition, and perception; and (4) to have minimal gender or cultural bias for children when performing the tasks.

We conducted a systematic review of 41 existing hand skill tests, focusing on the test items and contents. From the review, 81 potential activities were grouped according to task and tool similarity of those test items; these activities were then reduced to 32 according to the four established selection criteria. A panel of 10 experts (7 from Australia, 2 from Taiwan, and 1 from the United States) evaluated the suitability of the 32 activities. The panel members were either researchers or clinicians with at least 7 years’ experience in pediatric occupational therapy or research on children’s hand skills. They completed a questionnaire composed of 32 questions that involved rating the appropriateness of the activities on a 5-point Likert scale (5 = extremely and 1 = not at all ) as well as three checklists on whether the 49 discarded activities should be retained. The experts were also invited to provide additional written comments about the descriptions and content of the included activities as well as the justifications for the retention of the discarded activities.

The results of the experts’ feedback (nearly 200 comments were received) indicated a large degree of agreement on including the 32 listed activities (mean rating range of 2.8–4.4 points). Accordingly, 12 activities that were originally included in the activity pool were eliminated, because their main ratings were less than the cut-off value of 3 points (i.e., a moderate level), they were irrelevant for children’s current interests, or the item contents or hand skill requirements overlapped those of other activities. In addition, two originally eliminated activities were moved back into the activity pool following several experts’ suggestions. The final activity set for the ACHS consisted of 22 hand skill activities in the domains of Leisure and Play (8 activities), School-Related Work (8 activities), and Activities of Daily Living (6 activities). Each activity was operationally defined and described in terms of the essential goal as well as potential materials or ways in which children may use them, allowing flexibility in observing how these activities are to be performed by children naturalistically.

Completion of Performance Analyses.

We developed performance analyses of the 22 hand skill activities in which the identified 20 hand skills may be performed. Thereafter, the 14 content experts who participated in our previous reviews of the CHSF and hand skill activities (7 from Australia, 3 from Taiwan, 1 from each of the United States, United Kingdom, Canada, and Hong Kong) were invited to review the proposed performance analyses. Each expert completed a questionnaire confirming the suitability of the proposed performance analyses of the selected activities on 5-point Likert scales and providing qualitative written comments. Minor revisions on the performance analyses were made on the basis of the experts’ written feedback.

Construction and Testing of Rating Scales.

A 6-point Likert rating scale was determined to indicate the effectiveness level of each ACHS skill item. The rating scale format was based on the scale construction of the AMPS, which uses an even-numbered ordinal scale (e.g., 4 points). Different from the AMPS rating scale, however, the ACHS uses a 6-point rating scale to increase the discrimination level of the rating scale to better reflect children’s hand skill effectiveness. Within the proposed 6-point rating scale in the ACHS skill items, the top three rating categories (e.g., 6 = very effective, 5 = effective, and 4 = slightly effective) indicate that a child demonstrates effective hand skill performances that do not disrupt the activity processing (e.g., leading to unacceptable delay, increased efforts, activity breakdown, or the need for others’ assistance). Conversely, the lower three scores are rated as 3 = slightly ineffective, 2 = ineffective, and 1 = very ineffective, when the child’s demonstrated hand skill performances disrupt activity processing. For each ACHS skill item, specific scoring criteria and examples have been developed and provided in the test administration manual.

Pilot testing of the proposed rating scale structure was accomplished with 6 children presenting with different performance abilities in several of the activities selected from the determined activity pool. Three of the children were typically developing, and 3 had hemiplegic cerebral palsy or autism spectrum disorder; all were ages 2–12 yr. We each then observed the children’s hand skill performances in the selected activities and simultaneously scored their hand skills using the proposed 6-point rating criteria. Minor changes and clarifications were added into the formulation and refinement of the scoring criteria to create a preliminary research version. A further reliability study was conducted to examine the interrater and test–retest reliability of the ACHS.

Reliability Examination of the ACHS

Participants.

Fifty-four children participated in the study; 30 were in the interrater reliability component of the study and 44 were in the test–retest reliability part (see Table 1). Of the 54 participants, 30 were typically developing children who were recruited from one child care center and two preschools in Melbourne’s southern metropolitan regions; the remaining 24 children were recruited from two special schools if they presented (according to parent or therapist report) with hand skill difficulties caused by disorders such as autism spectrum disorder, Down syndrome, or developmental delay.

Table 1.

Demographic Characteristics of the Participating Children in the Reliability Study

Demographic Variables	Interrater Reliability Study (N = 30)	Test–Retest Reliability Study (N = 44)
Gender, n (%)
Boys	21 (70.0)	29 (65.9)
Girls	9 (30.0)	15 (34.1)
Average age in months, mean ± SD	64.3 ± 23.2	70.0 ± 26.6
Age in years, n (%)
2–4	18 (60.0)	23 (52.3)
5–7	7 (23.3)	13 (29.6)
8–10	5 (16.7)	6 (13.6)
11–12	—	2 (4.5)
Diagnosis, n (%)
No disability	18 (60.0)	24 (54.5)
Development delay	3 (10.0)	6 (13.6)
Autism	6 (20.0)	9 (20.5)
Down syndrome	3 (10.0)	4 (9.1)
Brachial plexus birth palsy	—	1 (2.3)
Education of the parents, n (%)
Primary school graduate	1 (3.3)	2 (4.5)
High school graduate	13 (43.3)	16 (36.4)
College/TAFE diploma	8 (26.7)	16 (36.4)
Undergraduate degree	3 (10.0)	4 (9.1)
Postgraduate degree	4 (13.3)	4 (9.1)
Unreported	1 (3.3)	2 (4.5)

Note. — = no child in the variable; TAFE = Technical and Further Education.

View Large

Table 1.

Demographic Characteristics of the Participating Children in the Reliability Study

Demographic Variables	Interrater Reliability Study (N = 30)	Test–Retest Reliability Study (N = 44)
Gender, n (%)
Boys	21 (70.0)	29 (65.9)
Girls	9 (30.0)	15 (34.1)
Average age in months, mean ± SD	64.3 ± 23.2	70.0 ± 26.6
Age in years, n (%)
2–4	18 (60.0)	23 (52.3)
5–7	7 (23.3)	13 (29.6)
8–10	5 (16.7)	6 (13.6)
11–12	—	2 (4.5)
Diagnosis, n (%)
No disability	18 (60.0)	24 (54.5)
Development delay	3 (10.0)	6 (13.6)
Autism	6 (20.0)	9 (20.5)
Down syndrome	3 (10.0)	4 (9.1)
Brachial plexus birth palsy	—	1 (2.3)
Education of the parents, n (%)
Primary school graduate	1 (3.3)	2 (4.5)
High school graduate	13 (43.3)	16 (36.4)
College/TAFE diploma	8 (26.7)	16 (36.4)
Undergraduate degree	3 (10.0)	4 (9.1)
Postgraduate degree	4 (13.3)	4 (9.1)
Unreported	1 (3.3)	2 (4.5)

Note. — = no child in the variable; TAFE = Technical and Further Education.

The interrater reliability study involved 69 observational assessments, whereas the test–retest reliability study involved 74 observations, each of which was based on the participating children’s engagement in 1 of 22 different activities (Table 2). The average retest interval between the initial and second observation periods was 13.9 days with a standard deviation of 2 days. In both studies, the observational assessments were relatively evenly distributed among the Leisure and Play, School-Related Work, and Activities of Daily Living domains.

Table 2.

Frequency of the Activities Observed in the Reliability Study

Activity Categories	Interrater Reliability Study (N = 69 assessments)	Test–Retest Reliability Study (N = 74 assessments)
Leisure and Play domain, n (%)	20 (29.0)	26 (35.1)
Construction (blocks)	1	1
Puzzle	4	5
Stringing beads	3	1
Catching and throwing objects	7	14
Card game	2	—
Play dough/clay	3	4
Folding paper	—	—
Handling money	—	1
School-Related Work domain, n (%)	28 (40.6)	22 (29.8)
Reading book	2	—
Drawing or coloring	10	9
Writing & copying	1	5
Cutting	10	4
Pasting	3	1
Using computer	1	3
School tool use (ruler)	—	—
Putting on backpack	1	—
Activities of Daily Living domain, n (%)	21 (30.4)	26 (35.1)
Drinking	2	6
Eating	6	10
Dressing upper body	6	—
Putting on socks and shoes	1	3
Washing hands	6	6
Brushing teeth	—	1

Note. — = no assessment in the activity category.

View Large

Table 2.

Frequency of the Activities Observed in the Reliability Study

Activity Categories	Interrater Reliability Study (N = 69 assessments)	Test–Retest Reliability Study (N = 74 assessments)
Leisure and Play domain, n (%)	20 (29.0)	26 (35.1)
Construction (blocks)	1	1
Puzzle	4	5
Stringing beads	3	1
Catching and throwing objects	7	14
Card game	2	—
Play dough/clay	3	4
Folding paper	—	—
Handling money	—	1
School-Related Work domain, n (%)	28 (40.6)	22 (29.8)
Reading book	2	—
Drawing or coloring	10	9
Writing & copying	1	5
Cutting	10	4
Pasting	3	1
Using computer	1	3
School tool use (ruler)	—	—
Putting on backpack	1	—
Activities of Daily Living domain, n (%)	21 (30.4)	26 (35.1)
Drinking	2	6
Eating	6	10
Dressing upper body	6	—
Putting on socks and shoes	1	3
Washing hands	6	6
Brushing teeth	—	1

Note. — = no assessment in the activity category.

Procedures.

The study was given approval by the university ethics committee, the Department of Education, and the committees of the participating child care centers, kindergartens, and schools. All the participants’ parents or caregivers signed informed consent forms before participation in the study.

Once consent was given, the participating parents completed a questionnaire before the ACHS observation was completed. The parent-report questionnaire asked for information about the children to help gauge which hand skill activities presented appropriate developmental challenges for the children being observed. On the basis of the questionnaire results, a selection of two or three activities that were appropriately challenging for the children to complete was proposed to make the ACHS observations of their hand skills. The current study, however, focused mainly on examining rater reliability when using the ACHS to score children’s hand skills. Therefore, in addition to observing the children performing the parent-reported challenging activities, the raters were able to observe and score the children’s hand skills performance in other, age-appropriate activities.

All the ACHS observations in this study were conducted in child care centers, kindergartens, or school settings. Two raters were involved in observing the children’s performing the potential activities that presented an appropriate challenge or matched the children’s age levels. Each activity was observed for a maximum time period of 10 min. For the interrater reliability study, the raters without previous knowledge of the children scored their hand skill performance on those observed activities simultaneously and independently using the skill items with the corresponding rating criteria. After a 2-wk interval, the same children’s hand skill performance on the same activities was observed and rated again by one rater who did not review the children’s previous ratings for the test–retest reliability study.

Raters.

The two raters who participated in the reliability study were occupational therapists. Rater A was the primary investigator (Chi-Wen Chien); Rater B was a therapist with 1 yr of professional experience working with children in a school setting. Before the study, Rater B was trained in an instructional 1-day course with seven subsequent videotaped ratings by the principal investigator (Chien). The training session involved a brief introduction of the ACHS content and observational procedure as well as a detailed explanation of the definitions and rating criteria of the hand skill items. The seven video clips, which included one typically developing child and three children with hand skill problems (e.g., hemiplegia cerebral palsy and autism spectrum disorder), were subsequently provided for the raters to practice completing the 20 ACHS skill items with the rating categories. The training videotapes consisted of seven activities: (1–3) playing with blocks, balls, and play dough; (4) drinking; (5) dressing upper body; (6) drawing; and (7) cutting. During the training, the raters scored the children’s hand skills using the ACHS independently and then discussed their ratings afterward to obtain a consensus. In addition, the raters evaluated two typically developing children by means of naturalistic observation with the ACHS while playing with puzzles and drawing pictures. The addition of the naturalistic observations aimed to familiarize the raters with the extraneous variables and contextual influences that may be present in real-life circumstances.

Instrumentation.

The ACHS is a naturalistic observational assessment to evaluate the effectiveness of 2- to 12-yr-old children’s hand skill performance in daily real-life environments. The ACHS research version consists of 20 hand skill items that are rated on a 6-point rating scale. A score of 6 indicates very effective hand skill performance, whereas a score of 1 indicates very ineffective hand skill performance. The total raw score has little meaning, because specific hand skill items can be reported as “not observed” when the child does not use them in activities. Therefore, a percentage score for the ACHS total score (scaling 0%–100%) is generated by following the formula:

Grahic Jump Location

The total percentage score is used for data analysis only when more than half of the 20 hand skill items are observed and scored by raters.

The parent-report questionnaire was also part of the ACHS. The questionnaire includes a 22-item checklist that requires children’s parents to report their perceptions regarding their child’s hand skill proficiency at carrying out the 22 selected activities of the ACHS on a 3-point scale: extremely difficult (1), difficult (2), or not difficult (3). An additional item involves determining which environments would provide the best opportunity to observe their child performing the activities as reported by them.

Data Analysis.

The interrater and test–retest reliability of the ACHS was examined at both the individual item level and the total scale level, respectively. At the individual item level, we first examined the extent to which the two raters identically observed or did not observe the occurrence of the specific hand skills in the same child. The Cohen’s κ statistics (Cohen, 1960) were used to examine the interrater agreement on reported observations of the hand skill items (κ > 0.80 = very good; 0.61–0.80 = good; 0.41–0.60 = moderate; and <0.40 = unacceptable;Altman, 1991; Andresen, 2000). In addition, percentage agreement of the raters’ observations was reported to assist with result interpretation, because the κ statistics may be reduced when chance agreement is also high (Brennan & Silman, 1992). Therefore, percentage agreement was provided for the analysis to justify the items with low κ statistics caused by high agreement.

Next, the agreement of the hand skill item ratings that were observed and then scored between the two raters or between the test–retest observations was further examined. Weighted κ statistics with quadratic weights were used for the 6-point ACHS rating scales. The quadratic weighted κ statistics are a common weighting method that can reflect the degree of disagreement in ordinal scales with more than two categories by attaching greater emphasis to large differences between ratings than to small differences (Sim & Wright, 2005). Interpretation of the quadratic weighted κ statistics is similar to that of the unweighted κ mentioned earlier. Moreover, the Wilcoxon signed-rank test (Portney & Watkins, 2000) was used to check whether there was a systematic difference in the agreement score of each item. The significance level was set at p < .05.

At total scale level, three statistical indexes were used to examine the interrater and test–retest reliability of the ACHS. First, the Spearman’s ρ correlation coefficient was calculated to investigate the level of agreement between the total percentage scores of the two raters or the two successive observations. Correlation coefficients were interpreted as follows: ρ ≥ 0.75 = strong relationship, ρ 0.50–0.74 = moderate relationship; and ρ ≤ 0.49 = weak relationship (Portney & Watkins, 2000). Second, the Wilcoxon signed-rank test was used to examine statistical significance across the raters or repeated observations. Third, the Bland–Altman plotting method (Bland & Altman, 1986) was completed to provide a visual illustration of the degree of percentage score agreement between the raters (for the interrater reliability analysis) and the test–retest observations (for the test–retest reliability analysis). The Bland–Altman method involves plotting the scores of the difference between the two measurements against the average of the measurements. The 95% limits of agreement (e.g., mean difference ± 1.96 SD of the difference) were calculated, and smaller values indicate higher agreement level.

Results

Interrater and Test–Retest Reliability of the Individual Items

Table 3 reports the results of the interrater reliability analysis of the agreement between the two raters on observing the occurrence of each hand skill item. Fifteen (75%) of 20 items had κ > 0.40, indicating at least a moderate agreement between the raters. Of the five hand skill items having potentially low interrater reliability, however, the carrying and moving hand skill items exhibited a high percentage of agreement (≥0.88). This finding implied that the children’s use of these two hand skill items during selected activities was consistently observed between the raters, thus causing false lower κ values. Therefore, only three ACHS skill items were found to demonstrate poor interrater agreement: body-contact hand skills, in-hand manipulating, and transferring.

Table 3.

Interrater Reliability Between the Two Raters on Observing the Occurrence of the Hand Skill Items

Items	No. of Reported Observations, Rater A (Rater B)	% Agreement	κ^a
01: Manual gesture	21 (14)	0.84	0.59
02: Body contact hand skills^b	14 (14)	0.77	0.28
03: Grasping	69 (67)	0.97	1.00
04: Holding	67 (67)	1.00	1.00
05: In-hand manipulating^b	49 (35)	0.57	0.13
06: Releasing	69 (69)	1.00	1.00
07: Isolated finger movement	69 (68)	0.99	1.00
08: Reaching	69 (69)	1.00	1.00
09: Turning	69 (68)	0.99	1.00
10: Carrying	66 (67)	0.96	0.38
11: Throwing	7 (6)	0.99	0.92
12: Catching	7 (7)	1.00	1.00
13: Moving	62 (68)	0.88	0.03
14: Stabilizing	58 (59)	0.87	0.50
15: Transferring^b	51 (28)	0.61	0.28
16: Using both hands simultaneously	43 (37)	0.77	0.53
17: Using both hands cooperatively	58 (64)	0.91	0.58
18: Accuracy	69 (69)	1.00	1.00
19: Pace	69 (69)	1.00	1.00
20: Movement quality	69 (69)	1.00	1.00

aκ statistics were used to examine the agreement between the two raters on whether the hand skill items were observed. Underlined values indicate poor agreement (weighted κ < 0.4).

bItems with substantially poor interrater agreement.

View Large

Table 3.

Interrater Reliability Between the Two Raters on Observing the Occurrence of the Hand Skill Items

Items	No. of Reported Observations, Rater A (Rater B)	% Agreement	κ^a
01: Manual gesture	21 (14)	0.84	0.59
02: Body contact hand skills^b	14 (14)	0.77	0.28
03: Grasping	69 (67)	0.97	1.00
04: Holding	67 (67)	1.00	1.00
05: In-hand manipulating^b	49 (35)	0.57	0.13
06: Releasing	69 (69)	1.00	1.00
07: Isolated finger movement	69 (68)	0.99	1.00
08: Reaching	69 (69)	1.00	1.00
09: Turning	69 (68)	0.99	1.00
10: Carrying	66 (67)	0.96	0.38
11: Throwing	7 (6)	0.99	0.92
12: Catching	7 (7)	1.00	1.00
13: Moving	62 (68)	0.88	0.03
14: Stabilizing	58 (59)	0.87	0.50
15: Transferring^b	51 (28)	0.61	0.28
16: Using both hands simultaneously	43 (37)	0.77	0.53
17: Using both hands cooperatively	58 (64)	0.91	0.58
18: Accuracy	69 (69)	1.00	1.00
19: Pace	69 (69)	1.00	1.00
20: Movement quality	69 (69)	1.00	1.00

aκ statistics were used to examine the agreement between the two raters on whether the hand skill items were observed. Underlined values indicate poor agreement (weighted κ < 0.4).

bItems with substantially poor interrater agreement.

Tables 4 and 5 report the interrater and test–retest agreement between the reported ratings of the individual items. For the interrater reliability study, 12 (60%) of the 20 hand skill items were found to have moderate agreement (weighted κ > 0.40), as reported in Table 4. Thirteen (65%) hand skill items did not exhibit systematic differences between the ratings of the two raters; however, high variability in the results of both statistical indicators of the 20 hand skill items was demonstrated. Three items (i.e., the holding, moving, and movement quality hand skill items) exhibited both low agreement and systematic difference between the two raters. Conversely, none of the hand skill items in the test–retest reliability study were found to have weighted κ values < 0.40 (Table 5). Only the accuracy hand skill item exhibited systematic difference between the ratings of the test–retest observations.

Table 4.

Interrater Reliability of the Observed Assessments in the Hand Skill Items

Items	N	Rater A, Mean (SD)	Rater B, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	12	4.4 (1.0)	4.9 (0.3)	0.33	.06
02: Body contact hand skills	6	4.5 (0.6)	4.8 (0.4)	0.33	.16
03: Grasping	67	4.3 (0.7)	4.1 (0.9)	0.49	.05
04: Holding	67	4.6 (0.5)	4.3 (0.8)	0.37	.02
05: In-hand manipulating	27	3.6 (0.7)	4.2 (0.7)	0.45	.01
06: Releasing	69	4.4 (0.6)	4.4 (0.7)	0.44	.72
07: Isolated finger movement	68	4.3 (0.7)	4.4 (0.7)	0.51	.37
08: Reaching	69	4.4 (0.6)	4.5 (0.7)	0.68	.25
09: Turning	68	4.2 (0.7)	4.2 (0.7)	0.57	.27
10: Carrying	65	4.3 (0.6)	4.5 (0.7)	0.51	.03
11: Throwing	6	4.3 (0.5)	4.2 (0.4)	0.09	.56
12: Catching	7	2.6 (1.3)	3.6 (0.8)	0.07	.10
13: Moving	61	3.9 (0.7)	4.2 (0.8)	0.37	.01
14: Stabilizing	54	4.1 (0.9)	4.5 (0.7)	0.52	<.01
15: Transferring	26	4.1 (0.6)	4.3 (0.5)	0.27	.25
16: Using both hands simultaneously	32	4.1 (0.8)	4.2 (0.9)	0.64	.32
17: Using both hands cooperatively	58	3.7 (0.9)	4.1 (0.8)	0.46	.01
18: Accuracy	69	3.9 (0.8)	3.9 (1.0)	0.51	.49
19: Pace	69	3.8 (0.8)	3.8 (0.8)	0.41	1.00
20: Movement quality	69	3.6 (0.7)	4.1 (0.8)	0.33	<.01

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests or poor agreement in weighted κ statistics (value < 0.40). SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test\x{2013}retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference between the raters’ reported ratings of the hand skill items between test\x{2013}retest observations.

View Large

Table 4.

Interrater Reliability of the Observed Assessments in the Hand Skill Items

Items	N	Rater A, Mean (SD)	Rater B, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	12	4.4 (1.0)	4.9 (0.3)	0.33	.06
02: Body contact hand skills	6	4.5 (0.6)	4.8 (0.4)	0.33	.16
03: Grasping	67	4.3 (0.7)	4.1 (0.9)	0.49	.05
04: Holding	67	4.6 (0.5)	4.3 (0.8)	0.37	.02
05: In-hand manipulating	27	3.6 (0.7)	4.2 (0.7)	0.45	.01
06: Releasing	69	4.4 (0.6)	4.4 (0.7)	0.44	.72
07: Isolated finger movement	68	4.3 (0.7)	4.4 (0.7)	0.51	.37
08: Reaching	69	4.4 (0.6)	4.5 (0.7)	0.68	.25
09: Turning	68	4.2 (0.7)	4.2 (0.7)	0.57	.27
10: Carrying	65	4.3 (0.6)	4.5 (0.7)	0.51	.03
11: Throwing	6	4.3 (0.5)	4.2 (0.4)	0.09	.56
12: Catching	7	2.6 (1.3)	3.6 (0.8)	0.07	.10
13: Moving	61	3.9 (0.7)	4.2 (0.8)	0.37	.01
14: Stabilizing	54	4.1 (0.9)	4.5 (0.7)	0.52	<.01
15: Transferring	26	4.1 (0.6)	4.3 (0.5)	0.27	.25
16: Using both hands simultaneously	32	4.1 (0.8)	4.2 (0.9)	0.64	.32
17: Using both hands cooperatively	58	3.7 (0.9)	4.1 (0.8)	0.46	.01
18: Accuracy	69	3.9 (0.8)	3.9 (1.0)	0.51	.49
19: Pace	69	3.8 (0.8)	3.8 (0.8)	0.41	1.00
20: Movement quality	69	3.6 (0.7)	4.1 (0.8)	0.33	<.01

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests or poor agreement in weighted κ statistics (value < 0.40). SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test\x{2013}retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference between the raters’ reported ratings of the hand skill items between test\x{2013}retest observations.

Table 5.

Test–Retest Reliability of the Observed Assessments in the Hand Skill Items

Items	N	First Evaluation, Mean (SD)	Second Evaluation, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	8	3.8 (1.0)	3.8 (1.3)	0.79	1.00
02: Body contact hand skills	6	4.8 (0.4)	4.7 (0.5)	0.57	.32
03: Grasping	74	4.4 (0.6)	4.4 (0.6)	0.56	.84
04: Holding	69	4.7 (0.6)	4.8 (0.6)	0.42	.84
05: In-hand manipulating	35	3.9 (1.1)	3.9 (1.0)	0.79	.80
06: Releasing	74	4.7 (0.6)	4.7 (0.6)	0.78	1.00
07: Isolated finger movement	74	4.5 (0.7)	4.5 (0.7)	0.68	.67
08: Reaching	74	4.7 (0.6)	4.6 (0.6)	0.59	.49
09: Turning	73	4.4 (0.7)	4.3 (0.6)	0.60	.22
10: Carrying	70	4.5 (0.6)	4.5 (0.5)	0.60	1.00
11: Throwing	14	4.0 (0.4)	3.7 (0.6)	0.50	.05
12: Catching	11	2.9 (0.8)	3.1 (1.0)	0.59	.48
13: Moving	50	3.9 (0.9)	4.0 (0.8)	0.54	.69
14: Stabilizing	54	4.3 (0.8)	4.2 (0.7)	0.61	.22
15: Transferring	43	4.5 (0.7)	4.4 (0.6)	0.70	.06
16: Using both hands simultaneously	34	4.3 (0.7)	4.4 (0.7)	0.43	.48
17: Using both hands cooperatively	53	4.0 (0.9)	4.0 (0.7)	0.61	.70
18: Accuracy	74	4.1 (0.8)	4.3 (0.7)	0.65	.03
19: Pace	74	4.3 (0.7)	4.4 (0.7)	0.56	.27
20: Movement quality	74	3.7 (0.8)	3.8 (0.7)	0.55	.18

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests. SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test–retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference in the rater's reported ratings of the hand skill items between test–retest observations.

View Large

Table 5.

Test–Retest Reliability of the Observed Assessments in the Hand Skill Items

Items	N	First Evaluation, Mean (SD)	Second Evaluation, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	8	3.8 (1.0)	3.8 (1.3)	0.79	1.00
02: Body contact hand skills	6	4.8 (0.4)	4.7 (0.5)	0.57	.32
03: Grasping	74	4.4 (0.6)	4.4 (0.6)	0.56	.84
04: Holding	69	4.7 (0.6)	4.8 (0.6)	0.42	.84
05: In-hand manipulating	35	3.9 (1.1)	3.9 (1.0)	0.79	.80
06: Releasing	74	4.7 (0.6)	4.7 (0.6)	0.78	1.00
07: Isolated finger movement	74	4.5 (0.7)	4.5 (0.7)	0.68	.67
08: Reaching	74	4.7 (0.6)	4.6 (0.6)	0.59	.49
09: Turning	73	4.4 (0.7)	4.3 (0.6)	0.60	.22
10: Carrying	70	4.5 (0.6)	4.5 (0.5)	0.60	1.00
11: Throwing	14	4.0 (0.4)	3.7 (0.6)	0.50	.05
12: Catching	11	2.9 (0.8)	3.1 (1.0)	0.59	.48
13: Moving	50	3.9 (0.9)	4.0 (0.8)	0.54	.69
14: Stabilizing	54	4.3 (0.8)	4.2 (0.7)	0.61	.22
15: Transferring	43	4.5 (0.7)	4.4 (0.6)	0.70	.06
16: Using both hands simultaneously	34	4.3 (0.7)	4.4 (0.7)	0.43	.48
17: Using both hands cooperatively	53	4.0 (0.9)	4.0 (0.7)	0.61	.70
18: Accuracy	74	4.1 (0.8)	4.3 (0.7)	0.65	.03
19: Pace	74	4.3 (0.7)	4.4 (0.7)	0.56	.27
20: Movement quality	74	3.7 (0.8)	3.8 (0.7)	0.55	.18

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests. SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test–retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference in the rater's reported ratings of the hand skill items between test–retest observations.

Interrater and Test–Retest Reliability of the Total Scale

For the interrater reliability study, the total percentage scores between the two raters exhibited a moderate level of correlation (ρ = 0.63, p < .01). The Bland–Altman plot (see Figure 1, Part A) exhibited no systematic trend (R² < .01) between the difference and the average of the two raters’ ratings. This finding indicated that the average ratings of the two raters were not significantly associated (e.g., neither increased nor decreased) with the difference between the raters. However, the limit of agreement of the ratings between the raters was notable (from –18.9 to 13.0), and two outliers beyond the identified limit of agreement were noted in the plot. The Wilcoxon signed-rank test also indicated a potential systematic difference (p =.05) of the percentage scores between the two raters.

Figure 1.

Bland–Altman plots for the interrater and test–retest reliability studies. (A) Interrater reliability study (N = 30 children, including 68 assessments); (B) test–retest reliability study (N = 43 children, including 73 assessments).

Note. The two bold dashed lines define the limits of agreement (mean difference ± 1.96 standard deviation).

Full Size | Slide (.ppt)

For the test–retest reliability study, the total percentage scores between the two repeated observations exhibited a strong level of correlation (ρ = 0.78, p < .01). No systematic difference of the total percentage scores between the test–retest observations was found on the basis of the Wilcoxon signed-rank test results. The Bland–Altman plot (Part B of Figure 1) also indicated no systematic trend (R² < .01) between the difference and the average of the two test–retest measurements. The limit of agreement of the ratings between the repeated measurements (from –14.0 to 13.0) was narrower when compared with that of the interrater reliability study. Three notable outliers were revealed in the Bland–Altman plot.

Discussion

This study described the development process of the ACHS, a scale designed to measure a comprehensive range of hand skills for use with different populations of children ages 2–12. The ACHS uses naturalistic observation and fits with an occupation-centered assessment approach to provide occupational therapists information about children’s hand skill performance on meaningful occupations that are completed in daily contexts. Its content validity was established through an extensive review of the literature, the conceptualization of a hand skill framework on which to base the ACHS skill items, several rounds of review and revision based on expert feedback, and preliminary field testing. Preliminary evidence for the reliability of the newly developed assessment, at the item and total scale level, was provided in the current study.

The study results exhibited satisfactory test–retest reliability of the ACHS at both the item and the total scale level, indicating that the hand skill items or total percentage scores rated by the same rater were relatively stable over time. Although the accuracy skill item exhibited systematic difference between the repeated observations, its mean difference between the observations was small (0.20 of 6 total item score), and its weighted κ value of 0.65 indicated a reasonable level of agreement. Regarding interrater reliability, several individual hand skill items displayed an unsatisfactory level of agreement between the two raters. Moderate interrater agreement of the total percentage scale scores was demonstrated. Moreover, less interrater agreement of the total percentage scale was confirmed by the Bland–Altman plots, where the limit of agreement was relatively larger than that of the test–retest agreement and indicated lower stability with higher natural variation. It is thus not advisable that multiple raters assess an individual child with the ACHS sequentially or interchangeably. We recommend that the total percentage scores rated by different raters may be used merely when compared or combined for data analysis.

Exploring why the ACHS exhibited acceptable test–retest reliability but inconsistent agreement between different raters is necessary for improvement of future reliability analyses. The effects of raters’ experience and skills may be one potential reason. Previous studies have reported that rater experience and the intensity of skills training provided for observational instruments might affect the reliability of the resulting test scores (Bernhardt, Matyas, & Bate, 2002; Cipriany-Dacko, Innerst, Johannsen, & Rude, 1997; Cusick, Vasquez, Knowles, & Wallen, 2005; Koch, Cruz, & Goodill, 2001). In the interrater reliability component of this study, the rater involved was Chi-Wen Chien, who has considerable experience and skill in completing ACHS ratings, whereas the second rater was a relatively recent graduate. This difference in experience would increase the likelihood of obtaining inconsistent interrater agreement of the hand skill item ratings between the two raters.

Two additional reasons may explain the inconsistent agreement between the two raters when perceiving the presence of certain hand skills or scoring them. First, the extraneous/contextual influence during naturalistic observation might lead to decreased interrater agreement. For example, any break, transition, and irrelevant activities that the child and others performed during real-life observations could interfere with the raters’ scoring or even their attention to the presence of specific hand skills under consideration. Second, the ACHS is a naturalistic assessment tool and does not use video recording to assist in item scoring. The raters did not have the option of watching the child’s performance repeatedly to confirm the hand skill item ratings. Observing and scoring of fast hand movements performed by children can become challenging for inexperienced raters (Sorsdahl, Moe-Nilssen, & Strand, 2008). In particular, body contact hand actions (e.g., scratching face), transferring an object between the hands in a relatively close distance, and manipulating small objects in the hand often occur very rapidly and last for a very short time. As might be expected, the format of naturalistic observation may limit the ability of raters to check their scoring or observe the presence of these quickly performed movements, impacting hand skill item interrater agreement. Moreover, the test–retest agreement of the ACHS, although adequate in the current study, was inferior to those of videotaped standardized assessments, such as the QUEST or the Melbourne assessment, which exhibited intraclass correlation coefficients of more than .90 in previous studies (Randall, Carlin, Chondros, & Reddihough, 2001; Sakzewski, Ziviani, & Van Eldik, 2001). Thus, a naturalistic observational instrument, by the nature of measurement, may not achieve the level of reliability that a standardized test would demonstrate.

Increasing the duration of ACHS rater training and providing more explicit instructions and examples in the test administration manual for raters are two possible ways to minimize inconsistent rating of hand skill items between raters. Therefore, the scope of future ACHS training courses for novices has been extended by proposing at least 10 video and naturalistic scoring practices each. Implicit descriptions and insufficient examples of a few items that exhibited poor interrater reliability were also revised to facilitate raters’ understanding and scoring. For example, the descriptions of body contact hand skills were reclassified as functional needs, and specific examples were offered. Revised instructions for the in-hand manipulating were emphasized to occur after an object had been grasped or picked up. Pictorial examples of the various in-hand manipulating subcategories were also provided. The description of the transferring hand skill item was made more explicit and specific to hand over individual object from one hand to the other.

In addition, more scoring explanations and general scoring guidelines were added or revised to assist raters to complete more accurate and consistent 6-point ratings. Providing further explicit revisions or examples and extensive training module, raters are expected to become more competent and accurate at using the ACHS. Future studies are warranted to examine the interrater reliability of the ACHS, by using the revised administration manual and offering extensive training opportunities for novice raters.

Limitations and Future Research

This study had several potential limitations. A modest sample of children, most of whom were male, of preschool age, and typically developing, was included in the study. Future research that investigates the interrater and test–retest agreement of the ACHS using a larger sample of children with a great diversity of characteristics is needed.

Moreover, the construct validity (or unidimensionality) of the 20 ACHS skill items is being researched; thus, it is unknown whether the hand skill items are valid or able to be summed together as a total percentage score. The internal consistency, commonly presented as Cronbach’s α, was not performed for the total scale, because missing values are allowed in the hand skill items. Use of an invalid or internally inconsistent total scale might overestimate or underestimate interrater and test–retest agreement of the assessment. Moreover, the total percentage scores proposed in the study represented the children’s hand skills on specific activities rather than the composite scores for their overall hand skills. The study's reliability results would be limited in the different activities involved. A potential solution to these limitations is to use the Many-Faceted Rasch model (Linacre, 1994) or other item-response theory models. These modeling analyses could generate children’s composite hand skill scores by considering all potential facets as well as providing statistics to examine the unidimensionality and item internal reliability of the ACHS. In addition, they allow for the determination of whether the 6-point rating scale of the ACHS is used in an appropriate manner. Evidence of the measurement properties of the ACHS performed by these models is warranted to assist appropriate interpretation in future reliability studies. Work to establish the construct validity of the ACHS is under way.

Acknowledgments

We thank all the content expert reviewers for their contributions in the development of the ACHS and acknowledge support from the Department of Education and Early Childhood Development, Vanessa Carson’s editing assistance, and Bronwyn Copping’s involvement in data collection. Chi-Wen Chien was supported by the Endeavour International Postgraduate Research Scholarship, Monash Postgraduate Scholarship, and Faculty Postgraduate Excellence Award from Monash University. Parts of this article were presented at the International Occupational Therapy Conference, November 14–17, 2008, Guangzhou, China.

References

Altman, D. G. (1991). Practical statistics for medical research. London: Chapman & Hall.

American Occupational Therapy Association. (2008). Occupational therapy practice framework: Domain and practice (2nd ed.). American Journal of Occupational Therapy, 62, 625–683. [Article] [PubMed]

Andresen, E. M. (2000). Criteria for assessing the tools of disability outcomes research. Archives of Physical Medicine and Rehabilitation, 81(Suppl. 2)S15–S20. doi: 10.1053/apmr.2000.20619 [Article] [PubMed]

Arnould, C., Penta, M., Renders, A., & Thonnard, J. L. (2004). ABILHAND–Kids: A measure of manual ability in children with cerebral palsy. Neurology, 63, 1045–1052. [Article] [PubMed]

Bernhardt, J., Matyas, T., & Bate, P. (2002). Does experience predict observational kinematic assessment accuracy?. Physiotherapy Theory and Practice, 18, 141–149. doi: 10.1080/09593980290058535 [Article]

Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1, 307–310. [Article] [PubMed]

Brennan, P., & Silman, A. (1992). Statistical methods for assessing observer variability in clinical measures. BMJ, 304, 1491–1494. doi: 10.1136/bmj.304.6840.1491 [Article] [PubMed]

Breslin, D. M., & Exner, C. E. (1999). Construct validity of the In-Hand Manipulation Test: A discriminant analysis with children without disability and children with spastic diplegia. American Journal of Occupational Therapy, 53, 381–386. [Article] [PubMed]

Brown, G. T., Rodger, S., Brown, A., & Roever, C. (2005). A comparison of Canadian and Australian paediatric occupational therapists. Occupational Therapy International, 12, 137–161. doi: 10.1002/oti.2 [Article] [PubMed]

Bruininks, R. H., & Bruininks, B. D. (2005). BOT-2: Bruininks-Oseretsky Test of Motor Proficiency (2nd ed.). Circle Pines, MN: AGS.

Case-Smith, J. (1996). Fine motor outcomes in preschool children who receive occupational therapy services. American Journal of Occupational Therapy, 50, 52–61. [Article] [PubMed]

Case-Smith, J., Heaphy, T., Marr, D., Galvin, B., Koch, V., Marcie, G. E., et al. (1998). Fine motor and functional performance outcomes in preschool children. American Journal of Occupational Therapy, 52, 788–796. [Article]

Chien, C.-W., Brown, T., & McDonald, R. (2009). A framework of children’s hand skills for assessment and intervention. Child: Care, Health and Development, 35(6), 873–884. [Article] [PubMed]

Cipriany-Dacko, L. M., Innerst, D., Johannsen, J., & Rude, V. (1997). Interrater reliability of the Tinetti Balance Scores in novice and experienced physical therapy clinicians. Archives of Physical Medicine and Rehabilitation, 78, 1160–1164. doi: 10.1016/S0003-9993(97)90145-3 [Article] [PubMed]

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. doi: 10.1177/001316446002000104 [Article]

Coster, W. (1998). Occupation-centered assessment of children. American Journal of Occupational Therapy, 52, 337–344. [Article] [PubMed]

Cusick, A., Vasquez, M., Knowles, L., & Wallen, M. (2005). Effect of rater training on reliability of Melbourne Assessment of Unilateral Upper Limb Function scores. Developmental Medicine and Child Neurology, 47, 39–45. doi: 10.1017/S0012162205000071 [Article] [PubMed]

DeMatteo, C., Law, M., Russell, D., Pollock, N., Rosenbaum, P., & Walter, S. (1992). QUEST: Quality of Upper Extremity Skills Test manual. Hamilton, Ontario: Neurodevelopmental Research Unit, Chedoke Campus, Chedoke-McMaster Hospital.

Exner, C. E. (1993). Content validity of the In-Hand Manipulation Test. American Journal of Occupational Therapy, 47, 505–513. [Article] [PubMed]

Fisher, A. G. (1997). Assessment of Motor and Process Skills (2nd ed.). Fort Collins, CO: Three Star Press.

Folio, M. R., & Fewell, R. R. (2000). Peabody Developmental Motor Scales: Examiner’s manual (2nd ed.). Austin, TX: Pro-Ed.

Haley, S. M., Coster, W. J., Ludlow, L., Haltiwanger, J. T., & Andrellos, P. J. (1992). Pediatric Evaluation of Disability Inventory. Boston: New England Medical Center Hospital.

Henderson, A. (2006). Self-care and hand skill. In A.Henderson & C.Pehoski (Eds.), Hand function in the child: Foundations for remediation (2nd ed., pp. 193–216). Philadelphia: Mosby/Elsevier.

Henderson, A., & Pehoski, C. (Eds.). (2006). Hand function in the child: Foundations for remediation (2nd ed.). Philadelphia: Mosby/Elsevier.

Hocking, C. (2001). Implementing occupation-based assessment. American Journal of Occupational Therapy, 55, 463–469. [Article] [PubMed]

Hofferth, S. L., & Sandberg, J. F. (2001). How American children spend their time. Journal of Marriage and the Family, 63, 295–308. doi: 10.1111/j.1741-3737.2001.00295.x [Article]

Koch, S. C., Cruz, R. F., & Goodill, S. W. (2001). The Kestenberg Movement Profile: Performance of novice raters. American Journal of Dance Therapy, 23, 71–87. doi: 10.1023/A:1013089606167 [Article]

Krumlinde-Sundholm, L., & Eliasson, A. C. (2003). Development of the Assisting Hand Assessment: A Rasch-built measure intended for children with unilateral upper limb impairments. Scandinavian Journal of Occupational Therapy, 10, 16–26. doi: 10.1080/11038120310004529 [Article]

Law, M., Cooper, B. A., Strong, S., Stewart, D., Rigby, P., & Letts, L. (1996). The person–environment–occupation model: A transactive approach to occupational performance. Canadian Journal of Occupational Therapy, 63, 9–23. [Article]

Linacre, J. M. (1994). FACETS: Many-faceted Rasch measurement computer program. Chicago: MESA.

Marr, D., Cermak, S., Cohn, E. S., & Henderson, A. (2003). Fine motor activities in Head Start and kindergarten classrooms. American Journal of Occupational Therapy, 57, 550–557. [Article] [PubMed]

McHale, K., & Cermak, S. A. (1992). Fine motor activities in elementary school: Preliminary findings and provisional implications for children with fine motor problems. American Journal of Occupational Therapy, 46, 898–903. [Article] [PubMed]

Portney, L. G., & Watkins, M. P. (2000). Foundation of clinical research: Application to practice (2nd ed.). Upper Saddle River, NJ: Prentice Hall Health.

Randall, M., Carlin, J. B., Chondros, P., & Reddihough, D. (2001). Reliability of the Melbourne Assessment of Unilateral Upper Limb Function. Developmental Medicine and Child Neurology, 43, 761–767. doi: 10.1017/S0012162201001396 [Article] [PubMed]

Randall, M., Imms, C., & Carey, L. (2008). Establishing validity of a modified Melbourne Assessment for children ages 2 to 4 years. American Journal of Occupational Therapy, 62, 373–383. [Article] [PubMed]

Randall, M. J., Johnson, L. M., & Reddihough, D. S. (1999). The Melbourne Assessment of Unilateral Upper Limb Function. Melbourne, Victoria, Australia: Royal Children’s Hospital.

Robinson, J. P., & Bianchi, S. (1997). The children’s hours. American Demographics, 19, 20–23.

Rogers, J., & Case-Smith, J. (2002). Relationships between handwriting and keyboarding performance of sixth-grade students. American Journal of Occupational Therapy, 56, 34–39. [Article] [PubMed]

Sakzewski, L., Ziviani, J., & Van Eldik, N. (2001). Test/retest reliability and inter-rater agreement of the Quality of Upper Extremities Skills Test (QUEST) for older children with acquired brain injuries. Physical and Occupational Therapy in Pediatrics, 21, 59–67. [PubMed]

Sim, J., & Wright, C. C. (2005). The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy, 85, 257–268. [PubMed]

Sorsdahl, A. B., Moe-Nilssen, R., & Strand, L. I. (2008). Observer reliability of the Gross Motor Performance Measure and the Quality of Upper Extremity Skills Test, based on video recordings. Developmental Medicine and Child Neurology, 50, 146–151. doi: 10.1111/j.1469-8749.2007.02023.x [Article] [PubMed]

Young, N. L., Williams, J. I., Yoshida, K. K., & Wright, J. G. (2000). Measurement properties of the Activities Scale for Kids. Journal of Clinical Epidemiology, 53, 125–137. doi: 10.1016/S0895-4356(99)00113-4 [Article] [PubMed]

Figure 1.

Note. The two bold dashed lines define the limits of agreement (mean difference ± 1.96 standard deviation).

Full Size | Slide (.ppt)

Table 1.

Demographic Characteristics of the Participating Children in the Reliability Study

Demographic Variables	Interrater Reliability Study (N = 30)	Test–Retest Reliability Study (N = 44)
Gender, n (%)
Boys	21 (70.0)	29 (65.9)
Girls	9 (30.0)	15 (34.1)
Average age in months, mean ± SD	64.3 ± 23.2	70.0 ± 26.6
Age in years, n (%)
2–4	18 (60.0)	23 (52.3)
5–7	7 (23.3)	13 (29.6)
8–10	5 (16.7)	6 (13.6)
11–12	—	2 (4.5)
Diagnosis, n (%)
No disability	18 (60.0)	24 (54.5)
Development delay	3 (10.0)	6 (13.6)
Autism	6 (20.0)	9 (20.5)
Down syndrome	3 (10.0)	4 (9.1)
Brachial plexus birth palsy	—	1 (2.3)
Education of the parents, n (%)
Primary school graduate	1 (3.3)	2 (4.5)
High school graduate	13 (43.3)	16 (36.4)
College/TAFE diploma	8 (26.7)	16 (36.4)
Undergraduate degree	3 (10.0)	4 (9.1)
Postgraduate degree	4 (13.3)	4 (9.1)
Unreported	1 (3.3)	2 (4.5)

Note. — = no child in the variable; TAFE = Technical and Further Education.

View Large

Table 1.

Demographic Characteristics of the Participating Children in the Reliability Study

Demographic Variables	Interrater Reliability Study (N = 30)	Test–Retest Reliability Study (N = 44)
Gender, n (%)
Boys	21 (70.0)	29 (65.9)
Girls	9 (30.0)	15 (34.1)
Average age in months, mean ± SD	64.3 ± 23.2	70.0 ± 26.6
Age in years, n (%)
2–4	18 (60.0)	23 (52.3)
5–7	7 (23.3)	13 (29.6)
8–10	5 (16.7)	6 (13.6)
11–12	—	2 (4.5)
Diagnosis, n (%)
No disability	18 (60.0)	24 (54.5)
Development delay	3 (10.0)	6 (13.6)
Autism	6 (20.0)	9 (20.5)
Down syndrome	3 (10.0)	4 (9.1)
Brachial plexus birth palsy	—	1 (2.3)
Education of the parents, n (%)
Primary school graduate	1 (3.3)	2 (4.5)
High school graduate	13 (43.3)	16 (36.4)
College/TAFE diploma	8 (26.7)	16 (36.4)
Undergraduate degree	3 (10.0)	4 (9.1)
Postgraduate degree	4 (13.3)	4 (9.1)
Unreported	1 (3.3)	2 (4.5)

Note. — = no child in the variable; TAFE = Technical and Further Education.

Table 2.

Frequency of the Activities Observed in the Reliability Study

Activity Categories	Interrater Reliability Study (N = 69 assessments)	Test–Retest Reliability Study (N = 74 assessments)
Leisure and Play domain, n (%)	20 (29.0)	26 (35.1)
Construction (blocks)	1	1
Puzzle	4	5
Stringing beads	3	1
Catching and throwing objects	7	14
Card game	2	—
Play dough/clay	3	4
Folding paper	—	—
Handling money	—	1
School-Related Work domain, n (%)	28 (40.6)	22 (29.8)
Reading book	2	—
Drawing or coloring	10	9
Writing & copying	1	5
Cutting	10	4
Pasting	3	1
Using computer	1	3
School tool use (ruler)	—	—
Putting on backpack	1	—
Activities of Daily Living domain, n (%)	21 (30.4)	26 (35.1)
Drinking	2	6
Eating	6	10
Dressing upper body	6	—
Putting on socks and shoes	1	3
Washing hands	6	6
Brushing teeth	—	1

Note. — = no assessment in the activity category.

View Large

Table 2.

Frequency of the Activities Observed in the Reliability Study

Activity Categories	Interrater Reliability Study (N = 69 assessments)	Test–Retest Reliability Study (N = 74 assessments)
Leisure and Play domain, n (%)	20 (29.0)	26 (35.1)
Construction (blocks)	1	1
Puzzle	4	5
Stringing beads	3	1
Catching and throwing objects	7	14
Card game	2	—
Play dough/clay	3	4
Folding paper	—	—
Handling money	—	1
School-Related Work domain, n (%)	28 (40.6)	22 (29.8)
Reading book	2	—
Drawing or coloring	10	9
Writing & copying	1	5
Cutting	10	4
Pasting	3	1
Using computer	1	3
School tool use (ruler)	—	—
Putting on backpack	1	—
Activities of Daily Living domain, n (%)	21 (30.4)	26 (35.1)
Drinking	2	6
Eating	6	10
Dressing upper body	6	—
Putting on socks and shoes	1	3
Washing hands	6	6
Brushing teeth	—	1

Note. — = no assessment in the activity category.

Table 3.

Interrater Reliability Between the Two Raters on Observing the Occurrence of the Hand Skill Items

Items	No. of Reported Observations, Rater A (Rater B)	% Agreement	κ^a
01: Manual gesture	21 (14)	0.84	0.59
02: Body contact hand skills^b	14 (14)	0.77	0.28
03: Grasping	69 (67)	0.97	1.00
04: Holding	67 (67)	1.00	1.00
05: In-hand manipulating^b	49 (35)	0.57	0.13
06: Releasing	69 (69)	1.00	1.00
07: Isolated finger movement	69 (68)	0.99	1.00
08: Reaching	69 (69)	1.00	1.00
09: Turning	69 (68)	0.99	1.00
10: Carrying	66 (67)	0.96	0.38
11: Throwing	7 (6)	0.99	0.92
12: Catching	7 (7)	1.00	1.00
13: Moving	62 (68)	0.88	0.03
14: Stabilizing	58 (59)	0.87	0.50
15: Transferring^b	51 (28)	0.61	0.28
16: Using both hands simultaneously	43 (37)	0.77	0.53
17: Using both hands cooperatively	58 (64)	0.91	0.58
18: Accuracy	69 (69)	1.00	1.00
19: Pace	69 (69)	1.00	1.00
20: Movement quality	69 (69)	1.00	1.00

aκ statistics were used to examine the agreement between the two raters on whether the hand skill items were observed. Underlined values indicate poor agreement (weighted κ < 0.4).

bItems with substantially poor interrater agreement.

View Large

Table 3.

Interrater Reliability Between the Two Raters on Observing the Occurrence of the Hand Skill Items

Items	No. of Reported Observations, Rater A (Rater B)	% Agreement	κ^a
01: Manual gesture	21 (14)	0.84	0.59
02: Body contact hand skills^b	14 (14)	0.77	0.28
03: Grasping	69 (67)	0.97	1.00
04: Holding	67 (67)	1.00	1.00
05: In-hand manipulating^b	49 (35)	0.57	0.13
06: Releasing	69 (69)	1.00	1.00
07: Isolated finger movement	69 (68)	0.99	1.00
08: Reaching	69 (69)	1.00	1.00
09: Turning	69 (68)	0.99	1.00
10: Carrying	66 (67)	0.96	0.38
11: Throwing	7 (6)	0.99	0.92
12: Catching	7 (7)	1.00	1.00
13: Moving	62 (68)	0.88	0.03
14: Stabilizing	58 (59)	0.87	0.50
15: Transferring^b	51 (28)	0.61	0.28
16: Using both hands simultaneously	43 (37)	0.77	0.53
17: Using both hands cooperatively	58 (64)	0.91	0.58
18: Accuracy	69 (69)	1.00	1.00
19: Pace	69 (69)	1.00	1.00
20: Movement quality	69 (69)	1.00	1.00

aκ statistics were used to examine the agreement between the two raters on whether the hand skill items were observed. Underlined values indicate poor agreement (weighted κ < 0.4).

bItems with substantially poor interrater agreement.

Table 4.

Interrater Reliability of the Observed Assessments in the Hand Skill Items

Items	N	Rater A, Mean (SD)	Rater B, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	12	4.4 (1.0)	4.9 (0.3)	0.33	.06
02: Body contact hand skills	6	4.5 (0.6)	4.8 (0.4)	0.33	.16
03: Grasping	67	4.3 (0.7)	4.1 (0.9)	0.49	.05
04: Holding	67	4.6 (0.5)	4.3 (0.8)	0.37	.02
05: In-hand manipulating	27	3.6 (0.7)	4.2 (0.7)	0.45	.01
06: Releasing	69	4.4 (0.6)	4.4 (0.7)	0.44	.72
07: Isolated finger movement	68	4.3 (0.7)	4.4 (0.7)	0.51	.37
08: Reaching	69	4.4 (0.6)	4.5 (0.7)	0.68	.25
09: Turning	68	4.2 (0.7)	4.2 (0.7)	0.57	.27
10: Carrying	65	4.3 (0.6)	4.5 (0.7)	0.51	.03
11: Throwing	6	4.3 (0.5)	4.2 (0.4)	0.09	.56
12: Catching	7	2.6 (1.3)	3.6 (0.8)	0.07	.10
13: Moving	61	3.9 (0.7)	4.2 (0.8)	0.37	.01
14: Stabilizing	54	4.1 (0.9)	4.5 (0.7)	0.52	<.01
15: Transferring	26	4.1 (0.6)	4.3 (0.5)	0.27	.25
16: Using both hands simultaneously	32	4.1 (0.8)	4.2 (0.9)	0.64	.32
17: Using both hands cooperatively	58	3.7 (0.9)	4.1 (0.8)	0.46	.01
18: Accuracy	69	3.9 (0.8)	3.9 (1.0)	0.51	.49
19: Pace	69	3.8 (0.8)	3.8 (0.8)	0.41	1.00
20: Movement quality	69	3.6 (0.7)	4.1 (0.8)	0.33	<.01

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests or poor agreement in weighted κ statistics (value < 0.40). SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test\x{2013}retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference between the raters’ reported ratings of the hand skill items between test\x{2013}retest observations.

View Large

Table 4.

Interrater Reliability of the Observed Assessments in the Hand Skill Items

Items	N	Rater A, Mean (SD)	Rater B, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	12	4.4 (1.0)	4.9 (0.3)	0.33	.06
02: Body contact hand skills	6	4.5 (0.6)	4.8 (0.4)	0.33	.16
03: Grasping	67	4.3 (0.7)	4.1 (0.9)	0.49	.05
04: Holding	67	4.6 (0.5)	4.3 (0.8)	0.37	.02
05: In-hand manipulating	27	3.6 (0.7)	4.2 (0.7)	0.45	.01
06: Releasing	69	4.4 (0.6)	4.4 (0.7)	0.44	.72
07: Isolated finger movement	68	4.3 (0.7)	4.4 (0.7)	0.51	.37
08: Reaching	69	4.4 (0.6)	4.5 (0.7)	0.68	.25
09: Turning	68	4.2 (0.7)	4.2 (0.7)	0.57	.27
10: Carrying	65	4.3 (0.6)	4.5 (0.7)	0.51	.03
11: Throwing	6	4.3 (0.5)	4.2 (0.4)	0.09	.56
12: Catching	7	2.6 (1.3)	3.6 (0.8)	0.07	.10
13: Moving	61	3.9 (0.7)	4.2 (0.8)	0.37	.01
14: Stabilizing	54	4.1 (0.9)	4.5 (0.7)	0.52	<.01
15: Transferring	26	4.1 (0.6)	4.3 (0.5)	0.27	.25
16: Using both hands simultaneously	32	4.1 (0.8)	4.2 (0.9)	0.64	.32
17: Using both hands cooperatively	58	3.7 (0.9)	4.1 (0.8)	0.46	.01
18: Accuracy	69	3.9 (0.8)	3.9 (1.0)	0.51	.49
19: Pace	69	3.8 (0.8)	3.8 (0.8)	0.41	1.00
20: Movement quality	69	3.6 (0.7)	4.1 (0.8)	0.33	<.01

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests or poor agreement in weighted κ statistics (value < 0.40). SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test\x{2013}retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference between the raters’ reported ratings of the hand skill items between test\x{2013}retest observations.

Table 5.

Test–Retest Reliability of the Observed Assessments in the Hand Skill Items

Items	N	First Evaluation, Mean (SD)	Second Evaluation, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	8	3.8 (1.0)	3.8 (1.3)	0.79	1.00
02: Body contact hand skills	6	4.8 (0.4)	4.7 (0.5)	0.57	.32
03: Grasping	74	4.4 (0.6)	4.4 (0.6)	0.56	.84
04: Holding	69	4.7 (0.6)	4.8 (0.6)	0.42	.84
05: In-hand manipulating	35	3.9 (1.1)	3.9 (1.0)	0.79	.80
06: Releasing	74	4.7 (0.6)	4.7 (0.6)	0.78	1.00
07: Isolated finger movement	74	4.5 (0.7)	4.5 (0.7)	0.68	.67
08: Reaching	74	4.7 (0.6)	4.6 (0.6)	0.59	.49
09: Turning	73	4.4 (0.7)	4.3 (0.6)	0.60	.22
10: Carrying	70	4.5 (0.6)	4.5 (0.5)	0.60	1.00
11: Throwing	14	4.0 (0.4)	3.7 (0.6)	0.50	.05
12: Catching	11	2.9 (0.8)	3.1 (1.0)	0.59	.48
13: Moving	50	3.9 (0.9)	4.0 (0.8)	0.54	.69
14: Stabilizing	54	4.3 (0.8)	4.2 (0.7)	0.61	.22
15: Transferring	43	4.5 (0.7)	4.4 (0.6)	0.70	.06
16: Using both hands simultaneously	34	4.3 (0.7)	4.4 (0.7)	0.43	.48
17: Using both hands cooperatively	53	4.0 (0.9)	4.0 (0.7)	0.61	.70
18: Accuracy	74	4.1 (0.8)	4.3 (0.7)	0.65	.03
19: Pace	74	4.3 (0.7)	4.4 (0.7)	0.56	.27
20: Movement quality	74	3.7 (0.8)	3.8 (0.7)	0.55	.18

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests. SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test–retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference in the rater's reported ratings of the hand skill items between test–retest observations.

View Large

Table 5.

Test–Retest Reliability of the Observed Assessments in the Hand Skill Items

Items	N	First Evaluation, Mean (SD)	Second Evaluation, Mean (SD)	Weighted κ^a	p^b
01: Manual gesture	8	3.8 (1.0)	3.8 (1.3)	0.79	1.00
02: Body contact hand skills	6	4.8 (0.4)	4.7 (0.5)	0.57	.32
03: Grasping	74	4.4 (0.6)	4.4 (0.6)	0.56	.84
04: Holding	69	4.7 (0.6)	4.8 (0.6)	0.42	.84
05: In-hand manipulating	35	3.9 (1.1)	3.9 (1.0)	0.79	.80
06: Releasing	74	4.7 (0.6)	4.7 (0.6)	0.78	1.00
07: Isolated finger movement	74	4.5 (0.7)	4.5 (0.7)	0.68	.67
08: Reaching	74	4.7 (0.6)	4.6 (0.6)	0.59	.49
09: Turning	73	4.4 (0.7)	4.3 (0.6)	0.60	.22
10: Carrying	70	4.5 (0.6)	4.5 (0.5)	0.60	1.00
11: Throwing	14	4.0 (0.4)	3.7 (0.6)	0.50	.05
12: Catching	11	2.9 (0.8)	3.1 (1.0)	0.59	.48
13: Moving	50	3.9 (0.9)	4.0 (0.8)	0.54	.69
14: Stabilizing	54	4.3 (0.8)	4.2 (0.7)	0.61	.22
15: Transferring	43	4.5 (0.7)	4.4 (0.6)	0.70	.06
16: Using both hands simultaneously	34	4.3 (0.7)	4.4 (0.7)	0.43	.48
17: Using both hands cooperatively	53	4.0 (0.9)	4.0 (0.7)	0.61	.70
18: Accuracy	74	4.1 (0.8)	4.3 (0.7)	0.65	.03
19: Pace	74	4.3 (0.7)	4.4 (0.7)	0.56	.27
20: Movement quality	74	3.7 (0.8)	3.8 (0.7)	0.55	.18

Note.Underlined values indicate significant difference (p < .05) in the Wilcoxon signed-rank tests. SD = standard deviation.

aWeighted κ statistics with quadratic weights were used to examine the agreement of the hand skill items between test–retest observations.

bWilcoxon signed-rank tests were used to examine the significance of difference in the rater's reported ratings of the hand skill items between test–retest observations.