Every day, health care professionals conduct evaluations and make recommendations regarding the most appropriate discharge plans and living environments for their clients. The overarching goal in this process is to ensure that clients can safely and efficiently manage, either independently or with the assistance of another, all the daily life tasks that are required for independent living (e.g., self-care, home maintenance, money management, medication adherence, leisure, social interaction, community transportation; e.g., Wehman, 2001).

In their exploration of this complex evaluation process, Meinow, Kåreholt, and Lagergren (2005)  found that the strongest predictors of the amount of assistance received in the home were dependency in personal activities of daily living (PADLs; R² = .34, p < .001), dependency in instrumental activities of daily living (IADLs; R² = .39, p < .001), and cognitive function (R² = .13, p < .001). Because PADL ability and IADL ability (hereinafter jointly referred to as activity of daily living [ADL] ability) are two of the key predictors of the need for assistance in the community, occupational therapists play a vital role in this evaluation process. If occupational therapists are to contribute to the decision-making process from an evidence-informed perspective, additional research must be conducted to document the validity and accuracy of using ADL measures to predict the need for assistance in the community.

The Assessment of Motor and Process Skills (AMPS; Fisher, 2006a, 2006b) is an internationally recognized, occupational therapy–specific assessment of the quality of a client’s ADL performance. The linear ADL process ability scale of the AMPS has shown promise when used to identify clients who need assistance to live in the community (Bernspång & Fisher, 1995; Fisher, 2006a; Hartman, Fisher, & Duran, 1999; Kottorp, 2008; Merritt, 2010); however, additional research is needed to determine the sensitivity and specificity of the AMPS ADL cutoff measures. Sensitivity is an estimate of the true positive rate and within this context represents the proportion of the independent sample correctly identified as independent. In contrast, specificity is an estimate of the true negative rate, or the proportion of the sample in need of assistance that is correctly identified as needing assistance. Identifying the sensitivity and specificity estimates that arise when using AMPS ADL ability measures to predict the need for assistance could provide occupational therapists with additional evidence to support their clinical judgments. For example, this evidence could potentially be used to acquire support from funding agencies to pay for a client’s care, establish the need for additional health care services to maximize a client’s independence and lessen caregiver burden, and document the efficacy of occupational therapy services and the impact they have had on the client’s need for assistance and burden of care.

To address this gap in the literature, this study addressed three issues: (1) whether significant association exists between ADL ability (motor ability, process ability, or both) and global functional level, (2) the sensitivity and specificity estimates that arise when using the current AMPS ADL motor and ADL process cutoff measures to differentiate clients who are independent in the community from those who are in need of assistance, and (3) the need to create new cutoff points that reflect more acceptable estimates of sensitivity and specificity (i.e., lower rates of error).

I conducted a retrospective, criterion-based validation study of the AMPS ADL motor and ADL process ability cutoff measures. Per standard protocol, before beginning this research project, I obtained approval from the Colorado State University regulatory compliance office.

I used existing data from the AMPS Project International database in Fort Collins, Colorado. The data were from people who were assessed by occupational therapy practitioners within typical occupational therapy settings (e.g., hospitals, client homes, rehabilitation clinics). The sample potentially included all available data records in the AMPS database that were evaluated between January 1, 1999, and January 1, 2005, and met the following criteria: (1) were age 16 yr or older; (2) were not associated with rater scoring error, as evidenced by artificially high ADL motor or ADL process ability measures; (3) were not scored by multiple raters as part of rater calibration; and (4) had known sex, age, and global functional level ratings. To clarify, only data from 1999 to 2005 were selected, because new criteria for rating each global functional level were created in 1999 and remained consistent until the data were extracted in 2006 (Fisher, 1999). Before 1999, specific criteria for rating functional level had not been developed.

For this study, 64,466 data records (58% of the potential sample in the AMPS database) met the criteria for inclusion. Data records eliminated from consideration were those of people younger than age 16 (23% of the potential sample), were associated with rater scoring error (10% of the potential sample), or had incomplete demographic data (9% of the potential sample). Data records included those for men (42.9%) and women (57.1%) from North America (14.8%), the United Kingdom (22.3%), the Nordic countries (34.2%), other European countries (11.6%), New Zealand and Australia (8.7%), Asia (8.0%), and unknown regions (0.4%). Data records were grouped into global diagnostic categories (e.g., the psychiatric diagnostic category included people with bipolar disorder, depression, schizophrenia, or personality disorder; see Table 1). Although specific diagnostic information could be lost by coding the data in this manner, it ensured that sample sizes in each diagnostic group were sufficient for statistical analyses.

The AMPS was administered by occupational therapists who attended a 5-day AMPS training workshop and subsequently were calibrated as AMPS raters (e.g., demonstrated valid and reliable administration and scoring of the AMPS). After observing the client perform at least two AMPS tasks, each rater used the criteria in the AMPS manual (Fisher, 2006b) to score the 16 ADL motor skill and 20 ADL process skill items. Globally, ADL motor skills are defined as the observable actions the person enacts when moving the self or task objects (Fisher, 2006a). ADL process skills are defined as “the observable actions of performance the person enacts to logically sequence the actions of the ADL task performance over time, select and use appropriate tools and materials, and adapt his or her performance when problems are encountered” (Fisher, 2006a, p. 4). According to the criteria in the AMPS manual, each ADL skill item was scored using a 4-point ordinal rating scale based on the global criterion of competence (1 = deficient performance and/or task breakdown, 2 = ineffective performance, 3 = questionable performance, 4 = competent performance). Each participant had at least two sets of ADL motor scores and two sets of ADL process scores: one set of scores for each task performed (i.e., two sets of scores were used to generate one ADL motor ability measure and one ADL process ability measure for each person).

Previous studies have supported using the AMPS cutoff measures as indicators of the need for assistance (Bernspång & Fisher, 1995; Hartman et al., 1999). Bernspång and Fisher (1995)  developed a “risk” zone, delineated by ADL measures that fall within ±0.3 logit of the ADL motor cutoff measure of 2.0 logits or ADL process cutoff measure of 1.0 logit. Their results indicate that approximately two-thirds of the clients who were frail or at risk of functional decline had ADL ability measures within the risk zones. With a sample of older adults (with and without dementia), Hartman et al. (1999)  documented a 94% overall correct classification rate when using the AMPS ADL process cutoff measure as an indicator of the need for assistance. Previous studies have also supported the reliability and validity of the AMPS ability measures across age groups (Hayase et al., 2004), between men and women (Merritt & Fisher, 2003), and with a variety of diagnoses (e.g., Doble, Fisk, Fisher, Ritvo, & Murray, 1994; Hartman et al., 1999; Rexroth, Fisher, Merritt, & Gliner, 2005). Last, the AMPS has also been shown to be a sensitive outcome measure (e.g., Chard, Liu, & Mulholland, 2009; Goverover, Johnston, Toglia, & Deluca, 2007; Kinnman, Andersson, Wetterquist, Kinnman, & Andersson, 2000; Oakley, Khin, Parks, Bauer, & Sunderland, 2002). Research has demonstrated that the characteristics of the test function independently of the person being tested (e.g., the item and task challenge calibrations do not vary between samples; Fisher, 2006b); thus, the linear ADL ability measures of men and women from different world regions and with different diagnoses can be validly generated and compared.

Considering the evidence that clinical judgment appears to be one of the most accurate means of determining global functioning within the community (Pinholt et al., 1987; Rogers et al., 2003), that direct observation of ADL performance tends to lead to more accurate estimates of global functioning within the community (Rogers et al., 2003), and that AMPS ADL ability measures and AMPS global functional level ratings are concurrently documented, I concluded that a practical and valid external criterion available for evaluating the accuracy of the AMPS cutoff measures was the AMPS global functional-level rating. Global functional level was rated by trained and calibrated AMPS raters according to the specific criteria outlined in the AMPS manual (Fisher, 2006b; see Table 2). More specifically, each occupational therapist used his or her clinical judgment to determine whether the client was able to live independently, in need of minimal assistance or supervision, or in need of moderate to maximal assistance to live in the community.

The AMPS global functional level was not based solely on the AMPS task observations; rather, the rating was based on all information the therapist gathered about the client. As a result, the therapist used multiple sources of information to determine the most accurate rating (e.g., interview, caregiver report, reports from other health care providers, observation of ADL performance, assessment of body functions). In support, other researchers have reported that using multiple sources of information, including direct observation of ADL ability, is the most accurate means of determining global functioning in the community (Jette, Grover, & Keck, 2003; Rogers et al., 2003). To minimize the risk of biased ratings, at the time of rating the client’s AMPS global functional level the raters were blind to the person’s final AMPS ability measures and unaware that the functional level ratings would be used to investigate the validity of the AMPS ability measures.

A specialized, many-facet Rasch analysis program, Facets (Linacre, 2006), was used to convert the raw ordinal ADL skill item scores into linear ADL motor and ADL process ability measures. Such analyses adjust the final ADL ability estimates to account for task challenge, skill item difficulty, and severity of the rater. Demographic information and ADL motor and ADL process ability measures were imported into SPSS Version 17.0 for Windows (SPSS, Inc., Chicago).

To evaluate the association between ADL ability measures and global functional level, two one-way analyses of variance (ANOVAs) were conducted (one for ADL motor ability and one for ADL process ability), and the size of the effect for each analysis was determined. Effect size (η²) was interpreted on the basis of Cohen’s measure of association (Cohen, 1988), in which .01, .06, and .14 indicate small, medium, or large effects, respectively.

To prepare the data for receiver operating characteristic (ROC) curve analysis, AMPS global functional level ratings were recoded into a dichotomous variable denoting independent (n = 10,458) or in need of assistance to live in the community (n = 54,008). Next, the data were prepared to conduct a split-sample validation of the predictive model (Kohavi, 1995). Therefore, approximately 60% of the sample (n = 38,540) was randomly selected using the SPSS random sampling function; this subsample became the test sample and was used to generate ROC curves and subsequently develop the predictive model. The remaining data records in the sample (n = 25,926) became the validation sample, which was used to validate the predictive model. The test and validation samples had the same proportion of data records in each functional level group as did the entire sample; 16% of data records in each sample were independent, 36% were in need of minimal assistance, and 47% were in need of moderate to maximal assistance.

Using the test sample (n = 38,540), the dichotomous state variable (need for assistance) was used to generate two ROC curves, one with ADL motor ability as the test variable and one with ADL process ability as the test variable. Normal distribution and homogeneity of variance were not assumed, and thus a nonparametric model was used to generate the ROC curves (Brown & Davis, 2006).

The area under the curve (AUC) was examined to globally determine whether the ADL motor and ADL process ability measures had merit in correctly categorizing clients who were independent from those who needed assistance to live in the community. Rough guidelines for determining the discriminating value of a test by examining the AUC values are as follows: fail = .50–.60, poor = .60–.70, fair = .70–.80, good = .80–.90, and excellent = .90–1.00 (Perneczky et al., 2006).

After verification that the ADL motor and ADL process scales had merit in predicting independence in the community (i.e., AUC ≥ .70), the data were examined more closely. First, the ROC curves were examined to determine the sensitivity and specificity estimates of the current ADL motor measure of 2.00 logits and ADL process ability cutoff measure of 1.00 logit. Next, the ROC curves were examined to determine whether a need existed to create new cutoff measures to reflect more accurate estimates of independence or need for assistance to live in the community.

With regard to identifying community independence, I found no specific recommendations for desired levels of sensitivity and specificity in the literature. Potential cutoff measures were thus determined by examining the ROC curve at the clinically relevant area of the curve, defined as the point on the curve where false positive rates (i.e., test incorrectly indicates that the person is able to live independently) are minimized without resulting in large false negative rates (i.e., test result incorrectly indicates that the person needs assistance in the community). In support of the decision to minimize false positive errors, Dijkstra, Tiesinga, Plantinga, Veltman, and Dassen (2005)  also chose cutoff values that minimized errors that could lead to clients not receiving the care, treatment, or support that they actually needed.

Once the optimal cutoff measures for ADL motor and ADL process ability were determined, the validation sample (n = 25,926) was used to create a predicted need for assistance variable that was based on each person’s ADL motor ability; people under the cutoff were coded as needing assistance, whereas those above the cutoff were coded as being independent in the community. In a similar manner, a second predicted need for assistance variable was created on the basis of the ADL process cutoff measure. Congruence between predicted need for assistance and the clinician’s judgment of the need for assistance (i.e., AMPS global functional level) was then analyzed to validate the predictive model. To determine whether categorical accuracy could be improved on, the accuracy achieved when decision points matched (i.e., when both ADL motor and ADL process abilities were either above or below the cutoff measures) was investigated.

Finally, from among all of the data records (n = 64,466), separate ROC curves for samples within each diagnostic category were generated using the occupational therapist’s original rating of need for assistance as the state variable and ADL motor ability or ADL process ability as the test variable. For this analysis, the AUC was examined to globally determine the relative predictive validity of the AMPS ability measures across the different diagnostic categories. Data were drawn from the entire data set to maximize the sample size of each diagnostic grouping.

Global functional level was found to be significantly associated with ADL motor ability, F(2, 64463) = 8,728, p < .01, and ADL process ability, F(2, 64463) = 15,916, p < .01. According to Cohen’s (1988)  criteria, ADL motor and ADL process ability measures have large effects with regard to global functional level; η² values were .213 and .331, respectively. Although age was also found to be significantly associated with functional level, F(2, 64463) = 526, p < .01, the effect was small (η² = .016).

The ROC curve for examining the utility of using ADL motor ability measures to determine community independence revealed an AUC estimate of .78, indicating fair discriminating value. Examination of the ROC curve revealed that the sensitivity (i.e., rate of those correctly identified as being independent within the community) of the current ADL motor cutoff measure of 2.00 logits was .40, and the specificity (i.e., rate of those correctly identified as needing assistance to live in the community) was .87. On further examination of the clinically relevant area on the ROC curve, an ideal cutoff measure for ADL motor ability was determined to be 1.50 logits, for which sensitivity and specificity estimates were .67 and .72, respectively. Using the validation sample, the accuracy estimates were validated; when the predicted need for assistance was compared with the clinician’s judgment of the need for assistance, the new ADL motor cutoff of 1.5 logits resulted in a sensitivity of .68 and a specificity of .72, thus validating the ROC curve estimates.

When ADL process ability measures were used to determine community independence, the AUC was .84, indicating good discriminating value. Examination of the ROC curve revealed that the sensitivity and specificity estimates of the current ADL process cutoff measure of 1.00 logits were .80 and .70, revealing an acceptable and ideal cutoff measure. Within the validation sample, when the predicted need for assistance was compared with the clinician’s judgment of the need for assistance, the ADL process cutoff of 1.0 logit resulted in a sensitivity rating of .81 and a specificity rating of .70, thus validating the ROC curve estimates.

To prepare the data to evaluate the accuracy of using matched ADL motor and ADL process decisions, only those participants in the validation sample (n = 25,926) with matched predicted ADL motor and ADL process decision points were selected (i.e., ADL motor and ADL process ability measures were either both above or both below the respective cutoff measures). Approximately 65% (n = 16,807) of the validation sample had matched predicted decisions, and the sensitivity and specificity estimates of this sample were .85 and .83, respectively. Thus, accuracy estimates were highest when matched ADL motor and ADL process decisions occurred.

Finally, separate ROC curves were generated for each diagnostic group. Because of invariant ratings of the need for assistance, the data coded with a diagnosis of either well or frail older adult were not analyzed. AUC estimates ranged from .68 to .85 (poor to good ratings) for ADL motor ability and from .72 to .92 (fair to excellent ratings) for ADL process ability (Table 3).

The analyses indicate that both ADL motor ability and ADL process ability have utility as indicators of the need for assistance to live in the community. At best, when the data records in the validation sample had matched ADL motor and ADL process decision points (n = 16,807), 85% of the independent sample and 83% of the sample in need of assistance were correctly categorized. Until now no study has investigated the use of matched ADL motor and ADL process decisions, and thus the findings have the potential to provide the clinical and research communities with a new means of examining the constellation of ability measures for the purpose of estimating the need for assistance to live in the community.

When decision points did not match, the most accurate predictions were obtained when using the ADL process ability cutoff measure of 1.0 logit; 80% of the independent sample and 70% of the sample in need of assistance were correctly classified. The ADL process scale of the AMPS continues to be more closely associated with global functioning in the community than the ADL motor scale, in line with previous studies (Fisher, 2006a; Hartman et al., 1999; Kottorp, 2008; Merritt, 2010).

Although the current ADL motor cutoff measure of 2.00 logits has not been used as an indicator of independence in the community, the data revealed that the current cutoff measure is too high for use in categorizing independence versus the need for assistance (i.e., 60% of the independent sample was incorrectly coded as needing assistance). As a result, a new ADL motor cutoff measure of 1.50 logits is proposed, resulting in more acceptable estimates of sensitivity and specificity. Although additional research is warranted, this study documents the first line of evidence that ADL motor ability has some promise for use as an indicator of community independence.

Global estimates of the accuracy of using AMPS ability measures to predict the need for assistance were evaluated across the various diagnostic groups (Table 3). The diagnosis-specific AUC estimates ranged from poor to good for ADL motor ability and from fair to excellent for ADL process ability. Although this global statistic does not provide conclusive evidence of an assessment’s utility and accuracy (Zweig & Campbell, 1993), the AUC statistics in Table 3 suggest that the ADL motor and ADL process ability scales are not equally accurate across all diagnostic categories. More specifically, this is the first line of evidence that suggests that the ADL motor ability scale may be more accurate than the ADL process ability scale for clients within specific diagnostic categories (e.g., musculoskeletal, medical), whereas the ADL process ability scale may be more accurate for clients with dementia, neurological developmental disabilities, and so forth.

Although the diagnosis-specific AUC statistics do not provide conclusive evidence, they do provide a foundation for comparing the accuracy estimates of other assessments. For example, Mausbach and colleagues (2008)  evaluated the accuracy of using either the University of California, San Diego, Performance-Based Skills Assessment (UPSA; Patterson, Goldman, McKibbin, Hughs, & Jeste, 2001) or the Dementia Rating Scale (DRS; Mattis, 1973) to predict residential independence of people with schizophrenia (n = 434). They determined that the AUC estimates were .74 for the UPSA (an evaluation of ADL ability) and .65 for the DRS (an evaluation of cognition). In contrast, the current findings indicate that with a sample of clients with psychiatric illness, the AUC estimate for the AMPS ADL process scale was .77 (Table 3). Additional studies are warranted to fully evaluate and compare the accuracy estimates of the AMPS cutoff measures across different diagnoses.

The results of this study are promising and indicate that the AMPS ability measures have merit when used to document the need for assistance to live in the community; however, the predictions are not 100% accurate. One explanation for this is the fact that the AMPS was not designed to measure global functioning within the community. The ability to live independently in the community requires the performance of tasks that extend beyond those included in the AMPS (e.g., participation in social tasks, community transportation, money management, phone use, computer use); thus, the AMPS ADL ability measures can be expected to explain some, but not all, of the variation in the construct of community independence.

Another area to explore is that of rater scoring error. For example, occupational therapists working with clients who have physical disabilities may overemphasize observation and scoring of ADL motor skill deficits relative to ADL process skill deficits, resulting in inflated ADL process ability measures and higher false positive errors. Likewise, rater scoring error can occur when occupational therapists working with clients who have cognitive or psychiatric deficits focus their observations and scoring on ADL process skills while failing to observe and accurately score ADL motor skills, resulting in inflated ADL motor ability measures and higher false positive errors. In such instances, rater error is introduced; rater severity should not change on the basis of the characteristics of the clients being assessed.

Another potential source of error within the current study is the accuracy of the occupational therapists’ ratings of global functional level. The possibility exists that clinicians are more accurate when determining the global functional levels of those who are on the extreme ends of the scale (e.g., clients who are clearly independent in the community and those who require substantial assistance to live in the community). When clients require some assistance to live in the community or have inconsistent needs for assistance, the possibility exists that clinicians’ judgments of global functional level are inaccurate. Additional research is warranted to verify this assumption and to further investigate the potential causes of error in determining global functional level.

Although the lack of evidence supporting the validity and reliability of the AMPS global functional ratings constitutes a limitation of this study, use of this rating scale created an economical opportunity to use a large existing database to generate initial evidence of the merit of using AMPS ADL ability measures to determine a person’s need for assistance in the community. Clearly, follow-up studies are warranted that prospectively gather ADL measures or AMPS global functional ratings and measure them against additional criteria (e.g., narrative reports from caregivers, overall time that assistance was provided over the course of a week, daily assistance logs). Such studies could seek to validate use of the AMPS cutoff measures as evidence of the need for assistance and could document the validity and reliability of the AMPS global functional level ratings.

Occupational therapists play a critical role in determining clients’ discharge needs and need for assistance in the community. This study provides evidence that occupational therapists can use the AMPS ability measures to support decisions regarding a client’s potential need for support or assistance in the community. The findings support using AMPS ADL motor and ADL process ability measures as evidence of a client’s need for assistance to live in the community, but it is important to stress that such decisions are rarely made on the basis of the results from one assessment. Although many occupational therapists use functional assessments to determine discharge needs, they also consider constructs beyond personal and domestic ADL performance, including the client’s wants and needs, the client’s life context, the severity and prognosis of the client’s impairment, and other professionals’ opinions (Dijkstra et al., 2005). Thus, AMPS ability measures should contribute to and support the clinician’s judgments and recommendations and should not serve as the sole piece of evidence when determining the need for assistance.

I thank Anne G. Fisher for her support and extensive expertise. Previous versions of this article were presented at the 2008 International Assessment of Motor and Process Skills Symposium: Measuring, Planning, and Implementing Occupation-Based Programs, Halifax, Nova Scotia, and the 2009 American Congress of Rehabilitation Medicine–American Society of Neurorehabilitation Joint Conference, Denver, Colorado.