Australian Occupational Therapy Driver Assessors’ Opinions on Improving On-Road Driver Assessment Procedures

Marilyn Di Stefano; Wendy Macdonald

doi:10.5014/ajot.64.2.325

Free

Research Article | March 2010

Australian Occupational Therapy Driver Assessors’ Opinions on Improving On-Road Driver Assessment Procedures

Marilyn Di Stefano; Wendy Macdonald

Author Affiliations

Article Information

American Journal of Occupational Therapy, March/April 2010, Vol. 64, 325-335. doi:10.5014/ajot.64.2.325

Abstract

OBJECTIVE. We collected practitioner opinions to improve the validity and reliability of the on-road driver assessment procedures used in Australia.

METHOD. We used focus groups to document the views of experienced occupational therapy driver assessors using a purposive sampling method.

RESULTS. Eight focus groups were conducted with 55 clinicians practicing in urban and rural regions. There was strong support for greater standardization of procedures for all tests. For drivers seeking unrestricted (open) licenses, use of standard routes with predetermined assessment points was important where practicable. Where use of a nonstandard route for this purpose was unavoidable, it was important to specify a minimum set of requirements related to route characteristics and assessment items.

CONCLUSION. Australian occupational therapy driver assessors support greater standardization of test routes and procedures to improve reliability and validity. However, the extent to which standardization can be achieved is limited by variable road traffic environments where assessments are conducted.

In industrially developed countries throughout the world, driver licensing authorities recognize the need for effective driver screening and assessment systems for drivers who may be functionally impaired, whether because of disability or for other reasons. Screening and assessment are viewed as particularly important in countries in which the population is aging and there is a high dependency on use of private vehicles for transportation to achieve community mobility (National Highway Traffic Safety Administration, 2001; Organisation for Economic Co-Operative Development, 2001; Road Safety Committee, Parliament of Victoria, 2003; Victorian State Government, 2004). More specifically, the value of evidence-based development and greater standardization has been widely acknowledged in such countries (Korner-Bitensky, Bitensky, Sofer, Man-Son-Hing, & Gelinas, 2006; Withaar, Brouwer, & Van Zomeren, 2000).

In Australia, on-road driver assessments for functionally impaired drivers fall into two main categories: an on-road assessment by a specialist license testing officer used by the licensing authority or a more comprehensive evaluation for drivers with more substantial impairments conducted by driver-trained occupational therapists (DTOTs) with specialist training (Fildes et al., 2000). The latter type of assessment is the focus of this article.

Licensing authorities vary in their policies concerning assessment of drivers with medical impairments and in whether the outcome of a formal, on-road assessment administered by a DTOT is acceptable in lieu of a licensing authority test. In jurisdictions in which the licensing authority has primary responsibility for on-road (re)testing, the DTOT might be more likely to focus his or her attention on the specific disability issues seen as likely to affect driving performance rather than on the need for universally applicable, standardized assessment procedures. In the Australian state of Victoria, standard driver assessment protocols for use by DTOTs were developed during the mid-1980s (Caust, 1988), and similar processes were undertaken in North America soon afterward (Korner-Bitensky, Sofer, Kaizer, Gelinas, & Talbot, 1994; Stav, Hunt, & Arbesman, 2006).

In Victoria, establishment of protocols was followed by development of a curriculum for postgraduate training of occupational therapists, and the specialist role of DTOTs was given legal status (Victorian State Government, 1988), such that DTOT driver assessments are accepted in lieu of testing conducted by the authority itself; more comprehensive DTOT competency standards were developed later (Schneider, 1998). In other Australian states, DTOT assessments do not have this legal status. The current study focused on on-road assessment and involved DTOTs in four states: Victoria, New South Wales, Queensland, and Tasmania.

A variety of procedures for conducting on-road assessments in both closed circuit and normal traffic environments have been described (Carr, Jackson, Madden, & Cohen, 1992; Ekelman, Mitchell, & O’Dell-Rossi, 2001; Fitten et al., 1995; Perryman & Fitten, 1996). Their adequacy for these applications warrants investigation. For example, some researchers have relied on entry-level license test procedures for use with experienced drivers, albeit occasionally with modified assessment criteria (Cushman, 1996; Snellgrove, 2005). More significantly in the present context, others have used assessors with particular expertise in the target driver population but have evaluated behavior only in terms of global pass-or-fail criteria for a small number of broadly defined behavioral categories, rather than adequately purposively sampling specific driving behaviors across a range of different test items (Akinwuntan et al., 2002; Klavora, Heslegrave, & Young, 2000; Korteling & Kaptein, 1996).

More recently, Korner-Bitensky et al. (2006) surveyed 114 North American clinicians; they found that although 78% indicated use of a standard driving route, only 24% applied a standardized scoring system, and fewer than <1% (2 participants) used standardized on-road tests. This is of some concern, particularly in jurisdictions where the assessment outcome directly influences the driver’s license status, because considerable research evidence indicates that reliable driver assessment requires adequate sampling and observations of prespecified aspects of driver behavior at predetermined locations along a standard test route, even when the assessors are highly experienced (see Macdonald, Bowland, & Triggs, 1992). Conversely, DTOT on-road assessments often have multiple goals, including diagnosis of impairment-related deficiencies in driving performance to develop a driving habilitation and rehabilitation program and to identify whether there is a need for vehicle modifications (Di Stefano & Macdonald, 2006a; Stav, 2004). On this basis, it can be argued that DTOTs need a more flexible approach to assessment so that they can achieve such goals not relevant in entry-level license testing.

In Australia, the on-road assessments of medically impaired drivers1 are usually conducted by a DTOT and a driving instructor in a vehicle with dual controls. In Victoria, testing for an open license (i.e., not geographically restricted) is conducted on a prespecified route with prespecified instructions to the driver at prespecified locations, all of which are detailed in the documentation used when recording on-road working notes. By contrast, testing for a geographically restricted license (local area driving only) is conducted on a nonstandard route between the driver’s home and common local destinations.

Study Goals and Research Questions

Our primary goal was to explore DTOTs’ views on a set of questions related to standardization compared with flexibility in their assessment procedures. Closely related to this goal, the study aimed to establish whether DTOTs themselves saw proposed changes to on-road assessment as desirable and compatible with existing professional competency standards for this procedure, because DTOTs are a key source of real-world evidence (Phillips et al., 2001). Also, the study was an important part of the ongoing consultative process with these professionals, particularly because their support would be essential to achieving successful implementation of any procedural changes.

The study is one of the intermediate steps in a wider program to develop and empirically test a more reliable and valid DTOT on-road assessment procedure for application in a variety of evaluation contexts. Earlier studies within the program had already identified the theoretical foundations and key requirements for achieving program goals, requirements for standardizing assessment procedures, and characteristics of drivers presenting for reassessments within a medical review system, along with a range of practical issues (Di Stefano & Macdonald, 1998, 2003, 2006b, 2007).

On the basis of these earlier studies and current practice guidelines, a set of questions was formulated for focus group discussions:

What do DTOTs see as the goals of their on-road assessment procedures?
Do DTOTs see a need for standardization of such procedures?
What factors do DTOTs see as limiting the real-world implementation of more standardized procedures?
What are DTOTs’ views on
- – The standard test items outlined in the Competency Standards for Occupational Therapy Driver Assessors, Victoria (OT-Australia Victoria, 1998; Schneider, 1998);
- – A specified set of on-road test features (current and proposed); and
- – The possibility and desirability of differentially weighting particular test items, as part of a revised test procedure.

Method

Study Design

As noted previously, this study is one of the intermediate steps in a wider research program that adopted a mixed-methods design. DTOTs’ views of driver assessment procedures were collected by means of focus groups, as reported here, and by means of questionnaires and semistructured personal interviews to enable data triangulation (Denzin, 1970). (Articles reporting results from the other studies are in preparation.) Focus groups permit documentation of the views of many people in a detailed and efficient manner (Stewart & Shamdasani, 1990) and are commonly used in both qualitative and quantitative research, although with somewhat different purposes. Within the framework of the current program, where they were used alongside questionnaires and semistructured interviews, their primary benefits were to gain consensus on some issues, to enable in-depth discussion of issues (both preidentified and others not previously documented; Morgan, 1993), and to enhance participant engagement with later stages of the research and its subsequent implementation (see Taylor & Keilhofner, 2006). Approval for the conduct of this study was obtained from La Trobe University Human Ethics Committee and from the national Occupational Therapy Association, OT-Australia.

Sampling

Purposive sampling was used in recruiting DTOTs to the study by means of professional and employer networks, state licensing authorities, special interest groups, advertising in professional publications, and word of mouth. This approach was appropriate because it was important to attract participants with relevant backgrounds (Minichiello, Sullivan, Greenwood, & Axford, 2003). Inclusion criteria included having completed a university-based postgraduate course in driver assessment and having delivered driver assessment services within the previous year (at minimum, one assessment per month). Also, to ensure viability of the focus groups, a minimum of 3 participants in that state was required.

Procedure

Information about the focus group(s), together with a copy of the self-completion questionnaire and a reply stamped and addressed envelope (for a parallel study, as previously mentioned), was mailed to 163 DTOTs. The cover letter included participation criteria, focus group arrangements (including a list of specific questions on which opinions would be sought during the focus groups), and informed consent forms; where necessary, one reminder letter was sent 2 wk later. An explanation for how the focus groups would be run was provided, and DTOTs were asked to complete the questionnaire before attending the focus group.

The focus group protocol, including specification of the moderator’s role, item content and presentation, and environmental ambience, was designed in accord with usual guidelines (Hollis, Openshaw, & Goble, 2002; St. John, 2003). A set of specific questions and related probes was developed and pilot tested with six occupational therapists, including three DTOTs, and wording was revised in response to feedback.

Each group was scheduled to take 1.5 hr. The moderator (Marilyn Di Stefano), an experienced group facilitator, followed a detailed script to ensure consistency across groups and took notes both during and immediately after the groups. Sessions were also audiotaped (with participant permission) and transcribed to enhance the rigor of data collection (DePoy & Gitlin, 1998). During discussion of some questions, participants referred to hard-copy prompts (e.g., extracts from competency standards, list of features that could be used in standardizing test routes). Where appropriate after discussion of a specific statement or question, the moderator counted and recorded the number of participants holding a particular view. The moderator’s field notes were reviewed in relation to the full transcripts, responses for similar items across groups were compared, and tallies were tabulated; in addition, quotes suitable for illustrating particular points were identified.

Results

Eight focus groups were conducted in four states: Victoria, three groups; New South Wales, two groups; Tasmania, one group; and Queensland, two groups. A total of 55 DTOTs participated, representing more than one-third of Australian DTOTs thought to be practicing at that time (N = 163). Twenty-three percent were based in rural areas, and most (75%) assisted clients with both physical and cognitive impairments. Their most common client diagnoses were stroke, acquired brain injury, dementia, and various neurological and musculoskeletal conditions. Almost half (46%) of participating DTOTs had ≥6 yr of experience in this role (range = 1–14 yr). Results are presented in the following sections, organized into key issues, and are summarized in Table 1.

Table 1.

Summary of Key Findings: Responses to Questions Posed to Participants (N = 55)

	Questions^a	Responses Total, n^b (%)
1	Assessment goals are multifaceted, including reference to personal impairment, disability, and life roles as well as driving performance and rehabilitation.	General agreement
2	Routes should incorporate all of the basic maneuvers and test items used in entry-level license testing.	30 (55%) agreed
3	A set route and documentation specifying route directions and particular traffic features should be used.	46 (84%) agreed
4a	Documentation to support recording and scoring set observations at set points along a route should be used.	32 (58%) agreed
4b	Vehicle orientation should be compulsory.	48 (87%) agreed
4c	A standard on-road test should last 40-50 min.	52 (95%) agreed
5	List of features that support standardizing test routes should be implemented.	39 (71%) agreed, remainder partially agreed
6	Competency standards; support for retention of features listed as compulsory/desirable test route inclusions (see Figure 1).	51 (93%) agreed
7a	The assessment procedure should be improved.	53 (96%) agreed
7b	A national on-road assessment protocol should be developed.	52 (95%) agreed
8	Additional hazard perception and memory-related test items should be added.	Majority indicated could be optional
9	Scoring system should incorporate differential weighting of test items considering varying low-vs. high-demand environments and item difficulty.	Unanimous agreement that development and implementation would be difficult

^aIn form of statements posed for discussion.

^bWhere it was possible to tally responses.

View Large

Table 1.

Summary of Key Findings: Responses to Questions Posed to Participants (N = 55)

	Questions^a	Responses Total, n^b (%)
1	Assessment goals are multifaceted, including reference to personal impairment, disability, and life roles as well as driving performance and rehabilitation.	General agreement
2	Routes should incorporate all of the basic maneuvers and test items used in entry-level license testing.	30 (55%) agreed
3	A set route and documentation specifying route directions and particular traffic features should be used.	46 (84%) agreed
4a	Documentation to support recording and scoring set observations at set points along a route should be used.	32 (58%) agreed
4b	Vehicle orientation should be compulsory.	48 (87%) agreed
4c	A standard on-road test should last 40-50 min.	52 (95%) agreed
5	List of features that support standardizing test routes should be implemented.	39 (71%) agreed, remainder partially agreed
6	Competency standards; support for retention of features listed as compulsory/desirable test route inclusions (see Figure 1).	51 (93%) agreed
7a	The assessment procedure should be improved.	53 (96%) agreed
7b	A national on-road assessment protocol should be developed.	52 (95%) agreed
8	Additional hazard perception and memory-related test items should be added.	Majority indicated could be optional
9	Scoring system should incorporate differential weighting of test items considering varying low-vs. high-demand environments and item difficulty.	Unanimous agreement that development and implementation would be difficult

^aIn form of statements posed for discussion.

^bWhere it was possible to tally responses.

Goals of the On-Road Assessment

Participants initially discussed some statements concerning the basic goals of occupational therapy on-road assessment. There was general agreement that DTOT assessment goals are usually multifaceted, taking account of each individual’s impairments, disabilities, and potential remediation needs. In addition, more wide-ranging personal needs (e.g., work or family roles), as well as driving-related abilities and performance, are considered. In relation to assessment outcomes, 1 participant said of her older driver clients,

Quite often … in the last 15 minutes of the assessment, I change my assessment to instruction [i.e., from performance assessment to driving instruction].… That gives you a better indication … [of whether] we go for a driving rehab program or … license cancellation.

Relationship Between Occupational Therapy On-Road Assessments and Entry-Level License Tests

Groups were asked whether DTOT on-road assessment should incorporate all of the basic maneuvers and test items used in entry-level license testing. More than half the participants (30 of 55) agreed.

I think those basic entry-level things are good … for client feedback so that you’ve got some concrete things to talk to them about … besides the OT stuff.

Conversely, some groups agreed that pass–fail criteria should not be the same as in entry-level license tests, arguing that occupational therapy assessments of experienced drivers should make allowance for common bad habits, not penalizing those who are not unduly risky. Participants also commented that not all test items needed to be identical to those in entry-level tests, for example,

To fail them because they can’t reverse park a car when they’re 75 years old seems a bit silly … they stop at all stop signs … they’re doing good observations at pedestrian crossings and they know the speed limit round schools and different things. That’s more important.

Use of a Set Route and Associated Documentation Specifying Route Directions and Particular Traffic Features

In principle, agreement was high (46 of 55 participants) that providing use of a set route was possible. Many DTOTs used established routes routinely, and these were seen as safer, allowing DTOTs to become familiar with the route and its associated high-error locations. A set route permits both the DTOT and the driving instructor to concentrate more on observing the driver at particular risky sections of the route, with resultant higher levels of safety awareness, objectivity, and reliability—particularly important when several different occupational therapists implement assessments through the same service. However, many DTOTs saw set routes as sometimes impracticable because of difficulties establishing a standard route in rural and some regional centers. As 1 rural-based participant said,

I’ve referred several people back [elsewhere] because I just cannot get enough into a route for somebody who has high-level cognitive stuff going on. There’s just no way I’m “calling it” based on a town that has no traffic lights and just four square blocks and they’re all roundabouts.…

In this context, DTOTs supported the specification of a core set of mandatory features for all on-road assessment, including test route characteristics, driving maneuvers, and assessment procedures, rather than always having prespecified routes. This specification was seen as particularly important for DTOTs who conduct occasional assessments in many different rural towns, where developing a set of specific routes would not be feasible.

Importance of Particular Features of a Set Route

Participants were asked to consider whether a specified set of test features should be mandatory for most DTOT standard routes. The most frequently discussed features are presented in the following sections.

Documentation to Support Recording and Scoring Set Observations at Set Points Along a Route, to Ensure Specific Observations Are Made of Driving Behavior at Relevant Route Locations.

There was consensus in favor of this approach within five groups, but participants in the other three groups had different views (32 of 55). This approach was seen as particularly useful for new DTOTs, whereas others thought that this level of documentation was unnecessary for experienced DTOTs who consistently use the same route. Several reservations were raised related to the need for flexibility (e.g., customized assessments for clients with specific physical disabilities).

Compulsory Orientation to the Vehicle.

Participants were asked whether there should be a standard requirement for the DTOT to formally orient the driver to the vehicle before commencement, including review of all controls, mirror and seat adjustments, and identification of blind spots. There were high levels of agreement (48 of 55 participants), and most DTOTs reported already doing this. However, some believed that it might be better to assess whether the client independently initiates the process of orientation to the vehicle because it would reveal driver insight into driver–vehicle setup requirements.

Duration of Approximately 40–50 Min for a Standard On-Road Test.

Only 3 occupational therapists disagreed with this timing, although participants in four of the eight groups commented that this duration should be the minimum required. There was agreement across all groups that the duration needed to be sufficiently long to test the client’s mental endurance and to be reflective of real-world travel distances, taking into account the individual’s location and how driving fits into his or her lifestyle and driving patterns (e.g., whether related to employment). One respondent explained how she had to modify test duration at times:

If you’ve got somebody that’s a long haul [truck] driver and has perhaps had an [acquired brain injury], we need to be looking at concentration. I’ve actually done 2-hour assessments because general feedback from family … health professionals … he just falls apart after an hour and a half, falls asleep … the assessment needs to be tailored.

Perceived Importance of Various Means of Achieving Standardization

The moderator asked whether participants agreed that a standard assessment should include all of the following:

A set sequence of procedures, including initial orientation, and then a familiarization driving period
Use of a standard route
Standard instructions to the driver
Set observations at set points along the route
Set pass–fail performance criteria
Use of normative data in evaluating performance criteria.

All participants in five of the eight groups (39 of 55) agreed in principle with the previously mentioned features, whereas members of the other groups disagreed or, more commonly, partially agreed. Having normative data was acknowledged as important for the development of valid pass–fail criteria, and the absence of normative data for a particular route was the most frequently mentioned barrier to implementing a standard assessment. The acceptability of “normal” bad habits was also questioned in relation to their potential safety implications, with some participants commenting that the frequency of a particular behavior among unimpaired drivers does not necessarily make it acceptable during a test. Also, two groups discussed the benefits of allowing the DTOT to make notations about driver performance at any location along the test route, rather than confining these notations to prespecified locations.

Comments on Current Victorian Occupational Therapy Competency Standards for Driving Assessors

As previously mentioned, Victorian DTOT driving assessors have a set of competency standards that include separate lists of compulsory or desirable features for assessments leading to an unconditional (open—driving not geographically restricted) license or a conditional (driving restricted to a local area) license (Figure 1). Each focus group reviewed the list contents, and most participants supported inclusion of all the items (51 of 55). However, some participants pointed out that for unconditional license assessments in some rural areas, it would probably be difficult to provide high-density traffic or intersections with parked cars occluding the view along the intersecting road. Also discussed was the need for clearer operational definitions of some items; inclusion of some disability-specific items (e.g., a hill start in a manual transmission car, when assessing those with certain physical disabilities); and standard test routes to include a speed zone higher than the general minimum (e.g., 80 kph).

Figure 1.

Extract from Appendix A: Assessment Route Specifications From the Competency Standards for Occupational Therapy Driver Assessors.

Description: The occupational therapist determines the number and location of traffic conditions and driving maneuvers and the grading of the traffic demands to facilitate a comprehensive on-road assessment. The compulsory inclusions, desirable inclusions and factors to consider listed below need to be applied to the following assessments: unconditional license (no geographical restriction) and conditional license (area restriction applied).

Note. From OT-Australia Victoria. (1998) . Competency standards for occupational therapy driver assessors (p. 36). Melbourne, Victoria, Australia: Author. Copyright ©1998 by OT-Australia Victoria. Used with permission.

Extract from Appendix A: Assessment Route Specifications From the Competency Standards for Occupational Therapy Driver Assessors.

Full Size | Slide (.ppt)

Comments on Proposed Standard Test Structure

Most participants indicated that the assessment procedure could be improved (53 of 55) and supported the development of a national on-road assessment protocol (52 of 55). In seven of the eight groups, there was unanimous agreement that the general form of a standard assessment should include an initial period of familiarization, a standard core set of driving task requirements, and, possibly, additional client-specific items chosen on the basis of the presenting impairment and disability issues. One of the groups did not reach consensus on this item, with some arguing that client-specific items should form the bulk of the test content.

Comments on Proposed Additional Inclusions in a Standard Assessment

Participants were asked to consider whether it would be both practicable and desirable to include the following items in a standard assessment:

A hazard perception task requiring the driver to park the vehicle at the curb and describe hazards in the road environment
A task to assess memory/planning ability, in which they are instructed to “pull into the next petrol station that we come to and position the car beside a petrol bowser.”

Responses to these proposals were mixed. The majority in four of the groups thought that the hazard perception task should be optional, whereas others said it would be practicable for all their assessments. According to one respondent,

You might go through a shopping center and nothing particularly untoward happens, so you’re not really sure whether the person has actually checked that pedestrian crossing or noticed that car pulling out. Whereas if [people are required to verbalize the kind of things they’re looking out for] it’s a way of confirming that even though they didn’t encounter it they actually were aware that it could happen.

However, many potential issues were raised, including client anxiety, language problems, and the varying effects of local area knowledge. There was also concern that such a task does not replicate real-world driving demands because it takes no account of the need for hazard perception to be integrated with other aspects of driving performance. One participant explained the reservations:

We don’t drive up to a complex situation, stop, think about it and then drive into it. We actually encounter it in the middle of whatever’s happening.… It’s not something that we do naturally when we’re driving.

Most participants thought the memory/planning task (recalling, then executing entry/exit into a petrol station) should be optional. Concerns expressed included client anxiety that might negatively affect performance, unavailability of service stations and possible issues from the viewpoint of their owners, and how driver performance on this task should be interpreted.

Weighting of Test Components

In Victoria, DTOTs are trained to tally various categories of driver error and driver instructor interventions to enable quantification and more reliable evaluation (Lovell & Di Stefano, 2008), whereas instructor interventions are considered automatic fail criteria (Roads Corporation, 2008). However, there is no basis for assuming that item scores should simply be added across different error categories; indeed, there are prima facie reasons to prefer a scoring system in which some test items are weighted more heavily than others. The set of questions that followed in the focus group agenda was designed to explore the feasibility of developing such a system. Issues related to this were a particular focus of the previously distributed questionnaire.

One issue concerned how it would be possible to determine appropriate weighting changes when superficially similar errors occurred in very different contexts that varied substantially in their demands on the driver, so that the context affects the probability of errors occurring. Focus group participants were asked to review specific questionnaire items concerning determinants of driving task demands or complexity and to discuss the issues raised. Discussion focused on the nature of low- versus high-demand driving environments and how they might affect weighting, including the difficulty of disentangling and taking account of the interacting effects of vehicle speeds, traffic density, different driving maneuvers, and the roadway environment (e.g., single vs. dual carriageways)—some of which typically vary with time of day. Other factors identified as relevant were driver familiarity with the test route; the presence of vehicles ahead following the same route that could simply be followed, making the task easier; and the perceived consequences of driving errors, which vary significantly with speed.

For these various reasons, participants were unanimous that although a standardized error weighting system might provide a clearer justification for test outcomes, it would be extremely difficult to develop and implement because of the complex and constantly changing contexts in which errors occur. Typical comments about contextual variables likely to affect error rate include the following:

Being out on a freeway sometimes … when you’re just driving ahead at a reasonable speed [is] less demanding than when you’re having to go slow.… It might not necessarily be a speed factor [that determines demand]. It’s whether there’s other things going on as well.
It would be dependent also on the task that they’ve been required to perform. So, if they’re in a low-demand environment but they’re self-navigating, that’s a higher cognitive demand than if they’re just following an instruction.

Members of six of the groups said that in fact they currently do weight errors differently—“subconsciously,” “automatically,” or as part of their professional judgment. In Victoria, we noted that DTOTs are in fact required to do this (albeit after the assessment) when completing the compulsory licensing authority report, which requires errors to be categorized as “critical” (directly threatening safety) or “noncritical” (perhaps reflecting poor habits but not creating an immediate risk). There was considerable discussion of the basis on which DTOTs currently weight different types of error in determining test outcomes. Typical examples reported were as follows:

Safety, safety, safety. I’d say it doesn’t really matter that they didn’t check their mirrors because there was nothing around but boy they missed that car that’s dangerous if they were in a four-lane carriageway or something.
Planning, judgment, and observation.… They’re the heavy weights!

The types of errors identified as most heavily weighted were those related to visual observation, attention, planning, and judgment. Consistent with this finding, some people recommended inclusion of a self-navigation task because this technique is particularly sensitive to planning and attentional impairments. There was a strong view that it would be very difficult to replace DTOTs’ professional judgment in weighting errors, particularly in relation to specific disability issues. However, there was also strong recognition of the potential benefits of a more standardized system that would facilitate more valid and reliable weighting by DTOTs of observed driver errors. For example,

I think it’s a good idea.… We are constantly juggling this … how relevant was this mistake, how risky was that, how does that compare to the other? … the weighting is just going to help us categorize it a little bit more coherently.

Summary and Discussion

We investigated the opinions of DTOTs regarding on-road assessment procedures and possible ways of improving them to enhance validity and reliability. There was strong support for the proposed developments of the current Australian DTOT on-road assessment procedure. Results highlighted several important issues that should be considered in the future development of these procedures and the professional competency standards and guidelines supporting their implementation.

First, there was clear in-principle support for use of prespecified test routes when assessing drivers who wish to gain or retain an unrestricted license, consistent with the views of many researchers (e.g., Duchek, Hunt, Ball, Buckles, & Morris, 1998; Justiss, Mann, Stav, & Velozo, 2006; Kay, Bundy, Clemson, & Jolly, 2008; Kowalski & Tuokko, 2007). However, it was agreed that in some circumstances, the use of prespecified test routes is not feasible, and it is therefore important to specify a standard set of core requirements for all routes used in an assessment, regardless of whether they are predeveloped, standard routes (Di Stefano & Macdonald, 2008). Such core requirements are already used in developing standard routes (Macdonald & Harrison, 2008). Results here highlight the need to more clearly identify core requirements for use when standard routes are not practical (OT-Australia Victoria, 1998; Roads Corporation, 2008).

Second, there was general support for some key features of standard routes. It was agreed that test duration should be a minimum of 40 to 50 min, and the procedure should entail an initial period of familiarization; a core set of mandatory maneuvers and other test items; and additional client-specific items should be included if and when necessary. Other researchers support most of these features, acknowledging that tests need to sample an adequate range of relevant behaviors and be sufficiently long and difficult to be valid (Fox, Bowden, & Smith, 1998; Korner-Bitensky, Gelinas, Man-Son-Hing, & Marshall, 2005; Kowalski & Tuokko, 2007; Stav, 2004; Withaar et al., 2000). The period of familiarization is important because drivers undertaking competency tests may perform more poorly in unfamiliar vehicles (Lundberg & Hakamies-Blomqvist, 2003). The core set of mandatory maneuvers and route design criteria currently specified in the Victorian Driver OT Competency Standards were seen as acceptable and mostly practicable. To enhance these standards, additional items specific to different assessment contexts (e.g., urban or rural environments) could be considered as optional, additional items for use when appropriate.

The additional items proposed for inclusion and discussed in the focus groups (hazard perception, petrol station memory–planning task) were seen as potentially useful and worthy of a trial, but some participants thought they were unsuitable for mandatory use—at least in the absence of further research evidence of their value. The petrol station item included a navigational component, which imposes a higher level of task demand compared with following scripted route instructions (Hunt et al., 1997). Lovell and Russell (2005) used such an item when they evaluated performance changes over time of a small group of drivers referred to a memory clinic. Drivers in that cohort displayed high rates of instructor interventions at six particular points along the route, and their results suggested that this task was not unduly difficult. The ecological validity of similar items improves face validity for the drivers themselves (Fox et al., 1998; Withaar et al., 2000), which is an advantage when providing feedback to drivers who fail the assessment.

Third, in discussing issues related to performance scoring, particularly differential weighting of driver errors, most agreed that it is important to take account of experienced drivers’ normal driving patterns and widely occurring bad habits, as previously identified (Dobbs, Heller, & Schopflocker, 1998; Janke & Eberhard, 1998). Most DTOTs believed that errors stemming from such habits do not warrant concern provided they present no immediate risk, and many reported already weighting such errors accordingly. Errors identified as most heavily weighted by the DTOTs were those related to visual scanning or observation, attention, planning, and judgment, consistent with other research (Kay, Bundy, & Clemson, 2008; Mallon & Wood, 2004).

Despite reporting that they informally weighted errors, the DTOTs acknowledged that this is a difficult process and were keen to trial procedures to improve standardization. Kowalski and Tuokko (2007, p. 14) recommended scoring procedures be “simple with specification of the correct responses to enhance reliability and ensure identical standards of performance are used for each assessment.” For some aspects of performance, altering dichotomous scoring (specific behaviors present or not) to a Likert scale rating might improve reliability. However, the introduction of such systems needs further empirical research to establish reliability and validity (Justiss et al., 2006). DTOTs were fully supportive of such a possibility but emphasized the need for it to take account of individual driver characteristics and of contextual issues such as the overall task demand level at the time (Di Stefano & Macdonald, 2003).

At a more general level, results confirmed the construct validity of the assessment procedure, consistent with occupational therapy conceptual frameworks (Macdonald, Pellerito, & Di Stefano, 2006). In relation to route design, item inclusion, and scoring, it was clear that DTOTs were taking account of driver characteristics, task characteristics, and the context of activity engagement, as emphasized in the World Health Organization’s (2001)International Classification of Functioning, Disability and Health, as well as in theoretical models underpinning occupational therapy practice (Dunn, McClain, Brown, & Youngstrom, 1998; Kielhofner, 2002; Law et al., 1996). Consistent with Fuller’s (2005) task-capability interface model of driving performance, it was evident that DTOTs consider the dynamic nature of the driving context and issues that influenced task demand such as the maneuver difficulty and the presence of other road users (Di Stefano & Macdonald, 2006a).

Study Strengths and Limitations

Focus group methods for collecting data have well-documented advantages and disadvantages (Taylor & Kielhofner, 2006). The limitations were managed adequately in this study because participants were briefed beforehand and implementation protocols were strictly applied, enabling generation of a rich set of data on issues concerning DTOT assessment practices drawn from a large and representative group of expert practitioners. This level of detail has not previously been documented, and it provides part of the necessary basis for the practical implementation of improvements to professional competency standards. Data collection was confined to Australian DTOTs, and their views on some issues are undoubtedly influenced by the legislative frameworks within which they work. Transference of findings to DTOT procedures in other jurisdictions would need to take account of specific driver licensing issues that may influence practice.

Conclusion

Practicing occupational therapy driver assessors strongly support amendments to their on-road assessment procedures to increase standardization, reliability, and validity, in accord with research-based evidence and practice guidelines in the United States and Canada, as well as in Australia. There was agreement regarding a minimum set of standard core requirements applicable to all tests, with fixed routes and predetermined assessment points important for standard assessments. However, some flexibility must be retained for local area tests in relation to route, procedure, and test items; trials of these and proposed new test features are needed.

Acknowledgments

We thank the clinicians, OT-Australia, and VicRoads for their assistance. This study was part of a project funded by the Australian Transport Safety Bureau. Marilyn Di Stefano was the recipient of a National Health and Medical Research post graduate scholarship in Public Health.

References

Akinwuntan, A. E., Feys, H., DeWeerdt, W., Pauwels, J., Baten, G., & Strypstein, E. (2002). Determinants of driving after stroke. Archives of Physical Medicine and Rehabilitation, 83, 334–341. [Article] [PubMed]

Austroads. (2003). Assessing fitness to drive for commercial and private vehicle drivers: Medical standards for licensing and clinical management guidelines (3rd ed.). Sydney, New South Wales, Australia: Author.

Carr, D., Jackson, T. W., Madden, D. J., & Cohen, H. J. (1992). The effect of age on driving skills. Journal of the American Geriatrics Society, 40, 567–573. [PubMed]

Caust, S. (1988). Clinical perspectives: Occupational therapy driver assessment course—A report by a course participant. Australian Journal of Occupational Therapy, 35, 181–185. [Article]

Cushman, L. A. (1996). Cognitive capacity and concurrent driving performance in older drivers. IATSS Research, 20, 38–45.

Denzin, N. K. (1970). The research act in sociology. Chicago: Aldine.

DePoy, E., & Gitlin, L. N. (1998). Introduction to research: Understanding and applying multiple strategies (2nd ed.). St. Louis, MO: Mosby.

Di Stefano, M., & Macdonald, W. (1998, June). Survey of Australian occupational therapists to determine disabled driver on-road assessment practice. Paper presented at the World Federation of Occupational Therapists Conference: Sharing a Global Perspective, Montreal, Quebec, Canada.

Di Stefano, M., & Macdonald, W. (2003). Assessment of older drivers: Relationships among on-road errors, medical conditions and test outcome. Journal of Safety Research, 34, 415–429. [Article] [PubMed]

Di Stefano, M., & Macdonald, W. (2006a). Advanced strategies for on-road driver rehabilitation and training. In J.Pellerito (Ed.), Driver rehabilitation and community mobility: Principles and practice (pp. 297–310). St. Louis, MO: Elsevier/Mosby.

Di Stefano, M., & Macdonald, W. (2006b). On-the-road evaluation of driving performance. In J.Pellerito (Ed.), Driver rehabilitation and community mobility: Principles and practice (pp. 255–274). St. Louis, MO: Elsevier/Mosby.

Di Stefano, M., & Macdonald, W. (2007). Driving assessment procedures of occupational therapists: Some results from a national survey of OT driver assessors. In Proceedings of the Road Safety, Research, Policing and Education Conference (pp. 17–19). Sydney, New South Wales: Australian College of Road Safety, Retrieved October 1, 2009, from www.rsconference.com/RoadSafety/detail/813

Di Stefano, M., & Macdonald, W. (2008). Design of on-road test routes: Validity issues for assessment of functionally impaired drivers. Paper presented at the Road Safety, Research, Policing and Education Conference, Adelaide, Australia.

Dobbs, A. R., Heller, R. B., & Schopflocker, D. (1998). A comparative approach to identify unsafe older drivers. Accident Analysis and Prevention, 30, 363–370. [Article] [PubMed]

Duchek, J. M., Hunt, L., Ball, K., Buckles, V., & Morris, J. C. (1998). Attention and driving performance in Alzheimer’s disease. Journals of Gerontology, Series B: Psychological Sciences, 53B, 130–141. [Article]

Dunn, W., McClain, L. H., Brown, C., & Youngstrom, M. J. (1998). The ecology of human performance. In M. E.Neistadt & E. B.Crepeau (Eds.), Willard and Spackman’s occupational therapy (9th ed., pp. 525–535). Philadelphia: Lippincott, Williams & Wilkins.

Ekelman, B. A., Mitchell, S., & O’Dell-Rossi, P. (2001). Driving and older adults. In M. B.Wagner (Ed.), Functional performance in older adults (2nd ed., pp. 448–472). Philadelphia: F. A. Davis.

Fildes, B., Pronk, N., Langford, J., Hull, M., Frith, B., & Anderson, R. (2000). Model Licence re-assessment procedure for older and disabled drivers (Report to Austroads AP–R 176/00). Melbourne, Victoria: Monash University Accident Research Centre.

Fitten, L. J., Perryman, K. M., Wilkinson, C. J., Little, R. J., Burns, M. M., Pachana, N., et al. (1995). Alzheimer and vascular dementias and driving: A prospective road and laboratory study. Journal of the American Geriatrics Association, 273, 1360–1365. [Article]

Fox, G. K., Bowden, S. C., & Smith, D. S. (1998). On-road assessment of driving competence after brain impairment: Review of current practice and recommendations for a standardized examination. Archives of Physical Medicine and Rehabilitation, 79, 1288–1296. [Article] [PubMed]

Fuller, R. (2005). Towards a general theory of driver behaviour. Accident Analysis and Prevention, 37, 461–472. [Article] [PubMed]

Hollis, V., Openshaw, S., & Goble, R. (2002). Conducting focus groups: Purpose and practicalities. British Journal of Occupational Therapy, 65, 2–8.

Hunt, L. A., Murphy, C. F., Carr, D., Duchek, J. M., Buckles, V., & Morris, J. C. (1997). Reliability of the Washington University Road Test: A performance-based assessment for drivers with dementia of the Alzheimer type. Archives of Neurology, 54, 707–712. [Article] [PubMed]

Janke, M. K., & Eberhard, J. W. (1998). Assessing medically impaired older drivers in a licensing agency setting. Accident Analysis and Prevention, 30, 347–361. [Article] [PubMed]

Justiss, M. D., Mann, W. C., Stav, W. B., & Velozo, C. (2006). Development of a behind-the-wheel driving performance assessment for older adults. Topics in Geriatric Rehabilitation, 22, 121–128. [Article]

Kay, L. G., Bundy, A. C., & Clemson, L. M. (2008). Predicting fitness to drive using the visual recognition slide test (USyd). American Journal of Occupational Therapy, 62, 187–197. [Article] [PubMed]

Kay, L., Bundy, A., Clemson, L., & Jolly, N. (2008). Validity and reliability of the on-road driving assessment with senior drivers. Accident Analysis and Prevention, 40, 751–759. [Article] [PubMed]

Kielhofner, G. (2002). A model of human occupation: Theory and application (3rd ed.). Baltimore: Lippincott Williams & Wilkins.

Klavora, P., Heslegrave, R. J., & Young, M. (2000). Driving skills in elderly persons with stroke: Comparison of two new assessment options. Archives of Physical Medicine and Rehabilitation, 81, 701–705. [PubMed]

Korner-Bitensky, N., Bitensky, J., Sofer, S., Man-Son-Hing, M., & Gelinas, I. (2006). Driving evaluation practices of clinicians working in the United States and Canada. American Journal of Occupational Therapy, 60, 428–434. [Article] [PubMed]

Korner-Bitensky, N., Gelinas, I., Man-Son-Hing, M., & Marshall, S. (2005). Recommendations of the Canadian Consensus Conference on Driving Evaluation in Older Drivers. Physical and Occupational Therapy in Geriatrics, 23, 119–140.

Korner-Bitensky, N., Sofer, S., Kaizer, F., Gelinas, I., & Talbot, L. (1994). Assessing ability to drive following an acute neurological event: Are we on the right road?. Canadian Journal of Occupational Therapy, 61, 141–148. [Article]

Korteling, J. E., & Kaptein, N. A. (1996). Neuropsychological driving fitness tests for brain-damaged subjects. Archives of Physical Medicine and Rehabilitation, 77, 138–146. [Article] [PubMed]

Kowalski, K., & Tuokko, H. (2007). On-road driving assessment of older adults: A review of the literature. Retrieved February 3, 2009, from www.jibc.ca/ptec/Customized/On-RoadDrivingOlderAdults%20_LiteratureReview.pdf

Law, M., Cooper, B. A., Strong, S., Stewart, D., Rigby, P., & Letts, L. (1996). The Person–Environment–Occupation model: A transactive approach to occupational performance. Canadian Journal of Occupational Therapy, 63, 9–23. [Article]

Lovell, R., & Di Stefano, M. (2008). Driver education and rehabilitation course manual. Melbourne, Victoria, Australia: La Trobe University.

Lovell, R., & Russell, K. (2005). Developing referral and reassessment criteria for drivers with dementia. Australian Occupational Therapy Journal, 52, 26–33. [Article]

Lundberg, C., & Hakamies-Blomqvist, L. (2003). Driving tests with older patients: Effect of unfamiliar versus familiar vehicle. Transportation Research Part F, 6, 163–173. [Article]

Macdonald, W. A., Bowland, L., & Triggs, T. J. (1992). Evaluation of driver testing and licensing in Victoria: Report on preliminary investigations (Monash University Accident Research Centre Report 24). Retrieved March 2009 from www.monash.edu.au/muarc/reports/muarc024.pdf

Macdonald, W. A., & Harrison, W. (2008, November). Construct validity of Victoria’s new license test. In Proceedings of the Road Safety, Research, Policing and Education Conference (pp. 270–281). Adelaide, South Australia: Australian College of Road Safety. Retrieved October 1, 2009, from www.rsconference.com/RoadSafety/detail/718

Macdonald, W., Pellerito, J., & Di Stefano, M. (2006). Introduction to driver rehabilitation and community mobility. In J.Pellerito (Ed.), Driver rehabilitation and community mobility: Principles and practice (pp. 5–35). Philadelphia: Elsevier.

Mallon, K., & Wood, J. M. (2004). Occupational therapy assessment of open-road driving performance: Validity of directed and self-directed navigational instructional components. American Journal of Occupational Therapy, 58, 279–286. [Article] [PubMed]

Minichiello, V., Sullivan, G., Greenwood, K., & Axford, R. (2003). Handbook of research methods for nursing and health science (2nd ed.). French Forest, New South Wales, Australia: Prentice Hall Health.

Morgan, D. L. (1993). Successful focus groups: Advancing state of the art. Newbury Park, CA: Sage.

National Highway Traffic Safety Administration. (2001). Older road user research plan. Retrieved April 18, 2006, from www.nhtsa.dot.gov/people/injury/olddrive/OlderRoad/index.htm

Organisation for Economic Co-Operative Development. (2001). Aging and transport: Mobility needs and transport issues (OECD Report). Geneva: Author.

OT-Australia Victoria. (1998). Competency standards for occupational therapy driver assessors. Melbourne, Victoria, Australia: Author.

Perryman, K. M., & Fitten, L. J. (1996). Effects of normal aging on the performance of motor-vehicle operational skills. Journal of Geriatric Psychiatry and Neurology, 9, 136–141. [Article] [PubMed]

Phillips, B., Ball, C., Sackette, D., Badenoch, D., Straus, S., Haynes, B., et al. (2001). Oxford centre for evidence-based medicine: Levels of evidence—May 2001. Retrieved February 3, 2009, from www.cebm.net/index.aspx?o=1025

Roads Corporation. (2008). VicRoads and OT Australia (Victoria): Guidelines for occupational therapy (OT) driver assessors (2nd ed.). Melbourne, Victoria, Australia: Author.

Road Safety Committee, Parliament of Victoria. (2003). Inquiry into road safety for older road users. Melbourne, Victoria, Australia: Victorian Government Printer.

Schneider, C. (1998). Competency standards for occupational therapy driver assessors. Paper presented at the Implementing Change: The role of Standards, Ergonomics Society of Australia Annual Conference, Melbourne, Victoria, Australia.

Snellgrove, C. A. (2005). Cognitive screening for the safe driving competence of older people with mild cognitive impairment or early dementia (Australian Transport Safety Bureau, Grant No. B2002/0204). Retrieved October 1, 2009, from www.infrastructure.gov.au/roads/safety/publications/2005/cog_screen_old.aspx

Stav, W. B. (2004). Driving rehabilitation: A guide for assessment and intervention. San Antonio, TX: Psychological Corp.

Stav, W. B., Hunt, L. A., & Arbesman, M. (2006). Occupational therapy practice guidelines for driving and community mobility for older adults. Bethesda, MD: AOTA Press.

Stewart, D. W., & Shamdasani, P. N. (1990). Focus groups: Theory and practice. Newbury Park, CA: Sage.

St. John, W. (2003). Focus group interviews. In V.Minichiello, G.Sullivan, K.Greenwood, & R. E.Axford (Eds.), Handbook of research methods for nursing and health science (2nd ed., pp. 419–430). French Forest, New South Wales, Australia: Prentice Hall Health.

Taylor, R., & Kielhofner, G. (2006). Collecting data. In G.Kielhofner (Ed.), Research in occupational therapy: Methods of inquiry for enhancing practice (pp. 530–547), Philadelphia: F. A. Davis.

Victorian State Government. (1988). Road Safety Act 27 and road safety procedures regulations. Melbourne, Victoria, Australia: Author.

Victorian State Government. (2004). Victorian government response to the report of the road safety committee of Parliament on road safety for older road users. Retrieved June, 23, 2005, from www.parliament.vic.gov.au/rsc/older/VicGov-Response_OlderRoadUsers_2004-04.pdf

Withaar, F. K., Brouwer, W. H., & van Zomeren, A. H. (2000). Fitness to drive in older drivers with cognitive impairment. Journal of the International Neuropsychological Society, 6, 480–490. [Article] [PubMed]

World Health Organization. (2001). International classification of functioning, disability and health. Geneva: Author.

1The term medically impaired driver is used throughout this article to refer to drivers with any aging, disease, acquired, or congenital related impairment that has resulted in a capacity reduction or functional limitation that affects their driving abilities. The inclusion of medical in this term acknowledges that within Australia, all drivers with such limitations must visit a medical practitioner for assessment before undertaking an open road assessment (Austroads, 2003).

Figure 1.

Extract from Appendix A: Assessment Route Specifications From the Competency Standards for Occupational Therapy Driver Assessors.

Full Size | Slide (.ppt)

Table 1.

Summary of Key Findings: Responses to Questions Posed to Participants (N = 55)

	Questions^a	Responses Total, n^b (%)
1	Assessment goals are multifaceted, including reference to personal impairment, disability, and life roles as well as driving performance and rehabilitation.	General agreement
2	Routes should incorporate all of the basic maneuvers and test items used in entry-level license testing.	30 (55%) agreed
3	A set route and documentation specifying route directions and particular traffic features should be used.	46 (84%) agreed
4a	Documentation to support recording and scoring set observations at set points along a route should be used.	32 (58%) agreed
4b	Vehicle orientation should be compulsory.	48 (87%) agreed
4c	A standard on-road test should last 40-50 min.	52 (95%) agreed
5	List of features that support standardizing test routes should be implemented.	39 (71%) agreed, remainder partially agreed
6	Competency standards; support for retention of features listed as compulsory/desirable test route inclusions (see Figure 1).	51 (93%) agreed
7a	The assessment procedure should be improved.	53 (96%) agreed
7b	A national on-road assessment protocol should be developed.	52 (95%) agreed
8	Additional hazard perception and memory-related test items should be added.	Majority indicated could be optional
9	Scoring system should incorporate differential weighting of test items considering varying low-vs. high-demand environments and item difficulty.	Unanimous agreement that development and implementation would be difficult

^aIn form of statements posed for discussion.

^bWhere it was possible to tally responses.

View Large

Table 1.

Summary of Key Findings: Responses to Questions Posed to Participants (N = 55)

	Questions^a	Responses Total, n^b (%)
1	Assessment goals are multifaceted, including reference to personal impairment, disability, and life roles as well as driving performance and rehabilitation.	General agreement
2	Routes should incorporate all of the basic maneuvers and test items used in entry-level license testing.	30 (55%) agreed
3	A set route and documentation specifying route directions and particular traffic features should be used.	46 (84%) agreed
4a	Documentation to support recording and scoring set observations at set points along a route should be used.	32 (58%) agreed
4b	Vehicle orientation should be compulsory.	48 (87%) agreed
4c	A standard on-road test should last 40-50 min.	52 (95%) agreed
5	List of features that support standardizing test routes should be implemented.	39 (71%) agreed, remainder partially agreed
6	Competency standards; support for retention of features listed as compulsory/desirable test route inclusions (see Figure 1).	51 (93%) agreed
7a	The assessment procedure should be improved.	53 (96%) agreed
7b	A national on-road assessment protocol should be developed.	52 (95%) agreed
8	Additional hazard perception and memory-related test items should be added.	Majority indicated could be optional
9	Scoring system should incorporate differential weighting of test items considering varying low-vs. high-demand environments and item difficulty.	Unanimous agreement that development and implementation would be difficult

^aIn form of statements posed for discussion.

^bWhere it was possible to tally responses.