The purpose of occupational therapy educational programs is to graduate clinicians possessing clinical competence in the skills and knowledge required to deliver entry-level evidence-based occupational therapy services (Avi-Itzhak & Krauss, 2014; Jedlicka, Mosley, Jaffe, & Kassner, 2004). Consequently, the aim of admissions committees of educational programs in occupational therapy and other health professions is to identify, from a pool of applicants, those who are most likely to successfully graduate from the program. Successful graduation from a health professions educational program indicates that the graduate was able to meet the academic and clinical requirements of the program, measured by the graduating grade point average (GPA; Auriemma, 2002; Avi-Itzhak & Kellner, 1996; Hansen-Suchy, 2011; Kirchner, Stone, & Holm, 2000; Lysaght, Donnelly, & Villeneuve, 2009; Posthuma & Sommerfreund, 1985) or was able to pass a licensure exam, measured by a first-time pass score (Aldridge, Keith, Sloas, & Mott-Murphree, 2010; Dadian, Geurink, Olney, & Littlefield, 2002; Edenfield & Hansen, 2000; Ennulat, Garrubba, & DeLong, 2011; Hagemaster, 2001; Sloas, Keith, & Whitehead, 2013).

This article reports the results of an outcome evaluation measuring the ability of the National Board for Certification in Occupational Therapy (NBCOT) Occupational Therapist Registered (OTR^®) practice test to predict first-time pass status on the NBCOT OTR national certification exam in a sample of students in an occupational therapy educational program in a public urban higher education institution. Credentialing of health professionals is a topic that has received increasing attention in recent years (Ebiasah, Schneider, Pedersen, & Mirtallo, 2002). Consequently, a body of literature has begun to emerge examining variables predicting student success in health professions educational programs using first-time pass rates on national or regional certification exams as an outcome measure for student success.

For example, the effects of teaching mode (in class, face to face vs. online) on performance on certification exams have been the focus of studies in various health professions, generating mixed results. Johnston (2008)  reported that national certification exam scores for the patient care section were higher for on-campus students in a radiological technology educational program. A trend analysis performed by Bearden, Robinson, and Deis (2002)  with students in a dental hygiene program indicated that students with a lower GPA who enrolled in online courses performed lower on the certification exam than on-campus students. Conversely, Hansen-Suchy (2011)  and Russell et al. (2007)  found no significant differences in performance on certification exams between students in a medical laboratory technician program or clinical laboratory science program receiving online compared with in-class, face-to-face instruction. Hagemaster (2001)  assessed the effectiveness of an online web-based curriculum pilot in substance abuse prevention in preparing professionals for certification exams meeting state and international certification criteria. Results showed a significantly higher pass rate for the pilot group compared with a national sample.

Although program accreditation is not mandatory in paramedic educational programs, several studies have assessed variables affecting performance on the National Registry Paramedic Certification Exam. Dickison, Hostler, Platt, and Wang (2006)  found that students who attended an accredited paramedic program were more likely to achieve a passing score on the certification exam. Findings by Fernandez, Studnek, and Margolis (2008)  supported those of Dickison et al. in that national program accreditation, instructor qualifications, student educational background, and student demographics were all significantly associated with the probability of success on the certification exam.

Williams and Hadfield (2003)  explored the relationship between attributes of an athletic training educational program curriculum and the pass rate on the National Athletic Training Association Board of Certification exam. The most effective predictors of pass rate included emphasizing the domains of athletic training, separating the clinical and academic responsibilities of faculty, and not focusing on hiring faculty with kindergarten through 12th-grade work experience.

Guffey, Farris, Aldridge, and Thomas (2002)  found that the Non-Cognitive Questionnaire–Revised was not effective in predicting performance on the physical therapy licensure exam. Aldridge et al. (2010)  examined the relationship between scaled scores on the Nelson-Denny Reading Test and scaled scores on the physical therapy licensure exam. Their findings yielded significant moderate correlations between reading comprehension and the ability to pass this exam. Goodyear and Lampe (2004)  examined several standardized tests that are part of the admissions criteria at a medical technology program to assess their ability to predict student success in the program and on the board of registry certification exam. They reported that scores on the Allied Health Professions Admission Test predicted both student academic success in the program and performance on the board of registry exam.

This literature has demonstrated the widespread use of first-time pass rates on national or regional certification exams as an effective outcome measure for student success in a variety of health professions educational programs. Indeed, first-time pass rates on certification exams is prominently posted on the websites of many health professions’ educational programs, including those of occupational therapy programs. However, no studies have examined first-time pass rates on the NBCOT exam as an outcome measure for student success in occupational therapy programs. In addition, the few studies that have assessed the predictive ability of practice tests designed by the agencies responsible for developing and administering national certification exams as a learning activity to help students gain familiarity with national certification exams have come from practice areas other than occupational therapy (Dadian et al., 2002; Edenfield & Hansen, 2000; Ennulat et al., 2011).

The purpose of the current study was to assess the relationship between performance ratios (i.e., ratio of items correctly answered) on each of the four NBCOT practice test domains and first-time pass status on the NBCOT exam for a sample of students in an occupational therapy department in a public urban institution of higher education. The aim was to apply the four NBCOT practice test performance ratios to develop a logistic regression model for estimating the probability of first-time pass status on the NBCOT exam and to identify the domains that have a significant predictive effect on this probability. The null hypothesis stated that the difference in the odds of the NBCOT practice test performance ratio for first-time pass status compared with first-time no-pass status would not be statistically significant.

Of the 71 students who graduated from the program during academic years 2010–2013, student records for 65 (92%) for whom information was available were used in the study. Of these students, 41 (63%) attained first-time pass status, whereas 24 (37%) attained first-time no-pass status.

The study used quantitative methodology (hypothesis driven) and was retrospective and cross-sectional.

NBCOT is an independent nonprofit organization responsible for the development and implementation of psychometrically sound and legally defensible occupational therapy exams. In 2008, NBCOT conducted a practice analysis study of entry-level occupational therapy practice and on the basis of the results adopted a format for both exams consisting of four practice domains. This format was used for exams administered beginning in academic year 2010. Descriptions of the four domain areas and examples of related tasks and skills appear in Table 1.

The NBCOT OTR exam is a timed online computer exam consisting of three simulation test items and 170 multiple-choice test items. The NBCOT practice test is also a timed online exam consisting of 100 multiple-choice test items reflecting the content of the current NBCOT exam. Passing score on both exams is 450. NBCOT provides test takers with their final score, the relative weight of each domain, and the percentage of correctly answered items in each domain and identifies domains of concern (<69% of items answered correctly) for possible focus in future NBCOT exam preparation strategies. This information is also provided to program directors. NBCOT (2008)  stated that although taking the NBCOT practice test, or using any other tool designed to help assess readiness for the certification exam, cannot guarantee success on the NBCOT exam, candidates may consider using the tool to guide their test preparation strategies.

This study used five measures. Performance on each of the four NBCOT domains was conceptualized as the predictors (independent variables) and operationally defined as the ratio of correctly answered items in each domain to the total number of items in the domain (NBCOT, 2011). At the time of this study, the four domains were as follows:

1. Gather information regarding factors that influence occupational performance

2. Formulate conclusions regarding the client’s needs and priorities to develop a client-centered intervention plan

3. Select and implement evidence-based interventions to support participation in areas of occupation (e.g., activities of daily living, education, work, play, leisure, social participation) throughout the continuum of care

4. Uphold professional standards and responsibilities to promote quality in practice.

The fifth measure, first-time pass status on the NBCOT OTR exam, was conceptualized as the outcome measure (dependent variable) and was defined as a categorical variable (first-time pass status, score of ≥450; first-time no-pass status, score of <450).

A mandatory graduate seminar was designed and incorporated into the occupational therapy curriculum in the last semester before graduation beginning in academic year 2007. The seminar was designed to assess and increase prelicensure skills and competence required for entry-level evidence-based practice and for successfully passing the NBCOT exam. As part of the seminar, each student was provided with two NBCOT practice tests, administered at the first (pre) and last (post) seminar sessions, that were paid for by the occupational therapy department (see Avi-Itzhak & Krauss, 2010, for additional information pertaining to the educational premise underlying the seminar). The data source for the four predictors (percentage of items answered correctly on each domain) was the postseminar NBCOT practice test.

Descriptive statistics were computed to summarize the study variables. A logistic (binary) regression statistical model was built to determine the probability of first-time no-pass status on the NBCOT exam using the performance ratio (percentage of items answered correctly in each domain) on the four domain areas. First-time no-pass status was determined by the odds ratio (OR) with 95% confidence interval. Level of significance was established at p < .05. All operations were carried out using IBM SPSS Statistics Version 20 (IBM Corporation, Armonk, NY).

Descriptive statistics pertaining to performance ratio on the four NBCOT practice test domains are summarized in Table 2. The mean performance ratio (ratio of correctly answered items in this domain) of Domain 1 was 75.82% ± 14.393%, indicating that this domain is not a domain of concern. The mean performance ratio for Domains 2 (65.34% ± 9.714%), 3 (68.24% ± 10.324%), and 4 (67.18% ± 11.794%), although close to the cutpoint (≤69%), were all within the ratio designation of domains of concern. The observed range for Domain 1 was the widest (69%); the observed ranges for Domains 2 (57%), 3 (57%), and 4 (55%) were slightly narrower and close.

Results of the logistic regression model for estimating the probability of first-time pass status on the NBCOT exam demonstrate that the model with all variables included has a –2 Log Likelihood (–2 LL) of 73.240, and the Hosmer–Lemeshow test generated a nonsignificant result (χ² = 7.777, p = .353; N = 71), indicating that the model is a good fit. The variance in NBCOT first-time pass status explained by the model ranged from 22% to 29% (Cox & Snell R² = 22%, Nagelkerke R² = 29%). An examination of the odds of first-time no-pass status indicated that it was associated with two variables: the ratio of correctly answered items in Domains 1 and 2. The beta weight associated with Domain 1 was β = −0.078, p = .032, and with Domain 2, β = −0.139, p = .020. The OR for Domain 1 was 0.925 and for Domain 2, 0.870. (OR values <1 indicate a negative relationship with first-time no-pass status). Both variables had odds with 95% confidence intervals that did not include 1, so for Domain 1, each 1% increase in performance ratio produced a 0.925 decrease in the odds of first-time no-pass status. For Domain 2, each 1% performance ratio increase produced a 0.870 decrease in the odds of first-time no-pass status. In terms of percentage change, for Domain 1, for each 1% performance ratio increase, the odds of first-time no-pass status decreased by 7.5% (100[0.925 − 1] = −7.5). For Domain 2, each 1% performance ratio increase produced a decrease of 13% (100[0.870 − 1] = −13) in the odds of first-time no-pass status (Table 3).

Results of the cross-classifying status cases show that 13 cases (19%) were misclassified by the model as follows: Four first-time pass status cases (6%) were classified as first-time no-pass status, and 9 first-time no-pass status cases (13%) were classified as first-time pass status cases. Thus, 92% of first-time pass status cases were correctly classified, as were 63% of first-time no-pass status cases. This multivariate prediction model achieved a prediction rate of 80%, which is 30% better than chance and gives each one of the two status cases a prior probability of 50%. The model was more effective for predicting first-time pass status cases than first-time no-pass status cases (Table 4).

The primary aim of this study was to apply NBCOT practice test performance ratios of correctly answered items in each domain area to develop a logistic regression model for estimating the probability of achieving first-time pass status on the NBCOT national certification exam. Contrary to our expectation, only the first two of the four domains were identified by the logistic regression model as having significant predicting effects. In addition, although this multivariate model achieved an overall prediction rate that was 30% better than chance, the estimates of the model’s explanatory contribution to the total variance in first-time pass status were modest.

Similarly modest findings were reported by previous studies assessing the predictive ability of tests designed by agencies in areas other than occupational therapy that were responsible for developing and administering national certification exams as a learning activity to help students gain familiarity with the exams and allow for student and program self-assessment and readiness. Ennulat and colleagues (2011)  found a positive correlation between the Physician Assistant National Certification Exam and the Physician Assistant Clinical Knowledge Rating and Assessment Tool, two tests developed by the Physician Assistant Education Association. The prediction equation generated by the analysis identified this test as the most accurately differentiating predictor between first-time pass and no-pass status groups on the Physician Assistant National Certification Exam. Dadian et al. (2002)  reported that the Mock Board Dental Hygiene Examination developed by the American Dental Association as a study aid for the Dental Hygiene National Board Exam accurately identified success or failure on the certification exam for 75% of the participants. Edenfield and Hansen (2000)  examined performance on the Dental Hygiene National Board Exam with a combination of grades and the Mock Board Exam. They found that the estimates of the model’s explanatory contribution to the total variance were low and that the combination of grades and, to a lesser extent, the Mock Board Exam affected candidates’ probability of passing the Dental Hygiene National Board Exam the first time.

Each of the four domains of the NBCOT exam carries a different relative weight in the formula used to compute the final NBCOT score. Yet the performance ratio for each domain area is reported on the NBCOT practice test in terms of absolute performance and not the relative weight carried by each domain. The total relative weight of the two significant predictors (Domains 1 and 2) amounts to 41% of the final score, whereas the total relative weight of the two nonsignificant predictors (Domains 3 and 4) amounts to 59%. A logistic regression prediction model (not reported here because of space constraints) using a performance ratio adjusted for the relative weight of each domain generated similar results. Thus, estimates of the probability of first-time pass status on the NBCOT exam using the absolute performance ratio and the relative performance ratio weight yielded similar results. Evidently, the 18% weight lead carried by the third and fourth domains was not reflected in the regression equations.

The tasks and skills entry-level clinicians routinely perform in each domain and the knowledge required to effectively perform these tasks are all embedded in the occupational therapy educational curriculum. Specific results of the research model show that students who were more likely to attain a first-time pass status could be identified by a high performance ratio on tasks and skills required for formulation of intervention plans (Domains 1 and 2). Exposure to these tasks and skills begins at an early stage in the curriculum and is incorporated into many courses. Consequently, students are provided with many opportunities to acquire and practice those tasks and skills throughout the curriculum, and the different performance ratios on these tasks mainly reflect differences in personal characteristics such academic aptitude.

Yet performance ratios on tasks and skills required for implementation of intervention plans (Domains 3 and 4) did not affect the odds of first-time pass status. Exposure to these tasks and skills begins at a later stage in the curriculum and is mainly limited to treatment courses. Consequently, students have fewer opportunities to acquire and practice these tasks and skills throughout the curriculum, and the different performance ratios on these tasks can be attributed to the limited opportunities the students had to acquire and practice these tasks and skills rather than to differences in personal characteristics such academic aptitude. To maximize students’ probability of attaining first-time pass status, occupational therapy faculty should consider ways to increase students’ exposure to these tasks and skills beyond treatment courses and at an earlier stage in the curriculum.

Successful graduation from occupational therapy educational programs has historically been measured by student graduating GPA (Auriemma, 2002; Avi-Itzhak & Kellner, 1995; Hansen-Suchy, 2011; Kirchner et al., 2000; Lysaght et al., 2009; Posthuma & Sommerfreund, 1985). In recent years, a growing body of research estimating the probability of first-time pass status on the national occupational therapy certification exam has emerged. In most of these studies, the probability of first-time pass status was conceptualized as the outcome measure rather than as a predictor for successful graduation from the program (Aldridge et al., 2010; Alzahrani, Thomson, & Bauman, 2007; Brown, Imel, Nelson, Hale, & Jansen, 2013; Dadian et al., 2002; Edenfield & Hansen, 2000; Ennulat et al., 2011; Hagemaster, 2001; Russell et al., 2007; Sloas et al., 2013; Williams & Hadfield, 2003).

In the current study, we chose to conceptualize successful graduation from occupational therapy educational programs by estimating the probability of first-time pass status on the NBCOT exam. This choice was based on the premise that graduating GPA measures students’ prelicensure level of clinical competence comprising the skills and knowledge that graduates from an occupational therapy educational program need to acquire and master to effectively deliver entry-level occupational therapy services. Yet to become a registered entry-level clinician, graduating students must successfully pass the NBCOT certification exam (Avi-Itzhak & Krauss, 2014). Thus, the ability to meet the academic and clinical requirements of a program is a potential predictor for estimating the probability of first-time pass status on the NBCOT exam. Successful graduation from an occupational therapy program is achieved when the student attains pass status, preferably first-time pass status, on the national certification exam.

Although the study’s prediction model achieved an overall prediction rate better than chance, the estimates of the model’s explanatory contribution to the total variance in first-time pass status were modest. The study’s conceptualization approach and results merit additional research. Next steps in research are to apply both graduating GPA and NBCOT practice test performance ratios on the four domains, as well as traditional cognitive and noncognitive admissions measures, to develop a logistic regression model for estimating the probability of first-time pass status on the NBCOT exam. The prediction equations generated by future studies would identify the configurations of effective predictors and their relative weight in these equations and the relative usefulness of the NBCOT practice test, a test developed by the same agency that develops the NBCOT certification exam, in predicting first-time pass status as a measure of successful program graduation.

These findings have educational implications because they can assist educators in identifying the most effective curricular models for instilling the prelicensure level of clinical competence that a student graduating from an occupational therapy educational program needs to acquire and master to attain a first-time pass status. These findings also have admissions policy implications because they can assist educators and administrators in formulating effective admissions-related strategies. Additional research on different occupational therapy educational programs is also needed to extend the limited external validity of the current study, which was performed with students in one occupational therapy program housed in a public urban institution.

Results of the logistic regression model for estimating the probability of first-time pass status on the NBCOT OTR certification exam demonstrate that the model with all four predictors is a good fit. The variance in NBCOT first-time pass status explained by the model ranged from 22% to 29%. The prediction model achieved a prediction rate of 80%, which is 30% better than chance. The model was more effective in predicting first-time pass status cases than first-time no-pass status cases. These results and the study’s conceptualization approach merit further research because of their interest to program directors, faculty, students, and members of admissions committees. Even though the current study targeted an occupational therapy educational program, this study can be used as an example for evaluating other health professions’ practice-based educational programs.

The author thanks the faculty who participated in and contributed to the seminar “Advanced Theory and Practice”: Andrea Krauss, Donald Auriemma, Sharon Faust, Beverly Horowitz, Lillian Kaplan, and Ivy Tilson.