Computer Adaptive Tests

In July 1996, the NMTCB began offering a computer adaptive test (CAT) for classification in association with ACT, Inc. The CAT for classification is designed to render a pass/fail decision. In a CAT of this type, examinees are NOT rank-ordered along a score scale in order to make a precise and accurate classification decision. In order to administer a CAT for classification, the items themselves are ranked at the decision point on the score scale according to their ability to classify accurately and quickly. Each item in the item pool is associated with the information on its difficulty (the proportion of examinees answering an item correctly) and discrimination (the ability of an item to distinguish between passing and failing individuals) levels. An item that has a difficulty level at or near the passing score and has good discrimination will be a better item for decision-making than another item that is too difficult or too easy or has little ability to discriminate between those examinees who should pass and those who should fail. ACT, Inc. psychometric staff obtained Item Response Theory (IRT) statistics for all items in the item pool.

A "classification" CAT is still adaptive in that an individual whose performance is significantly above or below the passing score will require fewer questions for classification. On the other hand, an individual whose performance is not clearly identified will need to answer more questions to demonstrate the required knowledge to pass the exam. The classification process will not be obvious to the examinee since there will be a certain number of unscored pretest questions that will be asked in order to obtain statistics on new questions for future use. For the classification CAT, the items that are administered to each and every candidate are the same type: there are no "difficult items for better examinees" or "easier items for poorer examinees." Each examinee answers a total of 90 items, and the items are different for each examinee.

Items for the classification CAT are selected from each of five content domains of nuclear medicine technology in the following approximate proportions: I. Radiation Physics & Detection (7%), II. Radiation Safety & Regulations (13%), III. Pharmaceutical & Radiopharmaceutical Agents (25%), IV. Instrument Operations & Quality Control (15%) and V. Clinical Procedures (40%).

The algorithm used in the classification CAT adjusts for differences in test form difficulty. For example, candidates that receive a CAT that is easier relative to the benchmark exam must answer more items correctly to receive a passing score. Conversely, if a candidate receives a set of items that is more difficult, they would be required to answer fewer questions correct to pass the exam. In essence, each CAT administered is equated so the passing level is appropriate for the set of items selected for administration to each candidate.

The CAT for classification, while providing a high degree of confidence in the pass/fail decision does not allow the same analysis of individual performance and subgroup performance obtained with the paper and pencil exam. In the past, because all examinees that sat at the same administration of a test took the same test, comparative information was obtained. Because the primary purpose of the CAT exam is to classify candidates as pass or fail, CAT for classification selects items that are optimal for minimizing errors in classification, a critical consideration in an occupation certification program. Candidates whose ability estimates are close to the passing score require more items to make a pass/fail decision; whereas, those that are clearly significantly above the pass/fail mark need fewer items. Candidates may not skip a test question, and are not permitted to return to an item. Candidates are permitted to change their answers before moving on, at which time the item is scored by the computer.