Key validation is a process whereby examination items (questions) are reviewed to ensure they are fair. After the administration of an examination, an item analysis is performed by a psychometrician. This analysis provides information about how many examinees answered each item correctly, which options (A, B, C, or D) were selected most often, and which items positively discriminated higher from lower performers. This analysis doesn’t in and of itself identify an unfair or “flawed” item, but it is used to identify possibly flawed items.
For example, imagine an item for which the answer (key) is A. Now, imagine that in the item analysis, it was determined that 30% of examinees selected A, 50% of examinees selected B, 10% of examinees selected C, and 10% of examinees selected D. We are also able to see that the individuals that selected option B tended to have high scores on the examination as a whole (as compared to those examinees who selected C and D. We would say option B is “positively discriminating”). This evidence would cause the psychometrician to send the examination item to one or more ophthalmologists (subject matter experts or “SMEs”) for a fairness review. The directions to the SME(s) would be to review the item carefully, especially option B which appears to possibly be a correct answer. If the SME determines that option B is also correct, all examinees that selected options A or B will be given credit for that item. Although it is rare, sometimes all examinees will receive credit for an item if it is not clear at all. In a case like this, no examinee will be penalized for incorrectly answering the item.
Key validation is a final step in the fairness review, but few items actually make it onto an examination that are flawed. Every item goes through several rounds of SME review prior to being on an examination. However, due to rapid changes in practice, occasionally an item that was fair a few months ago is no longer fair at the time of the examination. Ultimately, key validation provides a final fairness check on all examination items.