Note from the CEO: The ABO written and oral examinations are created through the dedicated collaboration of hundreds of volunteers along with ABO Directors and the Board’s office staff. A consistent comment from first-time volunteers is, “I had no idea how difficult it is to write good questions and how much work goes into this process.” To allow diplomates and candidates a glimpse “behind the curtain,” the ABO’s psychometrician, Sarah Schnabel, will contribute regular postings to Diplomate Digest. We hope that you will find them informative. G.B. Bartley, M.D.
Q: How does the American Board of Ophthalmology verify that examinations and assessments are reliable, valid, and fair to the test taker?
A: The American Board of Ophthalmology has administered certification examinations for more than 100 years—longer than any other medical specialty board in history. We set standards for these examinations based on the subject matter expertise of practicing ophthalmologists with the guidance of a psychometrician. A psychometrician is someone who practices the science of educational and psychological measurement, in other words, testing. As the ABO’s full-time psychometrician, I work closely with exam development staff and volunteers to measure and ensure the validity, reliability, and fairness of examinations. My role is to help subject matter experts define the knowledge, skills, and abilities (KSAs) that each exam is designed to measure. I also create test specifications, participate in the item-writing process, and conduct standard setting studies. While overseeing these areas, my chief objective is to ensure that each part of the process conforms to testing industry standards, such as those in the Standards for Educational and Psychological Testing, (which happens to be my favorite book). These standards are considered the primary source for test publishers, test takers, and test users. Psychometricians often joke that the reason we don’t have a board certification exam in our profession is because we’d never agree on how it should be designed or scored!
Do you have questions about psychometrics? Please submit them in the comments box below or send me an email and I’ll answer them in a future column. Thank you!