Diplomates often wonder “who writes the questions we see on tests?” Board-certified ophthalmologists volunteer annually to develop questions (items) for the ABO examinations. The process usually begins in February, when the volunteer item-writers receive training on developing good items. Psychometric research has found that the best way to capture someone’s knowledge in a multiple-choice question is to keep the item focused on a single topic, ask a straightforward question, and provide four options (one of which is the correct answer) that are clear and concise. We do our best not to try to trick examinees with wording or ask a question in a way that is ambiguous or requires re-reading multiple times. All items are mapped to the examination content outline or test blueprint, which is a list of the topic areas that appear on each examination. This requirement prevents item writers from developing questions on topics that were not deemed relevant and important through group consensus.
Item writers remotely develop approximately 10-15 items between February and late March. There are approximately 8 or 9 volunteer item writers that focus on developing content within their practice emphasis area (e.g., neuro-ophthalmology). The ABO examinations are divided into ten practice areas, which results in approximately 80-90 volunteer item writers annually. After this initial item development, an editorial review takes place and item writers are asked to re-review and revise their items. In early summer, item writers are given access to their peers’ drafted items to provide feedback. In mid-summer or early fall, the item writers come together for an in-person meeting to review and approve all of the items drafted. While it may seem like the process could be complete before this meeting, we have found that the in-person peer discussion has been invaluable in fine-tuning the items and identifying and correcting flaws. After this meeting, ABO staff completes a final editorial review of all items. These items are kept in a secure, digital item bank. When staff begins the process of developing an examination form, items are selected from the bank one by one to ensure that the content is representative of the full examination blueprint. The final step in the review process is subspecialist ophthalmologist review of the examination form (for example, a cataract specialist reviews the cataract questions on the WQE, a glaucoma specialist reviews the glaucoma items, etc.). This way, if an item was written two years ago but practice has changed since then, the subject-matter expert reviewer can remove the item and replace it with a better item. There is yet another layer of item review after an examination is administered, which you can read about here.