Thank you to the 972 early adopters who helped the American Board of Ophthalmology to pilot our new Quarterly Questions program. In the first three months of this initiative, participants answered 17,626 questions related to comprehensive ophthalmology and provided valuable feedback that the ABO will use to improve the Quarterly Questions pilot. New questions for the third quarter of 2017 are now available at https://questions.abop.org. (If you haven’t explored this optional pilot, now is a great time to jump in.)
The goal of this pilot is to develop a longitudinal assessment program that will ultimately replace the Demonstration of Ophthalmic Cognitive Knowledge (DOCK) Exam. We are committed to delivering an exceptional user experience, providing a useful and valuable activity for practicing ophthalmologists, and achieving psychometric validity comparable to (or better than) the traditional test format.
Here’s how the pilot performed in its first quarter:
User Experience
Overall, 96% of participants told us they would recommend participation in Quarterly Questions to a colleague.
The average time required to answer each question was 30.2 seconds with 98% of questions answered within the allotted 60-second timeframe. However, only about half of diplomates said they felt they had enough time to answer each question. We will continue to monitor the pros and cons of the timeframe, especially its effect on the psychometric validity of the assessment.
92% of participants reported no technical issues. In response to feedback, the ABO is working to improve the loading of images and enhancing the platform’s responsiveness to different devices.
Value/Use for Practicing Ophthalmologists
About 8 out of every 10 participants said the questions were appropriately challenging.
More than 90% said they did not review any study materials prior to beginning the activity.
At least 8 out of 10 appreciated knowing how their performance compared to the average scores of their peers.
When a question was answered incorrectly, 95% of participants found the question’s key point helpful and 93% appreciated the critique.
Psychometric Validity
The ABO employs a full-time psychometrician (test development scientist) who guides the development of our examinations to ensure they meet generally accepted standards. So far, our psychometrician is pleased with the way this new activity is performing, with a formal analysis slated to follow at the end of the year. In the first quarter, diplomates correctly answered an average of 3 of every 4 questions.
Given positive feedback so far, the ABO plans to introduce subspecialty content in 2018, allowing diplomates to customize their Quarterly Questions experience to their area(s) of practice.
Scroll through the quotes below to see what participants have said about the Quarterly Questions Pilot Program so far. For more information about the pilot, visit https://abop.org/maintain-certification/pilot-programs/#QQ, or, to begin or resume this activity, visit https://questions.abop.org/.
Dr. Lauer is a director of the American Board of Ophthalmology. He is the Kenneth C. Swan Professor of Ophthalmology at the Oregon Health and Sciences University School of Medicine and vice-chair for Education and chief of the Retina-Vitreous Division at Casey Eye Institute. Share Quarterly Questions Pilot feedback with Dr. Lauer via QuarterlyQuestions@abop.org.
