ChatGPT performance on pharmacology examination and board review questions: Implications for medical education and knowledge assessment

Abstract

Objectives: This study aimed to evaluate ChatGPT’s performance on pharmacology exam questions by assessing its accuracy in basic and clinical pharmacology, reasoning processes, and response consistency over time. Methods: A dataset of 583 multiple-choice questions from the Pharmacology Examination and Board Review (13th edition) was used. ChatGPT’s responses were evaluated for logical justification, use of internal question stem information, and integration of external knowledge. Statistical analyses, including chi-square and McNemar tests, assessed associations and changes in response accuracy over a four-week interval. Results: ChatGPT achieved 76.2% accuracy (444/583 questions), demonstrating logical reasoning in 97% of responses. Internal information was used in 99.7% of cases, while external information was incorporated in 98% of correct and 93.5% of incorrect responses (p = 0.008). Information errors were the most common reason for incorrect answers. A statistically significant improvement in accuracy upon re-evaluation (χ² = 37.3, p < 0.0001) was observed, suggesting potential temporal variation in performance. Conclusion: ChatGPT meets or exceeds typical passing standards in many educational settings, with evidence of improved response accuracy over time. These findings highlight its capabilities in processing pharmacological content, with potential implications for future research into AI-assisted educational tools.