Assessing medication counseling skills during transitioning from pre-clinical to clinical years among medical students: using generalizability theory to optimize reliable pharmacology exam design

Abstract

Background: Physician medication counseling (MC) skills are crucial for improving adherence, treatment outcomes, and minimizing preventable medical errors. However, studies on MC among medical students, particularly in the pre-clinical stage, are scarce. Objective: This study analyzed an MC examination to identify common errors and determine the number of assessment items and questions needed for reliable evaluation. Methods: Ninety-five third-year students took a written examination on MC at the end of their third year. The exam included 10 questions on common drugs used in different systems, each with five error types (item) that varied per question. There were eight assessment items: administration time, adverse drug reaction, drug interval, indication, drug interaction, compliance, dosage form, and drug-specific information. Each item scored 2 for 'Fully correct,' 1 for 'Partially correct,' and 0 for 'Incorrect.' A Kruskal-Wallis test was used to compare the scores for each item, and a generalizability study was conducted to determine the sources of variance and the optimal number of items and questions. Results: The Cronbach's alpha for the exam is 0.88. The unidimensionality of the questions was confirmed (Eigenvalue1:Eigenvalue2=4.95:0.97, λ=0.52-0.82). The median (IQR) score is 52 (40-63) out of 100. Significant differences were found in the mean rank of each item, H(7)=195.13, p<0.001. Items with relatively high medians (IQR) included dosage form (1.33 [1.00-1.67]) and drug interval (1.38 [1.13-1.50]), while drug interaction (1.00 [0.00-1.00]), compliance (0.80 [0.00-1.00]), and specific information (0.40 [0.00-0.80]) were lower. Most of the variance is attributable to students (11.60%), and items nested within the question are 20.70%. The current study had a Phi-coefficient of 0.85; at least eight questions are needed for reliable assessment using five items (Phi-coefficient = 0.82). Whereas utilizing all 8 items, 6 questions are required (Phi-coefficient = 0.84). For optimization, at least six questions using six items are needed for reliable assessment (Phi-coefficient = 0.80). Conclusion: This study identified MC errors and highlighted areas for improvement before transitioning from pre-clinical to clinical years. Moreover, most variance is due to items nested within questions, indicating that different types of errors should be assessed in each question, which reflects real-life counseling challenges.