TY  - JOUR
AU  - Kanzow, Philipp
AU  - Schmidt, Dennis
AU  - Herrmann, Manfred
AU  - Wassmann, Torsten
AU  - Wiegand, Annette
AU  - Raupach, Tobias
PY  - 2023
DA  - 2023/3/27
TI  - Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods
JO  - JMIR Med Educ
SP  - e43792
VL  - 9
KW  - dental education
KW  - education system
KW  - educational assessment
KW  - educational measurement
KW  - examination
KW  - k of n
KW  - Kprim
KW  - K’
KW  - MTF
KW  - Multiple-True-False
KW  - Pick-N
KW  - scoring
KW  - scoring system
KW  - Type X
KW  - undergraduate
KW  - undergraduate curriculum
KW  - undergraduate education
AB  - Background: Scoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data. Objective: This study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6). Methods: Items and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05). Results: We analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977). Conclusions: Educators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees’ scores and overall examination results. 
SN  - 2369-3762
UR  - https://mededu.jmir.org/2023/1/e43792
UR  - https://doi.org/10.2196/43792
UR  - http://www.ncbi.nlm.nih.gov/pubmed/36841970
DO  - 10.2196/43792
ID  - info:doi/10.2196/43792
ER  -