Background

JME

JMIR Med Educ

JMIR Medical Education

2369-3762

JMIR Publications

Toronto, Canada

v9i1e44084

37001510

10.2196/44084

Review

Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods

Leung

Tiffany

Lindner

Marlit Annalena

Feofanova

Elena

Kanzow

Amelie Friederike

MEd 1

https://orcid.org/0000-0002-1536-3248

Schmidt

Dennis

MSc 2

https://orcid.org/0000-0002-9828-5299

Kanzow

Philipp

MSc, Dr rer medic, PD Dr med dent 2

Department of Preventive Dentistry, Periodontology and Cariology University Medical Center Göttingen

Robert-Koch-Strasse 40

Göttingen, 37075

Germany 49 551 3960870 49 551 3960869 philipp.kanzow@med.uni-goettingen.de

https://orcid.org/0000-0002-2169-561X

1 Study Deanery University Medical Center Göttingen

Göttingen

Germany 2 Department of Preventive Dentistry, Periodontology and Cariology University Medical Center Göttingen

Göttingen

Germany

Corresponding Author: Philipp Kanzow philipp.kanzow@med.uni-goettingen.de

2023

19 5 2023

e44084

5 11 2022 5 3 2023 6 3 2023 31 3 2023

©Amelie Friederike Kanzow, Dennis Schmidt, Philipp Kanzow. Originally published in JMIR Medical Education (https://mededu.jmir.org), 19.05.2023.

2023

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on https://mededu.jmir.org/, as well as this copyright and license information must be included.

Background

Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees’ responses have to be analyzed and scored to derive information about examinees’ true knowledge.

Objective

The aim of this paper is to compile scoring methods for individual single-choice items described in the literature. Furthermore, the metric expected chance score and the relation between examinees’ true knowledge and expected scoring results (averaged percentage score) are analyzed. Besides, implications for potential pass marks to be used in examinations to test examinees for a predefined level of true knowledge are derived.

Methods

Scoring methods for individual single-choice items were extracted from various databases (ERIC, PsycInfo, Embase via Ovid, MEDLINE via PubMed) in September 2020. Eligible sources reported on scoring methods for individual single-choice items in written examinations including but not limited to medical education. Separately for items with n=2 answer options (eg, alternate-choice items, single true-false items) and best-answer items with n=5 answer options (eg, Type A items) and for each identified scoring method, the metric expected chance score and the expected scoring results as a function of examinees’ true knowledge using fictitious examinations with 100 single-choice items were calculated.

Results

A total of 21 different scoring methods were identified from the 258 included sources, with varying consideration of correctly marked, omitted, and incorrectly marked items. Resulting credit varied between –3 and +1 credit points per item. For items with n=2 answer options, expected chance scores from random guessing ranged between –1 and +0.75 credit points. For items with n=5 answer options, expected chance scores ranged between –2.2 and +0.84 credit points. All scoring methods showed a linear relation between examinees’ true knowledge and the expected scoring results. Depending on the scoring method used, examination results differed considerably: Expected scoring results from examinees with 50% true knowledge ranged between 0.0% (95% CI 0% to 0%) and 87.5% (95% CI 81.0% to 94.0%) for items with n=2 and between –60.0% (95% CI –60% to –60%) and 92.0% (95% CI 86.7% to 97.3%) for items with n=5.

Conclusions

In examinations with single-choice items, the scoring result is not always equivalent to examinees’ true knowledge. When interpreting examination scores and setting pass marks, the number of answer options per item must usually be taken into account in addition to the scoring method used.

alternate-choice best-answer education education system educational assessment educational measurement examination multiple choice results scoring scoring system single choice single response scoping review test testing true/false true-false Type A

Introduction

Multiple-choice items in single-response item formats (ie, single-choice items) require examinees to mark only 1 answer option or to make only 1 decision per item. The most frequently used item type among the group of single-choice items is the so-called best-answer items. Here, examinees must select exactly 1 (ie, the correct or most likely) answer option from the given answer options [1]. Often, best-answer items contain 5 answer options, although the number of answer options might vary (n≥2). Items with exactly 2 answer options are also referred to as alternative items (ie, alternate-choice items) [2]. In addition, single true-false items belong to the group of single-choice items. Examples of the mentioned single-choice items as well as alternative designations are shown in Figure 1.

Single-choice items have been used for more than 100 years to test examinees’ knowledge. The use of these items began among US school pupils, who were given alternate‑choice or best-answer items [3] or single true-false items [4] as a time-saving alternative to conventional open-ended questions (ie, essay-type examinations). Because of their character of only allowing clearly correct or incorrect responses from examinees, multiple-choice examinations were also called objective type examinations [5]. The term new type examinations was coined to distinguish them from previously commonly used open-ended questions [5,6].

The use of multiple-choice items did not remain exclusive to the setting of high schools but also extended to examinations in university contexts [7] and postgraduate medical education [8,9]. Today, multiple-choice items are frequently used in examinations of medical and dental students (eg, within the United States Medical Licensing Examination). Besides their usage in individual medical or dental programs, different multiple-choice item types found their way into examinations for medical students by the National Board of Medical Examiners [10]: within the context of single-choice items, those with n=5 were particularly used and referred to as Type A items.

Examinations aim at assessing examinees’ ability (ie, examinees’ true knowledge [k]) regarding predefined learning objectives. The downside when using multiple-choice examinations is that examinees might also mark an item correctly by guessing or by identifying the correct answer option through recognition. Thus, an active knowledge reproduction does not necessarily take place, and correct responses are not necessarily resulting from examinees’ true knowledge.

To grade examinees or to decide about passing or failing a summative examination based on a minimum required level of true knowledge, scoring algorithms are used to transfer examinees’ responses (ie, marking schemes) into a score. To assess examinees’ true knowledge, the obtained scores must either be reduced by the guessing factor, negative points (ie, malus points) must be assigned for incorrectly marked items, or the pass mark (ie, the corresponding cutoff score for the desired true knowledge cutoff value) must be adjusted based on the guessing probability [11]. The guessing probability for examinees without any knowledge (k=0, blind guessing only) amounts to 20% for single-choice items with n=5 and to 50% for alternate-choice items and single true-false items with n=2. Consequently, examinees without any knowledge score 20% or 50% of the maximum score on average, respectively [11]. However, it can be assumed that most examinees have at least partial knowledge (0<k<1) and that an informed guessing with remaining partial uncertainty occurs in most cases.

Since the introduction of multiple-choice items, numerous scoring methods have been described in the literature and (medical) educators are advised to choose an appropriate scoring method based on an informed decision. Therefore, the aim of this scoping review is (1) to map an overview of different scoring methods for individual single-choice items described in the literature, (2) to compare different scoring methods based on the metric expected chance score, and (3) to analyze the relation between examinees’ true knowledge and expected scoring results (averaged percentage score).

Figure 1

Examples of 3 different multiple-choice items in single-choice format and alternative designations used in the literature (no claim to completeness).

Methods Systematic Literature Search

The literature search was performed according to the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) checklist [12]. The checklist is available as Multimedia Appendix 1. As this review did not focus on health outcomes, the review was not registered at PROSPERO (International Prospective Register of Systematic Reviews) prior to its initiation.

Eligibility Criteria

Potentially eligible sources were scientific articles, books, book chapters, dissertations, and congress abstracts reporting scoring methods for individual single-choice items in written examinations including but not limited to medical examinations. Scoring methods for item groups and scoring on examination level (eg, with different weighting of individual items, with mixed item types, or considering the total number of items per examination) were not assessed. Further, scoring methods that deviate from the usual marking procedure (ie, a single choice of marking exactly 1 answer option per item) were not considered. These include, for example, procedures that assess the confidence of examinees in their marking (eg, confidence weighting), let examinees select the incorrect answer options (eg, elimination scoring), let examinees narrow down the correct answer option (eg, subset selection), or allow for the correction of initially incorrectly marked items (eg, answer-until-correct). No further specifications were made regarding language, quality (eg, minimum impact factor), or time of publication.

Information Sources

Four databases (ERIC, PsycInfo, Embase via Ovid, and MEDLINE via PubMed) were searched in September 2020. The search term was composed of various designations for single-choice items as well as keywords with regard to examinations. It was slightly adapted according to the specifications of the individual databases. The respective search terms for each database can be found in Table 1.

Table 1

Search terms used for each of the 4 databases.

Database	Search term
ERIC	(“single choice” OR “alternate choice” OR “single response” OR “one-best-answer” OR “single best response” OR “true-false” OR “Typ A”) AND (item OR items OR test OR tests OR testing OR score OR scoring OR examination OR examinations)
PsycInfo	(“single choice” OR “alternate choice” OR “single response” OR “one-best-answer” OR “single best response” OR “true-false” OR “Typ A”) AND (item OR items OR test OR tests OR testing OR score OR scoring OR examination OR examinations)
Embase via Ovid	((“single choice” or “alternate choice” or “single response” or “one-best-answer” or “single best response” or “true-false” or “Typ A”) and (item OR items or test or tests or testing or score or scoring or examination or examinations)).af.
MEDLINE via PubMed	(“single choice”[All Fields] OR “alternate choice”[All Fields] OR “single response”[All Fields] OR “one-best-answer” OR “single best response” OR “true-false”[All Fields] OR “Typ A”[All Fields]) AND (“item”[All Fields] OR “items”[All Fields] OR “test”[All Fields] OR “tests”[All Fields] OR “testing”[All Fields] OR “score”[All Fields] OR “scoring”[All Fields] OR “examination”[All Fields] OR “examinations”[All Fields])

Selection of Sources

Literature screening, inclusion of sources, and data extraction were independently performed by 2 authors (AFK and PK). First, the titles and abstracts of the database results were screened. Duplicate results as well as records being irrelevant to the research question were sorted out. For books and book chapters, however, different editions were included separately. In a second step, full-texts sources were screened, and eligible records were included as sources. In addition, the references of included sources were searched in an additional hand search for further, potentially relevant sources. After each step, the results were compared, and any discrepancies were discussed until a consensus was reached. Information with regard to the described scoring methods was extracted using a piloted checklist.

Data Extraction

The following data were extracted from included sources using a piloted spreadsheet if reported: (1) name of the scoring method, (2) associated item type, and (3) algorithm for calculating scores per item. The mathematical equations of each scoring method were adjusted to achieve normalization of scores up to a maximum of +1 point per item if necessary.

Data Synthesis

For all identified scoring methods, the expected scoring results in case of pure guessing were calculated for single-choice items with n=2 and n=5 answer options, respectively [13]. The expected chance score is described in the literature as a comparative metric of different scoring methods [11,13-15]. For its calculation, examinees without any knowledge (k=0) are expected to always guess blindly and thus achieve the expected chance score on average.

In addition, expected scoring results for varying levels of k (0≤k≤1) were calculated. For examinees with partial knowledge (0<k<1), a correct response can be attributed to both partial knowledge and guessing, with the proportion of guessing decreasing as knowledge increases. By contrast, examinees with perfect knowledge (k=1) always select the correct answer option without the need for guessing [11].

Examinees were expected to answer all items, and it was supposed that examinees were unable to omit individual items or that examinees do not use an omit option. Furthermore, all items and answer options were assumed to be of equal difficulty and to not contain any cues. The calculation of the expected scoring result is shown in the following equation:

where f are the credit points awarded for a correctly marked item (i=1) or an incorrectly marked item (i=0) depending on the scoring method used; k is the examinees’ true knowledge [0≤k≤1]; n is the number of answer options per item; x=1 if the correct answer option is selected by true knowledge, otherwise x=0; in the equation shown, 0⁰ is defined as 1.

MATLAB software (version R2019b; The MathWorks) was used to calculate the relation between examinees’ true knowledge and the expected scoring results using fictitious examinations consisting of 100 single-choice items (all items with either n=2 or n=5).

Results Overview

Within the literature search, a total of 3892 records were found through database search. Of these, 129 sources could be included. A further 129 sources were identified from the references of the included sources by hand search. The entire process of screening and including sources is shown in Figure 2. Reasons for exclusion of sources during full-text screening are given in Multimedia Appendix 2.

The included sources describe 21 different scoring methods for single-choice items. In the following subsections, all scoring methods are described with their corresponding scoring formulas for calculating examination results as absolute scores (S). In addition, an overview with the respective scoring results for individual items as well as alternative names used in the literature is presented in Table 2. All abbreviations used throughout this review are listed at the end of this review.

Figure 2

Flow diagram of systematic literature search.

Table 2

Identified scoring methods and algorithms for single-choice items.

Method number and sources	Scoring method	Algorithm^a-e
1 [5,6,16-172]	0-1 score [167] Zero-one scoring [146] Binary scoring [146] Dichotomous scoring [105,114] All-or-none scoring [166] Number-right (NR) scoring [6,20,21,24,25,27,29-31,37,39,50,54,56,66,67,71,73,76,79,80,85,87,95,97,99,100,111,128,132,140,145,147,153,157,160,164] Number of right (NR) rule [139] No. right score (No Rt) [42] NC^f scoring [144] Rights score [72,82,92] R method [24,29,39] Number correct scoring [101,106,114,124,138,151,154,155] Percentage-correct scoring [165] Raw score [44-46,48,51,54,57,68,86,102,118,125,131,135] Score=rights [23,24] Uncorrected score [91,122,137] Conventional scoring [98] Rights-only score [62,87] 3 right minus 0 wrong [17]	f=1 (if i=1)f=0 (otherwise)
2 [37,41,46,53,58,60, 65,67,79-81,87,91, 98,111,122,137,173-180]	Formula scoring [67] Omission-formula scoring [79] Omit-correction [180] Positive scoring rule [139] Adjusted score [91]	f=1 (if i=1)f=1/n (if o=1)f=0 (otherwise)
3 [154]	Fair penalty [154]	f=1 (if i=1)f=0 (if o=1)f = 1 – 1/n (otherwise)
4 [181]	N/A^g	f = 1/(n – 1) (if i=1)f=0 (if o=1)f=0 (otherwise)
5 [80,100,182]	N/A	f=1 (if i=1)f=0 (if o=1)f = –1/[2 (n – 1)] (otherwise)
6 [5,23-29,34,37,44, 46,48,50,51,53-57, 59-62,64,65,67,68, 70,71,74,75,79-81, 85-88,91,92,98-101, 105,106,111,113,120,122, 124-126,128,130,134,135, 137-139,144,145,160,169, 173-179,182-225]	Formula scoring [67,85,92,101,128,160,225] Conventional-formula scoring [79] Conventional correction-for-guessing formula [80,213] Conventional correction formula [201] “Neutral” counter-marking [88] CG^h scoring [144] Negative marking [130,145] Logical marking [130] Correction for blind guessing (CFBG) [135] Correction for guessing (CFG) formula [50,51,56,57,62,71,86,87,99,101,105,106,113,122,124,137,176,179,195,199,204,223] Correction for chance formula [56,87,174,188] Discouraging guessing [138] Rights minus wrongs correction [98] Corrected score [37,48,55,59,68,91] Classical score [207] Mixed rule [139]	f=1 (if i=1)f=0 (if o=1)f = –1/(n – 1) (otherwise)
7 [226]	N/A	f = 1/(n – 1) (if i=1)f=0 (if o=1)f = –1/(n – 1) (otherwise)
8 [41]	N/A	f = (n – 1)/n (if i=1)f=0 (if o=1)f = –1/n (otherwise)
9 [6,48,62,88,224,227,228]	3 right-wrong [6] Negative marking [228]	f=1 (if i=1)f=0 (if o=1)f=–1/3 (otherwise)
10ⁱ [229]	N/A	f=1 (if i=1)f=0 (if o=1)f=–0.48 (otherwise)
11 [18,23,41,62,69,224,229-234]	N/A	f=1 (if i=1)f=0 (if o=1)f=–0.5 (otherwise)
12ⁱ [229,231]	N/A	f=1 (if i=1)f=0 (if o=1)f=–0.6 (otherwise)
13 [4,6,16-19,21-25, 29-33,38,39,42,43, 45,49,52,55,69,72, 76,82,110,130,132,140, 143,154,157,164,172,190, 193,215,216,219,229,232, 233,235-267]	Formula scoring [157,164] Correct-minus-incorrect score [267] C-I score [132] R-W method [23,24,29,30,32,38,39,42,76,243,245,246,249,259] Number right minus number wrong method [39,45] Right-minus-wrong method [6,21,23,25,30,31,42,72,82,236,244,247] Rights minus wrongs method [29,253,254,256,258] Right-wrong [266] T-F formula [260] Guessing penalty [154] Correction-for-guessing [76,128] Negative marking [140] Logical marking [130] 1 right minus 1 wrong [17] Penal guessing formula [55] Corrected score [265]	f=1 (if i=1)f=0 (if o=1)f=–1 (otherwise)
14ⁱ [249,268]	N/A	f=1 (if i=1)f=0.7 (if o=1)f=–1 (otherwise)
15ⁱ [186]	N/A	f=1 (if i=1)f=0.7 (if o=1)f=–1.1 (otherwise)
16 [20]	N/A	f=1 (if i=1)f=0 (if o=1)f = –n/(n – 1) (otherwise)
17ⁱ [203,259]	N/A	f=1 (if i=1)f=0 (if o=1)f=–1.5 (otherwise)
18ⁱ [203]	N/A	f=1 (if i=1)f=0 (if o=1)f=–1.8 (otherwise)
19 [6,17,20,21,49,75,203,253,268-270]	Right – 2 wrong [6] 1 right minus 2 wrong [17] Rights minus two times wrongs [253] r-2w [253]	f=1 (if i=1)f=0 (if o=1)f = –2/(n – 1) (otherwise)
20ⁱ [17,41]	1 right minus 3 wrong [17]	f=1 (if i=1)f=0 (if o=1)f=–3 (otherwise)
21^j [259]	N/A	f=1 (if i=1)f=0 (if o=1)f=–62/38 (if i=0 and t_m=1)f=–38/62 (if i=0 and t_m=0)

^af: resulting score per item.

^bi=1 if the item was marked correctly; otherwise i=0.

^cn: number of answer options per item (n≥2).

^do=1 if the item was omitted; otherwise o=0.

^et_m=1 if the statement is true; otherwise t_m=0.

^fNC: number correct.

^gN/A: not applicable (ie, no explicit name was previously introduced in literature).

^hCG: correct for guessing.

ⁱOnly described for n=2.

^jOnly described for single true-false items.

Scoring Methods Without Malus Points (0 to a Maximum of +1 Point per Item) Method 1

One credit point is awarded for a correct response. Therefore, the examination result as absolute score (S) corresponds to the number of correct responses (R). No points are deducted for incorrect responses (W). The formula is S = R.

Method 2

One credit point is awarded for a correct response. In addition, 1/n credit points per item are awarded for each omitted item (O). No points are deducted for incorrect responses. The formula is S = R + O/n. This scoring method was first described by Lindquist [37] in 1951.

Method 3

One credit point is awarded for a correct response. For incorrect responses, 1 – 1/n credit points are awarded. The formula is S = R + (1 – 1/n)W. This scoring method was first described by Costagliola et al [154] in 2007 and named fair penalty by the authors. However, the term penalty is misleading because no points are deducted in case of incorrect responses.

Method 4

For each correct response, 1/(n – 1) credit points are awarded. Omitted items and incorrect responses do not affect the score. The formula is S = R/(n – 1). For example, 1 credit point is awarded for a correct response on single-choice items with n=2 (ie, alternate-choice items, single true-false items) but only 0.25 credit points are awarded for a correct response on best-answer items with n=5. This scoring method was first described by Foster and Ruch [181] in 1927.

Scoring Methods With Malus Points (Maximum –1 to +1 Point per Item) Method 5

One credit point is awarded for a correct response. For incorrect responses, 1/[2 (n – 1)] points are deducted. The formula is S = R – W/[2 (n – 1)]. This scoring method was first described by Little [182] in 1962.

Method 6

One credit point is awarded for a correct response. For incorrect responses, 1/(n – 1) points are deducted. The formula is S = R – W/(n – 1). This scoring method was first described by Holzinger [183] in 1924. For items with n=2, methods 6 and 13 result in identical scores; for items with n=4, methods 6 and 9 result in identical scores.

Method 7

For each correct response, 1/(n – 1) credit points are awarded. For an incorrect response, 1/(n – 1) points are deducted. The formula is S = (R – W)/(n – 1). This scoring method was first described by Petz [226] in 1978.

Method 8

For each correct response, (n – 1)/n credit points are awarded. For an incorrect response, 1/n points are deducted. Omissions do not affect the score. The formula is S = [(n – 1)/n]R – W/n. As a result, examinees achieve only 0.5 credit points for each correct response on single-choice items with n=2 and 0.8 credit points for each correct response on best-answer items with n=5. This scoring method was first described by Guilford [41] in 1954.

Method 9

One credit point is awarded for a correct response. For incorrect responses, 1/3 points are deducted. The formula is S = R – (1/3)W. Originally, this scoring method was described by Paterson and Langlie [6] in 1925 with the formula S = 3R – W for items with n=2 only. Later, the scoring method was also described for single-choice items with more answer options [88,203]. For items with n=4, methods 6 and 9 give identical results.

Method 10

One credit point is awarded for a correct response. For incorrect responses, 0.48 points are deducted. The formula is S = R – 0.48W. This scoring method was first described by Gupta and Penfold [229] in 1961 for single-choice items with n=2.

Method 11

One credit point is awarded for a correct response. Half a point is deducted for incorrect responses. The formula is S = R – 0.5 W. This scoring method was first described in 1924 by Brinkley [18] and Asker [230] for single-choice items with n=2, but was later also used for single-choice items with more answer options.

Method 12

One credit point is awarded for a correct response. For incorrect responses, 0.6 points are deducted. The formula is S = R – 0.6W. This scoring method was first described by Gupta [231] in 1957 for single-choice items with n=2.

Method 13

One credit point is awarded for a correct response. One point is deducted for incorrect responses. The formula is S = R – W. For items with n=2, methods 6 and 13 result in identical scores. This scoring method was first described by McCall [4] in 1920 for single-choice items with n=2, but was later also used for single-choice items with more answer options.

Method 14

This scoring method results in 1 credit point for a correct response, 0.7 credit points for an omitted item, and –1 point for an incorrect response. The formula is S = R + 0.7O – W. This scoring method was first described by Staffelbach [268] in 1930 for single-choice items with n=2.

Scoring Methods With Malus Points (Maximum –3 to +1 Points per Item) Method 15

This scoring method results in 1 credit point for a correct response, 0.7 credit points for an omitted item, and –1.1 points for an incorrect response. The formula is S = R + 0.7O – 1.1W. This scoring method was first described by Kinney and Eurich [186] in 1933 for items with n=2.

Method 16

One credit point is awarded for a correct response. For an incorrect response, n/(n – 1) points are deducted. The formula is S = R – nW/(n – 1). This scoring method was first described by Miller [20] in 1925. For items with n=2, methods 16 and 19 result in identical scores.

Method 17

For an incorrect response, 1.5 times as many points are deducted as credit points are awarded for a correct response. The original scoring formula is S = 2R – 3W. If a maximum of 1 credit point is awarded per item, 1 credit point is awarded for a correct response and 1.5 points are deducted for an incorrect response. This results in the following scoring formula: S = R – 1.5W. This scoring method was first described by Cronbach [259] in 1942 for items with n=2.

Method 18

One credit point is awarded for a correct response. For an incorrect response, 1.8 points are deducted. The scoring formula is S = R – 1.8W. This scoring method was first described by Lennox [203] in 1967 for items with n=2.

Method 19

One credit point is awarded for a correct response. For an incorrect response, 2/(n – 1) points are deducted. The formula is S = R – 2W/(n – 1). This scoring method was first described by Gates [269] in 1921 with the scoring formula S = R – 2W for items with n=2. Later, the scoring formula was also described for single-choice items [203,270]. In case of items with n=2, methods 16 and 19 result in identical scores.

Method 20

One credit point is awarded for a correct response. Three points are deducted for an incorrect response. The formula is S = R – 3W. This method was first described by Wood [17] in 1923 for items with n=2.

Specific Scoring Methods for Single True-False Items Method 21

One credit point is awarded for correctly identifying the statement of true-false single items as true or false. If the statement presented is marked incorrectly, 62/38 points are deducted on true statements (W_t, incorrectly marked as false), but only 38/62 points are deducted on false statements (W_f, incorrectly marked as true). The scoring formula is S = R – (62/38)W_t – (38/62)W_f. This scoring method was first described by Cronbach [259] in 1942 for single true-false items and differentiates in the scoring of incorrectly marked true/false statements.

Expected Chance Scores of the Identified Scoring Methods

The expected chance scores of examinees without any knowledge (k=0) vary between –1 and +0.75 credit points per item for single-choice items with n=2. For single-choice items with n=5, expected chance scores show a larger variability. Here, the expected chance scores vary between –2.2 and +0.84 credit points per item, depending on the selected scoring method. A detailed list is shown in Table 3.

Table 3

Overview of scoring results for single-choice items with either n=2 or n=5 answer option.

Method number	Scoring formula^a-f	n^g=2				n=5
Method number	Scoring formula^a-f	Credit for incorrect responses^h	Credit for correct responsesⁱ	Expected chance score	Credit for incorrect responses^h		Credit for correct responsesⁱ	Expected chance score
1	S = R	0	1	0.50	0		1	0.20
2	S = R + O/n	0	1	0.50	0		1	0.20
3	S = R + (1 – 1/n)W	0.50	1	0.75	0.80		1	0.84
4	S = R/(n – 1)	0	1	0.50	0		0.25	0.05
5	S = R – W/[2 (n – 1)]	–0.50	1	0.25	–1/8		1	0.10
6	S = R – W/(n – 1)	–1	1	0.00	–0.25		1	0.00
7	S = (R – W)/(n – 1)	–1	1	0.00	–0.25		0.25	0.15
8	S = [(n – 1)/n]R – W/n	–0.50	0.50	0.00	–0.20		0.80	0.00
9	S = R – (1/3)W	–1/3	1	1/3	–1/3		1	–2/30
10	S = R – 0.48W	–0.48	1	0.26	–0.48		1	–23/125
11	S = R – 0.5W	–0.50	1	0.25	–0.5		1	–0.20
12	S = R – 0.6W	–0.60	1	0.20	–0.6		1	–0.28
13	S = R – W	–1	1	0.00	–1		1	–0.60
14	S = R + 0.7O – W	–1	1	0.00	–1		1	–0.60
15	S = R + 0.7O – 1.1W	–1.10	1	–0.05	–1.10		1	–0.68
16	S = R – nW/(n – 1)	–2	1	–0.50	–1.25		1	–0.80
17	S = R – 1.5W	–1.50	1	–0.25	–1.5		1	–1.00
18	S = R – 1.8W	–1.80	1	–0.40	–1.8		1	–1.24
19	S = R – 2 W/(n – 1)	–2	1	–0.50	–0.5		1	–0.20
20	S = R – 3W	–3	1	–1.00	–3		1	–2.20
21	S = R – (62/38)W_t – (38/62)W_f	–62/38 or –38/62	1	N/A^j	–62/38 or –38/62		1	N/A^j

^aS: examination result as absolute score.

^bR: number of correct responses.

^cO: number of omitted items.

^dW: number of incorrect responses.

^eW_t: number of true statements incorrectly marked as false.

^fW_f: number of false statements incorrectly marked as true.

^gn: number of answer options per item.

^hR=0, O=0, W=1.

ⁱR=1, O=0, W=0.

^jExpected chance scores were not calculated for method 21, because these depend on the proportion of true-false items with correct or incorrect statements.

Relation Between Examinees’ <italic>true knowledge</italic> and the Expected Scoring Results

The relation between examinees’ true knowledge and expected scoring results for single-choice items with n=2 and n=5 is shown in Figure 3. For all identified scoring methods, there is a linear relation between examinees’ true knowledge and the expected scoring results. However, some scoring methods (ie, methods 4 and 7) award less than 1 point for correctly marked items if there are more than 2 answer options (n>2). One further method (method 8) awards less than 1 point for correctly marked items regardless of the number of answer options, so the maximum score for these scoring methods might be less than 100%. Depending on the scoring method and the number of answer options, the y-axis intercepts (expected chance scores, k=0) and the slopes differ. A low expected chance score results in a wide range of examination results that differentiate different examinees’ knowledge levels (ranging from the expected chance score as the lower limit to the maximum score as the upper limit). Only for methods 6 and 8 as well as method 7 in the case of n=2, the line starts from the pole (ie, examinees without any knowledge [k=0] achieve an examination result of 0%). Only for method 6, the relation between examinees’ true knowledge and the expected scoring results is independent of the number of answer options per item.

Figure 3

Relation between examinees’ true knowledge (%) and the expected scoring results for examinations with 100 single-choice items (either n=2 or n=5 answer options per item). In each case, the expected scoring result at 50% true knowledge is shown with the associated 95% confidence interval. Method 21 is not shown because the relation depends on the proportion of single true-false items with true or false statements. O: number of omitted items (O=0); R: number of correct responses; S: examination result as absolute score (max. up to 100 points); W: number of incorrect responses.

Discussion Principal Findings

In this review, a total of 21 scoring methods for single-choice items could be identified. The majority of identified scoring methods is based on theoretical considerations or empirical findings, while others have been arbitrarily determined. Although some methods were only described for certain item types (ie, single-choice items with n=2), most of them might also be used for scoring items with more answer options. However, 1 method is suitable for scoring single true-false items only.

All scoring methods have in common that omitted items do not result in any credit deduction. Some scoring methods even award a fixed amount of 0.7 points on omitted items (methods 14 and 15), which is, however, lower than the full credit for a correct response, or the score to be achieved on average by guessing (1/n, method 2).

For the identified scoring methods, the possible scores range from a maximum of –3 to +1 points. A correctly marked item is usually scored with 1 full point (1 credit point). Exceptions to this are 3 scoring methods that only award 1 credit point in case of single-choice items with n=2 (methods 4 and 7) or that never award 1 credit point (method 8). These scoring methods are questionable because as the number of answer options increases, the guessing probability decreases. Further, a differentiation between examinees’ marking on true and false statements (method 21) is not justified, because the importance of correctly identifying true statements (ie, correctly marking the statement as true) and false statements (ie, correctly marking the statement as false) is likely to be considered equivalent in the context of many examinations.

With the exception of method 6, the relation between examinees’ true knowledge and the resulting examination scores depends on the number of answer options per item (n). Therefore, the number of answer options per item must usually be taken into account when examination scores are interpreted.

Examinations are designed to determine examinees’ knowledge as well as to decide whether the examinees pass or fail in summative examinations. It can be generally assumed that examinees must perform at least 50% of the expected performance to receive at least a passing grade [271]. If examinees are to be tested on a true knowledge of 50%, adjusted pass marks must be applied depending on the scoring method used and the number of answer options per item. The theoretical considerations show that for an examination testing for 50% true knowledge, a pass mark of 0% or even negative scoring results might be appropriate, while other scoring methods would require pass marks up to 92%. Consequently, the examination’s pass mark must be considered or adjusted when selecting a suitable scoring method. However, the pass mark might be fixed due to local university or national guidelines resulting in a limited number of suitable scoring methods.

Correction for Guessing

To account for guessing in case of single true-false items, the scoring formula R – W (method 13) was originally propagated in the literature, where the number of incorrect responses is subtracted from the number of correct responses [4]. Since its first publication in 1920, this scoring method has been frequently criticized: the main criticism is that this scoring method assumes examinees to either have complete knowledge (k=1) or to guess blindly (k=0). However, especially in the context of university examinations, examinees are assumed to have at least some partial knowledge. Furthermore, the scoring method assumes that incorrect responses are exclusively the result of guessing. No differentiation is made between incorrect responses due to blind guessing (ie, complete lack of knowledge), informed guessing (ie, guessing with partial knowledge and remaining uncertainty), or other reasons (eg, transcription errors introduced when transferring markings to the answer sheet) despite complete knowledge. Because of the 50% guessing probability in case of alternate-choice items or single true-false items, it is assumed that for each incorrectly guessed response (W) 1 item is also marked correctly by guessing on average, so that the corrected result is obtained by the scoring formula R – W. Especially in the case of partial knowledge, examinees’ marking behavior not only depends on their actual knowledge but also on their individual personality (eg, risk-seeking behavior) [272]. Consequently, the construct validity of examinations must be questioned when using the scoring formula R – W. Another criticism is that a correction by awarding malus points does not change the relative ranking of the results of different examinees if all examinees have sufficient time to take the examination and all items are answered [44,46].

Therefore, alternative scoring methods and scoring formulas emerged in addition to the already discussed scoring formula R – W. In this context, the literature often refers to formula scoring. However, the term formula scoring is not used uniformly: on the one hand, it is used as a general umbrella term for various scoring methods to correct for the guessing probability. On the other hand, the term is used to refer to specific scoring methods (methods 2, 6, and 13). Using method 2, examinees receive 1/n points for each omitted item. This corresponds to the number of points they would have scored on average by blindly guessing. Method 6 is a generalization of the scoring formula R – W for variable numbers of answer options. In case of n answer options, there are n – 1 times as many incorrect answer options as correct answer options and it is assumed that for each incorrectly guessed response (W) also W/(n – 1) items are marked correctly by guessing on average. Therefore, the corrected score is given by the scoring formula R – W/(n – 1). Consequently, methods 6 and 13 yield identical scoring results in case of items with n=2.

Strengths and Limitations

So far, the relation between examinees’ true knowledge and the expected scoring result for single-choice items has been shown only for a small number of scoring methods [273]. Therefore, a systematic literature search was conducted in several databases as part of this review. As a result, a large number of different scoring methods have been identified and were compared in this review assisting (medical) educators in gaining a comprehensive overview and to allow for informed decisions regarding the scoring of single-choice items. However, limitations are also present: First, a number of assumptions (eg, equal difficulty of items and answer options, absence of cues) were required for simplification of the calculations and comparisons. However, these assumptions are likely to be violated in real examinations [15,274-276]. Second, calculations are based on classical test theory assumptions and did not employ item response theory models that might yield different results. Third, databases were already searched in September 2020 and potentially eligible sources published thereafter might not be included in this review. However, single-choice items have been used in examinations for over 100 years and further scoring methods are unlikely to have emerged in the past 2 years.

Comparison With Prior Work

Although some of the identified scoring methods might also be applied to other item formats (eg, multiple-select items), the presented equation for the calculation of the expected scoring result is limited to single-choice items. Analogous calculations for items in multiple-select multiple-choice formats with (eg, Pick-N items) or without (eg, Multiple-True-False items) mutual stochastic dependence have already been described in the literature [11,14].

Practical Implications

In practice, the evaluation of a multiple-choice examination should be based on an easy-to-calculate scoring method that allows for a transparent credit awarding and is accepted by jurisdiction. In this regard, scoring methods with malus points (ie, methods 5-21) may not be accepted by national jurisdiction in certain countries (eg, Germany) [277]. Furthermore, it does not seem reasonable to discourage examinees from marking an item by awarding malus points for the reasons already mentioned. Therefore, only 4 of the presented scoring methods can be versatilely used. Furthermore, it seems inconclusive to reward partial credit on incorrect responses or to refrain from awarding 1 credit point for correct responses in case of items with more than 2 answer options (n>2). As a result, only a dichotomous scoring method (1 credit point for a correct response, 0 points for an incorrect response or omitted items) is recommended. Within the context of this review, the outlined scoring method is referred to as method 1.

The scoring of examinations with different item types, item formats, or items containing a varying number of answer options within a single examination is more complicated. Here, the individual examination sections would have to be evaluated separately or the credit resulting from the respective item type or item format would have to be corrected to enable a uniform pass mark. For example, in the single-choice format, credit points resulting from items with n=2 would have to be reduced to compensate for the higher guessing probability compared with items with n=5 (ie, 50% vs 20% guessing probability).

Conclusions

Single-response items only allow clearly correct or incorrect responses from examinees. Consequently, the scoring should also be dichotomous and result in either 0 points (incorrect response) or 1 credit point (correct response) per item. Because of the possibility of guessing, scoring results cannot be equated with examinees’ true knowledge. If (medical) educators interpret scoring results and determine suitable pass marks, the expected chance score must be taken into account, which in the proposed dichotomous scoring methods depends on the number of answer options per item.

Multimedia Appendix 1

PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) checklist.

Multimedia Appendix 2

Excluded sources after screening of full texts.

Abbreviations

correct for guessing

resulting score per item

examinees’ true knowledge

number of answer options per item

number correct

number of omitted items

PRISMA-ScR

Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews

PROSPERO

International Prospective Register of Systematic Reviews

number of correct responses

examination result as absolute score

number of incorrect responses

W_f

number of false statements incorrectly marked as true

W_t

number of true statements incorrectly marked as false

The authors acknowledge support by the Open Access Publication Funds of Göttingen University. The funder had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability

All data generated during or analyzed during this study are included in this published article and its supplementary information files.

AFK and PK contributed to the study’s conception and design, performed the literature search and data extraction, and drafted the manuscript. PK performed statistical analyses. All authors interpreted the data, critically revised the manuscript, and approved the final version of the manuscript.

None declared.

Krebs

Prüfen mit Multiple Choice: Kompetent planen, entwickeln, durchführen und auswerten [Testing with Multiple Choice: Plan, Develop, Implement, and Evaluate Competently] 2019

Bern, Switzerland

Hogrefe

Ebel

Proposed solutions to two problems of test construction

J Educ Meas 1982 12 19 4 267 278

10.1111/j.1745-3984.1982.tb00133.x

Kelly

The Kansas silent reading test

J Educ Psychol 1916 02 7 2 63 80

10.1037/h0073542

McCall

A new kind of school examination

J Educ Res 1920 1 1 33 46

10.1080/00220671.1920.10879021

Ruch

Stoddard

Comparative reliabilities of five types of objective examinations

J Educ Psychol 1925 02 16 2 89 103

10.1037/h0072894

Paterson

Langlie

Empirical data on the scoring of true-false tests

J Appl Psychol 1925 12 9 4 339 348

10.1037/h0069813

Lindner

Strobel

Köller

Multiple-Choice-Prüfungen an Hochschulen? [Are multiple-choice exams useful for universities? A literature review and argument for a more practice oriented research]

Z Pädagog Psychol 2015 10 29 3-4 133 149

10.1024/1010-0652/a000156

Mathysen

DGP

Aclimandos

Roelant

Wouters

Creuzot-Garcher

Ringens

Hawlina

Tassignon

Evaluation of adding item-response theory analysis for evaluation of the European Board of Ophthalmology Diploma examination

Acta Ophthalmol 2013 11 91 7 e573 7

10.1111/aos.12135

23927770

Rutgers

van Raamt

van der Gijp

Mol

Ten Cate

Determinants of difficulty and discriminating power of image-based test items in postgraduate radiological examinations

Acad Radiol 2018 05 25 5 665 672

10.1016/j.acra.2017.10.014

29198947

S1076-6332(17)30440-3

Hubbard

Measuring Medical Education: The Tests and Test Procedures of the National Board of Medical Examiners 1971

Philadelphia, PA

Lea and Febiger

Schmidt

Raupach

Wiegand

Herrmann

Kanzow

Relation between examinees’ true knowledge and examination scores: systematic review and exemplary calculations on Multiple-True-False items

Educ Res Rev 2021 11 34 100409

10.1016/j.edurev.2021.100409

Tricco

Lillie

Zarin

O'Brien

Colquhoun

Levac

Moher

Peters

MDJ

Horsley

Weeks

Hempel

Akl

Chang

McGowan

Stewart

Hartling

Aldcroft

Wilson

Garritty

Lewin

Godfrey

Macdonald

Langlois

Soares-Weiser

Moriarty

Clifford

Tunçalp

Özge

Straus

PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation

Ann Intern Med 2018 10 02 169 7 467 473

10.7326/M18-0850

30178033

2700389

Albanese

Sabers

Multiple true-false items: a study of interitem correlations, scoring alternatives, and reliability estimation

J Educ Meas 1988 06 25 2 111 123

10.1111/j.1745-3984.1988.tb00296.x

Schmidt

Raupach

Wiegand

Herrmann

Kanzow

Relation between examinees’ true knowledge and examination scores: systematic review and exemplary calculations on Pick-N items

Educ Res Rev 2022 11 37 100483

10.1016/j.edurev.2022.100483

Kanzow

Schuelper

Witt

Wassmann

Sennhenn-Kirchner

Wiegand

Raupach

Effect of different scoring approaches upon credit assignment when using Multiple True-False items in dental undergraduate examinations

Eur J Dent Educ 2018 11 22 4 e669 e678

10.1111/eje.12372

29934980

Toops

Trade Tests in Education 1921

New York, NY

Teachers College, Columbia University

Wood

Measurement in Higher Education 1923

New York, NY

Teachers College, Columbia University

Brinkley

Values of New Type Examinations in the High School. With Special Reference to History 1924

New York, NY

Teachers College, Columbia University

Farwell

The new type examinations in physics

School Soc 1924 19 481 315 322

Miller

Formulas for scoring tests in which the maximum amount of chance is determined

Proc Okla Acad Sci 1925 5 30 42

Boyd

An exploration of the true-false method of examination

Forum Educ 1926 4 34 38

Christensen

A suggestion as to correcting guessing in examinations

J Educ Res 1926 14 5 370 374

10.1080/00220671.1926.10879703

Ruch

Degraff

Gordon

McGregor

Maupin

Murdock

Objective Examination Methods in the Social Studies 1926

Chicago, IL

Scott, Foresman and Company

Wood

Studies of achievement tests. Part I: the R versus the R–W method of scoring “do not guess” true-false examinations

J Educ Psychol 1926 01 17 1 1 22

10.1037/h0076061

Wood

Improving the validity of collegiate achievement tests

J Educ Psychol 1927 01 18 1 18 25

10.1037/h0070659

Greene

A new correction for chance in examinations of alternate-response type

J Educ Res 1928 17 2 102 107

10.1080/00220671.1928.10879818

Odell

Traditional Examinations and New-Type Tests 1928

New York, NY

The Century

Ruch

Charles

A comparison of five types of objective tests in elementary psychology

J Appl Psychol 1928 12 4 398 403

10.1037/h0075108

Cocks

The Pedagogical Value of the True-False Examination 1929

Baltimore, MD

Warwick and York

Dunlap

De Mello

Cureton

The effects of different directions and scoring methods on the reliability of a true-false test

School Soc 1929 30 768 378 382

Hevner

A method of correcting for guessing in true-false tests and empirical evidence in support of it

J Soc Psychol 1932 08 3 3 359 362

10.1080/00224545.1932.9919159

Melbo

How much do students guess in taking true-false examinations?

Educ Method 1932 12 485 487

Hawkes

Lindquist

Mann

The Construction and Use of Achievement Examinations: A Manual for Secondary School Teachers 1936

Bostan, MA

Houghton Mifflin

Rinsland

Constructing Tests and Grading in Elementary and High School Subjects 1937

New York, NY

Prentice-Hall

Lord

Reliability of multiple-choice tests as a function of number of choices per item

J Educ Psychol 1944 03 35 3 175 180

10.1037/h0061025

Engelhart

Suggestions for writing achievement exercises to be used in tests scored on the electric scoring machine

Educ Psychol Meas 1949 7 3 357 374

10.1177/001316444700700301

Lindquist

Educational Measurement 1951

Washington, DC

American Council on Education

Heston

How to Take a Test 1953

Oxford, UK

Science Research Associates

Keislar

Test instructions and scoring method in true-false tests

J Exp Educ 1953 21 3 243 249

10.1080/00220973.1953.11010457

Swineford

Miller

Effects of directions regarding guessing on item statistics of a multiple-choice vocabulary test

J Educ Psychol 1953 03 44 3 129 139

10.1037/h0057890

Guilford

Psychometric Methods 1954

New York, NY

McGraw-Hill

Sherriffs

Boomer

Who is penalized by the penalty for guessing?

J Educ Psychol 1954 02 45 2 81 90

10.1037/h0053756

Davis

Use of correction for chance success in test scoring

Educ Meas 1959 52 7 279 280

10.1080/00220671.1959.10882581

Hubbard

Clemans

Multiple-Choice Examinations in Medicine: A Guide for Examiner and Examinee 1961

Philadelphia, PA

Lea and Febiger

Durost

Prescott

Essentials of Measurement for Teachers 1962

New York, NY

Harcourt, Brace & World

Ebel

Measuring Educational Achievement 1965

Englewood Cliffs, NJ

Prentice-Hall

Mattson

The effects of guessing on the standard error of measurement and the reliability of test scores

Educ Psychol Meas 1965 25 3 727 730

10.1177/001316446502500305

Cooper

Foy

Guessing in multiple-choice tests

Br J Med Educ 1967 06 1 3 212 215

10.1111/j.1365-2923.1967.tb01699.x

6080737

Lennox

Multiple choice

Br J Med Educ 1967 12 1 5 340 344

5583311

Gronlund

Constructing Achievement Tests 1968

Englewood Cliffs, NJ

Prentice-Hall

Sax

Collet

The effects of differing instructions and guessing formulas on reliability and validity

Educ Psychol Meas 1968 28 4 1127 1136

10.1177/001316446802800411

Macintosh

Morrison

Objective Testing 1969

London, UK

University of London Press

Traub

Hambleton

Singh

Effects of promised reward and threatened penalty on performance of a multiple-choice vocabulary test

Educ Psychol Meas 1969 29 4 847 861

10.1177/001316446902900410

Cronbach

Essentials of Psychological Testing. 3rd ed 1970

New York, NY

Harper & Row

Houston

The Principles of Objective Testing in Physics 1970

London, UK

Heinemann Educational Books

Gronlund

Measurement and Evaluation in Teaching. 2nd ed 1971

New York, NY

Macmillan

Lyman

Test Scores and What They Mean. 2nd ed 1971

Englewood Cliffs, NJ

Prentice-Hall

Brandenburg

Whitney

Matched pair true-false scoring: effect on reliability and validity

J Educ Meas 1972 12 9 4 297 302

10.1111/j.1745-3984.1972.tb00961.x

Campbell

CVT

Milne

The Principles of Objective Testing in Chemistry 1972

London, UK

Heinemann Educational Books

Ebel

Essentials of Educational Measurement 1972

Englewood Cliffs, NJ

Prentice-Hall

Fraser

Gillam

The Principles of Objective Testing in Mathematics 1972

London, UK

Heinemann Educational Books

Diamond

Evans

The correction for guessing

Rev Educ Res 1973 43 2 181 191

10.3102/00346543043002181

Rust

Objective Testing in Education and Training 1973

London, UK

Pitman

Hill

Woods

Multiple true-false questions

Educ Chem 1974 11 3 86 87

10.1017/cbo9781107705623.002

Abu-Sayf

Relative effectiveness of the conventional formula score

J Educ Res 1975 69 4 160 162

10.1080/00220671.1975.10884861

Hakstian

Kansup

A comparison of several methods of assessing partial knowledge in multiple-choice tests: II. testing procedures

J Educ Meas 1975 12 12 4 231 239

10.1111/j.1745-3984.1975.tb01024.x

Lord

Formula scoring and number-right scoring

J Educ Meas 1975 03 12 1 7 11

10.1111/j.1745-3984.1975.tb01003.x

Brown

Principles of Educational and Psychological Testing. 2nd ed 1976

New York, NY

Holt, Rinehart and Winston

Harden

Brown

Biran

Dallas Ross

Wakeford

Multiple choice questions: to guess or not to guess

Med Educ 1976 01 10 1 27 32

10.1111/j.1365-2923.1976.tb00527.x

1263885

Albanese

Kent

Whitney

A comparison of the difficulty, reliability and validity of complex multiple choice, multiple response and multiple true-false items

Annu Conf Res Med Educ 1977 16 105 110

606061

Cross

Frary

An empirical test of Lord’s theoretical results regarding formula scoring of multiple-choice tests

J Educ Meas 1977 12 14 4 313 321

10.1111/j.1745-3984.1977.tb00047.x

Eakin

Long

Dodging the dilemma of true-false testing

Educ Psychol Meas 1977 37 3 659 663

10.1177/001316447703700308

Lord

Optimal number of choices per item—a comparison of four approaches

J Educ Meas 1977 03 14 1 33 38

10.1111/j.1745-3984.1977.tb00026.x

Reid

An alternative scoring formula for multiple-choice and true-false tests

J Educ Res 1977 70 6 335 339

10.1080/00220671.1977.10885018

Whitby

Marking systems for multiple choice examinations

Med Educ 1977 05 11 3 216 220

10.1111/j.1365-2923.1977.tb00596.x

865344

Aiken

Williams

Effects of instructions, option keying, and knowledge of test material on seven methods of scoring two-option items

Educ Psychol Meas 1978 38 1 53 59

10.1177/001316447803800108

Hubbard

Measuring Medical Education: The Tests and Test Procedures of the National Board of Medical Examiners. 2nd ed 1978

Philadelphia, PA

Lea and Febiger

Morgan

MKM

Irby

Evaluating Clinical Competence in the Health Professions 1978

St. Louis, MO

Mosby

Abu-Sayf

Recent developments in the scoring of multiple-choice items

Educ Rev 1979 31 3 269 279

10.1080/0013191790310308

Abu-Sayf

The scoring of multiple choice tests: a closer look

Educ Technol 1979 19 6 5 15

Ebel

Essentials of Educational Measurement. 3rd ed 1979

Englewood Cliffs, NJ

Prentice-Hall

Hsu

A comparison of three methods of scoring true-false tests

Educ Psychol Meas 1979 39 4 785 790

10.1177/001316447903900411

Newble

Baxter

Elmslie

A comparison of multiple-choice tests and free-response tests in examinations of clinical competence

Med Educ 1979 07 13 4 263 268

10.1111/j.1365-2923.1979.tb01511.x

470647

Skakun

Nanson

Kling

Taylor

A preliminary investigation of three types of multiple choice questions

Med Educ 1979 03 13 2 91 96

10.1111/j.1365-2923.1979.tb00928.x

431421

Bliss

A test of Lord’s assumption regarding examinee guessing behavior on multiple-choice tests using elementary school students

J Educ Meas 1980 06 17 2 147 153

10.1111/j.1745-3984.1980.tb00823.x

Ahmann

Glock

Evaluating Student Progress: Principles of Tests and Measurements. 6th ed 1981

Boston, MA

Allyn and Bacon

Hopkins

Stanley

Educational and Psychological Measurement and Evaluation. 6th ed 1981

Englewood Cliffs, NJ

Prentice-Hall

Anderson

Hand-scoring of multiple choice questions

Med Educ 1983 03 17 2 122 133

10.1111/j.1365-2923.1983.tb01111.x

6843390

Kolstad

Briggs

Bryant

Kolstad

Complex multiple-choice items fail to measure achievement

J Res Develop Educ 1983 17 1 7 11

Kolstad

Wagner

Kolstad

Miller

The failure of distractors on complex multiple-choice items to prevent guessing

Educ Res Quart 1983 8 2 44 50

Nitko

Educational Tests and Measurement: An Introduction 1983

New York, NY

Harcourt Brace Jovanovich

Angoff

Schrader

A study of hypotheses basic to the use of rights and formula scores

J Educ Meas 1984 03 21 1 1 17

10.1111/j.1745-3984.1984.tb00217.x

Diekhoff

True-false tests that measure and promote structural understanding

Teach Psychol 1984 11 2 99 101

10.1207/s15328023top1102_11

Kolstad

The construction of machine-scored examinations: MTF clusters are preferable to CMC items

Sci Paedagog Exp 1984 21 1 45 54

Norcini

Swanson

Grosso

Shea

Webster

A comparison of knowledge, synthesis, and clinical judgment. Multiple-choice questions in the assessment of physician competence

Eval Health Prof 1984 12 7 4 485 499

10.1177/016327878400700409

10269331

Kolstad

Multiple-choice test items are unsuitable for measuring the learning of complex instructional objectives

Sci Paedagog Exp 1985 22 1 68 76

Norcini

Swanson

Grosso

Webster

Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence

Med Educ 1985 05 19 3 238 247

10.1111/j.1365-2923.1985.tb01314.x

4010571

Crocker

Algina

Introduction to Classical and Modern Test Theory 1986

Orlando, FL

Holt, Rinehart and Winston

Jaradat

Sawaged

The subset selection technique for multiple-choice tests: an empirical inquiry

J Educ Meas 1986 12 23 4 369 376

10.1111/j.1745-3984.1986.tb00256.x

100

Aiken

Testing with multiple-choice items

J Res Develop Educ 1987 20 4 44 58

101

Friedman

Hopwood

Moulder

Cox

The potential use of the discouraging random guessing (DRG) approach in multiple-choice exams in medical education

Med Teach 1987 9 3 333 341

10.3109/01421598709034796

3683144

102

Carey

Measuring and Evaluating School Learning 1988

Newton, MA

Allyn and Bacon

103

Osterlind

Constructing Test Items 1989

Boston, MA

Kluwer Academic Publishers

104

Richards

Philp

Scoring the objective structured clinical examination using a microcomputer

Med Educ 1989 23 4 376 380

10.1111/j.1365-2923.1989.tb01563.x

105

Cangelosi

Designing Tests for Evaluating Student Achievement 1990

White Plains, NY

Longman

106

Popham

Modern Educational Measurement: A Practitioner’s Perspective. 2nd ed 1990

Needham Heights, MA

Allyn and Bacon

107

Moussa

MAA

Ouda

Nemeth

Analysis of multiple-choice items

Comput Methods Programs Biomed 1991 04 34 4 283 289

10.1016/0169-2607(91)90113-8

1873997

0169-2607(91)90113-8

108

Viniegra

Jiménez

Pérez-Padilla

El desafío de la evaluación de la competencia clínica [The challenge of evaluating clinical competence]

Rev Invest Clin 1991 43 1 87 98

1866504

109

Harasym

Price

Brant

Violato

Lorscheider

Evaluation of negation in stems of multiple-choice items

Eval Health Prof 1992 15 2 198 220

10.1177/016327879201500205

110

Nnodim

Multiple-choice testing in anatomy

Med Educ 1992 07 26 4 301 309

10.1111/j.1365-2923.1992.tb00173.x

1630332

111

Budescu

Bar-Hillel

To guess or not to guess: a decision-theoretic view of formula scoring

J Educ Meas 1993 12 30 4 277 291

10.1111/j.1745-3984.1993.tb00427.x

112

Fajardo

Chan

Evaluation of medical students in radiology. Written testing using uncued multiple-choice questions

Invest Radiol 1993 10 28 10 964 968

10.1097/00004424-199310000-00020

8262753

113

Gronlund

How to Make Achievement Tests and Assessments. 5th ed 1993

Needham Heights, MA

Allyn and Bacon

114

Haladyna

Downing

How many options is enough for a multiple-choice test item?

Educ Psychol Meas 1993 53 4 999 1010

10.1177/0013164493053004013

115

Harasym

Doran

Brant

Lorscheider

Negation in stems of single-response multiple-choice items

Eval Health Prof 1993 16 3 342 357

10.1177/016327879301600307

116

Pinckney

Borcher

Clemens

Comparative studies of true/false, multiple choice and multiple-multiple choice

NACTA 1993 37 1 21 24

117

Wolf

A comparison of assessment tasks used to measure FL reading comprehension

Mod Lang J 1993 77 4 473 489

10.1111/j.1540-4781.1993.tb01995.x

118

Bott

Testing and Assessment in Occupational and Technical Education 1995

Meedham Heights, MA

Allyn and Bacon

119

Downing

Baranowski

Grosso

Norcini

Item type and cognitive ability measured: The validity evidence for multiple true-false items in medical specialty certification

Appl Meas Educ 1995 04 8 2 187 207

10.1207/s15324818ame0802_5

120

Linn

Gronlund

Measurement and Assessment in Teaching. 7th ed 1995

Englewood Cliffs, NJ

Merril

121

Lumley

JSP

Craven

Introduction. MCQ’s in Anatomy: A Self-Testing Supplement to Essential Anatomy. 3rd ed 1996

New York, NY

Churchill Livingstone

122

Nitko

Educational Assessment of Students. 2nd ed 1996

Englewood Cliffs, NJ

Prentice Hall

123

Schuwirth

LWT

van der Vleuten

CPM

Donkers

HHL

A closer look at cueing effects in multiple-choice questions

Med Educ 1996 01 30 1 44 49

10.1111/j.1365-2923.1996.tb00716.x

8736188

124

Ben-Simon

Budescu

Nevo

A comparative study of measures of partial knowledge in multiple-choice tests

Appl Psychol Meas 1997 21 1 65 88

10.1177/0146621697211006

125

Thorndike

Measurement and Evaluation in Psychology and Education 1997

Upper Saddle River, NJ

Merrill

126

Gronlund

Assessment of Student Achievement 1998

Needham Heights, MA

Allyn and Bacon

127

Harasym

Leong

Violato

Brant

Lorscheider

Cuing effect of “all of the above” on the reliability and validity of multiple-choice test items

Eval Health Prof 1998 03 21 1 120 133

10.1177/016327879802100106

10183336

128

Agble

A Psychometric Analysis of Different Scoring Strategies in Statistics Assessment [PhD dissertation] 1999

2023-04-22

https://www.proquest.com/openview/9e0b28a2f2ff468cf635eff09c780fc4/1?pq-origsite=gscholar&cbl=18750&diss=y

129

Bandaranayake

Payne

White

Using multiple response true-false multiple choice questions

Aust N Z J Surg 1999 04 69 4 311 315

10.1046/j.1440-1622.1999.01551.x

10327124

130

Burton

Miller

Statistical modelling of multiple-choice and true/false tests: ways of considering, and of reducing, the uncertainties attributable to guessing

Ass Eval High Educ 1999 24 4 399 411

10.1080/0260293990240404

131

Joint Committee on Standards for Educational and Psychological Testing of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education

Standards for Educational and Psychological Testing 1999

Washington, DC

American Educational Research Association

132

Muijtjens

AMM

Mameren

Hoogenboom

RJI

Evers

JLH

van der Vleuten

CPM

The effect of a ‘don’t know’ option on test scores: number-right and formula scoring compared

Med Educ 1999 04 33 4 267 275

10.1046/j.1365-2923.1999.00292.x

10336757

med292

133

de Bruin

Fischhoff

The effect of question format on measured HIV/AIDS knowledge: detention center teens, high school students, and adults

AIDS Educ Prev 2000 06 12 3 187 198

10926123

134

Linn

Gronlund

Measurement and Assessment in Teaching. 8th ed 2000

Englewood Cliffs, NJ

Merril

135

Beeckmans

Eyckmans

Janssens

Dufranne

Van de Velde

Examining the yes/no vocabulary test: some methodological issues in theory and practice

Lang Test 2001 18 3 235 274

10.1177/026553220101800301

136

Blasberg

Güngerich

Müller-Esterl

Neumann

Schappel

Erfahrungen mit dem Fragentyp „k aus n“ in Multiple-Choice-Klausuren [Experiences with item type “k from n” in multiple-choice-tests]

Med Ausbild 2001 18 S1 73 76

137

Nitko

Educational Assessment of Students. 3rd ed 2001

Upper Saddle River, NJ

Merrill Prentice Hall

138

Alnabhan

An empirical investigation of the effects of three methods of handling guessing and risk taking on the psychometric indices of a test

Soc Behav Pers 2002 30 7 645 652

10.2224/sbp.2002.30.7.645

139

Bereby-Meyer

Meyer

Flascher

Prospect theory analysis of guessing in multiple choice tests

J Behav Decis Making 2002 10 15 4 313 327

10.1002/bdm.417

140

Burton

Misinformation, partial knowledge and guessing in true/false tests

Med Educ 2002 09 36 9 805 811

10.1046/j.1365-2923.2002.01299.x

12354242

1299

141

Griggs

Ransdell

Griggs

Misconceptions tests or misconceived tests?

Handbook for Teaching Introductory Psychology 2002

Mahwah, NH

Lawrence Erlbaum Associates

30 33

142

Rahim

Abumadini

Comparative evaluation of multiple choice question formats. Introducing a knowledge score

Neurosciences (Riyadh) 2003 07 8 3 156 160

23649110

20020430

143

Anderson

Multiple choice questions revisited

Med Teach 2004 03 26 2 110 113

10.1080/0142159042000196141

15203517

8PQKDQ7UHQUN01YF

144

Bradbard

Parker

Stone

An alternate multiple-choice scoring procedure in a macroeconomics course

Decis Sci J Innov Educ 2004 01 16 2 1 11 26

10.1111/j.0011-7315.2004.00016.x

145

Burton

Multiple choice and true/false tests: reliability measures and some implications of negative marking

Ass Eval High Educ 2004 10 29 5 585 595

10.1080/02602930410001689153

146

Haladyna

Developing and Validating Multiple-Choice Test Items. 3rd ed 2004

New York, NY

Routledge

147

Burton

Multiple‐choice and true/false tests: myths and misapprehensions

Ass Eval High Educ 2005 02 30 1 65 72

10.1080/0260293042003243904

148

Pamphlett

It takes only 100 true-false items to test medical students: true or false?

Med Teach 2005 08 27 5 468 472

10.1080/01421590500097018

16147803

J43T10781574K67W

149

Swanson

Holtzman

Clauser

Sawhill

Psychometric characteristics and response times for one-best-answer questions in relation to number and source of options

Acad Med 2005 10 80 S10 S93 S96

10.1097/00001888-200510001-00025

16199468

80/10_suppl/S93

150

MacCann

The equivalence of online and traditional testing for different subpopulations and item types

Br J Educ Technol 2006 01 37 1 79 91

10.1111/j.1467-8535.2005.00524.x

151

Shizuka

Takeuchi

Yashima

Yoshizawa

A comparison of three- and four-option English tests for university entrance selection purposes in Japan

Lang Test 2006 23 1 35 57

10.1191/0265532206lt319oa

152

Swanson

Holtzman

Allbee

Clauser

Psychometric characteristics and response times for content-parallel extended-matching and one-best-answer items in relation to number of options

Acad Med 2006 10 81 S10 S52 S55

10.1097/01.ACM.0000236518.87708.9d

17001136

00001888-200610001-00014

153

Afolabi

ERI

Effects of test format, self concept and anxiety on item response changing behaviour

Educ Res Rev 2007 2 9 255 258

154

Costagliola

Ferrucci

Fuccella

Oliveto

eWorkbook: a computer aided assessment system

Int J Distance Educ Technologies 2007 5 3 24 41

10.4018/jdet.2007070103

155

Downing

Yudkowsky

Assessment in health professions education 2009

New York, NY

Routledge

156

Tasdemir

A comparison of multiple-choice tests and true-false tests used in evaluating student progress

J Instruct Psychol 2010 37 3 258 266

157

Wakabayashi

Guskin

The effect of an “unsure” option on early childhood professionals’ pre- and post-training knowledge assessments

Am J Eval 2010 09 03 31 4 486 498

10.1177/1098214010371818

158

Bayazit

Aşkar

Performance and duration differences between online and paper–pencil tests

Asia Pacific Educ Rev 2012 13 2 219 226

10.1007/s12564-011-9190-9

159

Begum

A guideline on developing effective multiple choice questions and construction of single best answer format

J Bangladesh Coll Phys 2012 11 03 30 3 159 166

10.3329/jbcps.v30i3.12466

160

Arnold

Higham

Martín-Luengo

A little bias goes a long way: the effects of feedback on the strategic regulation of accuracy on formula-scored tests

J Exp Psychol Appl 2013 12 19 4 383 402

10.1037/a0034833

24341319

2013-43362-004

161

Schaper

Tipold

Ehlers

Use of key feature questions in summative assessment of veterinary medicine students

Ir Vet J 2013 03 07 66 1 3

10.1186/2046-0481-66-3

23497425

2046-0481-66-3

PMC3599329

162

Simbak

Aung

MMT

Ismail

Jusoh

NBM

Ali

Yassin

WAK

Haque

Rebuan

HMA

Comparative study of different formats of MCQs: multiple true-false and single best answer test formats, in a new medical school of Malaysia

Int Med J 2014 21 6 562 566

163

Patil

Item analysis of medicine multiple choice questions (MCQs) for under graduate (3rd year MBBS) students

Res J Pharma Biol Chem Sci 2015 6 3 1242 1251

164

Ravesloot

Van der Schaaf

Muijtjens

AMM

Haaring

Kruitwagen

CLJJ

Beek

FJA

Bakker

Van Schaik

JPJ

Ten Cate

The don’t know option in progress testing

Adv Health Sci Educ Theory Pract 2015 12 20 5 1325 1338

10.1007/s10459-015-9604-2

25912621

10.1007/s10459-015-9604-2

PMC4639571

165

Haladyna

Lane

Raymond

Haladyna

Item analysis for selected response test items

Handbook of Test Development. 2nd ed 2016

New York, NY

Routledge

166

Rush

Rankin

White

The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value

BMC Med Educ 2016 09 29 16 1 250

10.1186/s12909-016-0773-3

27681933

10.1186/s12909-016-0773-3

PMC5041405

167

Mafinejad

Arabshahi

SKS

Monajemi

Jalili

Soltani

Rasouli

Use of multi-response format test in the assessment of medical students’ critical thinking ability

J Clin Diagn Res 2017 09 11 9 LC10 LC13

10.7860/JCDR/2017/24884.10607

29207742

PMC5713764

168

Puthiaparampil

Assessment analysis: how it is done

MedEdPublish 2017 8 4 6 142

10.15694/mep.2017.000142

169

Vander Beken

Brysbaert

Studying texts in a second language: the importance of test type

Bil Lang Cog 2017 07 31 21 5 1062 1074

10.1017/s1366728917000189

170

Lahner

Lörwald

Bauer

Nouns

Krebs

Guttormsen

Fischer

Huwendiek

Multiple true-false items: a comparison of scoring algorithms

Adv Health Sci Educ Theory Pract 2018 08 23 3 455 463

10.1007/s10459-017-9805-y

29189963

10.1007/s10459-017-9805-y

171

Puthiaparampil

Rahman

Very short answer questions: a viable alternative to multiple choice questions

BMC Med Educ 2020 05 06 20 1 141

10.1186/s12909-020-02057-w

32375739

10.1186/s12909-020-02057-w

PMC7203787

172

May

Measuring achievement in elementary psychology and in other college subjects

School Soc 1923 17 435 472 476

173

Remmers

Gage

Educational Measurement and Evaluation. 2nd ed 1955

New York, NY

Harper & Brothers

174

Stanley

Hopkins

Educational and Psychological Measurement and Evaluation. 5th ed 1972

Englewood Cliffs, NJ

Prentice-Hall

175

Mehrens

Lehmann

Measurement and Evaluation in Education and Psychology. 3rd ed 1984

New York, NY

Holt, Rinehart and Winston

176

Ebel

Frisbie

The Administration and Scoring of Achievement Tests. Essentials of Educational Measurement 1986

Englewood Cliffs, NJ

Prentice-Hill

177

Ebel

Frisbie

Essentials of Educational Measurement. 5th ed 1991

Englewood Cliffs, NJ

Prentice-Hall

178

Mehrens

Lehmann

Measurement and Evaluation in Education and Psychology. 4th ed 1991

New York, NY

Holt, Rinehart and Winston

179

Rogers

Masters

Keeves

Guessing in multiple choice tests

Advances in Measurement in Educational Research and Assessment 1999

Kidlington, UK

Pergamon

180

Burton

Quantifying the effects of chance in multiple choice and true/false tests: question selection and guessing of answers

Ass Eval High Educ 2001 01 26 1 41 50

10.1080/02602930020022273

181

Foster

Ruch

On corrections for chance in multiple-response tests

J Educ Psychol 1927 01 18 1 48 51

10.1037/h0070562

182

Little

Overcorrection for guessing in multiple-choice test scoring

J Educ Res 1962 55 6 245 252

10.1080/00220671.1962.10882801

183

Holzinger

On scoring multiple response tests

J Educ Psychol 1924 15 7 445 447

10.1037/h0073083

184

Ruch

Degraff

Corrections for chance and “guess” vs. “do not guess” instructions in multiple response tests

J Educ Psychol 1926 09 17 6 368 375

10.1037/h0073222

185

Ruch

The Objective or New-Type Examination: An Introduction to Educational Measurement 1929

Chicago, IL

Scott, Doresman and Company

186

Kinney

Eurich

Studies of the true-false examination

Psychol Bull 1933 07 30 7 505 517

10.1037/h0070031

187

Lincoln

The Preparation of New Type Testing Materials. Testing and the Uses of Test Results 1935

New York, NY

Macmillan

182 205

188

Guilford

The determination of item difficulty when chance success is a factor

Psychometrika 1936 12 1 4 259 264

10.1007/BF02287877

189

Votaw

The effect of do-not-guess directions upon the validity of true-false or multiple choice tests

J Educ Psychol 1936 27 9 698 703

10.1037/h0055572

190

Wood

Objective test forms for school certificate physics

Br J Educ Psych 1943 13 3 141 146

10.1111/j.2044-8279.1943.tb02733.x

191

Varty

Guessing on examinations—is it worthwhile?

Educ Forum 1946 01 10 2 205 212

10.1080/00131724609342257

192

Cronbach

Essentials of Psychological Testing 1949

New York, NY

Harper & Brothers

193

Weitzman

McNamara

Scoring and Grading the Examination. Constructing Classroom Examinations: A Guide for Teachers. 2nd ed 1949

Chicago, IL

Science Research Associates

194

Lyerly

A note on correcting for chance success in objective tests

Psychometrika 1951 3 16 1 21 30

10.1007/bf02313424

195

Coombs

Milholland

Womer

The assessment of partial knowledge

Educ Psychol Meas 1953 16 1 13 37

10.1177/001316445601600102

196

Bradfield

Moredock

Measurement and Evaluation in Education 1957

New York, NY

Macmillan

197

Graesser

Guessing on multiple-choice tests

Educ Psychol Meas 1958 18 3 617 620

10.1177/001316445801800316

198

Anastasi

Psychological Testing. 2nd ed 1961

New York, NY

Macmillan

199

Glass

Wiley

Formula scoring and test reliability

J Educ Meas 1964 06 1 1 43 49

10.1111/j.1745-3984.1964.tb00150.x

200

Cureton

The correction for guessing

J Exp Educ 1966 34 4 44 47

10.1080/00220973.1966.11010953

201

Little

Overcorrection and undercorrection in multiple-choice test scoring

J Exp Educ 1966 35 1 44 47

10.1080/00220973.1966.11010968

202

Storey

A review of evidence or the case against the true-false item

J Educ Res 1966 59 6 282 285

10.1080/00220671.1966.10883357

203

Lennox

Marking multiple-choice examinations

Br J Med Educ 1967 06 1 3 203 211

10.1111/j.1365-2923.1967.tb01698.x

6080736

204

Nunnally

Psychometric Theory 1967

New York, NY

McGraw-Hill

205

Hill

Woods

Multiple true-false questions

Sch Sci Rev 1969 50 173 919 922

10.1017/cbo9781107705623.002

206

Weitzman

Ideal multiple-choice items

J Am Stat Assoc 1970 03 65 329 71 89

10.1080/01621459.1970.10481063

207

Collet

Elimination scoring: an empirical evaluation

J Educ Meas 1971 09 8 3 209 214

10.1111/j.1745-3984.1971.tb00927.x

208

Thorndike

Educational Measurement. 2nd ed 1971

Washington, DC

American Council on Education

209

Oosterhof

Glasnapp

Comparative reliabilities and difficulties of the multiple-choice and true-false formats

J Exp Educ 1974 42 3 62 64

10.1080/00220973.1974.11011479

210

Quereshi

Performance on multiple-choice tests and penalty for guessing

J Exp Educ 1974 42 3 74 77

10.1080/00220973.1974.11011481

211

Choppin

Guessing the answer on objective tests

Br J Educ Psychol 1975 45 2 206 213

10.1111/j.2044-8279.1975.tb03245.x

212

Robbins

Completion and true/false items

Nurs Times 1975 10 30 71 44 1751 1752

1196953

213

Frary

Cross

Lowry

Random guessing, correction for guessing, and reliability of multiple-choice test scores

J Exp Educ 1977 46 1 11 15

10.1080/00220973.1977.11011603

214

Benson

Crocker

The effects of item format and reading ability on objective test performance: a question of validity

Educ Psychol Meas 1979 39 2 381 387

10.1177/001316447903900217

215

Koeslag

Melzer

Schach

Inversions in true/false and in multiple choice questions—a new form of item analysis

Med Educ 1979 11 13 6 420 424

10.1111/j.1365-2923.1979.tb01201.x

537531

216

Bergman

Understanding Educational Measurement and Evaluation 1981

Boston, MA

Houghton Mifflin

217

Koeslag

Melzer

Schach

Penalties in multiple-choice and true-false questions

S Afr Med J 1983 01 01 63 1 20 2

6849146

218

Grosse

Wright

Validity and reliability of true-false tests

Educ Psychol Meas 1985 45 1 1 13

10.1177/0013164485451001

219

Ellington

Objective Questions. Teaching and Learning in Higher Education 1987

Aberdeen, Scotland, UK

Scottish Central Institutions Committee for Educational Development

220

Sax

Principles of Educational and Psychological Measurement and Evaluation. 3rd ed 1989

Belmont, CA

Wadsworth

221

Gronlund

Linn

Measurement and Evaluation in Teaching. 6th ed 1990

New York, NY

Macmillan

222

Ory

Ryan

Tips for Improving Testing and Grading 1993

Newbury Park, CA

Sage Publications Inc

223

Nunnally

Bernstein

Psychometric Theory. 3rd ed 1994

New York, NY

McGraw-Hill

224

Beullens

Jaspaert

Het examen met meerkeuzevragen [Multiple choice examination]

Ned Tijdschr Geneeskd 1999 55 7 529 535

10.2143/tvg.55.7.5000410

225

Oosterhof

Classroom Applications of Educational Measurement. 3rd ed 2001

Upper Saddle River, NJ

Prentice-Hall

226

Petz

Penalizirati ili ne penalizirati pogrešne odgovore u testovima znanja alternativnog tipa [To penalize or not to penalize false answers in the achievement tests of the alternative type]

Revija za Psihologiju 1978 8 1-2 49 56

227

Slakter

The effect of guessing strategy on objective test scores

J Educ Meas 1968 09 5 3 217 222

10.1111/j.1745-3984.1968.tb00629.x

228

Bush

A multiple choice test that rewards partial knowledge

J Further High Educ 2001 06 25 2 157 163

10.1080/03098770120050828

229

Gupta

Penfold

DME

Correction for guessing in true-false tests: an experimental approach

Brit J Educ Psychol 1961 31 P3 249 256

10.1111/j.2044-8279.1961.tb01714.x

230

Asker

The reliability of tests requiring alternative responses

J Educ Res 1924 9 3 234 240

10.1080/00220671.1924.10879451

231

Gupta

A new approach to correction in true false tests

Educ Psychol (Delhi) 1957 4 2 63 75

232

Sanderson

The ‘don’t know’ option in MCQ examinations

Br J Med Educ 1973 03 7 1 25 29

4723448

233

Anderson

Marking of multiple choice questions

The Multiple Choice Question in Medicine. 2nd ed 1982

London, UK

Pitman Books Limited

45 58

234

Gajjar

Sharma

Kumar

Rana

Item and test analysis to identify quality Multiple Choice Questions (MCQs) from an assessment of medical students of Ahmedabad, Gujarat

Indian J Community Med 2014 01 39 1 17 20

10.4103/0970-0218.126347

24696535

IJCM-39-17

PMC3968575

235

Kohs

High test scores attained by subaverage minds

Psychol Bull 1920 01 17 1 1 5

10.1037/h0064475

236

Chapman

Individual injustice and guessing in the true-false examination

J Appl Psychol 1922 6 4 342 348

10.1037/h0076011

237

Hahn

A criticism of tests requiring alternative responses

J Educ Res 1922 6 3 236 241

10.1080/00220671.1922.10879299

238

McCall

How to Measure in Education 1922

New York, NY

Macmillan

239

West

A critical study of the right minus wrong method

J Educ Res 1923 8 1 1 9

10.1080/00220671.1923.10879376

240

Batson

Reliability of the true-false form of examination

Educ Admin Supervision 1924 10 95 102

241

Miller

Tinkering with a true-false test

Proc Okla Acad Sci 1925 5 25 30

242

Weidemann

How to Construct the True-False Examination 1926

New York, NY

Teachers College, Columbia University

243

Palmer

New type examinations in physical education

Am Physical Educ Rev 1929 34 3 151 156

10.1080/23267224.1929.10652100

244

Jensen

An evaluation of three methods of presenting true-false examinations: visual, oral and visual-oral

School Soc 1930 32 829 675 677

245

Barton

Improving the true-false examination

School Soc 1931 34 877 544 546

246

Granich

A technique for experimentation on guessing in objective tests

J Educ Psychol 1931 02 22 2 145 156

10.1037/h0072728

247

Peters

Martz

A study of the validity of various types of examinations

School Soc 1931 33 845 336 338

248

Krueger

WCF

An experimental study of certain phases of a true-false test

J Educ Psychol 1932 02 23 2 81 91

10.1037/h0073943

249

Lee

Symonds

New-type or objective tests: a summary of recent investigations

J Educ Psychol 1933 01 24 1 21 38

10.1037/h0072226

250

Soderquist

A new method of weighting scores in a true-false test

J Educ Res 1936 30 4 290 292

10.1080/00220671.1936.10880670

251

Moore

Factors of chance in the true-false examination

J Genet Psychol 1938 09 53 1 215 229

10.1080/08856559.1938.10533806

252

Swineford

The measurement of a personality trait

J Educ Psychol 1938 04 29 4 295 300

10.1037/h0058735

253

Etoxinod

How to checkmate certain vicious consequences of true-false tests

Etoxin 1940 61 223 227

254

Moore

The rights-minus-wrongs method of correcting chance factors in the true-false examination

J Genet Psychol 1940 12 57 2 317 326

10.1080/08856559.1940.10534539

255

Cronbach

An experimental comparison of the multiple true-false and multiple multiple-choice tests

J Educ Psychol 1941 10 32 7 533 543

10.1037/h0058518

256

Weidemann

The “omission” as a specific determiner in the true-false examination

J Educ Psychol 1931 09 22 6 435 439

10.1037/h0074950

257

Cruze

Measuring the results of learning

Educational Psychology 1942

New York, NY

The Ronald Press Company

343 380

258

Gilmour

Gray

Guessing on true-false tests

Educ Res Bull 1942 21 1 9 12

259

Cronbach

Studies of acquiescence as a factor in the true-false test

J Educ Psychol 1942 09 33 6 401 415

10.1037/h0054677

260

Mead

Smith

Does the true-false scoring formula work? Some data on an old subject

J Educ Res 1957 51 1 47 53

10.1080/00220671.1957.10882437

261

Harden

Gleeson

Assessment of clinical competence using an objective structured clinical examination (OSCE)

Med Educ 1979 01 13 1 39 54

763183

262

Fleming

The profitability of ‘guessing’ in multiple choice question papers

Med Educ 1988 11 22 6 509 513

10.1111/j.1365-2923.1988.tb00795.x

3226344

263

Jacobs

Chase

Developing and Using Tests Effectively 1992

San Francisco, CA

Jossey-Bass Inc

264

Hammond

McIndoe

Sansome

Spargo

Multiple-choice examinations: adopting an evidence-based approach to exam technique

Anaesthesia 1998 11 53 11 1105 1108

10.1046/j.1365-2044.1998.00583.x

10023280

265

Chase

Contemporary Assessment for Educators 1999

New York, NY

Longman

266

Downing

The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education

Adv Health Sci Educ Theory Pract 2005 10 2 133 143

10.1007/s10459-004-4019-5

16078098

267

Dijksterhuis

MGK

Scheele

Schuwirth

LWT

Essed

GGM

Nijhuis

Braat

DDM

Progress testing in postgraduate medical education

Med Teach 2009 10 31 10 e464 e468

10.3109/01421590902849545

19877854

10.3109/01421590902849545

268

Staffelbach

Weighting responses in true-false examinations

J Educ Psychol 1930 02 21 2 136 139

10.1037/h0072266

269

Gates

The true-false test as a measure of achievement in college courses

J Educ Psychol 1921 05 12 5 276 287

10.1037/h0074436

270

Rao

A note on the evaluation of the true-false and similar tests of the new-type examination

Indian J Psychol 1937 12 176 179

271

Kirstges

Gerechte Noten: Zur Gestaltung von Notensystemen für die Beurteilung von Leistungen in Klausuren [Fair grades: designing grading systems for assessing performance in exams]

Neue Hochschule 2007 48 3 26 31

272

Frary

NCME instructional module: formula scoring of multiple-choice tests (correction for guessing)

Educ Meas 1988 06 7 2 33 38

10.1111/j.1745-3992.1988.tb00434.x

273

Lukas

Melzer

Much

Auswertung von Klausuren im Antwort-Wahl-Format [Evaluation of Multiple-Choice Examinations] 2017

Halle (Saale), Germany

Center for Media-Enhanced Learning and Teaching (LZZ) of the Martin Luther University of Halle-Wittenberg

274

Tarrant

Knierim

Hayes

Ware

The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments

Nurse Educ Today 2006 12 26 8 662 671

10.1016/j.nedt.2006.07.006

17014932

S0260-6917(06)00106-7

275

de Laffolie

Visser

Hirschburger

Turial

„Cues“ und „Pseudocues“ in chirurgischen MC-Fragen des deutschen Staatsexamens [Cues and pseudocues in surgical multiple choice questions from the German state examination]

Chirurg 2017 03 88 3 239 243

10.1007/s00104-016-0291-1

27678403

10.1007/s00104-016-0291-1

276

Kanzow

Schmidt

Herrmann

Wassmann

Wiegand

Raupach

Use of multiple-select multiple-choice items in a dental undergraduate curriculum: retrospective study involving the application of different scoring methods

JMIR Med Educ 2023 03 27 9 e43792

10.2196/43792

36841970

v9i1e43792

277

Kubinger

Gutachten zur Erstellung „gerichtsfester“ Multiple-Choice-Prüfungsaufgaben [Expert opinion on the creation of “lawful” multiple-choice items]

Psychol Rundschau 2014 07 65 3 169 178

10.1026/0033-3042/a000218