For better multiple-choice tests, avoid tricky questions, study finds

"Although people often think about multiple-choice tests as tools for assessment, they can also be used to facilitate learning," said Andrew Butler, a cognitive psychologist in Arts & Sciences who studies the brain processes behind learning and recall. "The act of retrieving information strengthens memory for that information, leading to better long-term retention, and changes the representation of the information, creating deeper understanding."

Butler's study, published in the September issue of Journal of Applied Research in Memory and Cognition, offers straightforward tips for constructing multiple-choice questions that are effective at both assessing current knowledge and strengthening ongoing learning.

Among key findings, educators should never include trick questions or offer "all of the above" or "none of the above" options among the list of possible answers.

Research on the format of multiple-choice questions is important, Butler noted, because the tests are widely used throughout the world, especially in the United States where they originated as part of early efforts to measure intelligence.

Fueled in the beginning by the need for an efficient way to measure characteristics of World War I soldiers and booming student enrollments, multiple-choice tests now influence important life decisions in areas such as college placement, workplace hiring, career advancement and even online dating.

As an associate professor in the Departments of Education and of Psychological & Brain Sciences, both in Arts & Sciences, Butler conducts research that explores the malleability of memory -- the cognitive processes and mechanisms that cause memories to change or remain stable over time.

Taking any form of test has the potential to alter our understanding of a topic, he said, because the process of recalling information requires important details to be freshly reconstructed from related memories.

While multiple-choice testing, especially repeated testing, has the potential to strengthen our recall, a poorly formatted test question can have the opposite effect, Butler said. Such an ill-formed question can muddy our recollection of the correct answer and reinforcing memories for inaccurate "distractor" answers, he added.

Butler's research review confirms that proper question formatting and presentation are critical to creating effective multiple-choice tests. It also suggests that many widely used multiple-choice tests still include lots of questions that fail to comply with research-based best practices.

"Fortunately, the best practices for creating multiple-choice tests that effectively assess understanding are much the same as those for supporting student learning," Butler said.

Butler's study explains the cognitive science behind five research-based recommendations for crafting more effective multiple choice questions:

• Create questions with simple formats. Complex question-and-answer formats have become popular as educators look for ways to test for deeper understanding and higher-level learning. Examples include offering answers such as "A and B, but not C," or allowing repeated answers until correct choice is made. Such formats may be detrimental to assessment since guessing is encouraged and processing may focus more on parsing question context and less on recalling correct information. Some complex formats, such as confidence-rated answers, may offer benefits, but more research is needed.
• Create questions that engage "real world" cognitive processes. To truly test abilities, questions must be structured so correct answers require use of the specific cognitive processes necessary to address similar problems in the real world. Questions that require higher-order thinking will enhance learning and improve future performance. For example, a multiple-choice question could require a test-taker to contrast two concepts (Which of the following is a way in which hawks differ from eagles?) or analyze a set of conditions to make a decision (Given a patient symptoms, which of the following diagnoses is most likely?).
• Avoid using "none of the above" and "all of the above" as answer choices. When "none-of-the-above" is correct, students may not need to retrieve correct information to answer the question and they are exposed a lot of incorrect information. Using "all of the above" exposes students to a lot of correct information, but answers may be more obvious, robbing students of potential learning that comes from recall processing. Both question types can be detrimental to accurate assessment and potential benefits to learning are small.
• Use three plausible response options. Question difficulty increases with each answer option offered. Students who correctly answer more difficult questions may learn more from rising to the challenge, but questions that offer too many plausible answers can have a negative effect on both learning and assessment. Use the Goldilocks principle: not too many, or too few.
• Make the test challenging, but not too difficult. Create tests that are hard enough to reveal how well students know the material, but easy enough that a majority (80 percent) get a passing grade. Retrieving information and answering questions correctly reinforces student learning; failing to answer correctly may strengthen memories for misinformation. Challenge students, but allow them to succeed.

Finally, because multiple choice questions expose students to lots of plausibly presented false information, it's important for students to review answers after grading is completed. Feedback enables test-takers to correct errors and avoid internalizing incorrect information. It also strengthens learning around correct answers that were low-confidence guesses at test time.

"One takeaway from these recommendations is that the most effective multiple-choice items get students to think in ways that are productive for learning and enable valid measurement of whether they have acquired the desired skills and knowledge," Butler said. "To maximize both effectiveness and efficiency, it is also best to keep the process of answering multiple-choice items simple -- added complexity often has a negative effect on both learning and assessment."