Designing Exam Questions

test-questions

Criteria for Questions

Writing exam questions can be challenging. Best Practices for Designing and Grading Exams suggests that good exams should have the following characteristics:
  • Valid: they should provide useful information about the concepts they were designed to test
  • Reliable: they should allow consistent measurement and should be able to discriminate between different levels of performance
  • Recognizable: instruction has prepared learners for the exam; it aligns with the exam content
  • Realistic: the time and effort required to complete the exam is realistic

Exams can comprise a variety of items including True/False, Multiple-Choice, Matching, Short Answer/Fill in the Blank, and Essay. Each of these is associated with advantages and disadvantages as outlined in the table below:

Type of Exam Item

Advantages Disadvantages
True-False
  • many items can be administered in a short period of time
  • moderately easy to write
  • easily scored
  • limited primarily to testing knowledge/recollection
  • easy to guess correctly on many items even if the material is not mastered
Multiple-Choice
  • can be used to assess a broad range of content in a brief period
  • skillfully written items can measure higher-order cognitive skills
  • easily scored
  • difficult and time-consuming to write good items
  • possible to assess higher-order cognitive skills, but most items assess only knowledge/recollection
  • some correct answers can be guessed
Matching
  • items can be written quickly
  • a broad range of content can be assessed
  • easily scored
  • higher-order cognitive skills are difficult to assess

Short Answer or 

Fill in the Blank

  • can be administered in a brief period of time
  • moderately easy to write
  • relatively easy to score
  • sometimes challenging to identify criteria for correct answers
  • limited to questions that can be answered or completed in very few words
Essay
  • can be used to measure higher-order cognitive skills
  • relatively easy to write 
  • difficult for respondents to get correct answers by guessing
  • time-consuming to administer and grade
  • challenging to identify criteria for scoring
  • only a limited range of content can be assessed during any one testing period

Adapted from Worthen et al., 1993

If exams, especially closed book/notes exams, are a significant part of your assessment strategy, it is important to intentionally design your course in a way that supports learners' ability to succeed and demonstrate their learning on your exams. By leveraging cognitive science principles that improve recall, building in repeated low-stakes practice quizzes, polling questions, homework assignments, and answering verbal questions in class, learners can improve recall, understanding, and comprehension. Writing exam questions for learners throughout a semester and receiving feedback from learners can also help instructors improve their own skills in developing good exams. For more information on organizing course material, enhancing memory, and metacognitive principles access the Center for Faculty Development and Excellence (CFDE) Learning Science webinar series available here. 

Developing Exams to Evaluate Higher-Order Thinking 

Exam questions can be written at different levels of Bloom’s Taxonomy  though some question types are better suited for assessing certain levels of cognitive complexity than others. For example, multiple-choice questions are best at assessing skills related to remembering or understanding while short answers and essays are better suited to allow learners to demonstrate their ability to evaluate, analyze and create. Bloom's Taxonomy is a useful framework that allows instructors to assure that exam questions are developed at the appropriate cognitive level to align with the cognitive complexity associated with learning tasks and course learning objectives. The Kansas State University Guide provides ideas on how exam questions of all types can be upgraded to cognitively challenge learners. 

pencil-banner

 Things to Consider When Developing an Exam

  • Align exam questions with course objectives; work with objectives that are well-suited to testing.
  • Construct exam questions that reflect the emphasis placed on the material during the course.
  • Questions should be stated clearly; don't write vague/ambiguous questions.
  • Develop instructions that preview the exam; give examples if needed for clarity.
  • Enlist other instructors or your Teaching Assistant to review your test questions, take the exam informally, and provide feedback on accuracy, clarity, and length.
  • Maintain learner attention and engagement in longer exams, which have more reliable outcomes with a variety of question types. 
  • Consider the point values of different question types and align them with the emphasis on and importance of the concepts and material.
  • Establish transparent grading criteria and publish those criteria.
  • Include questions that have a difficulty level appropriate to the learning objectives.
  • Review policies and procedures for accommodations and make learners aware of them.
  • Evaluate the test by also reviewing:
    • the cognitive, "thinking" level of questions 
    • the meaningfulness and value of what is tested
    • the clarity and inclusiveness of the language used in the questions
    • the fairness of the questions
    • the reliability of the test outcomes over time

Tips on Writing Individual Exam Questions

  • Write questions with one best answer.
  • Test a single idea per question.
  • Incorporate common learner misconceptions and errors as wrong answers.
  • Write more questions than you need, in case you want to create different exam versions.
  • Avoid extreme or vague modifiers or qualifiers (e.g. generally, all, frequently)
  • Randomize answers and avoid patterns in answer selections.
  • Keep the length of answer choices as short as possible.
  • Sparingly write questions with a negative statement.
  • Organize questions by question type (multiple-choice, true-false, matching, fill-in-the-blank)
  • Write explicit directions for each question type.
  • Eliminate grammatical clues that point to the correct answer.
  • Don't overuse “All of the above” and “None of the above” in responses. 
  • Review these tips specific to writing multiple-choice questions. 

 

References

Worthen, B.R., Borg, W.R., & White, K.R. (1993). Measurement and evaluation in the schools. New York: Longman.

Resources

A Short Guide to Writing Effective Test Questions - Kansas State University.

Center for Research on Learning and Teaching, University of Michigan. Framework for Designing Effective Exams

Center for Research on Learning and Teaching, University of Michigan. Best Practices for Designing and Grading Exams

Center for Research on Learning and Teaching, University of Michigan. Writing Questions

test-with-pencil