Use of Advanced Multiple Response Questions in Small Groups

For some time now, we have been touting the benefits of diversifying away from over-reliance on multiple choice test questions and encouraging educators to broaden their assessment toolkit.  The Advanced Multiple Response test item is one such tool.  It is designed to assess whether students can think critically and apply what they have learned.  Below we describe a way that these items can be used with students working in small groups.

Exam Master’s Advanced Multiple Response (AMR) test question may resemble at first glance a “Select All That Apply” (also referred to as a SATA) test question.  This is because both typically require an examinee to select more than one “correct” answer choice from a set of answer choices in order to get full credit for the question.  However, the Advanced Multiple Response includes two attributes not normally found on the SATA question-type.  Firstly, all of the answer choices in an AMR question are assigned point values (both positive and negative can be used).  Thus, greater or lesser degrees of “correctness” for each answer choice can be differentiated via the assigned point values.  The question, therefore, allows for faculty to designate a passing score on the question based on the points they’ve assigned to each individual answer choice.  Note, a passing grade on an AMR question may not require the student to earn all of the maximum available points.  That choice is up to the faculty person creating and assigning the question.

Another unique aspect of the way we handle AMR questions is to include the availability of a Critical Failure attribute that can be attached to any answer choice.  This is very important in the health sciences, and assessments at the clinical level should include this type of outcome option.  For example, while performing a clinical procedure for grade, if a student fails on one step of the rubric, the expectation should be that the student failed the entire procedure.  The logic behind this is that in many cases, poor clinical decisions or actions harm patients and negatively affect outcomes. So, with the AMR question, one or more of the answer choices can be assigned a critical failure attribute.  If the student selects that answer choice as part of what they believe to be the set of correct responses, they automatically get the question wrong.  Getting it wrong means they have not mastered the material sufficiently well to earn a passing grade.

Now imagine you have a small group of students working together, looking at one of these more complex questions.  They often will include a detailed clinical scenario or they might include other interpretive information such as charts, radiographs and the like.  Now imagine that you assign these students with the task of trying to reach a consensus on what set of answer choices would equate to a correct response, and earn them maximum credit.  Now imagine, also, that students are informed that they need to be aware of the land mines:  the answer choices tagged with critical failures.  Think about the value of having the students work through these questions, and as a team, trying to solve them.  Both negative point values and critical failures discourage guessing.  Students must constructively work together to identify the best responses.  Besides knowledge, comprehension and higher cognitive ability, think about professionalism, interpersonal skills, teamwork, collaboration and even how leadership qualities might play a factor in this type of interaction and problem-solving.  

Please let us know what you think.  If you are interested in seeing a sample of our latest AMR questions (Family Medicine) please click here.  Let us show you how these can be used to your students’ and program’s benefit.

Related Topics: Assessment // Formative Assessment // Multiple Response // Test items // Test Questions