Common Challenges with Vetting Test Items
Grammar
Even in novels published by major publishers such as Simon & Schuster or Penguin Random House, grammatical errors are found. If you’ve checked a book out of the library lately, you may have noticed that someone who read the book before you has pointed out typos or poor grammar in the book’s margins. English is known to be a tricky language, and the language’s grammar rules aren’t always rigid and agreed upon. For example, while many people use the “Oxford” comma, others find it unnecessary and never use it (we happen to like it here at Exam Master). English professors might argue that it is perfectly acceptable to end a sentence with a preposition under certain circumstances, as not doing so can create confusion for the reader.
At Exam Master, we are primarily concerned with grammar mistakes that might change the meaning of a sentence or paragraph, particularly as it relates to medical accuracy. A comma really can change the entire meaning of a sentence, after all. Missing words are even more problematic. Read one of our questions with the word “not” in it, and take away that “not”. At best, the question is somewhat unreadable; more likely, however, it is not at all medically accurate.
Spelling of Medical Terminology
When it comes to our questions, our chief concern at Exam Master is medical accuracy and relevancy. We hire qualified physicians and nurses to author/review our questions, but sometimes a drug or medical condition is spelled incorrectly. Why does this happen? Well, check out this list of the most commonly misspelled words (http://grammar.yourdictionary.com/spelling-and-word-lists/misspelled.html) and be honest when you answer this question: if you had to write an essay using all 100 of these words without using a spell checker, how many of them would you spell incorrectly? We often struggle to correctly spell words that we say out loud on a daily basis, and most of them aren’t nearly as long or complicated as Burkholderia pseudomallei or Thermodesulfobacterium hydrogeniphilum. Many of our test questions are dense with information, making grammar and spelling reviews particularly challenging.
Question Difficulty
While most of the questions our authors submit are appropriate, for every 100 receive, there are always a few that appear too easy or too difficult and for the resource and the audience they are intended. Of course, the true measure of whether a question is too easy or too difficult is through psychometric testing.
A question is generally deemed “too difficult” when it tests a disease or medical condition that is extremely rare, or more likely, the degree of education and training required to correctly answer the question is beyond the scope of what the learner is supposed to know at that point in their education and training. Thus all of our questions get what we call a relevance review where the subject matter expert makes a judgment about whether the question is suitable as to scope and difficulty for the intended learner.
On the flip side, there are several possible factors for why a question may be considered too easy, and therefore inappropriate for our products. Most of the questions found in our resources are case-based questions, often ones which require critical thinking, so-called “higher order” questions. The easier questions, on the other hand, are those which test factual knowledge and factual recall. While some of these are appropriate in a variety of assessments, these tend to be easier questions for the student to answer. Poorly written test items can also be easier in some cases because the range of possible answers does not present realistic options for the information presented, thus making the correct answer easier to spot. In addition, an author might inadvertently include keywords or “buzzwords” that could give away the answer to the question. Another common pitfall is when a test item author has made the correct answer choice the longest one!
References
For obvious reasons, references are often an important part of any exam preparation question. Our users often want to know that the medicine in our questions is valid, and where the information was sourced. However, medicine changes at a rapid rate. The best diagnostic test for a condition in 2012 might not be the best diagnostic test for the same condition in 2017. While the formatting of references is important, at Exam Master, our focus is on making sure that the texts and articles that comprise our references are from generally accepted primary authorities and are currently valid.
After reviewing thousands of test questions over twenty years of supporting the health sciences, we have gotten pretty good at separating the wheat from the chaff so to speak. Developing a good, psychometrically sound test item can be difficult, and which is why we help our clients with this important task. Even a good clinician or instructor might not be able to develop credible test questions without training, practice, and more practice. The true test of a quality test item is how it performs in an examination. Our processes are designed to filter out the good from the bad, the relevant from arcane, and the strong from the weak so that our learners and faculty can have the best possible outcomes.