A colleague and I had an interesting conversation about a quiz question of his. He asked students to put the four following quantities in order from least to greatest:
I don’t think I’ve ever written a problem like this for a test or quiz myself, which may explain why thinking about how to assess potential answers perplexed me so. How, exactly, would you grade this problem?
Suppose the correct order is A > B > C > D. A first attempt to grade this problem might be “One point for every item in the correct absolute position”. So
A > B > D > C
would earn two out of four points, as only A and B are in the correct absolute position, while
B > D > C > A
would earn one point, as only the C is in correct absolute position.
But there are some issues with this approach. This response:
B > C > D > A
earns zero points, since no item is in the correct absolute position. But in some sense this response is mostly correct: three of the items (B, C, and D) are correctly ordered relative to each other. Notice also that, in this scoring system, it is impossible to get 3 our of 4 points: the only possible scores are 0, 1, 2, or 4.
My suggestion was to consider all 
B > C > D > A
implies the following correct relationships: B > C, B > D, and C > D. The other three implied relationships are incorrect, thus this response would earn 3 out of 6 points.
The problem with this scoring system is that it is nearly impossible to get a zero! The only response that earns zero points under this scheme is
D > C > B > A
but this answer is compelling in its own way. The correct order is perfectly reversed, which strongly suggests the student knew something about the order of the items.
I like this problem, but I don’t think I”d put it on a test. It’s too hard to assess! But it was fun and mathematically interesting to think about, so maybe I’ll put the question “How would you grade this question?” on a test.
