Testing

Teaching Testing

Regents Recap — June 2015: Cubics, Conversions, and Common Core

Here is another installment in my series reviewing the NY State Regents exams in mathematics.

One of the biggest differences between the new Common Core Regents exams and the old Regents exams in New York state are the conversion charts that turn raw scores into “scaled” scores.

The conversions for the new Common Core exams make it substantially more difficult for students to earn good scores. The changes are particularly noticeable at the high end, where the notion of “Mastery” on New York’s state exams has been dramatically redefined.

Below is a graph showing raw versus scaled scores for the 2015 Common Core Algebra Regents exam.

As with last year’s Common Core Algebra Regents exam, there is a remarkable contrast between “Passing” and “Mastery” scores. To pass this exam (a 65 “scaled” score), a student must earn a raw score of 30 out of 86 (35%); to earn a “Mastery” score on this exam (an 85 “scaled” score), a student must earn a raw score of 75 out of 86 (87%). It seems clear that the new conversions are designed to reduce the number of “Mastery” scores on these exams.

Another curious feature of this conversion chart is what happens at the upper end. Consider the graph below, which shows the CC Algebra raw vs. scaled score in blue and a straight-percentage conversion (87% correct “scales” to an 87, for example) in orange.

At the very high end, the blue conversion curve dips below the orange straight-percentage curve. This means that, above a certain threshold, there is a negative curve for this exam! For example, a student with a raw score of 82 has earned 95% of the available points, but actually receives a scaled score of less than 95 (a 94, in this case). I suspect there are people who will claim expertise in these matters and argue that this makes sense for some reason, but it certainly doesn’t make common sense.

One final curiosity about this conversion. It’s no accident that the blue plot of raw vs. scaled scores looks like a cubic function.

Running a cubic regression on the (raw score, scaled score) pairs yields

$f(x) = 1.13 + 3.96x -0.76x^2 + .0005x^3$

$R^2 = 0.99$

That is a remarkably strong correlation. Clearly, those responsible for creating this conversion began with the assumption that the conversion should be modeled by a cubic function. What is the justification for such an assumption? It’s hard to believe this is anything but an arbitrary choice, made to produce the kinds of outcomes the testers know they want to see before the test is even administered.

These conversion charts are just one of many subtle ways these tests and their results can be manipulated. Jonathan Halabi has detailed the recent history of such manipulations in a series of posts at his blog. These are the kinds of things we should keep in mind when tests are described as objective measures of student learning.

Related Posts

By MrHonner, 9 years8 years ago

Teaching Testing

Hannah and Her Sweets

Much has been written and tweeted about this problem from a recent math exam administered in the UK:

After the exam, students took to social media to express their outrage at the absurdity of this question. This prompted some reaction pieces from mathematicians and math teachers defending this problem as a demonstration of a link between probability and algebra and as a non-routine problem-solving challenge.

The mathematical status of this problem is less interesting to me than its status as a test item. And as a test item, I think this is not only terrible, but also damaging.

The first eight sentences of this test item clearly indicate to the student that this is a probability problem. Then, it abruptly ceases to be a probability problem and becomes a problem about quadratic equations. No meaningful connection is made between the two concepts: the entire probability story simply exists to establish algebraic conditions on the number n. (And even in a world where contrived test questions are commonplace, this silly story stands out.)

For most students, this test question just reinforces the notion that math makes no sense. And I’m sure others come away feeling cheated, or deceived, by the exam-writers. High-stakes exam questions like this damage student attitudes about mathematics and learning, and have broad, long-term consequences that few people seem to think about.

This problem reminds me of the saga of “The Pineapple and the Hare“. A few years ago, a number of questions on an 8th-grade English exam referred to an absurd passage about a talking pineapple. The passage and the questions were published online, and the ensuing public outcry led to those items being nullified on the exam.

Yet test-writers defended the passage and the items as an effective discriminator: only the highest functioning test-takers could weave their way through the absurdity to answer the questions correctly. Thus, it effectively served to identify the highest performers.

Even if that were all true, why should the navigation of nonsense be a focus of our educational program? And what of the students who come away from such tests feeling demoralized and alienated because a probability abruptly became an algebra problem?

Regardless of what people think about this particular question, I’m glad that, more and more, we seem to be asking the question, “Are these tests any good?“.

By MrHonner, 9 years8 years ago

Statistics Teaching Testing

Regents Recap — January 2015: Questions with No Correct Answer

Here is another installment in my series reviewing the NY State Regents exams in mathematics.

This is question 14 from the Common Core Algebra exam.

Setting aside the excessive, and questionable, setup (do people really think about minimizing the interquartile range of daily temperatures when choosing vacation spots?), there is a serious issue with this question: it has no correct answer.

The student is asked to identify the data set that satisfies the following two conditions: median temperature over 80 and smallest interquartile range. No data set satisfies both these conditions. According to the diagram, the data associated with “Serene Shores” has the smallest interquartile range (represented by the width of the “box” in the box-and-whisker plot), but its median temperature (the vertical line segment in the box) is below 80.

The answer key says that (4) is the correct answer, but that data does not have the smallest interquartile range shown. Presumably, the intent was for students to evaluate a conditional statement, like, “Among those that satisfy condition A, which satisfies condition B?” But as written, the question asks, “which satisfies both condition A and condition B?” No set of data satisfies both.

Some may consider this nitpicking, but precision in language is an important part of doing mathematics. I focus on it in my classroom, and it is frustrating to see my work undermined by the very tests that are now being used to evaluate my job performance.

Moreover, this is by no means the only error present in these exams, nor is it the first example of errors in stating and evaluating compound sentences. If these exams don’t model exemplary mathematical practice, their credibility in evaluating the mathematical practice of students and teachers must be questioned.

By MrHonner, 10 years10 years ago

Teaching Testing Uncategorized

Regents Recap — January 2015: Admitting Mistakes

Here is another installment in my series reviewing the NY State Regents exams in mathematics.

This is question 27 from the Geometry exam.

This question has two correct answers: it is possible to map AB onto A’B’ using a glide reflection or a rotation. The original answer key indicated that (4) glide reflection was the correct answer. After the exam was administered, the state Department of Education issued a correction and told scorers to award full credit for both (2) and (4).

Mistakes happen, even on important exams that many people work hard to produce. But when mistakes are made, those responsible should accept responsibility, not equivocate.

Here’s the official correction from the state.

It’s hard to accept that the issue here was a lack of specificity in the wording of the question. The issue is that someone wrote a question without fully thinking through the mathematics, and then those tasked with checking the problem also failed to fully think through the mathematics. This isn’t a failure in communication; this is a failure in management and oversight.

And it has happened before. This example is particularly troubling, in which those responsible for producing these exams try to pretend that an egregious mathematical error is really just a lack of agreement about notation. Sometimes errors are just erased from the record with little or no explanation, and then, of course, there are the many mistakes that are never even acknowledged.

Mistakes are bound to happen. But by pretending that substantial errors are just misunderstandings, differences of opinion, or typos, the credibility of those responsible for these high-stakes exams suffers even further damage.

By MrHonner, 10 years10 years ago

Teaching Testing

Regents Recap — January 2015: It’s True Because It’s True

Here is another installment in my series reviewing the NY State Regents exams in mathematics.

This is question 25 from the Common Core Algebra exam.

I’ve already complained about the contrived, artificial contexts for these questions (why not just ask “Is the sum of these two numbers rational or irrational?”), so I’ll ignore that for now. What’s worth discussing here is the following sample student response provided by the state.

So, why is the sum of a rational number and an irrational number irrational? Because the sum of a rational number and an irrational number is always irrational. This circular argument is offered as an example of a complete and correct response.

I’m not sure there’s a way to rewrite this question so that it admits a sensible answer. That’s probably a good indication that it shouldn’t be on a high-stakes test.

As I’ve argued time and again, questions on these exams should stand as examples of proper mathematics. But questions like this actually encourage bad habits in students, and teachers too, who are being told that this constitutes an appropriate response to this question. This is yet another example of the danger of simply tacking on “Justify your reasoning” to a high-stakes exam question.

Related Posts

By MrHonner, 10 years8 years ago

Follow Mr Honner