More Meaningless Education Research

There is no shortage of dubious education research.  Reports “proving” that new teachers are better than old, charter schools are better than non-charter schools, and graduate schools of education are useless seem to pop up frequently.  If you have a loose-grasp of statistics and the willingness to tell someone what they want to hear, chances are there’s funding available for your study.

So it was no surprise to see exam schools finally make their way into the discourse.  The following study appeared in the New York Times, grabbing headlines with its claim that “the impact of attending an exam school school on college enrollment or graduation is, if anything, negative.”

http://artsbeat.blogs.nytimes.com/2011/08/16/thinking-cap-angst-before-high-school/

Exam schools grant admission based on a standardized test.  By achieving a minimum score on the test (the school’s “cutoff”), the student can choose to attend the school.  These public schools typically offer advanced courses and more rigorous instruction, and one would think that students would get a lot out them.  Not according to the authors of this study, who conclude that, in these schools, students’ “actual human capital essentially remains unchanged”.  In jargon common to these kinds of studies, exam school schools don’t add any value to the educational experience of students.

A cursory review of the study suggests some obvious problems, many of which are pointed out in the comments section of the original Times article.  However, a close review of the study revealed something so absurd, it makes the study seem not so much flawed as intentionally misleading by design.

The basic premise of the study is to compare students who just make the cutoff for an exam school with those who just miss that cutoff.  In theory, since these students have similar tests scores, they start with similar levels of ability.  Some of them enter the exam school, and some of them don’t.  By comparing their later achievement, we can get a sense of what, if anything, attendance at the exam school adds.

Let’s say that Student 1 just makes the cutoff for Exam School A, and Student 2 just misses that cutoff and thus attends a different school.  The study claims that Students 1 and 2 will go on to have similar SAT scores and have roughly the same chance of graduating college.  That is, attending the exam school does not add any value for Student 1.

What the study doesn’t take into account is that the school Student 2 ends up attending is also likely to be an exam school!  Student 2, who just missed the cutoff for Exam School A, might very well attend Exam School B, which has a lower cutoff.  In the eyes of this study, however, Student 2’s success at Exam School B counts as evidence that exam schools don’t add value!

In the New York City system, where this study was conducted, this situation arises frequently.  A student might miss the cutoff for one exam school but attend another exam school.  Indeed, the authors themselves note that in the case of one particular school, 40% of the students who miss the cutoff end up attending a second particular exam school.  And when they succeed, they all count as evidence against exam schools.

There are other serious issues regarding this study’s methodology, but to me this is the most significant.  Moreover, the obvious gap between what was actually done and what was purported to be done is very disturbing.

I wonder how closely such studies are read, and I wonder what this has to say about the state of current education “research” in general.

By patrick honner, ago

MIT OpenCourseWare

One of the true marvels of the internet-age, MIT’s OpenCourseWare project makes video lectures, notes, problem sets, and exams from over 2,000 MIT courses freely available to anyone with an internet connection.

http://ocw.mit.edu/index.htm

Naturally, MIT’s OCW project offers courses in technical fields like Mathematics, Engineering, Chemistry, and Computer Science.  But many humanities courses, in subjects such as Literature, Foreign Language, and Music, are also available.  You can browse all of the courses here:  http://ocw.mit.edu/courses/.

The Multivariable Calculus course has been especially helpful for me.  Not only can I brush up on techniques in vector calculus, but by watching the lectures I get a different teaching perspective on the material.  Browsing the problem sets and tests gives me ideas for my own assignments and assessments.  And, naturally, it’s a wonderful resource for the students!

I’ve also enjoyed making my way through some of the advanced statistics and physics courses MIT has to offer.  As a teacher, it’s always nice to remind yourself how lost one can be as a student!

By patrick honner, ago

Wrapped Trapezoid

Wrapping up this gift was much more challenging than I expected.

But it got me thinking about how this could be an interesting project.  Questions such as “What’s the smallest square piece of wrapping paper that could do the job?” or “What kind of rectangle would work best?” are compelling and  easy to investigate.  And maybe someone could make a triangle or trapezoid do the job efficiently.  There’s a lot of room for creativity and exploration here.

The usual restrictions on tearing and cutting would apply, although relaxing those restrictions might create interesting problems, too.

It wasn’t easy, but I did pick up some unexpected ideas along the way.  And the gift was well-received, too!

By patrick honner, ago

Are These Tests Any Good? Part 2

This is the second entry in a series that examines the test quality of the New York State Math Regents Exams.  In the on-going debate about using student test scores to evaluate teachers (and schools, and the students themselves), the issue of test quality rarely comes up.  And the issue is crucial:  if the tests are ill-conceived, poorly constructed, and erroneous, how legitimate can they be as measures of teaching and learning?

In Part 1 of this series I looked at three questions that demonstrated a significant lack of mathematical understanding on the part of the exam writers.  Here, in Part 2, I will look at three examples of poorly designed questions.

The first is from the 2011 Integrated Algebra Regents:  how many different ways can five books be arranged on a shelf?

This simple question looks innocent enough, and I imagine most students would get it “right”.  Unfortunately, they’ll get it “right,” not by answering the question that’s been posed, but by answering the question the exam writers meant to ask.

How many different ways are there to arrange five books on a shelf?  A lot.  You can stack them vertically, horizontally, diagonally.  You can put them in different orders; you can have the spines facing out, or in.  You could stand them up like little tents.  You could arrange each book in a different way.  The correct answer to this question is probably “as many ways as you could possibly imagine”.  In fact, exploring this question in an open-ended, creative way might actually be fun, and mathematically compelling to boot.

But students are trained to turn off their creativity and give the answer that the tester wants to hear.  A skilled test-taker sees “How many ways can five books be arranged on a shelf?” and translates it into  “If I ignore everything I know about books and bookshelves, stand all the books upright in the normal way, don’t rotate, turn or otherwise deviate from how books in math problems are supposed to behave, then how many ways can I arrange them?”

This question is only partly assessing the student’s ability to identify and count permutations.  This question mostly tests whether the student understands what “normal” math problems are supposed to look like.

This problem is an ineffective assessment tool, but there’s something even worse about it.  Problems like this, of which there are many, teach students a terrible lesson:  thinking creatively will get you into trouble.  This is not something we want to be teaching.

Here’s a question from the 2011 Algebra II and Trigonometry exam:

Solving equations is one of the most important skills in math, and this question pertains to a particular method (completing the square) used to solve a particular kind of equation (quadratic).  But instead of simply asking the student to solve the problem using this method, the question asks something like “if this procedure is executed normally, what number will be written down in step four?”.

This is not testing the student’s ability to do math; instead, it’s testing whether or not they understand what “normal” math looks like.  There are many ways to solve equations, and there are many ways a student might use this method.  Whether it looks exactly like what the teacher did, or what the book did, isn’t especially relevant.  So why is that being tested?  And like the question above, this reinforces the idea that thinking creatively can be dangerous by insisting that students see the “normal” solution as the only correct one.

Finally, here’s a problem from the 2011 Geometry Regents:

Once again, the student is not being tested on their knowledge of a concept or on their ability to perform a task.  Instead, they’re being tested on whether or not they recognize what “normal” math looks like, and that’s just not something worth testing.  There are lots of legitimate ways to construct a perpendicular bisector:  why are we testing whether the student recognizes if the “normal” way has been used?

These three problems showcase some of the dangers inherent in standardized testing.  Questions like these, and the tests built from them, discourage creative thinking;  they send students the message that there is only one right way to do things; they reinforce the idea that the “correct” answer is whatever the tester, or teacher, wants to hear; and they de-emphasize real skills and understanding.

At their worst, these tests may not just be poor measures of real learning and teaching; they may actually be an obstacle to real learning and teaching.

Related Posts

By patrick honner, ago
Follow

Get every new post delivered to your Inbox

Join other followers: