Regents Chancellor Merryl Tisch has been pushing New York’s new and (we’re assured) improved state testing program — but the public can have no confidence in these tests until we’ve had a thorough, independent investigation of the old, unimproved program.
Key data on the old tests, long under-exposed and unpublicized by the State Education Department, show signs of serious flaws.
To understand, consider the National Assessment of Educational Progress, considered “the gold standard” in testing. These exams consist of multiple-choice and open-ended items. Open-ended questions place a greater cognitive demand on students, asking them to interpret reading passages or solve math problems rather than select or guess at the right answer.
In well-constructed tests, students are expected to get a higher percentage of correct answers on the simpler multiple-choice questions. But performance on the two sets of items should also run fairly parallel — with students who do well on one type of item generally doing so on the other.
Stability and orderliness characterize New York’s NAEP results over time — for math and reading, in both Grade 4 and Grade 8. (NAEP does not test every grade, or every year.)
The English Language Arts and Math tests sold to New York by CTB/McGraw-Hill (under a five-year, $38 million contract) also consist of multiple-choice and open-ended questions. Resemblance to the NAEP ends there.
To put it simply, on New York’s tests, the underlying results (that is, the average percentage correct on each item) have varied widely from one year to the next. In one year, scores on multiple-choice items shoot up while scores on open-ended questions fall; in another, the reverse. In the 2008 Grade 4 math test, the open-ended questions actually proved easier than the multiple-choice. The 2011 English exams for Grades 5 and 7 show the same reversal.
All in all, the state exams taken every year by 1.2 million kids in Grades 3 to 8 show marked inconsistency — incoherent patterns that signal poorly developed exams.
Yet the State Education Department has effectively hidden these facts, by not reporting the separate information. Instead, it announces “composite” test results — muddled information that conceals messy contradictions. It is hard to see this as anything but willful deception.
Of course, revealing this mess would raise awkward issues:
* Why has the state spent so much for a defective product?
* Why are New York’s open-ended questions so much easier than the NAEP’s — still 26 percent easier this year, when our tests supposedly got harder? (Is it a scoring problem — that is, are teachers or whole districts being “generous” in order to make themselves look better?)
* Where were the Regents for the last decade, when officials were issuing misleading press releases and generally covering the obvious flaws in the exams?
Tisch and others insist that the important thing is to get the tests right in the future. But until we’re clear on what went wrong in the past — and adopt serious reforms, such the timely release of data for outside analysts to examine — the State Education Department is all too likely to make the same “mistakes” again.
Fred Smith is a retired Board of Education senior analyst.