The value of a college degree has perhaps never been more in question and yet we’re still struggling with whether and how to measure it.
But a new study suggests it may be possible to test what students actually learn in college, and the researchers behind it, including a former university president, are urging higher education leaders to use the findings as a call to quantify their schools’ effectiveness at helping students improve.
The researchers measured the writing abilities of more than 300 college students between 2000 and 2008, by repeatedly bringing them in for testing, to see whether they improved over the course of the students’ college career. The short answer is yes — between freshman and senior year, students improved on average by about 0.25 points (out of a possible four points) in persuasive writing and by about 0.33 points in expository writing. There was also some variation in writing ability between different groups — women were generally better than men and humanities and social studies students scored better than those majoring in science. But they didn’t improve more than other groups over the course of their college career.
“We found news that was encouraging, we found measurable improvements in writing,” said Daniel Oppenheimer, a psychology professor at University of California, Los Angeles, and one of the study’s authors. “This is the first evidence of that, but the writing improvements weren’t as big as maybe we would have hoped.”
Though intriguing, focusing on the 300 students’ writing improvement is a narrow interpretation of the study’s findings, the researchers say. Instead, the results prove a broader point: that it’s possible to measure a college’s ability to affect its students.
Having that evidence is important, said James Pomerantz, one of the authors of the study and the former provost and acting president of Brown University. Throughout his career in higher education, Pomerantz said he’s been struck by the lack of curiosity among his colleagues about whether their schools actually make a difference.
“For years, universities, unlike almost any other kind of enterprise have never compared the output of its system to the input to see whether there has been any change,” said Pomerantz, a psychology professor at Rice University. “We intended this just as a demonstration that it’s possible to do this kind of measurement not too expensively.”
Given the rising cost of college, it’s increasingly important that students and families know what they’re getting out of their schools. Right now, much of the way we measure institutions is focused on the students they bring in, the researchers say. Traditional college rankings place value on the grade point averages and the standardized test scores of incoming students instead of whether students actually see improvements after their four years on campus.
“If you happen to have really smart students coming in then you get ranked highly regardless of what you teach them,” Oppenheimer said. “Meanwhile you could have a school that has students who really aren’t at the top of the SAT score spectrum but is helping [students] a lot while they’re there. That’s a shame because students are going solely based on the marketing of the programs and how good they have traditionally been.”
Some organizations, including the government, are increasingly looking to measure college outcomes, but those ratings are focused on earnings and job placement, which may only be a valuable metric for some, said Pomerantz. Schools need to expand measures of success to include outcomes like the writing prowess tested in the study as well as quantitative reasoning and critical thinking skills.
“If you’re turning out students in classics or in other areas of the humanities, they’re probably not going to earn lot of money,” Pomerantz said. “Really what you want to know is do students come out better than when they went in.”