College Ratings: A Global Approach

A recent article in The Economist reported on efforts by the Organization for Economic Cooperation and Development, a think-tank funded by the world’s wealthiest countries, is trying out a new method for comparing colleges and universities around the world.Most college rankings, including the infamous annual ratings by US News & World Report (see my previous posts here and here) are based on things that are most easily measured. In most cases, these are the “inputs” into a college education. We can measure the average salary of professors. We can report ratios (students to faculty, athletes to athletic fields, Nobel Prize winners to faculty schleppers). We can measure the amount of money spent (on students, on landscaping, on science labs, on faculty salaries). We can also measure things we hope our students will not encounter on a campus (the number of rapes, the number of drug arrests, the number of bicycles stolen).
We can then invent some sort of algorithm that crunches all these measurable variables through a formula and…
Voilà!
A ranking or rating system that sells magazines!
As the Economist reports, there are other, relatively independent ranking systems that have popped up in the past few years, but neither of these has much public impact, nor do they really measure what we really want to know: which college provides the best education that money can buy?
The OECD’s new efforts are an attempt to measure educational outcomes. Instead of measuring inputs, the plan is to come up with new ways of measuring how much students actually learn as a result of their university education. This is similar to what US Secretary of Education Margaret Spellings has been talking about (and serves as the logic behind No Child Left Behind). We want to be able to measure how much learning is really taking place behind those Ivy-covered walls.
On the one hand, I applaud this effort. It would be very nice, indeed, to be able to better rate the quality of the education, not the quality of the facilities. It would be nice to know that the professors at this university are (measurably) better teachers than the teachers at the college down the road.
Take this example: Tokyo University is commonly considered to be the best university in Japan. It educates the upper crust of the secondary school pool. My grad school roommate had graduated from Tokyo University. But his analysis (corroborated by other anecdotal evidence I have come across) is that while Tokyo University is nearly impossible to get into, it is ridiculously easy once a student arrives there.
If we were able to measure the intellectual growth of a student at Tokyo University from matriculation to graduation, and assuming that my roommate was right, we’d a Tokyo University isn’t all that great.
On the other hand, if we were to look at a place like University of Dallas (see my post here) or Austin College (here) or any one of the “40 College That Change Lives” about which Loren Pope has written, we might very well find that the amount and the quality of education that a student receives in four years is much greater at these schools than at Harvard or Oxford or Tokyo University.
So the idea of an authoritative rating of universities based on educational outcomes is wonderful. I’m all for it (if only to see the US News rankings reduced to irrelevance).
But here’s the rub.
Measuring educational outcomes in a university context is devilishly hard to do. But comparing those outcomes against one another is nigh impossible, methinks, for many reasons, including:

1. Not all students start their college learning in the same place.
2. College students study a huge variety of courses and subjects.
3. Each college structures its program in different ways in order to achieve different learning outcomes.
4. Trying to get a national consensus on what learning outcomes ought to be in any field will be an exercise in futility (just trying to get the consensus within one department at one university is nearly impossible).

Then, add to this that the OECD is trying to measure learning across countries, across cultures, and across languages….
Believe me, I’m a sucker for Don Quixote, but this quest sure seems like an impossible dream. Oh, it’s a noble one, to be sure. And I do believe that colleges and universities can do a better job of measuring their success in achieving agreed-upon learning outcomes–based upon their own objectives, student populations, and and other particular circumstances.
So I’ll be watching the efforts by the OECD to create a meaningful index or rating system that is based on a measure of learning outcomes at the world’s most prominent colleges and universities. It would be great if we all had access to objective, complete, and simple-to-understand ratings of institutions of higher learning.
In the meantime, an ever-increasing number of secondary school students and their families will turn to me and to other independent educational consultants to help them find the college that is best for them.
Mark Montgomery
Montgomery Educational Consulting