College ratings are useful, rankings useless

Higher Education is a massive industry, including thousands of institutions offering certificates and degrees.

Almost a trillion dollars are spent each year on college decisions, which are so important, impactful, and complex that a whole subspecialty of advisors has risen to help parents and their students navigate the choices.

Prominent among these is the rating of these institutions and their ranking.

But are rankings the best way to choose a college? 

In this article, I will show why you should ignore those rankings and why college ratings and the return on investment (ROI) are far more valuable tools for making informed decisions.

Read on for the details.

“The ranking… it’s just a number.”

Angelique Kerber

Understanding College Rankings

Many big publishers have discovered the value of creating college rankings. These articles drive a lot of valuable ad traffic.

If these rankings appear based on a diligent review of college choices, they can become respected sources by parents and students considering college choices.

Americans also love ranking things, and colleges are not immune to this interest. These rankings correlate well with Prestige. 

Colleges encourage rankings as the winners can use them in their marketing efforts. 

Being ranked highly by The Wall Street Journal, the New York Times, or other raters can help create buzz and attract the attention of potential students and their parents.

A high ranking can attract new employers to a College. Graduates can see salary increases over time as their college rank increases.

The result is a reinforcing mechanism that drives competition by colleges for attention among the raters and behavior to influence better ranks.

However, are rankings helpful in answering the most critical questions parents and students should consider? 

Is there a better way to help parents and students find the right fit?

What are the critical questions?

For some, the interest is in getting into the most prestigious college. The assumption is that the cachet, expensive nature, and brand name recognition will lead to better outcomes and higher returns over time.

I will address the question of whether Prestige delivers better returns later.

For these parents and students, the higher-ranked, more prestigious rankings are the critical decision-making factor.

However, for most college decision-makers, there is a multifactorial consideration that is far more nuanced.

When considering college choices, parents and their children look at several key factors that ensure a well-rounded, valuable educational experience and an overall good investment.

The rational decision makers will want to choose their best fit. That is the best possible institution, where a student can do good work, where there is a likelihood of being accepted, where costs can be reasonable after grants and scholarships, and where the employment/earning outcomes are most favorable. 

In other words, most college decisions should optimize the return on investment for the student.

Here are the primary considerations:

1. Where the students will do their best work. The focus is on finding a good fit for the student and ensuring excellent academic quality.
   • One looks for the institution to have desired majors and specialties. 
   • Equally important is considering the faculty’s reputation in your desired field of study. Do storied professors teach classes, or are they doing mostly research, leaving much of the teaching to instructors?
   • Some students prefer more contact with the teaching staff. In those cases, a reasonable Class size and student-to-faculty ratio are desirable.
2. Likelihood of Acceptance.
   • The rational decision maker will want to balance the selectivity of desired institutions with the student’s academic performance and extracurricular track record.
   • Parents and students would do well to consider whether selective, competitive institutions, valued for their Prestige, are better if student performance is subpar.
3. Reasonable costs. Here, the focus is on finding value in an affordable, high-quality institution.
   • What is the level of costs adjusted for your student’s particular situation? Will the student be considered in-state or out-of-state, where applicable? Are they living on campus or attending from home?
   • How predictable and stable are tuition, fees, and other expected costs?
   • Are there reasonable quality housing and dining options, and what do they add to the overall budget?
   • How much cost relief is available to students based on their financial and academic standing?
4. Excellent Outcomes. The equation should focus on high graduation rates, employment rates, and salaries.
   • What are the graduation rates for the institution, and what are the job placement rates? 
   • Does the institution have an excellent track record and is a desirable source of talent for growing employers?
   • Are graduates from this institution desired in the employment marketplace, such that excellent salaries are typical?
   • What does the alumni track record suggest about the value of the institution?
5. Other fit variables. Students and their families might be interested in other variables to improve their preference fit. Many of these are desirable personal interests, but are not fundamental. For example, some students prefer large schools, while others prefer small ones. Maybe they want it close to home, or in a big city. They may be particularly attracted to schools with successful sports programs. Athletes, indeed, would find that variable important. Some important, though not necessarily rankable factors are:
   • Are the campus environment and resources high-quality, including libraries, labs, sports, entertainment, and other facilities?
   • Parents and students might be concerned about campus safety, security, and proximity to safe neighborhoods.
   • Some will look for cultural, ethnic, and socio-economic diversity in the student body.
   • Decision-makers might be interested in the availability of social organizations, clubs, and other activities for students. 
   • Many other quality-of-life considerations may also be necessary to an individual, though not primary to the decision.

Parents and their children often seek a balance of these factors, aiming to find a college that offers a quality education and aligns with personal values, financial capabilities, and long-term career goals. 

Each family may prioritize these factors differently based on unique circumstances and aspirations.

What Are College Rankings?

Think of college rankings like a sports league table, where colleges are like teams. The higher the ranking, the better the college is assumed to be based on various factors. 

There are three broad categories of rankings in higher education:

1. The Global or International Ranking. These rankings seek to identify and rank the best institutions around the world. They tend to be heavily influenced by research metrics and are followed primarily by governments. Prospective students and employers interested principally in research will also follow these. Examples include the Academic Ranking of World Universities (ARWU), Times Higher Education (THE), and Quacquarelli Symonds (QS) rankings. They only rank a limited number of institutions around the world.
2. National Rankings. Numerous publications build proprietary measurement systems and unique ranking methodologies applied to universities in a specific country. Examples include the U.S. News and World Report, Bloomberg, New York Times, Princeton Review, and Wall Street Journal in the USA. The majority of our discussions will center on these U.S.-based ranking systems.
3. Field of Study or specific Rankings. Several academic fields produce rankings for their fields of study. A prominent example is the Financial Times MBA ranking. All the national ranking sources also provide rankings for law schools, engineering, and other specific areas of study.

Like a sports league, which considers wins, losses, and points, college rankings consider several criteria to decide which colleges are at the top.

Unlike the sports league tables, college rankings use many variables that are not necessarily rankable.

Consider what your sports league table ranking might look like if you added staff costs spent, the size of stadiums played in, a qualitative measure of coaching staff, or more. 

Combining many variables into a compound ranking value can make the ranking meaningless, especially when the variables cannot be measured objectively, and the decision maker weighs those variables’ importance differently.

The Ranking Challenge

Many traditional ranking publishers perpetuate chronic flaws with lists focused on resources and reputation. 

Many college Admissions consultants use the Shanghai Ranking or Academic Ranking of World Universities (ARWU). They say, “ARWU considers every university that has any Nobel Laureates, Fields Medalists, Highly Cited Researchers, or papers published in Nature or Science. In addition, universities with a significant number of papers indexed by Science Citation Index-Expanded (SCIE) and Social Science Citation Index (SSCI) are also included.”(1)

One thousand of the world’s 18 thousand are included. This ranking may be relevant if you want to research or pursue the Nobel Prize. 

The ranking systems end up being about historic strengths and mostly about Prestige. 

Their biggest flaw is the notion that there is a single best.

Is Princeton better than MIT, Yale, and Stanford? 

They each have their strengths and appeal to a prospect differently. The ratings on the factors that matter to the prospect are incredibly valuable.

The compound all-in-one-number ranking tries too much.

Complexity in Rankings and Ratings Hurt Usability

A ranking implies the institution ranked 1 is better than that ranked 2, and so on.

One presumes that rankers have a clear view as to what is best and what is worse, and the ranking methodology should mirror those relative positions. It should make sense and be relatively obvious.

Bloomberg has been creating rankings of business schools for some time. They say “we rank schools based on five indexes that capture fundamental elements of business school education: Compensation, Learning, Networking, Entrepreneurship, and Diversity (for US schools only).”(2)

The Diversity index has been added relatively recently, since 2021. For U.S. Schools, it represents a minor 7.1% of the overall score. For international schools, that 7.1% is spread over the other four categories. That makes the comparisons very different between U.S. and International schools.

Please review the attached image, which shows the top 9 Institutions and the information used to calculate a  Diversity Score. 

I am pleased that my Alma Mater Baruch College is ranked 6th here. I think it’s a school that has an excellent record with diversity. But I struggle to understand for example why NYU is rated higher. Is it better that on the gender side they have 11% more women and non-binary? Does that more than make up for the fact that on the Ethnicity side they also have 45% more Whites? 

Also, the top school is a HBCU with 100% Black students on the ethnicity side. I am not sure many Americans would consider a 100% male, or White school to be diverse. My guess is that would also apply to 100% women, or Black school. I am confident not many parents would view Howard as nearly perfect (97.6 Score) from a diversity perspective.

These rankings are the result of complex mathematical manipulations of factors that usually end up not reflecting believable results to informed observers.  

Consider  this sample of the methodology described: “For the final diversity score, log-adjusted scores for race/ethnicity and gender are averaged, then rescaled from 1-7 for a final number (7 would be awarded to a theoretical school with the maximum minority and women/nonbinary populations of all schools in the ranking;”

Improvements in Ranking Methods and Ongoing Challenges

The New York Times and The Wall Street Journal have adopted a better methodology that addresses several flaws.

For one, they provide filters or mechanisms that allow a prospect to build a personal ranking prioritizing their interests.

The second change they make is a higher focus on outcomes.

The WSJ introduces its methodology by saying: “Some college-ranking methodologies tend to have the effect of splitting universities into the haves and the have-nots by evaluating the resources a college has at its disposal.” They say the WSJ ranking “put colleges on a more level playing field, with a focus on comparing the outcomes of each school’s graduates to what those students were likely to achieve no matter where they went to school.”(4)

Although the NYT and WSJ only provide rankings of a small subset of four-year colleges (400 for WSJ and 883 for NYT), they do an admirable job of shifting away from the prestige dynamic. Their filtering and tailoring methodologies are also beneficial.

Still, there are several challenges.

They also end up with absolute rankings. We prefer shifting the dialogue to “great choices” rather than absolute ranks. 

Further, the use of outcomes is not exempt from issues. Some suspect institutions are achieving higher graduation rates through lower performance standards. Further, an institution’s mix of fields of study heavily influences its median salaries. Schools focusing on Engineering, STEM, or Business will perform better than liberal arts schools.

Do Rankings Work?

College admission officers and administrators pay attention to rankings, even when they don’t make sense. Students and their families also pay attention to rankings.

Two Cornell Economics researchers, Amanda Griffith and Kevin Rask, examined high school students’ decisions over nine years. Their report, published in 2005, concluded: “Our results suggest that there is a benefit to a positive change in a school’s USNWR rank. We find that full-pay applicants are more likely to attend a school that is higher ranked by even a few places. Aided applicants are less responsive, but still systematically prefer higher-ranked schools.”(5) 

They also found the rankings were more important even though they did not make sense to use: “More importantly, these preferences for the USNWR rank are independent of other measures of quality (student-faculty ratio and expenditures per student), and estimates of school fixed effects themselves. This would be less distressing if the USNWR rank were a widely accepted measure of quality. However, the measures included as components of the rank, especially the weights attached to those components, are somewhat arbitrarily chosen in terms of being measures of educational quality.”(5)

That same year, Stephanie Owings-Edwards examined the effect of attending a US News and World Report Best College on wages. Her Conclusion? “I  find no  statistically  significant  relationship  between  attending  a  U.S.  News and World Report “best college” and wage earned.”(6)

More recently, in an article titled Why New ‘U.S. News’ Rankings Are Flawed, Daniel Diermeier, the Chancellor of Vanderbilt University, says: “All college rankings are problematic. But for several years, U.S. News has adopted increasingly questionable practices for gathering and interpreting its data, which have prompted some universities and professional schools to stop participating in the process altogether. With the version released Sept. 18, U.S. News’s rankings process has gone from bad to worse.”(7)

He counsels “…universities should seriously examine our role in a process that misleads students and families.”(7)

The UNESCO global forum on rankings summarizes our view on the issue of rankings: “By their very nature indicators are selective and not exhaustive. As such, they responsibly caution that, none of the current global ranking systems can provide a complete view of universities; taking any single ranking as a standard to judge a university’s overall performance is improper (Liu, 2012).”(8)

Ratings, not Rankings

The notion that an institution is absolutely better needs to be revised. While they may be better at a specific objectively measured thing, the compound measure implies something inaccurate.

Alexander C. McCormick is a former admissions officer at his alma mater, Dartmouth College, and associate professor of Education at Indiana University at Bloomington. He says: “Everything we know from studying college student experiences and outcomes tells us that there is more variability within schools than between them.” And, “This is the irony, given the dominance of the rankings mentality of who’s No. 5 or No. 50. The quality of that biology major offered at School No. 50? It may exceed that at School No. 5.”(9)

Different ranking systems use other criteria, leading to inconsistent results. This variability only contributes to confusion around the most critical issues to decision-makers.

The advent of ranking systems has been valuable, though. “Notably, rankings have also encouraged transparency of information and accountability of these hallowed institutions, which hitherto have been cloaked in exclusivity, academic freedoms and even restrictive prestige. More and more, universities find themselves having to explain to the public their performance on set criteria used by rankers and other quality monitoring bodies. Rankings ‘have led to a revolution in the availability of data on higher education institutions and intelligence to guide institutional and government strategies for higher education’ (Sowter, 2012).”

In other words, ranking systems introduced us to essential ratings data!

We need an approach that encourages students and their parents to make choices that best suit their needs, preferences, and situations, using the most relevant ratings data. 

How valuable is Prestige?

If most rankings end up being Prestige metrics, that raises the question of whether Prestige adds value.

If Prestige contributes to dramatically better outcomes, then we should not be so quick to discard its value.

Does attending a prestigious college result in higher salaries?

In 2010, the New York Times published Is Going to an Elite College Worth the Cost? They discussed a report published in the Journal of Labor Research exploring this question. One of the authors, Scott Thomas, says: “Prestige does pay. But prestige costs, too. The question is, is the cost less than the added return?”(9)

Another set of researchers examined the data and discovered that the argument in favor of Prestige was less compelling when considering students with “seemingly comparable capability.”(10)

The authors found that “The earnings of graduates in the two groups were about the same, perhaps shifting the ledger in favor of the less expensive, less prestigious route.”(9)

In 2016, Eric Eide and his co-authors wrote a piece in WSJ titled Do Elite Colleges Lead to Higher Salaries? Only for Some Professions

Their finding? Diplomas from prestigious schools only boost future earnings in specific fields, while they don’t make a difference in others.(11)

But here’s the problem: They needed to complete the analysis by determining whether the cost was less than the added return. 

When considering costs, the ROI generated by Prestige all but disappears. 

There is one exception, though. Students from disadvantaged families attending prestigious colleges will generate excellent returns. That is because grants and scholarships eliminate most attendance costs.

A Fatal Flaw – The Missing Rating

College is an investment in time and resources by individuals, their families, and society, with some broad expectations of future benefits.

If it works, we should see long-term gains for the investors (students, their families, and society).

We have yet to see publishers properly deal with rating college investment returns.

In recent years, the ROI of colleges has become increasingly debated. However, many published reports have used simplistic measures, such as a college wage premium, to conclude that going to college is a no-brainer. 

Unfortunately, the reality may be starker than you believe. 

Some college degrees are incredible investments. Others could be better.

A college investment is a considerable commitment and must receive the requisite due diligence.

We recommend that every college rater calculate the net present value of lifetime earnings after considering the costs and taxes of the institutions they are interested in. They should do this calculation for their chosen field of study and compare the result to a High School Graduate benchmark.

If your chosen field of study or college cannot provide a return more significant than that of a high schooler, why is that a good investment decision? 

A Better Way – ROI and Tailored Ratings

There is a better way to find an ideal college. 

Ignore rankings and Prestige.

Here is what we suggest:

1. Do your research: Use multiple sources of information.
2. Tailor your search: Look for colleges that align with your personal and financial needs.
3. Use ratings: Narrow down choices.
4. Review the ROI: Ensure your choice of college and field of study provides a good return on your investment.
5. Visit campuses: Experience the college environment firsthand.
6. Talk to the experts: Get insights from those already attending the school, graduates, and your college counselor.

Conclusion

When making decisions about higher Education, it’s essential to look beyond college rankings and consider factors contributing to the best return on investment for the student and family. 

Rankings reflect Prestige and create a seemingly definitive but flawed view of colleges. 

Students and their families can make more informed decisions about their college choices by focusing on rating factors such as academic fit, likelihood of acceptance, reasonable costs, and excellent outcomes. Ultimately, the goal should be to find a college that offers both a valuable educational experience and a good investment for the future.

Reference Sources

1. Shanghai Ranking. ”ShanghaiRanking’s Academic Ranking of World Universities Methodology 2023.” Shanghairanking.Com. Accessed July 13, 2024. https://www.shanghairanking.com/methodology/arwu/2023.
2. Bloomberg. ”Best B-Schools 2023-24 Methodology.” Bloomberg.Com. Accessed July 13, 2024. https://www.bloomberg.com/business-schools/methodology/.
3. Bloomberg. ”Best B-Schools 2023-24 Diversity Index.” Bloomberg.Com. Accessed July 13, 2024. https://www.bloomberg.com/business-schools/diversity-index/.
4. The Wall Street Journal. ”The 2024 Best Colleges in America: Princeton, MIT and Yale Take Top Spots.” Wsj.Com. September 6, 2023. https://www.wsj.com/us-news/education/wsj-best-college-rankings-princeton-12c44c47.
5. Owings-Edwards, Stephanie. “The Effect On Wage Of Attending A US News And World Report Best College.” Journal of College Teaching & Learning 2, no. 10 (2005). Accessed July 22, 2024. https://doi.org/10.19030/tlc.v2i10.1870.
6. Griffith, Amanda, and Kevin Rask. “The Influence of the US News and World Report Collegiate Rankings on the Matriculation Decision of High-ability Students: 1995–2004.” Economics of Education Review 26, no. 2. Accessed July 22, 2024. https://doi.org/10.1016/j.econedurev.2005.11.002.
7. Diermeier, Daniel. “Why New ‘U.S. News’ Rankings Are Flawed.” Insidehighered.Com. Inside Higher Ed, October 9, 2023. https://www.insidehighered.com/opinion/views/2023/10/09/why-new-us-news-rankings-are-flawed-opinion.
8. P.T.M., Peter J. Wells, and Ellen Hazelkorn. “Rankings and Accountability in Higher Education Uses and Misuses.” Unesdoc.Unesco.Org. UNESCO Global Forum, January 1, 2013. https://unesdoc.unesco.org/ark:/48223/pf0000220789.
9. Steinberg, Jacques. “Is Going to an Elite College Worth the Cost?” Nytimes.Com. The New York Times, December 17, 2010. https://www.nytimes.com/2010/12/19/weekinreview/19steinberg.html
10. Dale, Stacy B., and Alan B. Krueger. “Estimating the Payoff to Attending a More Selective College – An Application of Selection on Observables and Unobservables.” NBER Working Paper Series, no. Working Paper 7322 (1999). Accessed July 13, 2024. https://www.nber.org/system/files/working_papers/w7322/w7322.pdf
11. Eide, Eric R., and Michael J. Hilmer. “Do Elite Colleges Lead to Higher Salaries? Only for Some Professions.” Wsj.Com. The Wall Street Journal, January 31, 2016. https://www.wsj.com/articles/do-elite-colleges-lead-to-higher-salaries-only-for-some-professions-1454295674?st=coece2shs6vnx5o&reflink=article_copyURL_share.