Is It Fair to Judge a College by Its ROI?

2.3.20 by Bennett Leckrone for The Chronicle of Higher Education

Recent headlines proclaiming the long-term payoff of liberal-arts colleges were greeted enthusiastically by advocates for those institutions, which are often derided as impractical destinations for students concerned with finding jobs and paying off debt after they graduate.

But the analysis prompting the headlines, which was conducted by Georgetown University’s Center on Education and the Workforce and calculated the 40-year return on investment for 210 liberal-arts colleges across the country, also cast a harsh light on a subgroup of institutions — those that serve higher numbers of Pell Grant recipients or are located in less-prosperous regions of the country.

For example, Talladega College, a historically black college in Alabama, had the lowest ROI of the 210 colleges surveyed, at $432,000 over 40 years. The highest long-term ROI, $1.85 million, belonged to Harvey Mudd, a renowned STEM-focused institution in Los Angeles. But is it fair to compare Talladega, where more than 93 percent of students receive Pell Grants, according to a report by the Georgetown center, to Harvey Mudd, where just 13 percent of students do?

Officials at colleges with lower returns on investment don’t think so. Here’s why.

ROI Is Noisy

The center’s original analysis, in November, calculated long-term earnings using net present value — how much a sum of money in the future is valued today — and compared it with median debt to measure the 10- and 40-year ROI for 4,500 colleges. But those numbers don’t exist in a vacuum, say critics. They reflect other factors, such as race, class, and geographic region.

The liberal-arts institutions with the highest long-term returns were generally located on the coasts and had significantly fewer Pell Grant recipients. That’s not to imply that every student at every well-to-do coastal college is well off. Colleges with high long-term ROIs have Pell Grant recipients on their campuses; but they generally have fewer than do HBCUs and other colleges with lower ROIs. LaToya Russell Owens, director of learning and evaluation at the United Negro College Fund’s Frederick D. Patterson Research Institute, said the study was a better measure of inequity and systemic racism in the United States than of student success.

Raw financial data are also skewed by privilege, Owens said, noting that student bodies at elite colleges often have a sharply different makeup from those at HBCUs, which serve less-advantaged populations. “They are educating a mostly homogenous student body,” Owens said of the elites. “The top tier is a list of institutions that are educating the most elite students.”

Raw data can’t account for a history of systemic racism, Owens said. HBCUs have often faced an uphill battle when compared with historically white and privileged colleges. “Black people are just a few generations from not being allowed to read,” Owens said. “But now we’re having conversations about achievement gaps … without addressing that.”

Dillard University, an HBCU in New Orleans that has been praised for the social mobility it offers its students, had a long-term ROI that was more than $200,000 below the median for all liberal-arts colleges — $712,000 compared with the median of $918,000.

“There is no way to have an authentic comparison of institutions if you pull out a subset like HBCUs and you don’t control all the other factors,” said Walter M. Kimbrough, Dillard’s president.

“If you have an income gap based on race, an income gap based on gender, income gap based on geographic location, and HBCUs are mostly black, mostly women, and in Southern states, how can we have this conversation in an authentic way?” Kimbrough asked. “The data is woefully inadequate to have any real meaning.”

Anthony Carnevale, author of the Georgetown report and director of the Center on Education and the Workforce, agreed that return on investment isn’t the only way to measure student success. He noted that race and class do influence ROI and said he expected debate about measuring return on investment.

“ROI measures everything that’s important to an economist, but doesn’t measure anything else,” Carnevale said. “This is one indicator.”

Other institutional differences can get magnified by ROI. Colleges with the highest ROI tend to be private nonprofit institutions. That was noted in Georgetown’s November study, which calculated the ROI for more than 4,000 institutions nationwide. According to the report: “Even though the median debt levels at public institutions ($7,000) are less than half those at private nonprofit institutions ($17,000), the higher earnings for students who attended private colleges are sufficient to drive the long-term gain in favor of private nonprofit institutions.”

Such findings reinforce the narrative that private colleges are superior to public ones, said Virginia Sapiro, a professor of political science at Boston University and a vocal critic of studies like the Georgetown report. She doesn’t doubt the report’s accuracy, but said an analytical approach needs to be taken when evaluating colleges, rather than a “purely descriptive” one.

What Is ‘Return,’ Anyway?

To Kimbrough, the Dillard president, numbers can do only so much.

“People are trying to quantify the higher-education experience, which, based on the cost, makes a lot of sense,” Kimbrough said. But, in doing so, he continued, “we try to simplify something that is very complex.”

The value of a college can’t always be spelled out on paper, and Kimbrough said trying to compare the value of colleges on a macro level is flawed because of the many differences, geographical and otherwise, among colleges. The report, he said, isn’t comparing apples to apples.

Owens, at the United Negro College Fund, said the unseen factors that affect students before and after they graduate must be factored in for the comparisons to be meaningful: Income gaps, family wealth, and other factors can influence returns and student outcomes, she said.

The fund has studied its own institutions when it comes to returns, and Owens said “return on investment is generally consistent across demographics, meaning that students are getting the same education.”

While their return on investment might not be glamorous or eye-catching, HBCUs have been prized as a pathway to the middle class for many. Sapiro said a careful analysis of how students’ incomes and living situations change compared with their parents could be one useful way to see the true value of a college education.

ROI can also mean a lot of things, said Sapiro. “The increase in skills one might have, the happiness one might have, the ability to have a more satisfactory life and all sorts of things,” she said. “All of those have always been part of what’s understood to be the return on investment in higher education, and not simply income.”

Use and Misuse

When it was rolled out, the College Scorecard, from which the Georgetown study drew its data to calculate ROI, was hailed as a way to raise consumer choice in higher education. It was thought that the data tool would change how students pick the institutions they attend and what they choose to major in.

No matter how compelling such data are, they might not catch the eye of prospective students. Economists, college administrators, and the faculty are more likely than an incoming student to read and understand a study, said David A. Hawkins, executive director for educational content and policy at the National Association for College Admission Counseling.

Projected future earnings, potential debt, and student aid can all be spelled out on paper, Hawkins said, but those figures don’t account for certain “intangibles” in the world of admissions.

Where students go to college doesn’t hinge on a single factor like return on investment, but rather a slew of variables that change for individual students. Most students attend an institution that’s within 100 miles of their home. A student choosing among several options might winnow down choices based on his or her financial situation and desired major, and then select a college based on campus life or athletics. It’s often more an emotional process than a rational one.

Students found the College Scorecard easy to use and useful when it was rolled out, but they told The Chronicle at the time that it was missing data on student life. While the scorecard has since been expanded to include more information, Hawkins said, the data still can’t account for all of the factors that can sway a student’s choice.

“What really makes the difference that data doesn’t quite capture is … every student is different, and the students are going to have individual needs,” Hawkins said.

Owens emphasized that HBCUs have built themselves around educating first-generation black college students, and are well equipped to help them succeed. For those students and many others, the institutions’ mission and history matter — perhaps more than a single metric like ROI.

But prospective students and their parents aren’t the only ones who might look at reports on ROI, said Sapiro, an expert in political psychology. Legislators and political figures who are responsible for funding state colleges and universities are watching, too, she said. State funding to public institutions has dropped off over the last 30 years. Sapiro said those cuts are “responsible for maintaining so much inequality.”

She worries that if legislators “look at this kind of report and just take it at face value, what they could end up saying is, ‘Look, we’ve been right to cut back on our support of state institutions, because in the long run, they just don’t give as much ROI … we shouldn’t be wasting our valuable state funds on these institutions that don’t do so well over time.’”

A version of this article appeared in the February 14, 2020, issue.

If you have questions or concerns about this article, please email the editors or submit a letter for publication.