The following post is from site editor John Willingham.

In the aftermath of the “Varsity Blues” college admissions scandal that included cheating on entrance exams, three social scientists recently weighed in on the continued importance of those same examinations, arguing that “No one likes the SAT” but “It’s still the fairest thing about admissions.”

“It has become a mantra in some quarters to assert that standardized tests measure wealth more than intellectual ability or academic potential, but this is not actually the case. These tests clearly assess verbal and mathematical skills, which a century of psychological science shows are not mere reflections of upbringing. Research has consistently found that ability tests like the SAT and the ACT are strongly predictive of success in college and beyond, even after accounting for a student’s socioeconomic status.”

For years, U.S. News has used test scores and selection rates as ranking data for the annual “Best Colleges” report. The publication has slightly reduced the impact of test scores in recent editions.

Below I will explain why we do not include test scores as a metric and argue that, for honors and non-honors students, other factors are more important in predicting success. (High school GPA is certainly a major factor; but since almost all honors students have high GPAs, I do not discuss the impact of GPA in this post.)

In their published scholarly work, the authors argue that test scores by themselves correlate very strongly ( r= -.892) with the annual U.S. News Best Colleges rankings for national universities even though the test scores count for only 7.75 percent of the total ranking score. (The authors do not cite the impact of test scores on other ranking factors such as graduation and retention rates, which together account for 22 percent of the total ranking score.)

Our own work for the past eight years, however, shows that test scores do not have a similar correlation to quantitative assessments of honors programs. In our publications we list minimum and average admissions test scores for all programs we rate, but we do not count the scores alone as a rating factor.

Here’s why we do not use test scores as a measure: The factors that make for an excellent honors program are primarily structural. The major building blocks are the credits required for honors completion; the number of honors class sections offered, by type and academic discipline; the availability of priority registration and honors housing; the size of honors class sections; and the number of staff to assist students.

So, don’t the test scores drive the university graduation rates of honors program entrants, just as they do in elite colleges? The answer is not so much; the correlation is r= .50

Admittedly, it is probably difficult for a student with, say, a 1050 SAT score to succeed in an elite college or in most honors programs. But within a fairly large range of SAT scores (~1280-1510), the opportunities for success are more often present given a conducive structure. With every biannual review of honors data, I find great pleasure in discovering outstanding honors programs that are not housed in highly- ranked and extremely selective universities. The golden nuggets of excellence in higher education are scattered much farther and wider than many would have us believe.

I am strongly opposed to the numerical ranking of colleges or their honors programs, whether or not test scores are included in the methodology. I ranked honors program one time, in 2012, and regret doing so. Yes, I have data that allows me to numerically differentiate the total rating scores earned by honors programs. But anyone who wants to provide some kind of assessment of colleges or programs needs to do so with the assumption that their methodology is subjective and imperfect. Ordinal rankings based on distinctions of one point or fractions of a point give readers a veneer of certitude that a qualitative difference exists even if it (often) does not.

Although we do not rank honors programs, we do place them in one of five rating groups, a process that is similar to rating films on a five-star basis but based on quantitative rather than completely subjective data. The seven honors programs in the top group in 2018 (out of 41) had average SAT scores (enrolled students) ranging from 1280 to 1490, a sizable range.

Honors completion rates are something of an issue these days. An honors completion rate is the percentage of first year honors entrants who complete at least one honors program graduation requirement by the time of graduation from the university. About 42 percent of honors students do not complete honors requirements before graduation, although a very high percentage of honors entrants (87 percent) do graduate from the university.

The seven honors programs with honors completion rates of 75 percent or higher in our 2018 ratings had average SAT scores ranging from 1340 to 1510; the mean for this group was 1420. The mean SAT for the 31 (of 41) programs that provided completion rates was 1405, not much lower. And another seven programs with mean SAT scores of 1420 or higher had completion rates below 58 percent, the group mean.

The mean SAT score for all 41 rated programs was 1407; the mean SAT for the top seven programs was only one point higher at 1408.

It is clear, at least with respect to honors programs, that average SAT scores are not the best predictors of program effectiveness. What does this mean for the value of test scores nationwide, if anything?

I think it means that for students who are in the 1280 to 1500 SAT range, success depends as much or more on mentoring, smaller interdisciplinary sections, student engagement, course availability, community (including housing), and advising support than it does on test scores.

The good news here is that even for students who are not in honors programs, high levels of achievement are accessible to students who do not begin college with extremely high test scores, although non-honors students will probably have to assert themselves more in order to benefit from the strongest attributes of their university.