The Curious Case of U.S. News and the High School Counselor Metric

Except for the nuts and bolts metrics used by U.S. News in its annual college rankings (grad and retention rates, class sizes) all of the other ranking categories receive strong criticism from education writers and the academic community. A category since 2009, the high school counselor rankings of colleges’ reputations fly a bit under the radar. But the fact is, they do appear to have a curious impact on the rankings.

A recent, excellent article about the rankings on the website Politico argues that the counselor rankings rely heavily on “guidance counselors from highly ranked high schools, while many high schools in less affluent areas have few or no counselors.”

According the the Washington Post, the rankings do include “surveys of 2,200 counselors at public high schools, each of which was a gold, silver or bronze medal winner in the 2016 edition of the U.S. News Best High Schools rankings.” U.S. News also surveys “the largest private independent schools nationwide.”

This already elite group of respondents is even more restrictive than it seems: “The counselors’ one-year response rate was 7 percent for the spring 2017 surveys,” according to U.S News.

Using the nuts and bolts categories and reputation rankings alone, as in this recent post, and separating out the peer reputation rankings from the high school counselor rankings, we can see the impact the counselor rankings have.

Using a sample of 60 national universities that are either in the top 50 nationally or have at least 7 nationally rated academic departments, we found that the high school counselor rankings of private colleges were about 11% higher than those of university peer rankings of the same colleges. (Twenty-five of the schools are public, while 35 are private.)

The fact is, high school counselor rankings on the whole run higher than those of peer reviewers. But counselor rankings of public colleges were only 6.5% higher than peer rankings.

The main question at hand is, do these (few) counselors have more useful knowledge about national universities that peer reviewers have? Peer reviewers have a response rate of more than 40%; this much broader response rate (in absolute percentages and, almost certainly, demographically) should yield a more accurate assessment from peers. (Even more accurate would be the academic departmental rankings, but those are not included.)

Related questions are, how much marketing information do counselors receive, and do they receive a disproportionate share from private colleges? Do they tour private colleges more frequently? Peer reviewers are not without biases, either, but they are not recipients of marketing information from other colleges. Finally, do counselors rely more on…U.S. News rankings?

Again using the same data set we cite above, a side by side comparison of peer and counselor assessments reveals the following:

–Of the 14 universities that rose in rankings at least two places, three were public universities (21.4%) while 11 (78.6%) were private universities. (The percentage of universities in the sample is 41.7% public and 58.3% private.)

–Of the 17 universities that fell in rankings at least two places, 14 (82.4%) were public while three (17.6%) were private.

Below is a table showing the side-by-side comparison. Please bear in mind that the rankings are our adjusted rankings, not the actual U.S. News rankings.

University Peer Only Peer + Counselors Dif +,-
Princeton 1 1 0
Harvard 1 1 0
Yale 1 1 0
Stanford 4 5 -1
Columbia 4 4 0
MIT 4 6 -2
Chicago 7 7 0
Johns Hopkins 8 8 0
Caltech 9 9 0
Penn 9 9 0
Northwestern 11 11 0
Cornell 11 14 -3
Brown 11 11 0
UC Berkeley 11 16 -5
Duke 11 11 0
Dartmouth 16 14 2
Michigan 17 17 0
Vanderbilt 18 17 1
Carnegie Mellon 18 21 -3
Notre Dame 18 17 1
Rice 18 17 1
Virginia 18 21 -3
UCLA 23 25 -2
Wash U 23 21 2
Georgetown 23 21 2
USC 26 25 1
Emory 27 27 0
Georgia Tech 28 30 -2
North Carolina 28 28 0
Tufts 30 28 2
NYU 31 32 -1
Wisconsin 31 34 -3
Boston College 33 31 2
Brandeis 34 33 1
Wake Forest 34 34 0
Illinois 36 38 -2
Florida 36 36 0
Boston Univ 38 36 2
UC Davis 38 38 0
UT Austin 38 46 -8
UCSD 41 43 -2
Washington 41 46 -5
UC Irvine 43 38 5
Case Western 43 43 0
Maryland 43 43 0
Rochester 46 38 8
Ohio State 46 50 -4
Northeastern 48 38 10
UCSB 48 46 2
Penn State 48 50 -2
Tulane 51 46 5
RPI 52 50 2
Lehigh 53 50 3
Purdue 53 55 -2
U of Miami 55 54 1
Minnesota 55 56 -1
Pitt 57 56 1
Texas A&M 58 58 0
Michigan State 58 60 -2
Indiana 58 60 -2
Rutgers New Bruns 61 58 3

 

Advertisements

U.S. News Rankings, Minus the Financial Padding Metrics

The critics of the annual–and hugely popular–U.S. News Best Colleges rankings are vocal, large in number, well-armed with data, and mostly unavailing. Here is another attempt, based on the idea that the “financial” metrics used in the rankings distort the results. If Harvard has a zillion dollars, Harvard will have smaller classes than Mammoth State University with its meager funding per student. But why give Harvard credit for the zillion dollars and the smaller classes, when the smaller classes are the “output” that really matters?

So…the adjusted rankings below use the major non-financial metrics only: Peer assessment of academic reputation; high school counselor recommendations; graduation rates; retention rates; and class sizes. No acceptance rates or test score-related metrics are used. The impact of both are reflected in the output metric of graduation rates. (A separate post will discuss the curious disparities in high school counselor recommendations.)

Each of the universities on the list is in the top 50 in the 2018 U.S. News rankings with at least 7 ranked departments or has an aggregate academic department ranking of 50 or better across a minimum of 7 departments. The departments ranked are business and engineering (undergrad); biology, chemistry, computer science, earth sciences, economics, education, English, history, math, physics, political science, psychology, and sociology (graduate level).

Therefore, even though department ranking data are not included in the adjusted rankings below, they are used as part of the eligibility requirements for inclusion.

Below are the adjusted rankings of 60 national universities, in the order of the adjusted ranking. Also shown are the U.S. News rankings for 2018 and the difference between the adjusted rankings and those of the magazine.  We used data from U.S News for the categories listed above, with the same weight assigned to each category. All categories were then standardized and aggregated. After the first fifteen or so schools, some of the disparities are striking, especially for the last half.

University Adj Rank US News Dif +, –
Yale 1 3 2
Harvard 1 2 1
Princeton 1 1 0
Columbia 4 5 1
Stanford 5 5 0
MIT 6 5 -1
Chicago 7 3 -4
Johns Hopkins 8 11 3
Penn 9 8 -1
Caltech 9 10 1
Brown 11 14 3
Northwestern 11 11 0
Duke 11 9 -2
Dartmouth 14 11 -3
Cornell 14 14 0
UC Berkeley 16 21 5
Notre Dame 17 18 1
Rice 17 14 -3
Vanderbilt 17 14 -3
Michigan 17 28 11
Georgetown 21 20 -1
Carnegie Mellon 21 25 4
Virginia 21 25 4
Wash U 21 18 -3
UCLA 25 21 -4
USC 25 21 -4
Emory 27 21 -6
Tufts 28 29 1
North Carolina 28 30 2
Georgia Tech 30 34 4
Boston College 31 32 1
NYU 32 30 -2
Brandeis 33 34 1
Wake Forest 34 27 -7
Wisconsin 34 46 12
Boston Univ 36 37 1
Florida 36 42 6
Illinois 38 52 14
Northeastern 38 40 2
Rochester 38 34 -4
UC Irvine 38 42 4
UC Davis 38 46 8
UCSD 43 42 -1
Maryland 43 61 18
Case Western 43 37 -6
UT Austin 46 56 10
Washington 46 56 10
UCSB 46 37 -9
Tulane 46 40 -6
Ohio State 50 54 4
Lehigh 50 46 -4
RPI 50 42 -8
Penn State 50 52 2
U of Miami 54 46 -8
Purdue 55 56 1
Pitt 56 68 12
Minnesota 56 69 13
Rutgers 58 69 11
Texas A&M 58 69 11
Michigan State 60 81 21
Indiana 60 90 30