The Curious Case of U.S. News and the High School Counselor Metric

Except for the nuts and bolts metrics used by U.S. News in its annual college rankings (grad and retention rates, class sizes) all of the other ranking categories receive strong criticism from education writers and the academic community. A category since 2009, the high school counselor rankings of colleges’ reputations fly a bit under the radar. But the fact is, they do appear to have a curious impact on the rankings.

A recent, excellent article about the rankings on the website Politico argues that the counselor rankings rely heavily on “guidance counselors from highly ranked high schools, while many high schools in less affluent areas have few or no counselors.”

According the the Washington Post, the rankings do include “surveys of 2,200 counselors at public high schools, each of which was a gold, silver or bronze medal winner in the 2016 edition of the U.S. News Best High Schools rankings.” U.S. News also surveys “the largest private independent schools nationwide.”

This already elite group of respondents is even more restrictive than it seems: “The counselors’ one-year response rate was 7 percent for the spring 2017 surveys,” according to U.S News.

Using the nuts and bolts categories and reputation rankings alone, as in this recent post, and separating out the peer reputation rankings from the high school counselor rankings, we can see the impact the counselor rankings have.

Using a sample of 60 national universities that are either in the top 50 nationally or have at least 7 nationally rated academic departments, we found that the high school counselor rankings of private colleges were about 11% higher than those of university peer rankings of the same colleges. (Twenty-five of the schools are public, while 35 are private.)

The fact is, high school counselor rankings on the whole run higher than those of peer reviewers. But counselor rankings of public colleges were only 6.5% higher than peer rankings.

The main question at hand is, do these (few) counselors have more useful knowledge about national universities that peer reviewers have? Peer reviewers have a response rate of more than 40%; this much broader response rate (in absolute percentages and, almost certainly, demographically) should yield a more accurate assessment from peers. (Even more accurate would be the academic departmental rankings, but those are not included.)

Related questions are, how much marketing information do counselors receive, and do they receive a disproportionate share from private colleges? Do they tour private colleges more frequently? Peer reviewers are not without biases, either, but they are not recipients of marketing information from other colleges. Finally, do counselors rely more on…U.S. News rankings?

Again using the same data set we cite above, a side by side comparison of peer and counselor assessments reveals the following:

–Of the 14 universities that rose in rankings at least two places, three were public universities (21.4%) while 11 (78.6%) were private universities. (The percentage of universities in the sample is 41.7% public and 58.3% private.)

–Of the 17 universities that fell in rankings at least two places, 14 (82.4%) were public while three (17.6%) were private.

Below is a table showing the side-by-side comparison. Please bear in mind that the rankings are our adjusted rankings, not the actual U.S. News rankings.

University Peer Only Peer + Counselors Dif +,-
Princeton 1 1 0
Harvard 1 1 0
Yale 1 1 0
Stanford 4 5 -1
Columbia 4 4 0
MIT 4 6 -2
Chicago 7 7 0
Johns Hopkins 8 8 0
Caltech 9 9 0
Penn 9 9 0
Northwestern 11 11 0
Cornell 11 14 -3
Brown 11 11 0
UC Berkeley 11 16 -5
Duke 11 11 0
Dartmouth 16 14 2
Michigan 17 17 0
Vanderbilt 18 17 1
Carnegie Mellon 18 21 -3
Notre Dame 18 17 1
Rice 18 17 1
Virginia 18 21 -3
UCLA 23 25 -2
Wash U 23 21 2
Georgetown 23 21 2
USC 26 25 1
Emory 27 27 0
Georgia Tech 28 30 -2
North Carolina 28 28 0
Tufts 30 28 2
NYU 31 32 -1
Wisconsin 31 34 -3
Boston College 33 31 2
Brandeis 34 33 1
Wake Forest 34 34 0
Illinois 36 38 -2
Florida 36 36 0
Boston Univ 38 36 2
UC Davis 38 38 0
UT Austin 38 46 -8
UCSD 41 43 -2
Washington 41 46 -5
UC Irvine 43 38 5
Case Western 43 43 0
Maryland 43 43 0
Rochester 46 38 8
Ohio State 46 50 -4
Northeastern 48 38 10
UCSB 48 46 2
Penn State 48 50 -2
Tulane 51 46 5
RPI 52 50 2
Lehigh 53 50 3
Purdue 53 55 -2
U of Miami 55 54 1
Minnesota 55 56 -1
Pitt 57 56 1
Texas A&M 58 58 0
Michigan State 58 60 -2
Indiana 58 60 -2
Rutgers New Bruns 61 58 3

 

Advertisements