Except for the nuts and bolts metrics used by U.S. News in its annual college rankings (grad and retention rates, class sizes) all of the other ranking categories receive strong criticism from education writers and the academic community. A category since 2009, the high school counselor rankings of colleges’ reputations fly a bit under the radar. But the fact is, they do appear to have a curious impact on the rankings.

A recent, excellent article about the rankings on the website Politico argues that the counselor rankings rely heavily on “guidance counselors from highly ranked high schools, while many high schools in less affluent areas have few or no counselors.”

According the the Washington Post, the rankings do include “surveys of 2,200 counselors at public high schools, each of which was a gold, silver or bronze medal winner in the 2016 edition of the U.S. News Best High Schools rankings.” U.S. News also surveys “the largest private independent schools nationwide.”

This already elite group of respondents is even more restrictive than it seems: “The counselors’ one-year response rate was 7 percent for the spring 2017 surveys,” according to U.S News.

Using the nuts and bolts categories and reputation rankings alone, as in this recent post, and separating out the peer reputation rankings from the high school counselor rankings, we can see the impact the counselor rankings have.

Using a sample of 60 national universities that are either in the top 50 nationally or have at least 7 nationally rated academic departments, we found that the high school counselor rankings of private colleges were about 11% higher than those of university peer rankings of the same colleges. (Twenty-five of the schools are public, while 35 are private.)

The fact is, high school counselor rankings on the whole run higher than those of peer reviewers. But counselor rankings of public colleges were only 6.5% higher than peer rankings.

The main question at hand is, do these (few) counselors have more useful knowledge about national universities that peer reviewers have? Peer reviewers have a response rate of more than 40%; this much broader response rate (in absolute percentages and, almost certainly, demographically) should yield a more accurate assessment from peers. (Even more accurate would be the academic departmental rankings, but those are not included.)

Related questions are, how much marketing information do counselors receive, and do they receive a disproportionate share from private colleges? Do they tour private colleges more frequently? Peer reviewers are not without biases, either, but they are not recipients of marketing information from other colleges. Finally, do counselors rely more on…U.S. News rankings?

Again using the same data set we cite above, a side by side comparison of peer and counselor assessments reveals the following:

–Of the 14 universities that rose in rankings at least two places, three were public universities (21.4%) while 11 (78.6%) were private universities. (The percentage of universities in the sample is 41.7% public and 58.3% private.)

–Of the 17 universities that fell in rankings at least two places, 14 (82.4%) were public while three (17.6%) were private.

Below is a table showing the side-by-side comparison. Please bear in mind that the rankings are our adjusted rankings, not the actual U.S. News rankings.