Are peer reviews a waste of time?

The report of the Commission on Race and Ethnic Disparities, which was published on 31 March, was greeted by an avalanche of hostile commentary. Precisely how seriously we should take the criticisms is debatable. Iain Dale, an informed commentator, noted, “The number of people who castigated and slammed it within minutes of its release can’t possibly have read all of its 258 pages.” He concluded, “They just jerked their knees in the time-honoured fashion. Basically, the criticism was based on the fact that Boris Johnson had commissioned it, ergo it must be biased, useless or bad, or all three.” Very unpleasant ad hominem attacks, such as that by the Cambridge professor, Priyamavada Gopal, did little to raise the tone of debate.

One of the more rationally-argued criticisms, on the website of the British Medical Journal, noted that, “The 30 page section on health in the report claims to undo several decades of irrefutable peer reviewed research evidence on ethnic disparities…” (Emphasis added.) I aim to explore the use of peer review, particularly in the social sciences. Its connection to contentious areas of research, like racism, has had an eventful recent history.

Peer review is one of the cornerstones of academic research. It means that an article which has been submitted for publication has to be reviewed by experts in the field, to ensure that the article’s arguments are robust and its conclusions reliable. Peer review is generally regarded as the gold standard in academic research quality. It is a system which has been tried and tested for centuries. It works. 

However, it is not foolproof. The medical journal The Lancet, for instance, uses peer review rigorously. Even so, it still managed to publish Andrew Wakefield’s now notorious and debunked article in 1998, which linked autism with the MMR vaccine. Sadly, this was not an isolated example. The frauds listed here did not all involve the defeat of the peer review system, but many did. Moreover, this list covers just the hard sciences, where ideology and politics are absent, and where input from postmodernist theory is minimal. It is a very different picture in the social sciences and humanities.

Peer review is excellent at catching individuals or small groups who “turn rogue”. But what happens if an entire field of research, perhaps a field which came into existence very recently, is colonised by people who share the same ideological or political convictions? We can surmise that peer review would become useless. It would merely ask people of the same ideological world view to mark each other’s homework. Worse, it would become a mechanism to silence those who do not share the True Faith. Papers which fundamentally challenge the accepted orthodoxy would never see light of day in the core periodicals of the field. All this is, of course, speculation. Is it borne out by reality?

The academic rigour that peer review implies was always assumed, but until recently no one thought to test it. Then, in 1994, an American physicist called Alan Sokal decided to see how easy it would be to perpetuate an academic fraud. He submitted an article to the Duke University periodical Social Text. We might fairly describe it as a periodical in the postmodernist avant garde. In his article, Sokal used postmodernist ideas to challenge the fundamental basis of science. To put it slightly crudely, he argued that something like the theory of gravity is merely a figment of the imagination of the colonialist, white supremacist patriarchy, and it is therefore nothing more than a tool to keep the oppressed masses in their place, on terra firma. (After all, it was first put forward by a white, English-speaking male. That alone has to be grounds for suspicion.) Sokal quoted all the right theorists and used all the right jargon to make it look as though he knew what he was talking about, but took care to include enough deliberate errors in basic science to make it clear that his article was rubbish. Social Text published Sokal’s article. When he revealed the hoax, his fellow scientists enjoyed the joke. Social scientists were rather less amused.

However, Social Text was not peer reviewed (although it is now). This is why a trio of scholars, most notably Peter Boghossian, decided to conduct a larger-scale test in 2017-18, but aimed purely at peer reviewed periodicals. They, like Sokal, had doubts about the quality of scholarship in the social sciences and humanities. They chose the most politically contentious areas of social science research, such as race, gender and sexuality, which they dubbed “grievance studies”. 

Boghossian et al set themselves the goal of getting twenty fictitious papers published in the most prestigious, peer reviewed periodicals in their fields. To do this, they had to produce papers which would not raise suspicion in the mindsets of the people who edited and reviewed papers for these periodicals, but which would be obviously bogus to the general public. To that end, they studied the papers these periodicals were publishing, until they could mimic the style word-perfectly. Then they pushed the envelope slightly, to come up with ideas for their own papers that were either absurd or unethical.

Their first attempt, which argued that the male penis conceptually causes climate change, was not a success. It was too obviously a hoax. Later attempts were much more successful. Papers which were accepted for publication argued: that society’s prejudice against unhealthy, obese bodies was an example of oppression; that men could be cured of homophobia and transphobia by encouraging them to masturbate anally with dildos; that men who visited restaurants like “Hooters” to gaze at waitresses with big breasts were examples of the oppressive patriarchy; and that social justice scholarship should be immune to criticism. They managed to get a feminist reworking of a chapter from Hitler’s Mein Kampf accepted for publication, and even a rambling, misandrist poem, which was written by a random poetry generator on the internet. 

One of their papers, which was still under consideration when the experiment was ended prematurely, argued that artificial intelligence is too male-oriented, in that it is rationalist and masculinist, and that it needs to be female-oriented. It is noteworthy that another paper, written by academics at New York University, and which argues almost exactly the same thing, was published in May 2019. One fears the NYU academics were being entirely serious.

The pièce de résistance was a paper on dogs which rape other dogs in the park. It argued that the lessons learned here could be used to train men to refrain from sexual violence and misogyny. Not only was this accepted for publication, it was acclaimed for its outstanding scholarship. (The underlying aim was to see if a paper whose argument was ludicrous and unfalsifiable would be accepted. Clearly, it was.) It was only when journalists took an interest, and wondered why neither the purported author nor the academic institution which employed her existed, that Boghossian et al came clean and admitted what they were doing. That ended prematurely the experiment that has become known as “Sokal Squared.” The final tally was: seven papers accepted for publication, six rejected, and the remaining seven still under consideration for publication.

The ramifications of Sokal and Sokal Squared will reverberate in academia for some time to come. In the meantime, politicians and civil servants, who rely on academic research for a secure platform for formulating public policy, have some thinking to do. Just how reliable is the peer reviewed research in ideology-driven fields, and can we take it on someone’s say-so that it is “irrefutable”? I am not suggesting that any specific piece of research, or whole fields of research have been corrupted by ideological conviction. On the contrary, I am arguing that we cannot assume they have not – even when they have been subjected to the purity test of peer review.

A Message from TheArticle

We are the only publication that’s committed to covering every angle. We have an important contribution to make, one that’s needed now more than ever, and we need your help to continue publishing throughout the pandemic. So please, make a donation.