editors, reporters, tyrannosaurs, and dirty professional secrets.
Last week, we were all treated to the news that Tyrannosaurus rex, everyone’s favorite dinosaur, was actually three distinct species. This was according to a paper that was published in a peer-reviewed journal, though associated news reports suggested the idea was “controversial.”
I think this incident reveals a lot about weaknesses in the peer-review system and the way the media report scientific developments. I’d like to discuss these with y’all’s indulgence.
First, my qualifications – I’m a professional vertebrate paleontologist. My career has largely been devoted to crocodyliforms (crocodiles, alligators, gharials, and their extinct relatives), but after finishing grad school, I spent time as a post-doc at a large natural history museum working on a very infamous tyrannosaur skeleton. I won’t divulge much more except to say this particular theropod’s common name rhymes with “due” and that they shouldn’t have hired me. They should have hired a priest. I still say it screams when you throw holy water on it.
Anyway –
The paper arguing that T. rex is three separate species is terrible. Awful. Laughable. One of the worst papers I’ve ever seen in the peer-reviewed literature. It makes mistakes the average undergraduate with one or two biology or historical geology courses under their belt would know to avoid. There are portions that actually qualify as “unintentionally funny.” It demonstrates nothing, when one considers the actual evidence we have.
A lot of us are still trying to figure out how, exactly, it got through peer-review. I know one person who submitted a review, but the recommendations in that review were ignored. We don’t know if the celebrity status of the lead author – not an academic, but an accomplished and well-known (if iconoclastic) paleoartist – influenced the editors, or if someone on the editorial team knew the authors. It wasn’t published in an open-access journal, so the whole “pay to play” scenario is very unlikely to be the case. It’s entirely possible the authors submitted this to multiple journals until they found one willing to run it. But whatever the reason, this was a lapse of editorial oversight. The paper should never have been published in a peer-reviewed journal.
This highlights a problem we academics understand, but the general public generally does not – that, to corrupt a quote from Winston Churchill, peer review is the worst possible form of editorial oversight, except for all of the alternatives that have been tried from time to time. Peer review is run by fallible human beings. It’s no better than the fallible human beings involved. It's something of a dirty secret that no one's tried to hide, but which hasn't gotten a whole lot of attention.
I’ve heard people equate peer review with censorship, but in fact, it’s a whole lot easier to get bad research into the peer-reviewed literature than to prevent something cutting edge and controversial, but prescient, from being published. In fact, it’s gotten much, much easier because of the proliferation of journals, and especially of open-access journals in which authors pay a publication fee, in recent years. Some open-access journals are legit, but a lot of them aren’t – they’ll publish anything just so long as the author pays up.
But even without this, peer review has its limitations. I encounter this all the time in my own little field because of its cross-disciplinary nature. Papers using molecular data to work out the relationships and divergence timing of crocodylians are regularly published. They often make basic mistakes in using the fossil record to calibrate rates of evolution, or they make comparisons between their results and outdated paleontological work. There’s nothing nefarious about this; the authors, in these cases, were trained as molecular biologists and don’t really have the background in paleontology needed to fully grasp the literature. These are honest mistakes, not rhetorical gymnastics. (We paleontologists, by contrast, usually have some training in molecular biology and can swim through their literature reasonably well.)
The problem is with the editors. Papers on molecular systematics are usually reviewed by other molecular systematists. It doesn’t dawn on the editors that it might be good to have the manuscript reviewed by a paleontologist. Papers that could have been excellent are diminished because of some honest mistakes a reviewer should have caught.
There are other reasons peer review fails. Sometimes, it’s because some of us are assholes. They may feel threatened by something written by a younger scholar that challenges their work. Or they may be sexists, bigots, racists, or some other species of dirtbag. If they have some sort of clout, they can submit an unnecessarily harsh review that kills the manuscript. And because some fields are small, it may be impossible to avoid such fuckwits as reviewers.
Peer-review can, indeed, fail if a manuscript contradicts the consensus. But if it does, this is rarely because it bucks consensus. It's usually because it doesn't have enough support to do so. Wegener's theory of continental drift (which, granted, was presented before peer review was a thing) was rejected not because it ran against common wisdom, but because the evidence he marshalled, though interesting, wasn't enough to shift opinion. That he had no mechanism to drive it didn't help. (He actually thought continents plowed through oceanic crust. In fact, oceanic and continental crust move together.) The consensus shifted when a sufficient volume of new observation met a workable mechanism. Geologists weren't closed-minded; they just didn't see enough to force a revisit of the paradigm.
Some journals use a double-blind system for peer review, but in my experience, it generally doesn’t work. Our fields are fairly tight; it’s all but impossible to remain anonymous. Most authors can figure out it’s my review whether I sign my review (and I usually do) or not. And an established jerk is going to go nuclear on a paper challenging their work no matter who the authors are.
I’m not saying we should abandon peer-review. We need some sort of quality control in science, and this seems to be the best we can manage. But the mere fact that a paper was peer-reviewed doesn’t necessarily mean it’s reliable. We must all keep this in mind.
The T. rex paper also laid another phenomenon bare – the way reporters try to convey what looks like exciting science.
News reports uniformly expressed that this paper had generated “controversy.” In fact, the only controversy it’s generated is how, exactly, it got through peer review. Scientifically, it’s been more or less universally rejected by experts in the field. But news sources can’t get much traction from “bad paper elicits dismay from professionals.”
I wasn’t contacted about this particular paper, but I’ve been asked for comments before. Reporters are cool with disagreement, but what they want is something like “this is so cool! We have to re-write our textbooks!” As long as you include something like that, you might get quoted and, just maybe, a qualifier will squeak through.
I’m pretty sure reporters in this case were shocked at what they encountered. Every single person they called for a comment said what I would have said – that this is an awful paper, it doesn’t demonstrate a damned thing about tyrannosaurs, and it shouldn’t be given air time.” They were literally unable to get “cool” or “re-write the textbooks.” So they went with “controversial.”
And you know the worst part? It’s entirely possible that T. rex might be divisible. I HIGHLY doubt it’s divisible into more than two, and it’s extremely unlikely either of these would have co-occurred. That could be a cool study. It would involve a modern approach to variation and a thoughtful approach to the nature of species. Unfortunately, that’s not what happened with this paper. We’re learning some lessons, but this isn’t one of them.
(And no, I'm not going to do that study. Crocodiles are objectively better than dinosaurs - we don't need special effects to see them eat people. I've got plenty enough to do with them.)
= new reply since forum marked as read
