What do you mean, all different? Most are exactly the same. The first 4 are a bit low and the last 3 a bit high, but last 2 and first also extremely wide, so irrelevant anyway. Everything else agrees, most within >99 % confidence with only slight differences on the absolute values.

The chart sure makes it look like there was an overall consensus that refs are about 1.5x as likely, though.

Obligatory link to Statistics Done Wrong: The Woefully Complete Guide, a book on how statistics can and has been abused in subtle and insidious ways, sometimes recklessly. Specifically, the chapters on the consequences of underpowered statistics and comparing statistical significance between studies.

I'm no expert on statistics, but I know enough that repeated experiments should not yield wildly different results unless: 1) the phenomenon under observation is extremely subtle so results are getting lost in noise, 2) the experiments were performed incorrectly, or 3) the results aren't wildly divergent after all.

Just eyeballing the linked image... it looks like most of them agree?

The bias almost certainly exists, according to nearly all analysis here. They just disagree on its magnitude. And for the most part they don't disagree by much.

I really found this out while writing my essay. If I wanted to I could interpret it slightly differently, resulting in totally different results.

Ah, the ol' art and science of statistics.

Scientists who fiddle around like this — just about all of them do, Simonsohn told me — aren’t usually committing fraud, nor are they intending to. They’re just falling prey to natural human biases that lead them to tip the scales and set up studies to produce false-positive results.

Since publishing novel results can garner a scientist rewards such as tenure and jobs, there’s ample incentive to p-hack.

I mean really, making claims they aren't committing fraud yet in the very next paragraph demonstrates their motivation... To commit fraud

Nevermind the numerous cases of published papers being bunk. And that something like 80% of published science isn't reproduceable...which is part of what publishing is to enable.

Why have 4 of the studies seemingly not used error bars at all‽ Like I get that different analyses will arrive at different results, but they should always have error bars, right?