The headline ‘A new study says [x]’ can often be found in newspapers and the media generally. Why do journalists use it? And what is the general status of such studies? These questions relate to our attitudes to science itself, as well as to our attitudes to particular studies, published at particular times on particular subjects. One study has shown that the “academic system incentivises journals and researchers to publish exciting findings”. An important consequence of this is that many studies end up being just plain “wrong”.
Scientists and academics are human beings.
Scientists and academics have strongly-held values and strongly-held political beliefs.
Scientists and academics have strong emotions and feelings.
Scientists and academics have good careers and are usually tenured.
Scientists and academics need to pay their mortgages, bills, the school fees of their children, go on holidays, buy new homes, cars, computers, clothes, etc.
Most scientists and academics are funded by various and many bodies and institutions. They also work for — or actually within — governments, political parties, university departments (some with strong political and/or scientific biases), corporations, global institutions (such as the Intergovernmental Panel on Climate Change), thinks tanks (or policy institutes) and activist/pressure groups (such as The Heartland Institute and the Union of Concerned Scientists), the European Union, pharmaceutical/oil/etc. companies…
So it’s worth bearing all that in mind in the context of the following little story.
At the end of July I was reading a copy of the free newspaper Metro. In it, there were three short news items side by side. Two included the headline ‘A new study says [x]’, and the other one used the words “research shows” in the subtext.
These phrases are used all the time in national newspapers, on websites, and on social media. In fact, when it comes to national newspapers, it can safely be assumed that each publication will include at least one item every day which has the headline ‘A new study says [x]’.
There are also some odd examples of news items which have the headline ‘A new study says [x]’. Take these examples (which were easy to find):
‘Groundbreaking new study says time spent playing video games can be good for your well being’, ‘ChatGPT’s Performance Is Slipping, New Study Says’, ‘Some Extra Heft May Be Helpful, New Study Says’, ‘Climate change is leading to more home runs, says new study’, ‘People aged under 40 should not drink alcohol and consumption guidelines should be changed, study says’, ‘Bradford beats London for business startups, says new study’, ‘A new study says only working out on the weekend is just fine (and here’s how to do it)’, ‘Tip or no tip? Over 65% of U.S. have negative view on tipping, new study’, ‘Can You Be Overweight & Healthy? A New Study Says It’s Possible’, ‘A new study says it’s okay to eat red meat. An immediate uproar follows’, ‘New Jersey is one of the best places to live and work in the US, new study says’, ‘Turn off that camera during virtual meetings, environmental study says’, ‘Is Coffee Good For You? A New Study Says It Is’, ‘Children Who Are Exposed to Awe-Inspiring Art Are More Likely to Become Generous, Empathic Adults, a New Study Says’, ‘Is sitting always bad for older adults? A new study says maybe not’…
Ironically enough, we also have the following titles:
‘Most scientific papers are probably wrong’, ‘Study: half of the studies you read about in the news are wrong’, ‘Why Most Published Research Findings Are False’, ‘This is why you shouldn’t believe that exciting new medical study’, ‘New Study Says There Are Too Many New Studies, New Study Finds’, ‘Over half of psychology studies fail reproducibility test’, ‘Research findings that are probably wrong cited far more than robust ones, study finds’, ‘Is Most Published Research Really False?’…
‘A new study of a new study which says that most new studies are wrong is wrong’…
That last one is a joke!
Yet we do have this study: ‘Ioannidis (2005) was wrong: Most published research findings are not false’.
So now readers can wait for a study called, ‘The study which claimed that ‘Ioannidis was wrong to claim that most studies are wrong’ was itself wrong’. (Incidentally, Professor John Ioannidis’s research findings will be discussed later.)
In any case, every day there will be dozens — perhaps hundreds — of studies published. There’s also vast amounts of research being carried out all the time by numerous scientists, academics and graduates. Hundreds of studies and papers are published every month…
And all that largely explains why newspaper editors and journalists find it so easy to write news items with the headline ‘A new study says [x]’.
So I suggest that readers put the words ‘A new study says…’ into the Google search engine. (One set of results can be seen here.)
I personally found 50 ‘A new study says [x]’ headlines, and then I gave up counting. There are 20 Google pages of links with the headline ‘A new study says [x]’. And then, for some reason, the Google search just comes to an end. (Readers can presume that it could have gone on for a lot longer.)
So what are we meant to make of the news headline ‘A new study says [x]’?
A New Study Says [x]
This essay is going to stick to the words ‘A new study says [x]’, rather than going into the details of particular studies, or the details of the news items about particular studies. That said, there’ll be a small amount of commentary on the studies themselves.
That aforementioned distinction between scientific studies and new items about those studies is relevant here because journalists may simply get things wrong.
A news item on a scientific study may also be extremely biased.
What’s more, the news item (or journalist) may ignore various important parts of the study, and play up various unimportant parts.
All this is very easy to do, either wilfully or through basic ignorance.
On another theme.
What a new study says may be false.
Some parts of what a study says may be true, and other parts may be false.
What a study says may also be irrelevant, trivial, plagiaristic, biased, a waste of time, etc.
What’s more, what a new study says is often contradicted at a later point. Indeed, sometimes new studies are strongly contradicted within weeks (or less), but usually the contradictions come much later.
In addition, the headline ‘A new study says [x]’ is often meant to give legitimacy to whatever the news item is about. Basically, sometimes it’s the journalist himself who’s using the headline ‘A study says [x]’ to give credence to what he believes anyway. Editors may also publish stories about studies (or research) simply to fill space. Who knows, they may sometimes do so to enlighten their readers.
A Scientific Study About Scientific Studies
There’s a study called ‘Why Most Published Research Findings Are False’, which was written in 2005 by Professor John Ioannidis of the Stanford School of Medicine.
[When I Googled the words ‘A new study says…’, as mentioned earlier, I couldn’t find this particular new study. It was discovered via other means.]
Some readers may spot that there’s a self-referential problem here. The Guardian newspaper, for example, picks up on the self-reference when it says that this study “is itself not exempt from the need for scrutiny”.
In any case, one passage in ‘Why Most Published Research Findings Are False’ says something which many laypersons have known for decades. Professor Ioannidis wrote:
“There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims.”
What’s more:
“Published research findings are sometimes refuted by subsequent evidence, with ensuing confusion and disappointment.”
All this has been especially true about dietary matters or issues relating to food and health.
There’s also “controversy” over and above what is “refuted” in these studies. As Ioannidis put it:
“Refutation and controversy is seen across the range of research designs, from clinical trials and traditional epidemiological studies to the most modern molecular research.”
Is it really the case that, as Dr Marta Serra-Garci (who studies behavioural and experimental economics at the University of California) puts it, some studies “conclude that something is true or not based on one study and one replication”?
Readers may also wonder why certain (scare-quoted) “scientific” disciplines are more prone to bullshit than others.
Professor Brian Nosek on Scientific Studies
Professor Brian Nosek (at the University of Virginia) tells us that “[w]e presume that science is self-correcting”. Indeed, many scientists do say that science is self-correcting. (Perhaps Nosek meant fellow scientists — not people as a whole — by his word “we”.)
For a start, it’s not really the case that most laypersons presume that science is self-correcting. Most laypersons may well (as it were) believe in science (whatever that means). However, that probably isn’t because they also believe that science is self-correcting…
Actually, it depends.
It depends on how much a particular (to put it crudely) pro-science layperson thinks about the nature of science.
Put it this way.
Many of those people who see themselves as being pro-science think less about the actual (philosophical and otherwise) nature of science (or read less science books and papers) than many of those who’re critical of science. Anecdotally, this is something I’ve come across many times — especially in political contexts. Particularly, claiming to be “pro-science” is often simply a way of also saying that one’s political opponent is anti-science.
To get back to Professor Nosek’s words:
“[E]rrors will happen regularly, but science roots out and removes those errors in the ongoing dialogue among scientists conducting, reporting, and citing each others research.”
This is a simple point about the communal nature of (all? most? much?) science.
In basic terms.
On the large-scale and over (relatively) long periods, science often is self-correcting. However, particular scientists, particular groups of scientists, particular scientific journals, particular scientific departments, etc. are often not self-correcting. Or, at very least, they aren’t self-correcting to a satisfactory degree. (It would be hard to even imagine any scientist, scientific group or scientific institution never indulging in self-correction.)
In detail.
Even most scientists accept the problem of political and other kinds of bias in scientific work. So in this particular case, biased scientists don’t (to use Professor Nosek’s words) “root out and remove errors” when those errors work in support of their political — and otherwise — biases. Or, as it’s often been put, the errors will be retained if they work toward the conclusion (or thesis) that these scientists wanted to arrive at from the very beginning.
More relevantly, when a particular news headline tells us that ‘A new study says [x]’, then this may be a study by a group of scientists who weren’t sufficiently self-correcting. Indeed, in isolated cases, some scientists aren’t self-correctors at all.
All this ties in with Nosek’s words that
“dialogue among scientists conducting, reporting, and citing each others research”.
But what if many (even all) the other scientists engaged in a particular “dialogue” are also biased in the same way or ways?
What if they adhere to the same theories, worldviews, etc?
Thus, how will dialogue work (or help) in this case?
If anything, “citing each others research” would only compound (or solidify) the bias and/or lack of self-correction.
Professor Nosek himself admits that scientists aren’t always self-correcting. He says:
“If more replicable findings are less likely to be cited, it could suggest that science isn’t just failing to self-correct; it might be going in the wrong direction.”
Nosek also picks up on a point that undergraduate philosophy students - and almost everyone else with a logical mind — does (or should) spot. He tells us that it’s often the case that when it comes to the conclusions of particular studies, the “evidence is not sufficient to draw such a conclusion”. (Nosek believes this is true of many studies.)
The problem here, Nosek argues, is about the “social systems of science” which don’t “foster self-correction”.
Interestingly and relevantly, many scientists have been critical of what the philosophers Thomas Kuhn and Paul Feyerabend (along with many others) have said about the (to use Nosek’s words) social systems of science. However, we must add here that actual scientists have said similar things too. The thing is, many scientists will admit to various shortcomings when it comes to scientific practice, but virulently deny such things when it comes to science itself.
[To take just one example of a scientist who’s aware of scientific bias and the sociology of science, see theoretical physicist Lee Smolin’s ‘How Do You Fight Sociology?’ chapter in his book The Trouble With Physics. This specifically deals with the decades-old social systems of string theory.]
So now take this very Kuhnian passage from the aforementioned Guardian article:
“The academic system incentivises journals and researchers to publish exciting findings, and citations are taken into account for promotion and tenure. But history suggests that the more dramatic the results, the more likely they are to be wrong.”
So do we really have science on the one hand, and the social systems of science on the other hand?
Some “radical” philosophers of science (such as the ones just mentioned), as well as historians of science, have strongly questioned this bifurcation or binary opposition.
More relevantly, when we read the phrase ‘A new study says [x]’, we should be aware — at least to some degree — of the (to use Professor Nosek’s words again) social systems of science in which that study was embedded.
Other Media Clichés About Scientific Studies
The headline ‘A new study says [x]’ can even be taken to be a journalistic cliché. So now let’s looks at some other examples.
Take this headline: ‘A new study says that [x] might lead to [y]’.
A lot of other headlines (or actual studies) say that ‘[x] might be [y]’, ‘[x] could be [y]’ or ‘[x] could occur if [y]’.
The (very) important word here is might.
To be extreme for a moment. x might lead to all sorts of things. The probability of x leading to y may also be fantastically small. What’s more, the fact that it’s a fantastically small probability may not even be mentioned in the news item (or even the study) itself.
Indeed, there could be a flying teapot floating around the Sun (see ‘Russell’s teapot’). The Royal Family may be controlled by alien reptiles (see here). The world could come to an end on August the 20th, 2023. There could be philosophical zombies. Etc.
So saying that something could happen (or that something could be the case), may not be much of claim. Of course, the news item (or actual study) will still have the veneer of being important — even very important.
So all this has to be placed within the context of the infinite number of things which could be the case. What’s more, if ‘A new study says [x] is going to happen’, then another study (perhaps published the day after) may say that ‘[x] is unlikely to happen’.
In other news headlines about studies (as well as the studies themselves) we also have the common, “[x] has been associated with [y]”. Again, and at an extreme, anything can be associated with anything else. [See note.]
Yet many journalists and editors don’t care about all this.
Indeed, even if journalists don’t sympathise with the explicit or implicit politics of the study (i.e., if they don’t want to “weaponize” it), or even care about the science of the study, then they may still write a titillating and saleable news item with the headline ‘A new study says [x]’. After all, most editors love this kind of stuff.
Conclusion
It must now be said that criticising particular scientists and particular studies is sometimes — indeed often — deemed to be what’s called “anti-science”. Yet seeing all such criticisms as being anti-science is itself very… anti-science. Or, at the very least, it’s very unscientific. In other words, this conflation of science itself with particular scientists and particular studies saying particular things at particular times is very unscientific.
More relevantly, this conflation of science itself with whatever a new study says is scientifically naïve.
However, it can be acknowledged that taking such a view of science may lead some people to arbitrarily pick and choose which scientists or scientific studies to believe. Yet that’s also certainly the case with many of those people who conflate particular scientists, particular scientific institutions or particular studies with (what’s now come to be called) “the science”.
Finally, all the above isn’t to argue that all studies are false, biased or a waste of time…
However, there are many good reasons (some just discussed) to believe that many studies are.
Of course, it would need to be explained why certain studies are false, biased or a waste of time, and others aren’t. And that explanation may itself display biases or simply show personal interests or concerns. It can also be suspected that most people do deem various (even many) studies to be false, wrong, biased or simply a waste of time. Of course, these conclusions too will probably involve at least some bias or personal interest.
Yet these controversial — often fierce — debates about what studies say are a very good thing… for science.
Note
(1) The guilt by association trick is used a hell of a lot in politics and on social media. For example, Person A is — as the phrase has it — “linked to” (nefarious) Group Y or Person Z in order to discredit that person. Yet that link may well be very weak or completely irrelevant. Still, the journalist’s (or activist’s) purpose is to plant that link into the minds of his viewers or readers.
No comments:
Post a Comment