Avoiding 'muddled science' in the newsroom
On April 23, I was part of a webinar called ProtoCall, organised by Pro.to with the support of International Centre for Journalists and the International AIDS Vaccine Initiative. It happens once a week and is hosted by Ameya Nagarajan and Nayantara Narayanan. Every week there’s a theme which, together with the discussion around it, is picked to help non-science and non-health journalists cover the coronavirus pandemic. The session before the one I was part of discussed the role of data, the gaps in data and how journalists could help fill them. My session was entitled ‘How muddled science drives misinformation’, and my fellow panelists were Shruti Muralidhar and Shahid Jameel, neither of whom should need introduction on the pages of this blog.
Given a brief ahead of the session (available to read here), I prepared some notes for the conversation and which I’m pasting below in full. Note that the conversation itself panned out differently (as military historians have noted, “no plan survives contact with the enemy”), so you could watch the full video if you’re interested or read the transcript when it comes out. Both Shruti and Dr Jameel made some great points throughout the conversation, plus the occasional provocative opinion (by myself as well).
§
1. Science journalists should continue to do what we’ve always had to do — empower our readers to decide for themselves based on what data they have available. Yes, this is a slow process, and yes, it’s tedious, but we shouldn’t have to adopt radical tactics now just because we haven’t been doing our job properly before. Introduce the relevant concept, theories, hypotheses, etc. as well as introduce how scientists evaluate data and keeping what in mind.
I can think of at least three doctors I’ve spoken to recently – all three of very good standing in the medical research community, and one is pro-lockdown, one is anti-lockdown, and one argues that there’s a time and place to impose a lockdown. This is a new virus for everybody and there is disagreement between doctors as well. But this doesn’t imply that some doctors are motivated by ideologies or whatever. It means the story here is that doctors disagree, period.
2. Because this is a new disease for everybody, be skeptical of every result, especially those that claim 100% certainty. No matter what anyone says, the only thing you can know with 100% certainty is that you cannot know anything with 100% certainty. This is a pandemic and suddenly everyone is interested in what scientific studies have to say, because people are desperately looking for hope and there will be a high uptake for positive news – no matter how misinformed or misguided.
But before everyone was interested in scientific studies, it was always the case that results from tests and experiments and such were never 100% accurate. They all had error rates, they were all contingent on replication studies, they were and are all works in progress. So no matter what a study says, you can very safely assume it has a caveat or a shortcoming, or a specific, well-defined context in which it is true, and you need to go looking for it.
3. It’s okay to take time to check results. At a time of such confusion and more importantly heightened risk, misinformation can kill. So take your time, speak to doctors and scientists. Resisting the pressure to publish quickly is important. If you’re on a hard deadline, be as conservative in your language as possible, just go with the facts – but then even facts are not entirely harmless. There are different facts pointing to different possibilities.
Amitabh Joshi said a couple years back at a talk that science is not about facts but about interpreting collections of facts. And scientists often differ because they’re interpreting different groups of facts to explain trends in the data. Which also means expertise is not a straightforward affair, especially in the face of new threats.
4. Please become comfortable saying “I don’t know”. I think those are some of the most important words these days. Too many people – especially many celebrities – think that the opposite of ‘true’ is ‘false’ and that the opposite of ‘false’ is ‘true’. But actually there’s a no man’s land in between called ‘I don’t know’, which stands for claims, data, etc. that we haven’t yet been able to verify yet.
Amitabh Bachchan recently recorded a video suggesting that the coronavirus is transmitted via human faeces and by flies that move between that faecal matter and nearby food items. The thing is, we don’t know if this is true. There have been some studies but obviously they didn’t specifically study what Amitabh Bachchan claimed. But saying ‘I don’t know’ here wouldn’t mean that the opposite of what Bachchan said is true. It would mean Bachchan was wrong to ascribe certainty to a claim that doesn’t presently deserve that certainty. And when you say you don’t know, please don’t attach caveats to a claim saying ‘it may be true’ or ‘it may be false’.
We need to get comfortable saying ‘we don’t know’ because then that’s how we know we need more research, and even that we need to support scientists, etc.
5. Generally beware of averages. Averages have a tendency to flatten the data, which is not good when regional differences matter.
6. Has there been a lot of science journalism of the pandemic in India? I’m not sure. A lot of explanations have come forth as background to larger stories about the technology, sampling/testing methods, governance, rights, etc. But I’ve seen very little of the mathematics, of the biology and research into the virus as such.
I don’t think this is a problem of access to scientists or availability of accessible material, which to my mind are secondary issues, especially from journalists’ point of view. Yes, you need to be able to speak to doctors and medical researchers, and many of them are quite busy these days and their priorities are very different. But also many, many scientists are sitting at home because of the lockdown and many of them are keen to help.
To me, it’s more a problem of journalists not knowing which questions to ask. For example, unless you know that something called a cytokine storm exists, to you it remains an unknown-unknown. So the bigger issue for me is that journalists shouldn’t expect to do a good job covering this crisis without knowing the underlying science. A cytokine storm is one example, but I’d say not many journalists are asking more important questions, from my point of view, about statistical methods, clinical trials, scientific publishing, etc. and I suspect it’s because they’re not aware these issues exist.
If you want to cover the health aspects like a seasoned health journalist would, there are obviously other things you’re going to have to familiarise yourself with, like pharmaceutical policy, clinical trials, how diseases are tracked, hospital administration, etc.
So I’d say learn the science/health or you’re going to have a tough time asking the right questions. You can’t expect to go into this thinking you can do a good job just by speaking to different doctors and scientists because sooner than later, you’re going to miss asking the right questions.
7. Three things have worked for The Wire Science, vis-à-vis working with freelancers and other editors.
First, there needs to be clear communication. For example, if you disagree with a submission, please take time out to explain what you think is wrong about it, because it often happens that the author knows the science very well but may just not have laid it out in a way that’s completely clear. This is also exhausting but in the long run it helps.
Second, set clear expectations. For example at The Wire Science, I insist on primary sources to all claims to the extent possible, so we don’t accidentally help magnify a dubious claim made by a secondary source. I don’t accept articles or comments on papers that have not been published in a peer-reviewed journal or in a legitimate preprint repository. And I insist that any articles based on scientific papers must carry an independent voice commenting on the merits and weaknesses of the study, even if the reporter hasn’t spoken to the paper’s authors themselves.
Interestingly enough, in our internal fact-check filters, these ‘clear expectations’ criteria act as pre-filters in the sense that if an article meets these three criteria, it’s also factually accurate more than 90% of the time. And because these criteria are fairly simple to define and identify in the article, anyone can check for them instead of just me.
Third, usually the flow of information and decisions in our newsroom is top-down-ish (not entirely top-down), but once the pandemic took centerstage, this organisation sort of became radial. Editors, reporters and news producers all have different ideas for stories and I’ve been available as a sort of advisor, so before they pursue any story, they sometimes come to me to discuss if they’re thinking about it the right way.
This way automatically prevents a lot of unfeasible ideas from being followed up. Obviously it’s not the ultimate solution but it covers a lot of ground.
8. The urgency and tension of a pandemic can’t be an excuse to compromise on quality and nuance. And especially at a time like now, misinformation can kill, so I’m being very clear with my colleagues and freelancers that we’re going to take the time to verify, that I’m going to resist the temptation to publish quickly. Even if there’s an implicit need to publish stuff quickly since the pandemic is evolving so fast, I’d say if you can write pieces with complexity and nuance, please do.
The need for speed arises, at least from what I can see, in terms of getting more traffic to your site and which in turn your product, business and editorial teams have together decided is going to be driven by primacy – in terms of being seen by your readers as the publication that puts information out first. So you’re going to need to have a conversation with your bosses and team members as well about the importance at a time like this of being correct over being fast. The Wire Science does incur a traffic penalty as a result of going a bit slower than others but it’s a clear choice for us because it’s been the lesser price to pay.
In fact, I think now is a great time to say to your readers, “It’s a pandemic and we want to do this right. Give us money and we’ll stop rushing for ads.”
Full video:
