Science Media Centre
Dr Samir Bhatt
FEBRUARY 3, 2022
terrabrasil
A preprint, an unpublished non peer reviewed study posted on the John Hopkins Krieger School of Arts and Sciences website, has looked at the impact of lockdowns.
Prof Neil Ferguson, Director of the MRC Centre for Global Infectious Disease Analysis, Jameel Institute, Imperial College London, said:
“This report on the effect of “lockdowns” does not significantly advance our understanding of the relative effectiveness of the plethora of public health measures adopted by different countries to limit COVID-19 transmission.
First, the policies which comprised “lockdown” varied dramatically between countries, meaning defining the term is problematic.
In their new report, Herby et al appear to define lockdown as imposition of one or more mandatory non-pharmaceutical interventions (NPIs); by that definition, the UK has been in permanent lockdown since 16 th of March 2021, and remains in lockdown — given it remain compulsory for people with diagnosed COVID-19 to self-isolate for at least 5 days.
“A second and more important issue is that the statistical methods used to estimate the impact of NPIs using observational data need to be appropriate.
Such interventions are intended to reduce contact rates between individuals in a population, so their primary impact, if effective, is on transmission rates.
Impacts on hospitalisation and mortality are delayed, in some cases by several weeks. In addition, such measures were generally introduced (or intensified) during periods where governments saw rapidly growing hospitalisations and deaths.
Hence mortality immediately following the introduction of lockdowns is generally substantially higher than before.
Neither is lockdown a single event as some of the studies feeding into this meta-analysis assume; the duration of the intervention needs to be accounted for when assessing its impact.
“A consequence of NPIs affecting transmission (rather than total deaths directly), is that interventions cannot be assumed to have fixed additive effects on outcome measures such as deaths over a certain time window — interventions affect transmission rates, and therefore the appropriate outcome measures to consider are growth rates (of cases or deaths) over time, with appropriate time lags — not total cases or deaths.
Many studies of the effects of NPIs (e.g. https://www.thelancet.com/pdfs/journals/eclinm/PIIS2589-5370(20)30208-X.pdf ) fail to recognise this important issue, but notable and methodologically rigorous exceptions have been published by both economists (e.g. Chernozhukov et al — https://www.sciencedirect.com/science/article/pii/S0304407620303468, which studied NPIs in the US) and public health researchers (e.g. Brauner et al. https://www.science.org/doi/10.1126/science.abd9338, an analysis of NPIs in 41 countries).
Interestingly, both of the latter papers (only one of which is included in the Herby et al meta-analysis) reach similar qualitative conclusions, despite their results not being directly comparable.
Namely, that the effectiveness of “lockdowns” came from the combined impact of the multiple individual interventions which made up that policy in different countries and states: limiting gathering size, business closure, mask wearing, school closure and stay at home orders.
While removing any one of the measures making up “lockdown” is predicted by most studies to have a relatively limited effect on the effectiveness of the overall policy, that does not mean that the combined set of measures in place during times countries were in “lockdown” were not highly effective at driving down both COVID-19 transmission and daily deaths.
“Disentangling the precise impact of individual NPIs remains extremely challenging, not least because the most socially and economically disruptive measures (closing all non-essential businesses, stay at home orders) were generally used in combination and as last resorts on top of longer-term measures such as mask wearing.
Analysis has been further complicated by the accumulation of immunity (from infection and vaccination) in populations together with the emergence of new COVID-19 variants.
Distinguishing the relatively effectiveness of mandates versus government recommendations — while clearly of political interest — is even more challenging, given the large between- (and even within-) country differences in population responses to both types of measures.”
Dr Seth Flaxman, Associate Professor in the Department of Computer Science, University of Oxford, said:
“Smoking causes cancer, the earth is round, and ordering people to stay at home (the correct definition of lockdown) decreases disease transmission.
None of this is controversial among scientists. A study purporting to prove the opposite is almost certain to be fundamentally flawed.
“In this case, a trio of economists have undertaken a meta-analysis of many previous studies. So far so good.
But they systematically excluded from consideration any study based on the science of disease transmission, meaning that the only studies looked at in the analysis are studies using the methods of economics.
But they systematically excluded from consideration any study based on the science of disease transmission, meaning that the only studies looked at in the analysis are studies using the methods of economics.
These do not include key facts about disease transmission such as:
- later lockdowns are less effective than earlier lockdowns, because many people are already infected;
- lockdowns do not immediately save lives, because there’s a lag from infection to death, so to see the effect of lockdowns on Covid deaths we need to wait about two or three weeks.
(This was all known in March 2020 — we discussed it in a paper released that month, and later published in Nature. Our paper is excluded from considerationin this meta-analysis.)
Prof Samir Bhatt, Professor of Statistics and Public Health, Imperial College London:
“It’s as if we wanted to know whether smoking causes cancer and so we asked a bunch of new smokers: did you have cancer the day before you started smoking? And what about the day after?
If we did this, obviously we’d incorrectly conclude smoking is unrelated to cancer, but we’d be ignoring basic science.
The science of diseases and their causes is complex, and it has a lot of surprises for us, but there are appropriate methods to study it, and inappropriate methods.
This study intentionally excludes all studies rooted in epidemiology-the science of disease.”
This study intentionally excludes all studies rooted in epidemiology-the science of disease.
Prof David Paton, Chair of Industrial Economics, Nottingham University Business School, said:
“I find this paper has flaws and needs to be interpreted very carefully.
Two years in, it seems still to focus on the first wave of SARS-COV2 and in a very limited number of countries.
“I find this paper has flaws and needs to be interpreted very carefully.
The most inconsistent aspect is the reinterpreting of what a lockdown is. The authors define lockdown as “as the imposition of at least one compulsory, non-pharmaceutical intervention”.
This would make a mask wearing policy a lockdown.
For a meta-analysis using a definition that is at odds with the dictionary definition (a state of isolation or restricted access instituted as a security measure) is strange.
The authors then further confuse matters when in Table 7 they revert to the more common definition of lockdown.
Many scientists, including myself, quickly moved on from the word “lockdown” as this isn’t really a policy (Brauner et al 2020, and my work in Sharma et al 2021).
It’s an umbrella word for a set of strict policies designed to reduce the reproduction number below one and halt the exponential growth of infections.
Lockdown in Denmark and Lockdown in the UK are made up of very different individual policies.
Aside from issues of definitions there are other issues such as
(a) It’s not easy to compare Low and High income countries in terms of the enforcement and adherence of policies,
(b) Many countries locked down before seeing exponential growth and therefore saw no reduction in deaths,
© There are lags — interventions operate on transmission but mortality is indirect and lagged — comparing mortality a month before and after lockdown is likely to have no effect (e.g Bjørnskov 2021a),
(d) As i have mentioned it looks at a tiny slice of the pandemic, there have been many lockdowns since globally with far better data,
(e) There are many prominent studies that cover the period in question looking at infections included including Brauner et al 2020, Alfano et al 2020, Dye et al 2020, Lai et al 2020, Hsiang et al 2020, Salje et al 2020 etc.
The list of such studies is very long and suggests a highly incomplete meta-analysis. “
The list of such studies is very long and suggests a highly incomplete meta-analysis. “
“First the paper is not yet peer-reviewed. It looks to be of good quality so I suspect it will end up in a peer-reviewed journal but obviously results need to be interpreted with that caveat.
“Meta analysis is becoming a bit more common in social sciences. There is a whole literature on pros & cons of that approach, e.g. can be difficult to combine studies with quite different measures & methodologies.
This is a bit tricker in the social sciences than when dealing with RCTs in the medical field. However, the authors do start with a more standard systematic literature review before going on to the MA.
“Both parts of the paper (systematic review and the meta analysis) make a significant contribution to our understanding of lockdown effects.
“Key to a systematic review like this are the sets of search & exclusion criteria. The paper is very transparent about this which is good. They focus on difference-in-difference empirical studies. i.e. they look at papers which compare the impact of an intervention on mortality by looking before & after, but relative to other areas which did not have the intervention.
As a result, modelling studies (like the well-known Flaxman Nature paper) are excluded. That is not controversial.
More marginal in my view is their exclusion of synthetic control method (SCM) papers. Some of these paper find a significant impact of NPIs on mortality so including them might have led to somewhat higher average mortality effects. The paper gives a robust defence of their exclusion, but I think you would get people on both sides of that debate.
“One point on their search criteria is that they focus on SIPO/lockdowns and that means studies of very specific NPIs are not necessarily included. They discuss this openly but it does mean that their finding on specific NPIs are less robust than those on lockdowns & SIPOs.
One point on terminology, when they talk about lockdowns, they are discussion the overall stringency of interventions as a whole. I think most people (in this country at least) would understand SIPOs as being closer to the 3 actual lockdowns that we had in the UK.
“On the other side of the coin, I would query their inclusion of mortality effects which are observed very soon after the interventions. In particular Fowler et al (2021) and Dave et al (2020). The authors note that the timing of the big mortality effects is impossible given the lag between infection & death. However, the studies meet their inclusion criteria so they keep them in. If these were excluded, the SIPO average mortality effect would certainly be less than 2.9%.
“I guess that is quibbling about the details, though. The headline of the paper is that the central estimate of the impact of SIPOs is that they reduced Covid-19 mortality by 2.9% is their central estimate. Based on their sensitivity analysis, we can’t rule out a bigger impact, but we also can’t rule out that there was no impact at all on Covid mortality. Same goes for overall stringency of interventions, though the central estimate is even small.https://joaquimcardoso.blog/media/c72beb3434069c0c01dfd149bd828b82
All our previous output on this subject can be seen at this weblink:www.sciencemediacentre.org/tag/covid-19Declared interests
“That is pretty consistent with other, non- systematic reviews (e.g. Herby & Allen) which is reassuring. It is also consistent with the (few) studies which look at the impact on overall excess mortality. There are only 3 of which I am aware: Glass, Bjornskov and Agrawal. All three find that the net impact of lockdowns on excess mortality is either zero or positive. It is not completely impossible that those findings are explained by a very large negative impact of lockdowns on Covid mortality and an equally large impact on non-Covid mortality. However, much more likely is that lockdowns have a small negative impact on Covid mortality but a positive (& similar or slightly larger) impact on overall mortality. In that context, the central estimates of the Hanke paper are unsurprising to those who know the literature (though obviously very newsworthy in terms of the general perception).”
Declared interests
Dr Samir Bhatt: I am the senior author on the first ever study comparing the effect of lockdowns, ( https://www.nature.com/articles/s41586-020-2405-7) and several other studies on this topic.
None others received.
Originally published at https://www.sciencemediacentre.org.
