institute for
continuous health transformation (CHT)
Joaquim Cardoso MSc
Senior Advisor for
Continuous Health Transformation
and Digital Health
EXECUTIVE SUMMARY
The Bill and Melinda Gates Foundation has identified “uninformative research” as a global health crisis where clinical trials are conducted but don’t provide enough information to determine whether the interventions being tested are successful or not.
- This problem results in billions of dollars and millions of hours wasted, resources that the global health sector is in short supply of.
- The foundation cites a 2021 study of COVID-19 drug trials, where only 5% of study arms had enough participants to be statistically valid, …
- … and a study of U.S. clinical trials in heart disease, diabetes, and lung cancer where only 26% ended informatively.
The foundation has launched a program called “Design, Analyze, Communicate” (DAC) to address this problem by training researchers to better design and conduct studies to improve the quality and usefulness of the data generated.
- The Gates Foundation’s Integrated Development Team created the Design, Analyze, and Communicate (DAC) program in 2020 to help grantees optimize their clinical trials for informativeness.
- The program provides feedback on draft study protocols using 16 best practices, and can also provide funding to assist with technical expertise and tools.
- The program has collaborated with several trials, including the DeWorm3 trial, which was able to save money by adjusting its strategy for analyzing samples and also added additional research to help inform the implementation of deworming programs in the future.
- The ultimate goal of the program is to improve the quality and usefulness of data generated by clinical trials to inform policy and practice.
DEEP DIVE
“Uninformative research” is the global health crisis you’ve never heard of
Bill and Melinda Gates Foundation
Jan 11, 2023
Historians like to fight about whether Francis Bacon or Galileo Galilei invented the scientific method.
Either way, we’ve been relying on it for going on 500 years.
Here’s how it works: You
- 1.Observe the world around you,
- 2.make a hypothesis based on your observations,
- 3.run an experiment to test your hypothesis,
- 4.get a result, and
- 5.repeat the process until you can explain what you observed in step 1.
The global health community relies on the scientific method, in the form of clinical trials, to continually improve health care.
As of 2022, at least 329,830 trials were in progress, seeking to learn whether a new drug, vaccine, device, procedure, or behavior is better than the alternatives.
More than a million researchers are devoting their time and energy to these studies. Many millions of people are participating as subjects. All in the pursuit of knowledge that, we hope, will save lives.
As of 2022, at least 329,830 trials were in progress, seeking to learn whether a new drug, vaccine, device, procedure, or behavior is better than the alternatives.
More than a million researchers are devoting their time and energy to these studies.
Many millions of people are participating as subjects. All in the pursuit of knowledge that, we hope, will save lives.
But there’s a major problem. Most clinical trials neither prove nor disprove the hypothesis.
It’s not that the intervention being tested failed. It’s that we don’t know whether it failed or succeeded.
These studies are what some scholars call “uninformative.” In lay terms, it’s almost like they never happened.
Except that they collectively required many billions of dollars and millions of hours, two resources the global health sector doesn’t have nearly enough of.
According to a 2021 study of thousands of COVID-19 drug trials, only 5% of the “study arms” included enough people to generate a statistically valid result.
Chaos is to be expected during a crisis like a pandemic, but the norm during “peacetime” is nearly as worrisome.
A recent study of U.S. clinical trials related to heart disease, diabetes, and lung cancer found that only 26% ended informatively.
To put it bluntly: Three out of four were a waste of time and money.
To put it bluntly: Three out of four [U.S. clinical trials] were a waste of time and money.
How is it possible that the vast majority of these trials don’t answer the question they set out to investigate?
One reason that some trials yield no useful findings is that the people who run them, while experts in their fields, aren’t necessarily experts at designing trials, analyzing the results, and communicating the findings.
The Gates Foundation has launched a program called Design, Analyze, Communicate (DAC) to understand this challenge and begin to address it.
But before we delve into DAC and its work, one piece of context is critical: Clinical trials can be exceedingly complex.
Consider an ongoing clinical trial called DeWorm3 that our foundation supports with grant funding.
It asks a simple question: Is it possible to eliminate the transmission of parasitic worms (called soil-transmitted helminths)?
The question may be simple, but answering it in an informative way is anything but, so DeWorm3 was designed to run a gauntlet of potential challenges.
The scientists working to protect kids from parasites
Dr. Sitara Swarna Rao Ajjampur has built her career by subverting other people’s expectations.
“In India, when you get into medical school,” she said, “It’s almost impossible to tell someone that you want to do research. You’re expected to be happy just being a doctor.”
But Dr. Ajjampur wasn’t happy, because she was “bitten by the research bug” during a summer fellowship as a medical student.
When she chose research following her M.D., “virology was considered the cool science.” But, says Dr. Ajjampur, “I was fascinated by parasites.”
After her Ph.D. and research fellowship, Dr. Ajjampur, now at Christian Medical College (CMC) in Vellore, India, became the principal investigator of the Indian site of the DeWorm3 trial.
The origins of DeWorm3 lie in public health policies to prevent infection by intestinal parasites called soil-transmitted helminths.
In India and dozens of other countries, school-aged children are given the drug albendazole twice a year.
This relatively recent effort has successfully helped communities get control of hookworms, roundworms, and other worms.
This is a public health breakthrough. Before school-based deworming started, hundreds of millions of children were infected by worms that stunted their neurological and physiological development.
When they received their first doses of albendazole, donated by the pharmaceutical company GlaxoSmithKline, many children shed worms not only in their stool but also through their mouths and noses.
The days of widespread infection by intestinal parasites are over, but the worms haven’t disappeared entirely.
Many countries, including India, have to keep administering albendazole year after year after year to keep them at bay. It’s expensive, it’s labor intensive, and it can be a logistical nightmare.
So experts in the field of neglected tropical diseases started to ask, What if it were possible to eliminate worms altogether and stop administering the drugs?
This would give back two lost school days each year, redirect the work of millions of frontline health workers, repurpose manufacturing capacity and supply chains for other lifesaving medicines, and otherwise free up limited resources.
Through mathematical modeling, researchers developed a promising hypothesis: elimination could be accomplished by giving albendazole not just to children at school but also to all adults in the community.
DeWorm3, led by Dr. Judd Walson and his team at the University of Washington (UW), is now testing that hypothesis at three study sites: Benin (led by Dr. Moudachirou Ibikounlé and Dr. Adrian Luty), Malawi (led by Dr. Khumbo Kalua and Dr. Robin Bailey), and India (led by Dr. Sitara Swarna Rao Ajjampur).
How to design a rigorous clinical trial
In India, the DeWorm3 team is trying to show a statistically significant change from “very low” incidence of parasites to “eliminated.”
A substantial sample size — the number of people included in the study — increases the odds that they’ll obtain clear results.
Inadequate sample size is the leading reason that so many studies end uninformatively.
Estimating the required sample sizes is not straightforward. Furthermore, recruiting study participants costs a lot of money and takes a long time.
As a result of these two factors, many researchers underestimate — and end up with “underpowered” and therefore uninformative studies.
The 140,000 participants in Dr. Ajjampur’s part of the study are spread across 477 square kilometers in the state of Tamil Nadu and are divided into 20 clusters.
Before her team could give a single person a deworming tablet, they had to take a detailed census of the whole area — that is, precisely who lives there and precisely where they live.
Once they enumerated the people they intended to treat, they needed to make sure those people would want to be treated.
Although adults in Tamil Nadu are used to children getting dewormed at school, the idea of getting dewormed themselves is new.
No one will just swallow a tablet if someone shows up at their door and says it’s for the good of science — and they shouldn’t.
So, again, before Dr. Ajjampur’s team could give a single person a deworming tablet, they had to conduct a vast awareness-raising and education campaign, in all local languages.
They even wrote their own jingle and played it from auto rickshaws driving through the villages.
“We were very prepared to answer any questions that they had,” says Dr. Ajjampur. “It wasn’t just a banner and giving them some information and then leaving. It was very, very interactive.”
Finally, they launched the intervention:
They gave albendazole to the trial participants — children in schools, adults in homes — twice a year for three years. After each round, the team surveyed the community.
Running any one of the three sites in the study is the rough equivalent of running a mid-size company.
Dr. Ajjampur’s team consists of 75 to 125 field workers, plus about 25 coordinators, managers, supervisors, and laboratory staff.
Working in close coordination with the Indian government (the Ministry of Health and Family Welfare and the Indian Council of Medical Research, New Delhi) and the state of Tamil Nadu (the Directorate of Public Health and Preventive Medicine), the India team completed all the deworming rounds — even finding ways to operate safely through the COVID-19 pandemic, which threatened to upend their plans.
After three years of deworming and two years of waiting, the next step is to find out whether any worms remain in the community. That means testing 40,000 stool samples for soil-transmitted helminths. And therein lies another challenge.
When a sample is full of worms, a lab technician can just slide it under a microscope and start counting.
But the whole point of DeWorm3 is getting to a very tiny number of worms. Microscopy isn’t sensitive enough for that work. A PCR test is.
But, just as there was no PCR test for COVID-19 when the pandemic began, there was no automated, high-throughput PCR test for detecting intestinal parasites when the trial began.
So the researchers at UW and Dr. Ajjampur’s team at CMC worked at a furious pace with partners at Smith College and the biotech company Quantigen to develop a high-throughput molecular diagnostic test in time to test their large number of samples.
Those samples are being tested now, with the results expected late in 2023.
The clinical trial advisory service
Since its founding in 2000, the Gates Foundation has funded grantees to support thousands of clinical trials, many of which have hit some kind of obstacle and ended uninformatively.
In Malawi, for example, the foundation funded grantees running a trial to discover whether eating one egg a day would improve the health of malnourished children. (A previous trial in Ecuador had shown that it would.)
However, none of the people associated with the trial adequately considered the ramifications of doing the study next to a lake. The children there ate a steady diet of fish, so they already consumed adequate protein. These were not the specific children who should have been studied.
To help grantees ensure that their research studies are as informative as possible, the foundation’s Integrated Development Team created the Design, Analyze, and Communicate (DAC) program in 2020 specifically for that purpose.
DAC advisors are what most principal investigators and their teams are not: trained and experienced in optimizing different aspects of clinical trials for informativeness.
Grantees that choose to work with DAC submit a draft of their study protocol — the roadmap that researchers produce before they embark on the trial — to the DAC team for feedback.
They receive a list of recommendations based on 16 best practices for informativeness, and they have an opportunity to adjust the protocol before finalizing it.
The recommendations can be highly technical. Not everyone is up to date on the Kato-Katz testing methodology, for example, and some grantees don’t have the cutting-edge expertise or tools to implement DAC’s ideas.
In those cases, DAC can make grants to help the researchers get technical assistance to fill in the gaps, run the trial according to best practices, and be better prepared for future studies.
Back in 2020, when DAC was just getting started, one of its earliest collaborations was with the DeWorm3 team.
The DeWorm3 trial was already past its midpoint, so the University of Washington team was not looking to implement radical changes.
But because the trial is big and expensive, everyone agreed that it was worth having the DAC team take a look to see if any adjustments could make the results even more useful.
“It can be a difficult process to subject your work to a critical review,” says Dr. Walson. “But we all have the same goal, and together we can drive maximum impact.”
The DAC team concluded that the DeWorm3 trial was well designed, but they pinpointed a couple of areas where relatively small changes could make a big difference.
First, because of the complexities described earlier, the trial was extremely expensive, and the DAC team suggested saving money by adjusting the strategy for analyzing the tens of thousands of samples collected during the study.
The UW team decided to test the endline samples (after two years) but not the midline samples (after one year). The midline results are not needed to determine whether transmission has been interrupted, and testing the midline samples would cost US$3 million. The researchers have the option to test the midline samples later, however, depending on the results of the study.
Second, the DAC team suggested conducting additional research to inform the implementation of deworming programs.
The ultimate goal of the study is not to answer an academic question, but rather to inform policy and practice.
What policymakers want to know is not just whether elimination is theoretically possible, but what the real-life considerations are, such as how much it would cost to implement community-wide deworming or how to keep deworming coverage high when regular health workers, and not a dedicated team, are doing deworming as part of their job.
Typically, answering these questions requires an entirely new trial, which means that it can take 20 years or more to go from someone’s idea to actually saving lives.
“But we all have the same goal, and together we can drive maximum impact.” — Dr. Judd Walson
UW is a pioneer in the emerging field of implementation studies — studies that answer these cost and delivery questions — and the trial was already focusing on implementation to an unusual degree.
DeWorm3 was designed as what is known as a hybrid trial: It tested implementation strategies at the same time as the efficacy of the intervention.
Based on the DAC team’s feedback, and with the support of a small grant, the DeWorm3 team interviewed experts in the field about what it would take for global and national guidelines and policies to be changed.
According to Dr. Arianna Means, the UW team’s implementation lead, the researchers used a tool created by DAC called a Target Policy Profile to survey and help build agreement among academics, nongovernmental organizations, and government offices in the study countries about the evidence they would need to “feel comfortable…making a big, expensive change.”
Turning clinical trials into progress for people
It’s easy to imagine research and development as a series of eureka moments. Archimedes in his bathtub.
Tu Youyou discovering artemisinin, an anti-malarial drug that has saved millions of lives, based on her knowledge of traditional Chinese medicine. These flashes of discovery do happen from time to time.
But usually, it’s the slow and steady accumulation of knowledge that leads to the gradual evolution of health policy and practice.
Such is the slow, steady work of global health.
Let’s assume for a moment that the DeWorm3 results will show that elimination is achievable.
Then the World Health Organization (WHO) will consider whether to change the global guidelines on deworming.
Separately, countries with deworming programs will consider whether to change their national strategies.
If WHO changes its guidelines, that doesn’t mean countries will change their strategies.
If WHO doesn’t change its guidelines, that doesn’t mean countries won’t change their strategies. These actions tend to align, but it’s not automatic.
The hope is that countries that adopt community-wide deworming will eventually eliminate all infections, stop deworming altogether, and transfer those resources to meet other urgent needs — perhaps as soon as a decade from now.
In the meantime, researchers will continue running hundreds of thousands of clinical trials as they slog their way toward better health for everyone in the world.
Every uninformative trial makes these slow processes that much slower.
Money is wasted.
People die waiting for answers.
That’s why we need researchers like the hundreds of people across three continents working on DeWorm3.
Their work may not win a Nobel Prize, but it can save lives.
DAC and other programs like it can help maximize their impact. Such is the slow, steady work of global health.
Originally published at https://www.gatesfoundation.org.
