Lack of clinical evidence ‘major gap’ in digital health: study

Nick Paul Taylor
June 22, 2022

Edited by

Joaquim Cardoso MSc.
The Health Foundation

Digital Health Institute
June 22, 2022

Dive Brief:

  • Most digital health companies have a low level of “clinical robustness” as measured by their number of regulatory filings and clinical trials, according to a paper published in the Journal of Medical Internet Research.

  • Researchers at Rock Health and Johns Hopkins University analyzed the activity of 224 companies that have collectively raised $8.2 billion since 2011 to determine their clinical rigor. 

  • The median score on the robustness scale was one, reflecting the fact that almost half of the companies had no regulatory filings or clinical trials to their names.

  • The researchers framed the low clinical robustness scores as evidence of “a major gap in health care technology,” adding that there is a “significant opportunity” for companies that differentiate themselves through a more rigorous approach.

Dive Insight:

Funding for U.S. digital health companies surged last year despite studies finding little clinical validation and a lack of robust evidence that digital health solutions are effective, according to Rock Health, a seed fund that tracks the space. 

Rock Health noted that digital health firms raised more than $29 billion in 2021, up from $15 billion in 2020 and $8 billion in 2019, though funding levels started to decline at the beginning of 2022.

To understand the clinical rigor of digital health companies, the researchers looked at 224 companies in the Rock Health database of businesses. 

The companies had an average age of 7.7 years. Around half of the companies offered treatments, with a similar proportion having diagnostic solutions. A smaller group of 25 companies offered solutions for prevention.

The researchers calculated the clinical robustness of the companies by adding up the market applications they have made to FDA and the clinical trials they have either sponsored or collaborated on. 

The average clinical robustness score was 2.5. That figure was supported by the 1.8 clinical trials the companies have worked on, on average, and dragged up by a small band of businesses that scored highly.

One-fifth of the companies had a clinical robustness score of five or higher. Six percent scored 10 or higher. Yet, the largest group of companies, 44%, had no regulatory filings or clinical trials. The presence of a large group of companies with a score of zero explains the fact that the median score was one.

Another part of the study looked at the claims, defined as unique quantitative statements about product outcomes, that the companies made on their websites. Again, the researchers found little activity. The average number of claims was 1.3, reflecting an almost even split of clinical, economic and engagement claims.

The researchers found no correlation between clinical robustness and the number of clinical claims. 

Thirty two companies made one or more clinical claims despite having a clinical robustness score of zero. Similarly, the study found no correlation between clinical robustness and total funding. That lack of a link between funding and robustness drew the attention of the researchers.

“The lack of overall correlation between a company’s total venture funding and its clinical robustness score similarly highlighted a significant asymmetry in how companies are potentially valued in today’s marketplace (ie, no correlation between clinical impact and funding). 

However, it is possible that funding amounts reflect future anticipated value rather than current value,” the authors wrote.


Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis

Sean Day 1, 
Veeraj Shah 1, 2 
Sari Kaganoff 1,
Shannon Powelson 3,
Simon C Mathews 4 



The digital health sector has experienced rapid growth over the past decade. However, health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry.


This analysis aimed to examine the clinical robustness and public claims made by digital health companies.


A cross-sectional observational analysis was conducted using company data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. Companies were included if they sell products targeting the prevention, diagnosis, or treatment phases of the care continuum. Clinical robustness was defined using regulatory filings and clinical trials completed by each company. Public claims data included clinical, economic, and engagement claims regarding product outcomes made by each company on its website.


  • A total of 224 digital health companies with an average age of 7.7 years were included in our cohort. 

  • Average clinical robustness was 2.5 (1.8 clinical trials and 0.8 regulatory filings) with a median score of 1. 

  • Ninety-eight (44%) companies had a clinical robustness score of 0, while 45 (20%) companies had a clinical robustness score of 5 or more. 

  • The average number of public claims was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement); the median number of claims was 1. 

  • No correlation was observed between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18).


  • Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online. 

  • Companies and customers may benefit from investing in greater clinical validation efforts.

Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online.

Companies and customers may benefit from investing in greater clinical validation efforts.

J Med Internet Res 2022;24(6):e37677


Authors Affiliations

Sean Day 1, BS, MPhil; Veeraj Shah 1, 2, BS; Sari Kaganoff 1, MBA; Shannon Powelson 3, BS; Simon C Mathews 4, MD

1 Rock Health Inc , San Francisco, CA, US

2 Department of Public Health and Primary Care, University of Cambridge , Cambridge , GB

3 Whiting School of Engineering, Johns Hopkins University , Baltimore, MD, US

4 Division of Gastroenterology & Hepatology, Johns Hopkins University School of Medicine , Baltimore, MD, US

Original publication

Deixe um comentário

O seu endereço de e-mail não será publicado.

Related Posts