How to Critically Evaluate the Evidence Base of your Digital Health Solution

Evidence-based Practice Matters_Blog – 1Recent digital innovations in mental health care present an enormous opportunity to mitigate the global burden of mental health disorders by enabling timely, accessible, scalable, and affordable interventions. Early evidence showed that digitized versions of evidence-based psychological treatments were as effective as face-to-face delivery of these same interventions1, and the balance of existing evidence over the past decade continues to substantiate this2

To learn more about how to critically evaluate your digital health solution click below. 

Download our Information Sheet

Indeed, many people report a preference for internet-delivered over face-to-face interventions, appreciating the convenience, ease of access, and greater relative anonymity, that they provide3. However, despite the irrefutable promise of technology-enabled mental healthcare, there are also several challenges facing the digital mental health industry, particularly in relation to false and misleading claims, which not only seriously jeopardize the reputation and future of the industry, but also expose vulnerable end-users in need of the mental health care to risks.

A dramatic surge in digital mental health solutions

The glaring demand-supply discrepancy in mental health care has been widely recognized as an area that is ripe for innovation. And, consistent with this, venture capital companies have in turn been fervently pursuing digital mental health and wellness-oriented technologies as investment opportunities. A record-breaking $5.1 billion has been invested in mental health start-ups in 2021 by the venture capital industry, which constituted a fourfold increase over 2019, and a tenfold increase over 20174.

However, these phenomena have also simultaneously cultivated an unprecedented landscape where there is a striking clash between the ideologies of the venture-backed industry and the need for the responsible development of safe and robust evidence-based mental health interventions. In Silicon Valley, for instance, tech entrepreneurs are encouraged to “fail fast, fail often,” regardless of risk, which promotes minimal viable product launches and aggressive marketing without an investment of time and resources in rigorous validation. This stands in sharp contrast with the risk-averse ethos of clinical medicine, which is centered on the ethical care of the patient, and advocates a cautious, conscientious, and tested approach. The core tenet of this approach, “first, do no harm,” is invariably disregarded in the prevailing venture-backed entrepreneurial landscape, where companies are formed over very short time periods, unvalidated claims are made, and products are sold.

Critically evaluate a Digital Mental Health Intervention’s claims

It is concerning that so few of the commercially available digital mental health interventions (DMHIs) are supported with published evidence. However, even when published research does exist it is often of questionable quality5,6. Hence, we encourage stakeholders to empower themselves with an understanding of how to critically evaluate the reliability and validity of published research, as well as the veracity of “claims” pertaining to DMHIs. When reviewing evidence and claims relating to a DMHI it’s important to ask the following five questions:

  1. Is the product truly evidence-based or merely “evidence-informed”?

  2. What type of research trial was employed?

  3. Were the trial participants appropriately representative?

  4. Has the research been published in a peer-reviewed journal?

  5. Has there been an appropriate level of transparency about how outcome variables are defined?

How is SilverCloud different?

At SilverCloud Health we are of the firm opinion that the capacity to harness the full potential of technology-enabled mental health care is critically contingent on building and sustaining a strong legacy of rigorous research and scientific inquiry. This means that innovations must continue to be evaluated in a variety of contexts using a range of designs to ensure that they are not only effective in treating the symptoms and disorders which they target, but are also feasible and acceptable to our end-users, and are ultimately structured to have the best possible balance of increasing access, maximizing clinical outcomes, and minimizing costs. This is not an easy endeavor to prioritize, as the DMHI landscape is changing at lightning speed, and significant time, effort, and resources are required to conduct this research. But, ultimately as demand and competition continue to accelerate, we can be confident that our commitment to scientific rigor and clinically proven outcomes will have a lasting effect on how digital behavioral health interventions are implemented and used to treat and improve the quality of life of people throughout the world.

Interested in learning more about how to critically evaluate the evidence base of your digital health solution? Download our entire information sheet – Five questions to ask when reviewing the evidence and claims of digital mental health companies.

Download our Information Sheet

About the authors

Siobhan Harty

 

Dr. Siobhán Harty is a Digital Health Scientist on the team, with both a primary degree and PhD in Psychology, and a number of years experience as a postdoctoral researcher at the Department of Experimental Psychology, University of Oxford and Trinity College Dublin. She has a particularly keen interest in experimental design and inter-individual differences in responsiveness to interventions.

 

 

Jorge Palacios-1

 

 

Dr. Jorge Palacios is a Senior Digital Health Scientist and leads our research strategy and dissemination, working both internally and with external partners to ensure the maximum impact of our evidence-based science. Originally from Mexico City, he now resides in London and holds a PhD in Psychological Medicine from the Institute of Psychiatry, Psychology & Neuroscience (IoPPN) at Kings College London. 

 

 

 

References

1. Ström, M., Uckelstam, C. J., Andersson, G., Hassmén, P., Umefjord, G., & Carlbring, P. (2013). Internet-delivered therapist-guided physical activity for mild to moderate depression: a randomized controlled trial. PeerJ, 1, e178.

2. Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., ... & Sander, L. B. (2021). Digital interventions for the treatment of depression: A meta-analytic review. Psychological bulletin, 147(8), 749.

3. Boydell, K. M., Hodgins, M., Pignatiello, A., Teshima, J., Edwards, H., & Willis, D. (2014). Using technology to deliver mental health services to children and youth: a scoping review. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 23(2), 87.

4. Krasniansky. A. (2022). Q1 2022 digital health funding: Staying the course in choppy waters. Rock Health. https://rockhealth.com/insights/q1-2022-digital-health-funding-staying-the-course-in-choppy-waters/ 

5. Byambasuren, O., Sanders, S., Beller, E., & Glasziou, P. (2018). Prescribable mHealth apps identified from an overview of systematic reviews. NPJ Digital Medicine, 1, 12.

6. Morley, J., Powell, J., & Floridi, L. (2021). What’s the Evidence that Health Apps Work? A Scoping Study of Evidence of Effectiveness for Direct-to-Consumer Apps on the App Store. A Scoping Study of Evidence of Effectiveness for Direct-to-Consumer Apps on the App Store. Available at: http://dx.doi.org/10.2139/ssrn.3851242