Mental health: developing predictive modeling tools to inform clinical decisions and improve outcomes

At the recent DTx East conference in Boston, Dr. Derek Richards, Chief Science Officer at SilverCloud Health (SCH), accompanied by SCH Senior Digital Health Scientists, Dr. Jorge Palacios and Dr. Angel Enrique, together with Microsoft Research (MSR) Senior Researcher, Dr. Anja Thieme, delivered a keynote on the topic of next generation of digital therapeutics for mental health.

As Dr. Richards pointed out when multidisciplinary teams come together, “magic can happen”, and Project Talia is a perfect example of that: a patient-centric, multidisciplinary collaboration between SCH and MSR, bringing together domain-specific expertise in machine learning, digital mental health, and clinical intervention to explore how we can leverage machine learning to improve the effectiveness of mental health services. There were several key points discussed that will drive digital solutions in the future.

1. We can leverage big data and develop machine learning models to help us better understand engagement and how to personalise the user experience.

At a time when digital interventions are a growing area of interest, a recent review of literature reviews including 166 individual studies suggested that these interventions are effective and cost-effective in most populations. However, as Dr. Enrique pointed out, most often these interventions are treated as a black box, and there is a need to explore how these interventions work with a higher level of granularity. With big data, we now have the ability to understand not only how much users engage with these interventions (e.g., looking at metrics such as time spent online or number of logins), but also what content is being accessed, how much users engage with the tools, and what is the temporality of that engagement (order and time of usage).

Dr. Enrique presented several studies that the SCH research team has been involved in recently in an attempt to better understand how these interventions work:

• In a randomized controlled trial (RCT), 216 users were divided into two categories based on whether they improved or not at the end of therapy. Overall, the key finding was that more usage was associated with better outcomes, especially in the first half of the intervention.
  
  
• The collaboration with MSR has allowed SCH to further explore this topic by applying machine learning to identify behavioral patterns based on how users engage with the SCH platform. For this study, machine learning (ML) models were applied on a naturalistic, de-identified dataset of more than 54,000 users and the results suggested there are 5 different patterns of engagement (low engagers, late engagers, high engagers with rapid disengagement, high engagers with moderate decrease, and highest engagers). 
  
  
• A separate study on this topic aimed to understand how different support strategies correlate with clinical outcomes. For this purpose, 234,735 supporter messages were analysed, and the key finding was that concrete, positive, and supportive feedback was strongly associated with better outcomes. Moreover, messages containing more positive and less abstract words, related to social behaviors, were related to higher client outcomes. 

2. Applying these machine learning models in research and understanding how to take this newly developed prediction tool from an idea to a large-scale trial, to real-world development

The teams at SCH and MSR focused on using machine learning for outcome prediction based on measures for depression and anxiety (PHQ-9 and GAD-7). Dr. Thieme explained how the prediction model was developed using data from 46,000 SCH users. Information from these individuals, specifically their trajectories of symptoms over time, was used to train the algorithm, which then could be used on new users to predict whether they will achieve reliable improvement at the end of treatment. There were in fact two algorithms developed, one for depression (based on the PHQ-9 scores) and one for anxiety (based on the GAD-7 scores), both with high levels of accuracy.

Having created these two robust prediction models, a series of interviews with digital behavioral health coaches was conducted to better understand how these predictions could be useful and what challenges they may introduce. Several design mock-ups were created to illustrate how the predictions could be displayed and explained to the coaches. Based on the feedback received, the SCH team created a simple design, with more content/details being available on demand. During interviews, coaches also identified a few areas of concern regarding the prediction tool that could be targeted during training in the future. These included the concern of over-relying on the prediction due to time constraints for the coaches to properly ponder over it, a feeling of losing the humane aspect and making sure these predictions don’t replace the coach nor affect the interaction with users, and the importance of training and the need to properly explain to the coaches that these models are based on probabilities, therefore they can be wrong and should be interpreted with caution/responsibly.

Since at SCH “we believe in scientific excellence and transparent dissemination of our research” as Dr. Palacios mentioned, the intention is to formally test this prediction tool in an upcoming RCT. To inform the RCT, besides the feedback from the coaches, the SCH research team also ran a small pilot study during which 6 coaches in the US were interviewed on 3 separate occasions on their use of the prediction tool. Overall, coaches reported a high level of trust in the accuracy of the prediction, and they mentioned that the positive predictions reaffirmed the clinical support they were giving, whereas the negative predictions led to increased time spent on the reviews to help users get back on track. The pilot study also indicated the prediction was more likely to help novice coaches whereas experienced coaches had routines that made it harder to adapt to new tools/features.

In the upcoming RCT, the SCH team has built on the knowledge gained from all these prior studies and will focus on three key concepts: feedback-informed psychotherapy (FIT) (i.e. providing feedback to therapists to support clinical decision making/coaching decisions), deliberate practice (i.e., activities meant to improve therapeutic skills), and performance of the model. The hypothesis is that this model, which delivers FIT, will lead to increased levels of deliberate practice of coaches who in turn will deliver better outcomes. The performance of the model will also be validated in a new, larger setting. All coaches will be recruited from a large UK provider, will be split into novices and experienced, and randomized to control and intervention (i.e., access to use the prediction tool). They will be asked to complete questionnaires assessing the usefulness of the prediction tool, as well as the deliberate practice and overall experience using the tool.

The RCT will serve to validate how a unique collaboration between SCH and MSR led to the implementation of a first-of-its-kind digital machine learning model for FIT and evidence-base generation in a digital mental health intervention within a real-world setting. In terms of clinical contributions, the expectation is that it will lead to better outcomes due to clinicians making more informed decisions and perhaps prioritizing cases that need more attention or tailoring treatment to better fit each client’s needs. In terms of scientific contributions, we are leveraging machine learning to employ FIT and enhance digital health delivery and effectiveness.