Triangulating Success Outcomes for Online Therapy Platforms

Extensive research has been published on the conversational factors of effective volunteer peer counseling on mental health platforms. These studies have focused on predicting counseling expertise using mostly singular outcomes, but little work has been done on examining multiple success outcomes for platform design. To address concerns of validity with reported outcomes and predictors of counseling expertise, we document a process for triangulating among multiple outcome measures available on a single therapy platform to select four outcome metrics: retention in the community, following up in a conversation, ratings for counselors, and user mood scores. Then, we model the relationship between previously reported linguistic predictors of effective counseling with these outcomes. Our findings reveal that predictor variables relate to outcomes differently based on nuances in measurement construct and operationalization details. We suggest actionable insights for therapy platform design based on triangulation of multiple outcomes.

People in need of mental health support have reported benefits from interacting with peers online through online mental health platforms (OMHPs). These platforms have been growing in popularity, are accessible and cost-effective, reduce stigma about mental health by building anonymous connections among individuals, empower the sharing of individual journey, and enable individuals in times of need to find advice and support for their problems. Prior research has studied the impact of peer counseling on a variety of metrics such as satisfaction, mental health outcomes, community participation, and language, providing valuable insights on effective support strategies for those in need using social platforms.

Diverse outcomes offer an opportunity for platforms to better track the impact of peer counseling online, but prior work has tended to use singular outcomes reflecting theory-driven approaches towards finding the causes of effective peer counseling rather than examining the value of these metrics. Examining a single outcome may lead to non-robust and non-generalizable findings. Moreover, factors related to peer counseling could correlate with multiple success outcomes in inconsistent or conflicting patterns, limiting potential applications in the design of OMHPs. Some prior studies have used multiple outcomes in a composite outcome measure, but they did not study the relationship between individual outcomes and their associations with behavioral factors from either peer counselors or people who have mental health concerns.

Triangulation of key success outcomes is valuable for online platforms because multiple outcomes provide a comprehensive overview of user engagement, and has been leveraged in digital user experience, human factors, medical informatics, and healthcare quality for designing and evaluating products. In the related area of predicting mental health states from social media data, Ernala et al. Previous work has suggested triangulation of diagnostic signals for predictive models to remedy issues in the validity and contextualization of predictor variables. However, to the best of our knowledge, no studies have paid attention to triangulating outcomes in studying peer-counseling on OMHPs despite the large body of research in connecting success outcomes with users' behaviors in OMHPs.

Given the lack of comparison between various proposed outcomes for peer counseling, we hypothesize that a multilevel view of platform metrics can illuminate tensions among outcomes and predictors. Our investigation is centered on two questions:

• RQ1: Does triangulating across multiple outcomes provide novel insights for finding counseling success indicators?
• RQ2: Do widely used predictor variables of counseling success track multiple outcome metrics consistently?

To answer these questions, we first review prior research and note ambiguities in the operationalization of variables across the literature. Next, we document our process for organizing outcomes to identify four signals of peer counseling success for therapy-focused platforms and compare these signals across widely used predictor variables of counseling success. Using regression modeling of a large dataset of one-to-one chats between support seekers and support providers, we show that triangulation across multiple levels of outcomes enables fine-grained discussion of design goals for OMHPs. For brevity, we refer to online peer support seekers as seekers and online peer support providers as providers.

Contribution

In this project, we make an original contribution by providing a multilevel view of different success outcomes and how they correlate with widely used predictors for counseling. Concretely, we leverage a dataset of 1.7M chat sessions between support seekers and support providers to conduct large-scale statistical analysis of peer counseling on 7 Cups of Tea, an online therapy platform. We validate and extend prior work by investigating the relationship between a large set of widely-used linguistic predictors of effective counseling and multiple outcome metrics to examine validity of different reported metrics at scale. We discuss the consistency and differences when multiple outcomes are used, which provide novel insights for therapy platform design and generalization of social computing research on peer counseling.