A systematic review of engagement reporting in remote measurement studies for health symptom tracking

Characteristics of the included papers are described in Supplementary Table 1. The most common conditions of study were bipolar disorder (n = 23), depression and/or anxiety disorder (n = 11), chronic pain (n = 8), cancer (n = 5), and psychosis (n = 4). Forty one papers (54%) reported on RMT use in a research context, 22 (29%) on RMTs for self-management and 13 (17%) on the use of RMTs for clinical support. aRMT only was used in 45 (59%) papers, where 31 (45%) papers used a combination of aRMT and pRMT methods. Symptoms were tracked for a median of 56 days (range = 794 days). Common symptoms tracked included mood, pain, sleep, fatigue, exercise, and daily activity levels.

Of the 76 papers included, a large majority (n = 74, 97%) reported on engagement in some form. The two papers that did not report on engagement^30,31 are secondary analyses and cite previous feasibility papers. Of the 74 papers that did report on engagement, 48 (65%) did not include a definition of engagement. Twenty-six (35%) papers included both a definition and corresponding measurement of engagement. All but one of the papers that included a definition and measurement noted engagement as a main aim of the paper; Greer et al.³² aimed to test the effect of a smartphone app, of which symptom monitoring was one component, on adherence to cancer treatment. Of the 30 papers (39.5%) that noted engagement as a main aim, 5 (17%) did not include a definition of engagement.

Papers used a range of, often interchangeable, definitions for engagement. The most cited term was ‘feasibility’. For many studies, feasibility was conceptualised as objective compliance with, or adherence to, symptom monitoring^33–37, such that higher completion rates signalled greater feasibility. Depending on the context of RMT use, feasibility either represented (i) the ability of users to use an app to complete the monitoring, or (ii) the ability of RMTs to collect sufficient data for assessment of the cohort in question. Terms such as ‘usability’, ‘acceptability’ and ‘satisfaction’ were often used in addition to feasibility. These tended to be more subjective concepts, often paired with rating scales and questionnaires to understand how far users felt the technologies were appropriate for use. Two studies further understood engagement as ‘perceived utility’ or ‘usefulness’ of the RMT for disease management^38,39. Other studies defined app usage data as a proxy for engagement. Selter et al.³⁸ sought to understand ‘patient engagement’ on the basis of the frequency and quality of interaction (active vs. passive) with components of the self-monitoring system, and⁴⁰ identified patient preference for RMT type by usage data. One paper defined engagement as “participant willingness to engage in mobile research protocols”⁴¹ (pg. 14). Overall, a large proportion of studies used multiple conceptualisations of, and terms for, engagement in the same paper, often with little justification for why they had been chosen or how they complemented each other.

What follows comprises a narrative synthesis of each synthetic construct for the measurement of engagement across the included papers.

A large majority of papers (88%, n = 65) reported on engagement in reference to recruitment, retention and withdrawal from RMT studies. Where generic reasons for declining research participation, such as lack of time and/or lack of interest, were widely reported, some studies also reported on RMT-specific reasons: unwilling to use/switch to trial smartphone^32,42–45, concerns about remote data collection^42,45,46, having ‘no need’ to monitor symptoms alongside treatment⁴⁷. One study⁴⁸ calculated the percentage of participants who declined to participate explicitly based on the RMT nature of the research, using data on previous non-response to clinic and research opportunities.

Reasons for participant dropout followed a similar vein. Technological issues, for example an inability to download the study app⁴⁹, phone updates³³ and software malfunctions with study apps^33,36,37, were the most reported reasons for non-completion of a study. However, many papers did not differentiate these from non-RMT related reasons for dropout, e.g., death or hospitalisation, or failed to provide reasons at all. Seven studies^{2,35,41,49–52} investigated whether dropout was associated with sociodemographic or clinical variables, and one further explored the association between dropout and daily access to data, as a main aim of the study⁵⁰. Two studies reported on likelihood of dropout over time, finding opposing results^35,53. Understanding the factors associated with decline and dropout over time provided insights into interest in RMTs for support alongside clinical care^47,48.

There was little consensus over what constituted researcher-initiated withdrawal from the study. Again, there was a lack of distinction made between RMT and non-RMT related factors. Some studies reported on the percentage of participants completing the full number of days of remote monitoring^34,54,55, where others focused on completion of a sufficient, often arbitrary, number of daily assessments. Participants were withdrawn or excluded from analyses if they did not submit any daily assessments^33,37,56 or never used the app³⁶. Two papers^51,57 defined withdrawal based on failure to complete a minimum amount of ESM assessments during the study period: 33% and 30% respectively. A failure to attend appointments for additional, in-person questionnaires was also a reason for withdrawal^36,43,58. Retention rates were largely discussed in terms of the resulting impact on the final dataset for analyses. Overall, a lack of standardisation in the measurement of engagement with the research protocol, coupled with the merging of technological and non-technological factors, suggests that attempts to provide accurate information on dropout and retention rates in RMT studies would be difficult.

Alongside engagement with the research protocol, studies also reported engagement with the RMTs themselves. 87% (n = 64) of papers conceptualised engagement with RMTs as an objective measure, e.g., direct behaviour whilst using the device. First, studies reported on the total, raw number of assessments sent and responded to, or the number of symptoms tracked, during the study period in the sample as a whole. This was further broken down into ‘per participant’ measures: the mean or median average number of daily aRMT assessments completed across the study period^{32,33,49,53,54,59,60}, or the average number of assessments completed within each day^61–63, depending on the frequency of the aRMT tasks. Some studies also reported the number of days in which participants provided self-monitoring data^37,46,53,54, though at times it was unclear whether this referenced only the days where data was recorded, or the total number of days during which the participant remained in the study.

Second, symptom tracking compliance rates were reported. There was little standardisation in how these were measured. ‘Compliance rate’ was generally conceptualised as the percentage of aRMT assessments completed over the total available to be completed and was used interchangeably with the terms ‘response rate’, ‘adherence’, and ‘completion’. Many studies simply reported the average percentage compliance, often with accompanying measures of variance, across the sample for aRMT assessments and associated clinical measures. Some instead reported on the percentage of days in which self-monitoring was adhered to^64–66 (assuming that self-monitoring was assessed daily). Some also measured compliance against set criteria, for example the percentage of participants who completed all assessments^42,67–69, or percentage completion at longitudinal time points; Jamison et al.³³ reported the percentage of participants who completed at least 30, 60 and 90 daily assessments respectively. Four studies assessed the variation in compliance across time of day^69–72. Of the 26 studies which included passive tracking, five reported on some form of pRMT compliance^{33,40,56,73,74}. These included the percentage of participants logging daily Fitbit data and the percentage of days in which ‘complete’ actigraphy/activity data was logged. One study defined multi-parametric adherence as the proportion of study days with at least 50% of daily self-report questionnaires completed and 12 h of logged activity tracking, chosen to align with future research and clinical goals⁴⁰. Crucially however, most studies gave little or no justification for their measurement of compliance, or the conclusions that were drawn from it.

Third, app usage statistics were used when discussing engagement. For studies which required participants to remotely enrol, percentage of successful study app downloads was a fundamental indicator of engagement^33,49,50. Other reported usage statistics throughout the study period included total app use (per day, per week or per the study period)^{32,42,52,71,75,76} and number of times the app was launched by the user, be that self-initiated⁷⁵ or in response to prompts^36,60. One multi-parametric study directly compared the number of days spent engaging with the app with that of the Fitbit⁴⁰. Studies which included RMT symptom tracking as a component of a behaviour change app also reported on in-app module viewing^38,47,77,78. The impact of app usage was considered by three studies: (i) minutes and days of app use accounted for a large percentage of variance in an ‘app engagement factor’³², (ii) viewing in-app symptom visualisations correlated with aRMT and pRMT adherence⁴⁰, and (iii) longitudinal app use was considered to reflect ‘satisfaction and interest’⁷⁶.

An exploration of subjective engagement, e.g., indirect user experience during or after interacting with the device, was included as an adjunct to objective engagement in 27 (37%) papers. Many studies used quantitative measures, usually in the form of Likert scales, to measure participant experiences of engaging with remote self-monitoring. These were administered at study end, apart from one study⁷² which issued a usability scale after week one in order to assess early technological problems. Four studies also assessed corresponding clinicians^33,44,68,79. Scales included a wide range of questions surrounding two main themes: (i) usability of, or satisfaction with, the technology itself, and (ii) utility of the technology for symptom management. Technology usability assessed ease of use, helpfulness of reminders and navigation of the app interface. Utility of the technology focused more on the use of the technology for symptom management and communication with clinical care teams, and intentions for future use. Most studies included questions covering both themes to some extent, however crucially these were generally assessed as a combined ‘satisfaction’ variable (apart from⁴² which used two distinct scales to measure technology obtrusiveness and clinical utility respectively). Studies either combined several, validated scales in one paper, including the System Usability Scale (SUS)^36,69,75,80 and Technology Assessment Model (TAM)^75,81, or used author-developed scales, due to a lack of ‘psychometrically evaluated measures of user acceptance’ of RMTs⁸¹.

Qualitative measures of subjective engagement also explored participants’ experiences. Five studies conducted semi-structured interviews with participants^{34,40,42,78,82}, and one with care managers⁴². Other studies reported more vaguely on ‘evaluative feedback’^49,65,83, ‘response to open-ended questions’^36,39,84 or ‘in-app free text’⁷⁸ collected throughout follow-up. Qualitative methods were nearly always collected in conjunction with quantitative usability scales, or as a section of the usability scale itself, however one study⁸⁰ recorded questions and difficulties reported by participants to the researchers during the initial demonstration of the technology. Emerging themes covered overviews of experience, challenges and benefits of RMTs for symptom management, and suggestions for improvement.

Most studies reported on objective and subjective engagement either in isolation, or as one, combined ‘engagement’ variable. However, 22 papers (30%) explored the link between the objective and subjective measures used.

Some studies reported on the resulting, subjective effects of using RMTs for symptom monitoring in the study (either as a main aim or an additional outcome). Where the technology took the form of an intervention, quantitative analyses explored associations between RMT use and changes in main outcome variables, including symptom severity^{32,33,37,43,49,62,76,85}, physical activity^33,37,49, receipt of treatment^32,86, and medication uptake³². Other studies reported on the impacts of remote self-monitoring from a more exploratory standpoint. The following qualitative themes were suggested: identification, self-awareness and mindfulness of symptoms and/or emotional health^34,36,40,42, adapting self-management strategies^42,65, being seen ‘as a person’⁷⁸, having access to a ‘safety net’ of professional support^51,78. Only one study suggested a perceived negative effect of RMT use; viewing real-time mood forecasting might lead to a self-fulfilling prophecy, worsening mood⁸⁷.

Five studies conducted correlational analyses on the relationship between objective and subjective variables^{33,37,49,53,84}. Participants who reported ‘liking’ an RMT app, or a higher degree of satisfaction, tended to submit more daily assessments⁸⁴ and have a higher duration of compliance (consecutive days with recorded data)⁵³. Two studies from the same author found significant Pearson product moment correlations between higher daily assessment count and app satisfaction, across both a 3-month and 6-month time point^33,37. A consideration of the interaction between both forms of engagement was considered essential for encouraging future use of RMTs for remote medical and psychological assistance³⁷.

Understanding how engagement is reported in the RMT literature is imperative to ensure reproducibility in the field, to allow studies to build upon results of previous work, and to implement findings in real world settings. The first and second aims of this review were to explore the current state of defining and measuring engagement by using both quantitative and CIS data synthesis methods. The majority of papers reported on engagement in some form. However, these represented a large range of incoherent and often unjustified measures. Many studies employed several measures in one paper, resulting in a lack of distinction between measure types, explanation of why some were chosen over others, and understanding of what conclusions might be drawn from the engagement findings. A much lower proportion of studies included a corresponding definition of engagement, including many of those which described engagement as a main aim of the study. Where engagement was defined, concepts and phrases were interchangeable across papers. Indeed, even across papers with similar engagement definitions, such as ‘feasibility’, cut-offs used to measure RMT compliance differed hugely. Thus, though there is potential for the field to evaluate engagement, a lack of standardised reporting is impeding progress.

The third aim of this review was to present recommendations for the standardisation of future work. The first step towards this has been to provide clarity on the engagement measures that are currently used in the literature. The integrative framework depicted in Fig. 2 is split into two core themes of engagement measures: engagement with the research protocol and engagement with RMTs themselves. This distinction is important given that many studies did not differentiate between these concepts when reporting on engagement; using correlates of dropout from RMT studies as a proxy for gauging wider interest in RMT implementation is only relevant if the reasons for dropout were specific to the RMT aspects of the study. The engagement with RMTs section is further split into objective and subjective measurements. Interestingly, a wide range of measures were used here, but very few studies acknowledged a distinction between objective and subjective engagement or the possible interactions between the two. Under each section of the framework lies a series of options for measuring each engagement type. This framework should aid in the classification and selection of measurements when reporting on an RMT study.

To our knowledge, this is the first review that has explored the extent of engagement reporting across the entirety of the RMT literature. This work complements and expands on previous reviews that have found heterogeneity in reporting of RMT studies^22–24. It also might provide explanation as to why previous studies have found a wide range of dropout rates^20,21. A lack of provision of engagement definitions in this work reflects the findings of Sieverink and colleagues in the eHealth literature²⁷; a small minority of studies included an operationalisation of the engagement measure used, making comparisons difficult at best and futile at worst.

A key link to previous work is the parallel between the objective and subjective engagement concepts highlighted in Perski and colleagues’ work with DBCIs²⁸. Both the definition and measurement synthetic constructs found in this review loosely map on to this distinction, with two exceptions. First, subjective engagement with RMTs is largely focused on the usability or utility of the technology, or on the subjective effects of tracking symptoms, e.g., self-awareness of symptoms, feeling monitored by a ‘safety net’. This is in comparison to the typical measures of ‘flow’ or immersion that are seen as a mechanism of action towards behaviour change in DBCIs. Second, engagement with the research protocol is a novel finding that has not been acknowledged in the DBCI literature. This is likely owing to the simultaneous use of RMTs as both a method for data collection and a tool for symptom self-management. Such differences warrant the exploration of the RMT field in its own right, as separate from DBCI or general eHealth literature.

This review was deliberately extensive in nature, including papers spanning physical and mental health conditions and aRMT or multiparametric RMT measures. It did not exclude papers for not reporting on engagement. This allowed for an overview of the scope of reporting across the field, in contrast to previous reviews which pre-defined engagement for the purpose of inclusion^20,22,27,88. However, one limitation of such a broad focus is the lack of sub-typing by condition, journal type, RMT type or RMT purpose. For example, objective RMT engagement might be more important to consider in studies using RMTs for research purposes, whereas subjective RMT engagement might be more insightful for RMTs implemented into clinical practice. A specific focus on pRMT studies might also uncover additional measures of objective and subjective engagement. A second limitation is the decision to include secondary analyses papers in data synthesis. Six of the included papers reviewed analyses from the MONARCA I trial^{43,45,46,51,58,89}, and a subsequent 5 on the MONARCA II trial^{64,66,85,87,90}. This was justified given the tendency to report on feasibility in standalone papers, however, may have resulted in an over-representation of definitions or measures chosen by these authors. It might also be the case that papers reporting on the same dataset publish multiple analysis papers but only one engagement paper, resulting in a skew towards measures of engagement with the research protocol in these findings. It should also be considered that the searches for this review were undertaken in July 2020. However, the authors have no reason to believe that engagement reporting standards have radically changed between this date and publication of this manuscript.

This review provides the foundation for future RMT studies to define, measure and report on engagement in a standardised and reproducible way. Authors should use both the engagement reporting process (Fig. 3) and the integrative framework (Fig. 2) as a basis for assessing engagement. At the same time, further work should be undertaken to understand how engagement reporting might differ by condition, journal type, RMT type and RMT purpose. Such findings will build on the proposed framework, making it applicable to a wider range of RMT work as the field continues to grow.

This review suggests that where authors are generally interested in measuring engagement with RMTs, the primary emphasis is placed on engaging with the research protocol or objectively with the RMTs themselves. This is logical for two reasons. First, the CONSORT-EHEALTH guidelines²⁵ focus mainly on reporting of engagement with the research protocol, e.g., participant attrition, and technology process outcomes, e.g., metrics of use. Second, these types of engagement directly impact on missing data and resulting statistical analyses. However, a deeper exploration of subjective engagement with RMTs might aid further understanding. A handful of papers in this review have already examined the effects of monitoring symptoms remotely on both clinical outcome variables and feelings of self-awareness and safety. A few have also begun to acknowledge the interaction between RMT usability and future use. Future work should focus on uncovering the links between objective and subjective RMT engagement, which could have huge implications for understanding how and why users engage with these technologies for research, clinical practice, or self-management.

To conclude, the current review provides an exploration of the current state of engagement reporting in studies which use RMTs for symptom tracking in physical or mental health conditions. Where there is clearly interest, the growth of the field is currently impeded by a lack of engagement definitions, incoherent measures, and an inability to compare findings. Recommendations for the standardisation of reporting guidelines and the integration of engagement definitions and measures into the study design process have been put forward. In extending existing reporting guidelines, engagement with RMTs should also be considered as distinct from general eHealth interventions.

Future work should aim to contribute to the ongoing framework that has been provided, moving towards a unified understanding of the impact of engagement in this field, and what can be done to promote it.

Following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines⁹¹, we report on the development of the search strategy, exclusion criteria, and selection and data extraction process for this systematic review. Data synthesis was informed by critical interpretive synthesis (CIS)⁹². CIS is an analysis technique that combines both quantitative and qualitative data, and promotes an inductive approach to the development of a theoretical framework from the available literature⁹³. There is currently a lack of understanding of reporting of engagement in RMTs, thus CIS was deemed an appropriate method. CIS has previously been used in reviews of engagement with eHealth^28,92. This review has been registered on PROSPERO [CRD42020192652].

A systematic search of the following seven databases was conducted in July 2020: Embase, MEDLINE and PsycINFO (all via Ovid), PubMed, IEEE Xplore, Web of Science Core Collection (comprising Science Citation Index Expanded and Social Sciences Citation Index), and Cochrane Central Register of Controlled Trials (CENTRAL). No date limits were imposed on the searches. Search terms included a combination of synonyms of i) symptom monitoring (e.g., self-monitor*), ii) remote measurement technology (e.g., smartphone app*), iii) health disorder (e.g., chronic disease*), and (iv) core symptomatology across physical and mental health conditions (mood, depress*, pain, fatigue, mobility) using the Boolean operator ‘AND’. Terms were searched for as free text in the title, abstract, or full text if available, using subject headings where appropriate (see Supplementary Note). 1

Papers were included if they met the following criteria: 1) a peer-reviewed, full-text publication that observes the use of aRMT or aRMT and pRMT for the purpose of symptom monitoring for a physical or mental health condition, 2) participants of 18+ years recruited on the basis of having a physical or mental health condition (clinical cohorts with diagnosed disorders or non-clinical samples with validated measures), and 3) an RMT smartphone app that prompts users to track symptoms at least three times during the study period (in order to capture at least one time point in between enrolment and study end). Studies with or without comparison groups were included. Papers were excluded for the following reasons:Not written in English, Spanish, or German (languages spoken by co-authors);A sole focus on pre-implementation of RMT use, e.g., purely qualitative user-experience workshops;Description of an RMT system without actual observed data;General population cohorts, e.g., fitness tracking, wellbeing studies in non-clinical samples;RMT properties: use of a generic symptom monitoring app not developed specifically for the research on which the paper reports, e.g., PsyMate^TM94, only pRMT;Symptom tracking properties: data inputted through a web-link or SMS, non-prompted, user-initiated symptom monitoring only.

Papers reporting secondary analyses were included, given that they have the potential to report on engagement information that might not have been presented in the initial paper²⁹. General wellbeing or fitness studies were excluded as users might have different motivations to engage than if the symptom tracking was linked with specific disease management. Generic apps were excluded on the assumption that they might carry pre-defined engagement metrics. Engagement reporting was not included in the search strategy or eligibility criteria in order to evaluate the consistency of reporting across the literature²⁹. No papers were excluded on the basis of quality, in order to describe the overall state of the research field⁹³.

Papers identified through the search process were merged using EndNote 20⁹⁵. Duplicate records were removed. Two pairs of reviewers independently completed the title and abstract screening stage, whereby each record was screened by at least two reviewers. Two reviewers (KMW and CW) completed the full text screening stage. Any disagreements at each stage were resolved through discussion or by consulting a third reviewer.

A data extraction form was developed and refined, informed by recent systematic reviews on engagement in eHealth^28,29 and a preliminary review of included papers. Due to the large volume of eligible studies, three team members performed data extraction; KMW independently checked the extraction data for accuracy. For each paper, data were extracted on i) study characteristics, ii) symptom monitoring, and iii) engagement reporting. ‘RMT use’ reflected the main aim of the RMT and was coded as (1) self-management (used to manage a condition by the individual), (2) clinical support (used in conjunction with clinical care), or (3) research (asked to use as part of a research study only). Engagement reporting was split into definition, measurement and reporting sections. For the purpose of data extraction, ‘engagement’ was loosely defined as any definition or measure pertaining to how or why participants interacted with any element of the study. No pre-determined codes were enforced onto the engagement data, in keeping with the inductive approach of CIS.

No papers were excluded on the basis of quality. Following the principles of CIS, the process of data synthesis itself represents a critique of the literature and interpretations of credibility or methodological standards⁹². The data synthesis in this review therefore treats the literature as an ‘object of scrutiny in its own right’⁹², acting as a form of quality appraisal.

The ultimate goal of CIS is the formulation of one or more synthesising arguments that integrate evidence from synthetic constructs in the literature into a coherent theoretical framework⁹². A detailed process for conducting CIS can be found in⁹² and²⁸. We took the following steps in synthesising the qualitative constructs of engagement definition and measurement:Text identified in the ‘engagement reporting’ stage of data extraction for each individual paper was coded using nVivo software⁹⁶. The specific research aims were used to guide the coding frame.Codes were reviewed to develop synthetic constructs, whereby similar codes were grouped together to form themes.Each paper was retrospectively categorised on the presence of each synthetic construct. This gave the opportunity to present a complementary quantitative overview of the data.An overarching synthesising argument was developed (i.e., an integrative framework for reporting on engagement with RMTs) by combining all synthetic constructs.The synthesising argument was refined through discussion between co-authors.

Further information on research design is available in thelinked to this article. Nature Research Reporting Summary

Reporting Summary Supplementary Information