Objective monitoring of loneliness levels using smart devices: A multi-device approach for mental health applications

Though frequently used and described in the same contexts, loneliness is distinct from social isolation. Loneliness refers to a subjective feeling wherein the individual experiences a lack of actual connection within social interactions [1, 2]. It is also important to note distinctions between loneliness from being alone, in that individuals may be alone but not feel lonely. Loneliness has been defined as a universal human experience, yet highly subjective in the way it is felt [3, 4]. Social isolation, on the other hand, refers to the specific lack of social interaction or access to one’s social networks [5]. A further distinction to consider is that loneliness is separate from an emotion, and can be viewed as a judgment about one’s social connections or lack thereof, that elicit negative emotions and feelings [3, 4]. Being alone or experiencing social isolation may not include the subjective adverse negative feelings or judgment of one’s social relationships that loneliness entails.

A substantial number of researchers have delved into diverse approaches to accurately define loneliness as an accessible issue. Sullivan proposed that individuals may not be aware of their own or others’ states of loneliness or social deficiency because of a number of factors, including defense mechanism, lack of self awareness and cultural norms [6]. Young highlighted the temporal aspect of loneliness, suggesting that studies that dynamically collect individual information over extended periods offer exceptional reference value [7]. He argued that loneliness is a dynamic experience that can fluctuate over time, and that in order to fully understand it, we need to study it longitudinally. Rook characterized loneliness as an ongoing emotional turmoil experienced in scenarios of isolation, miscommunication, or rejection, marked by an insufficient level of meaningful social engagement and participation in uplifting activities. Alleviation can be achieved through avenues that foster emotional intimacy. He defined loneliness index based on six items assessing different aspects of loneliness [8].

Both loneliness and social isolation have profound health implications and are linked with increased morbidity and mortality rates [9, 10]. Concerns for the health implications of loneliness and social isolation were heightened during the COVID-19 pandemic [11, 12], during which people around the world were requested to minimize contact and comply with stay-at-home orders to reduce transmission of the disease.

When focusing on loneliness, individuals with lower socioeconomic status, younger adults, and students in particular were found to have high feelings of loneliness during the pandemic [13, 14], though empirical evidence regarding the differences in loneliness over time and the psychological implications of lockdown remain unclear [15–17]. However, a recent meta-analysis by Ernst and colleagues [18] suggested a small yet significant effect size in the differences between feelings of loneliness before and after the pandemic. Several researchers argue that there are differential associations between loneliness and social isolation with health outcomes, such that loneliness contributes more to mental health outcomes while social isolation is more important in predicting declines in cognitive and physical health [19–22]. However, other studies suggest that the differential health implications of loneliness and social isolation are much more difficult to disentangle; for example both constructs are associated with increased risk for cardiovascular disease and interact in their relation with mortality [5, 23, 24]. In summary, given their crucial link to health-risks, both loneliness and social isolation are necessary to address. To target ways of reducing these feelings, it is important to first determine when and how they occur.

Although a robust body of literature has established a strong link between loneliness and health outcomes [25, 26], this work is constrained by the fact that the majority of this work has relied on self-reported measures of loneliness and social isolation that require that participants report on their symptoms spanning an extended period of time (e.g., in the past two weeks). It can be difficult to accurately report on symptoms over such a long time period, and such symptom reports can be influenced by current mood [27–29]. An on-going and exciting avenue to examine loneliness, both its occurrence and association with health and social outcomes, is through its occurrence in daily life [30]. By assessing loneliness with greater ecological validity, we may be able to understand how and when loneliness plays a role in physical and mental health outcomes. For example, existing literature has found that loneliness in daily life is linked to increased feelings of negative affect, greater negative appraisals of social interactions, and more time spent alone; furthermore, these associations had lagged associations with loneliness and depression at a later time point [30, 31].

In addition to outcomes associated with loneliness, several studies have been conducted to assess the feasibility of detecting and modeling loneliness and social isolation in real-world settings. Previous approaches using intensive longitudinal self-reports can put a heavier load on users, resulting in unsatisfactory experiences for the users due to the reliance on frequent engagement [32, 33]. An attempt to resolve this issue involves leveraging users’ behavioral data as collected by continuous mobile sensing. Using continuous and objective sensing devices (e.g., smartwatches) to monitor loneliness allows for a more accurate and representative assessment when taking into account contextual factors in a user’s environment. Smartphones can monitor certain aspects of a user’s life using multiple embedded sensors (e.g. GPS, accelerometer, microphone, light sensor and phone usage logs). Furthermore, passive sensing can quantify the behavioral patterns of participants, including the type and frequency of activities performed, changes in geographic location, and phone usage. Such work has already begun to leverage this type of data to determine features that contribute to loneliness with high predictive accuracy [34–37]. For example, mobility and digital media use are found to be features linked to feelings of loneliness [32, 38]. Additionally, the presence of other people and differences in phone usage were also shown to be predictors of loneliness [38, 39]. Doryab and colleagues [38] have attempted to classify loneliness levels amongst college students before and after a semester by leveraging data collected with smart wristbands and phones. Using these devices, Doryab and colleagues were able to extract features about calls, locations and screen usage collected from phones along with sleep and activity level features from smart wristbands. Nevertheless, the loneliness scale labels obtained in this study were limited to only two timepoints: pre- and post-semester, which obscures the temporal dynamics of the loneliness data. The wristband used in this study (Fitbit Flex 2) assessed sleep information using the same motion sensors used for activity level recognition. This is while recent devices are capable of leveraging heart rate and other physiological sensors for sleep quality assessment [40, 41].

Despite existing literature that has leveraged the capabilities of smartphones in monitoring loneliness and well-being more broadly, a few novel directions in this approach have yet to be fully explored. First, ubiquitous sensing and advances in technology can allow us to potentially monitor and predict loneliness with higher granularity. Most studies have assessed loneliness as a trait measure or with few assessments throughout the study duration [32, 33, 38, 39] as opposed to intensive assessments of loneliness across multiple days over a long period of time. For example, work by [39, 42] measured loneliness, social anxiety, and affect at the beginning and end of their study. Although measuring these levels at two time points can provide general trends, such approaches likely omit the temporal changes in loneliness during the course of the study or at the daily level. There are, however, a few studies that have assessed loneliness and mental health states multiple times a day [43, 44]. In a study by Ben-Zeev et. al. [43], participants completed a 10-week study that passively collected their speech patterns, location, kinesthetic activities, and sleep duration using smartphones alongside self-reported assessments of daily stress, depression, and loneliness. StudentLife [44], was another study conducted by Wang and colleagues [44], participating 48 students during a 10 week period collecting various contextual data from phone and comprehensive ecological momentary assessments (EMA) assessing variables including stress, mood, and loneliness etc. The authors performed detailed analysis on the correlation of contextual information extracted by smartphones and different EMA data; however, they did not report loneliness detection/classification tasks. While these works provide a relational analysis between the loneliness scales and data, our study aims to take a predictive approach to loneliness.

Lastly, an exciting avenue for the field is the use of a comprehensive approach to monitor and predict loneliness and mental states that involves multiple modalities of passive sensing. A limitation of previous work that relies on passive sensing using smartphones is the expectation of continuous user engagement. While individuals do frequently access their smartphones, there may be times where data collection through smartphones may lack granularity in assessment (e.g., leaving smartphones in bags while at the gym or doing an activity). With the progression of technological tools for entertainment, users, particularly younger adults, may use other modalities for their online presence, such as tablets, game consoles, and smartwatches. Furthermore, smartphone assessments are often limited to assessment of behaviors, unlike other ubiquitous devices such as smart wearables that can provide physiological state measurements. Indeed, prior literature that has used smartphones in real time typically focuses on usage of phone features (e.g., incoming and outgoing calls or messages), geolocation, and participant self-reported responses to assess how these behaviors relate to health and social outcomes [42, 44]. Incorporating other ubiquitous devices such as the OuraRing and Samsung smartwatch allows for the continuous collection of several features such as sleep (e.g., sleep duration, quality, restlessness) and cardiovascular (e.g., heart-rate, heart-rate variability, blood pressure) indicators [40]. The ability to assess sleep and cardiovascular indicators is important, given existing literature suggesting the links between loneliness with sleep disturbances and lower cardiac output [31, 45, 46]. Moreover, the effectiveness of each of these devices in modeling loneliness levels in individuals is a research gap that has been inadequately explored. Assessing the strengths and limitations of different passive sensing tools in predicting loneliness not only is important for determining best practices and feedback for tools to improve their predictive accuracy, but could also enable us to gain a more profound understanding of how various behavioral and physiological features impact the user’s loneliness and in the future, mental health more broadly. It would also assist in optimizing the modeling and inference processes across various dimensions, such as accuracy and energy consumption, among others. Therefore, leveraging passive sensing using multiple devices with the capacity to detect unique factors such as sleep and cardiovascular activity would build upon existing literature to provide a more comprehensive and granular understanding of when and how loneliness occurs.

Altogether, the intersection of mental health and technology offers an exciting opportunity to build upon foundational work in the field [38, 44, 47] by developing a more comprehensive monitoring framework capable of assessing the occurrence and predictability of loneliness. In this study, we develop a method of passively and continuously monitoring loneliness using smart wearables and smartphone devices. Using these devices together allows us to assess a wide range of behavioral patterns such as phone usage, communications, and locations as well as physiological patterns such as sleep indices and cardiovascular data. We leverage these devices to propose and evaluate a fully objective loneliness detection method that does not rely on user engagement. We also aim to examine which behavioral and physiological features were most predictive of loneliness for each individual. Finally, our third aim is to determine device efficiency of loneliness monitoring and detection.

Full-time college students (N = 30) at a west coast university between the ages of 18–22 were recruited to participate in an intensive longitudinal study investigating loneliness and mental health. Participants were eligible if they spoke fluent English and used an Android smartphone compatible with the Oura Ring and Samsung Active 2 watch. Students were ineligible to participate if they were married, had children, were returning to school after a three year hiatus, or if they experienced any severe forms of psychopathology (i.e., diagnosed with clinical depression or substance use disorders, psychosis, or any form of suicidal ideation). The exclusion criteria were intended to ensure a relatively homogenous sample of college students and individuals in emerging adulthood. We reasoned that students whose demographics were different (parents, older adults, and individuals returning to school) may have a different experience than emerging adults as they develop their identity and social networks. Students were recruited during September 2021 until January 2022 through faculty outreach where professors may then make announcements to their classes. Social media posts on institutional platforms (e.g., Facebook, Reddit, and Discord) were also used to recruit students. Interested participants reached out via email and were administered a screening survey to assess for moderate to severe depression or suicidal ideation. Individuals who met these criteria were then reached out by the clinical psychologist on our team (JB) for additional support and follow-up. Campus and local wellness resources were provided to all individuals who reached out or completed the screening survey.

The study participants comprised a diverse group in terms of gender, with 57% identifying as female, 40% as male, and 3% as non-binary. In relation to race, the sample exhibited a balanced distribution, with 40% identifying as Asian, another 40% as Hispanic/Latino, 13% as White, and 7% as biracial. Participants also spanned multiple school years, with 17% being in their first or second year, 40% in their third year, 23% in their fourth year, and 3% in their fifth year. The mean age of the participants was 19.90 years, with a standard deviation of 1.21, reflecting the age diversity within the cohort. The data collection occurred simultaneously among participants, spanning from January to March 2021.

This research received approval from the Institutional Review Board (HS#2019–5153) overseen by Dr. Elizabeth Cauffman, at the University of California, Irvine. Prior to their involvement, participants gave written consent. As part of a broader investigation into mental health trajectories, potential participants underwent screening for severe depression or suicidal thoughts. A clinical psychologist on the project team provided consultation during this screening. Participants had the option to withdraw from the study, and the principal investigators had the authority to halt their participation at any time to safeguard their well-being.

For the purpose of this investigation, only relevant procedures and our monitoring phase will be described from the larger study. If participants met the inclusion criteria, they were then contacted to schedule their first in-person lab session in which they completed a baseline battery assessment of mental health, emotions, well-being related measures and a written consent. The mental health assessment included measures of psychopathology, on the basis of which participants were withdrawn from the study if the research team determined in consultation with JB that continued participation would be unsafe. During the first lab session, Oura Ring, Samsung Gear Sport watch, and smartphone apps related to the study were set up and downloaded for the participants. Participants were then instructed on how to use the devices and were guided through tasks that they would be completing for the next four weeks of their participation. Participants were instructed to wear their devices at all times throughout the study period, except under certain conditions (e.g., while charging devices or while performing intensive activities that risked damage to the watch). For the smartphone apps, participants downloaded AWARE that monitored their phone usage and a study-designed app, mSavorUs, which would prompt individuals to complete a brief survey five times per day, using an interval-based ecological momentary assessment (EMA) design. The survey included questions about affect, interactions with others, and feelings of loneliness, social isolation, and connectedness. Apps relating to the wearable devices were also downloaded onto participants’ phones for proper device functionality. Participants completed this monitoring phase for about 8-weeks and were compensated for their involvement in all components of the study.

To support our longitudinal study design, we built a platform capable of collecting and storing data efficiently [48]. We designed a research dashboard that allowed the researchers to have constant access to the data being collected in order to monitor participants’ progress. The dashboard visually displayed the collected data and a summary of each participant’s daily activities. The information was then used to track and assess possible connectivity issues with the devices throughout the study duration. The collected data fell under three categories: subjective self-report questionnaires (used as target labels), objective physiological data, and objective behavioral data. Participants’ objective physiological data was collected using the Oura ring and Samsung watch. The Oura ring was used to assess information regarding the participant’s sleep patterns, and the Samsung watch was employed to conduct continuous physiological assessments throughout the day. The objective behavioral data was collected using the AWARE phone application. AWARE provides us with mobile sensory data and participants’ phone usage details, which can then be used to derive behavioral features. Subjective self-report data was collected using the mSavorUs phone application that was developed for the purpose of collecting subjective experiential data and delivering interventions in this study.

For intensive longitudinal studies assessing experiences in real-time, many factors could interrupt continuous data collection and result in missing data. These factors may result from participants forgetting to wear their devices or charge them overnight, or may result due to technical issues (e.g., server congestion or permission issues over the phone). Using the monitoring tools in the data collection server, we were able to track most of these issues over the course of study and solve them in the shortest time possible. After all these arrangements, having a certain degree of missing data is inevitable and must be taken care of in the analysis stage. We removed the features that had more than 30% missing data. Because of varying windowing policies for different features, it was necessary to apply the dropping criteria to each feature individually, rather than to each modality as a whole. In total, 19 features were omitted due to a significant amount of missing values. In our analysis, we tested two data imputation methods to handle missing data. We replaced the missing values with the average of: A) two preceding and succeeding valid samples, or B) all valid values [55] of that participant. The latter method yielded a higher correlation with the target questionnaire response values for the majority of the features. Therefore, for consistency, we used this method to handle missing data for all of the features.

We defined the binary classification labels (true/false values) based on the participant’s self-reported loneliness rating (scaled from 0–100, 0 = not at all, 100 = extremely) as below or above the median to have a balanced number of labels. We developed a random forest [56] method to predict these classification labels using the 84 selected features. Random Forests is a machine learning algorithm that combines multiple decision trees to make more accurate predictions. Leveraging multiple decision trees enables this algorithm to process high dimensional data efficiently. Before training predictive models, we Z-normalized the features using each participant’s data in the training set to reduce interpersonal bias in feature space. For model evaluation, we used a one-subject-out policy for predictive modeling evaluation, where we used the most recent 50% of each participant’s data as a test set and the remaining data of that participant with all the other participants for the training model. We calculated the accuracy, F1-score, precision, recall, and mean squared error for all the test entries combined. We defined true positive and false positive predictions if our model detected the loneliness class respectively correct and incorrect (according to EMA labels). Similarly, true negative and false negative predictions were instances where the model did not detect the loneliness class while they were correct and incorrect, respectively. Given these terms we can define aforementioned performance metrics as:

These reporting metrics are important for data interpretation because they can help us to understand the strengths and weaknesses of a classification model. For example, a model with high accuracy may have low precision or recall, meaning that it is good at predicting the majority class but not good at predicting the minority class. Or higher precision and lower recall means that the model is more likely to predict positive instances correctly, but it is less likely to identify all positive examples. In our case this means the model detects the majority of loneliness instances in participants but not all of them.

A limitation of using classic model evaluation metrics is that they lack insight into the dynamics of prediction. To address this limitation, the SHapley Additive exPlanations (SHAP) method was used to investigate the contribution of features to predictions given a model [57]. SHAP offers explainability and insight into the contribution of each feature used by a given model in the prediction stage. Given a trained model, this method generates numerical estimates (called SHAP values) that represent how much a feature value affects the output value of the model and what direction (toward either of the classification labels) this effect is. Variations of this method have been proposed for different machine learning models. Some of them consider the trained model as black-box and extract local explanations using efficient sampling in the feature space [58]. Other variations of SHAP analysis methods are proposed for specific machine learning models. In this work, we leveraged path dependent feature perturbation algorithms [59] developed for tree-based models. This method splits the feature space while recursively following the decision path for a given datapoint. This approach enabled us to enhance the accuracy and interpretability of our models, making them more effective tools for decision-making and analysis.

We first present the performance of our proposed loneliness detection method enabled by using the multiple ubiquitous sensing devices AWARE, Oura ring, and Samsung watch. As previously mentioned, we conducted a two-month monitoring study with 30 participants who completed five self-report assessments per day, totaling to about 7,300 data points. The trained model was evaluated by comparing the estimated loneliness values with the corresponding ground truth values (i.e., collected by subjective questionnaires). This procedure was repeated for every participant; in other words, 30 different personal models were built. In the following, we report the aggregated results obtained from all the trained models. Our loneliness detection models obtained an accuracy of 82% Table 2, precision score of 81%, and value of 0.83 Area Under Curve (AUC). Higher AUC generally represents better classification capability (ranging from 0.5 showing random classification to 1 that is for a perfect classification). Fig 3 shows the confusion matrix of the loneliness detection models extracted from about 3,600 tested samples. The obtained true positives (detecting loneliness correctly) and true negatives (estimating no loneliness correctly) were considerably lower compared to the false positives and false negatives.

As discussed, we obtained loneliness detection accuracy of 82% using all the modalities from three devices (smartwatch, ring, and phone). In this section, we explore further into the analysis to investigate the effect of the features on the performance of loneliness detection using explainable machine learning techniques. To this end, we carried out SHapley Additive exPlanations [57] (SHAP) analysis to gain deeper insight into the detection models. SHAP values serve as a measure showing the significance of each feature in the model’s ability to perform the detection. As previously mentioned, we trained a personal model for each participant. To assess the impact of each feature on the loneliness detection among participants, we calculated the mean SHAP values across the test samples.

Fig 4 depicts the absolute SHAP values of the 20 features for each participant, with dark blue and light green hues showing highest and lowest values, respectively. Overall, the 6 most influential features from our prediction model were related to behavioral markers extracted from smartphones (i.e., the AWARE platform), while the HRV features—collected from the smart watch—had less impact. However, the influence of each feature varied across participants. For example, the number of notifications from lifestyle and communication applications had a significant impact on Participants 15 and 20, respectively, whereas these features did not exert as similar of an impact on the remaining participants. This inter-individual difference indicates the importance of personalization in loneliness detection methods.

The findings shown in Fig 4 further offer a broad understating of the impact of individual features on the loneliness detection. However, due to averaging the absolute values, this representation obscures the distribution and direction of the effects of each feature on each participant. Specifically, a positive SHAP value denotes that the corresponding feature shows a positive impact on the loneliness class detection, while a negative value contributes to the non-loneliness class detection. Thus, Fig 5a and 5b shows the positive and negative SHAP values of two randomly selected participants, revealing the variation in the importance of the features. Loneliness class is represented on the right side of the y axis in this Figure, while non-loneliness is represented on the left side. The red and blue colors illustrate negative associations and positive associations respectively for a given feature. For example, physiological features were the most influential for Participant 8 (Fig 5a), whereas Participant 10’s most important features (Fig 5b) were behavioral features. It also shows that on average, Participant 8’s lower values of HRV metrics (represented by the blue color) are associated with higher feelings of loneliness (left side of the vertical line). These figures also indicate how the order and impact of features vary between the participants for loneliness detection. For example, for Participant 8, the most influential features are extracted from the smartwatch. However, this effect is less evident for Participant 10 (Fig 5b). For instance, decreases in the HRV CVSD feature (presented in blue color) results in loneliness for Participant 10. This is while higher values of this feature (red) results in non-loneliness class.

We investigated the effects of different modalities on the performance of our loneliness detection method. To this end, in the training phase, we built four random forest models, three of which were trained with data collected from a single device (i.e., smart ring, smart watch, or smartphone), and one of which was trained using data from all three devices. Subsequently, the models were evaluated using test data from the same device(s). This assessment strategy allowed us to select the optimal set of devices based on the desired criteria, including performance, costs, or user burden. The performance of the models is indicated in Table 2. The smart ring (Oura) is small, lightweight, and easy-to-use, with a battery life of approximately one week following a full charge. The device provides sleep quality, physical activity, and nocturnal HR and RMSSD parameters with a rather high accuracy [40, 60, 61]. Although Oura obtained the highest recall value compared to all other cases, it showed the poorest accuracy, F1-score, and precision for loneliness detection. The second-best accuracy of a single device was obtained using the smart watch (Samsung), which enabled us to acquire HR and multiple HRV measurements at all times. In comparison to Oura, the watch obtained better precision but worse recall. The AWARE framework, which only uses smartphone logging and sensing features, obtained the highest accuracy and precision but the lowest recall compared to the Oura ring and Samsung watch. AWARE runs as a background application on the user’s smartphone. This passive sensing eliminates the need for user input, in contrast to wearable devices that require continuous wear. It should be noted however, that the AWARE data collection is limited to the behavioral and contextual parameters. For instance, our previous study [62] indicated that COVID-19 lockdown had an adverse impact on such passive smartphone-based data collection, as user mobility decreased (e.g., fewer location changes). Our results showed that multimodality has a positive impact on the overall performance of loneliness detection, providing an improved trade-off between precision and recall. Specifically, the three devices used in the study together obtained higher precision values compared to the Oura ring alone, and slightly higher recall values compared to Samsung Watch and AWARE alone.

Despite this growing concern surrounding loneliness, similar to affect and emotional states—particularly negative ones, loneliness is part of the human experience [3, 4]. It would be naive to simply suggest that we are finding a way to predict loneliness to absolve it in its entirety. However, in finding ways to objectively predict and monitor loneliness, we may be able to help individuals and their clinical providers become aware of the contexts and behaviors that particularly elicit feelings of loneliness. Doing so aligns with prior theory that awareness offers a first step to acceptance or enacting any behavioral change in a way that fits with the individual’s goal [77, 78]. It can help individuals view their situations as less lonely, or to reduce the negative feelings that arise in states of loneliness. Our work altogether builds upon and contributes to the field of precision medicine and individualized approaches to acknowledge that an individual’s own contexts and way they perceive their experiences matter in addressing mental health [63, 79].

Loneliness is associated with several other negative feelings and emotional states, including anxiety and depression [2, 4, 80]. While the focus of our study was considering the ways we could objectively measure behavior and experiences tied to loneliness, it could also be beneficial to consider other subjective states that are experienced alongside feelings of loneliness. The associations between loneliness and other emotions may play a role in the kinds of behaviors and physiological experiences individuals have that we aimed to have collected.

The method we propose also faces technical challenges and can pave the way for subsequent research endeavors. Mobile devices like smartphones and smartwatches have battery constraints, and continuous mental health monitoring can deplete them rapidly. Regular data synchronization may also use significant network bandwidth, which can become costly, especially during roaming. These issues indicate the need for refining the proposed approach for greater feasibility. Potential refinements might include optimizing data collection, processing, and inference in a decentralized manner.