Validation of a Neurophysiological-Based Wearable Device (Somfit) for the Assessment of Sleep in Athletes

Twenty-seven participants completed the study [14 female; 13 male; age = 22.3 ± 5.1 years (mean ± SD)]. To be included, participants had to be (a) well-trained athletes (n = 4), i.e., participating in at least 2 h of training per day, at least 3 days per week, for at least 3 years [16], or (b) highly trained athletes (n = 23), i.e., participating in at least 3 h of training per day, at least 5 days per week, for at least 5 years [16]. Participants were recruited from the sports of soccer (n = 10), beach volleyball (n = 6), athletics (n = 3), basketball (n = 3), diving (n = 2), orienteering (n = 1), track cycling (n = 1), and weightlifting (n = 1). Wearable devices may be used to assess athletes’ sleep at any phase of the yearly competition cycle, so the sample in this study included participants in all three phases of the cycle, i.e., pre-season (n = 10), in-season (n = 14), and off-season (n = 3). Participants provided written informed consent and received a nominal honorarium for their involvement (in the form of a gift voucher). This study was approved by the CQUniversity Human Research Ethics Committee following the guidelines of the National Health and Medical Research Council (Australia).

The study was conducted at The Sleep Lab at CQUniversity’s Appleton Institute for Behavioural Science in Wayville, South Australia. The Sleep Lab comprises two accommodation suites fitted out as serviced apartments. In total, the suites contain six bedrooms, six bathrooms, two kitchens, two dining rooms, a gymnasium, and laundry facilities. The accommodation suites are sound-attenuated, windowless, and temperature-controlled, such that, during time in bed, background noise was 30 dB, bedrooms were completely dark, and the target ambient temperature was 21–23 °C.

Each participant attended The Sleep Lab on a single night in groups of 4–6 people. Participants arrived after dinner in the evening (~20:00), were fitted with the PSG and Somfit equipment from 21:30 to 22:30, were given a 9 h sleep opportunity in bed with lights out from 23:00 to 08:00, and departed at ~08:30 after the sleep monitoring equipment was removed. To ensure the timing of the PSG and Somfit equipment was synchronised, all devices were set to the same time prior to each night of use.

An analysis of the data was conducted using Microsoft Excel for Mac version 16.74 and IBM SPSS Statistics version 27.0.1.0. The analysis was consistent with a standardised procedure for assessing the performance of sleep wearables [19].

Twenty-seven participants completed the study. All data from one participant were excluded because no Somfit data were recorded. It could not be determined whether this instance of complete data loss was due to human error (by researchers or the participant) or failure of Somfit hardware, software, firmware, or systems.

For the 26 participants for whom Somfit data were available, there were large differences between participants in (a) the number of 30-s epochs for which data were captured in the Somfit records and (b) the number of captured 30-s epochs for which data were of sufficient quality to be scored (as wake or a particular stage of sleep) by Somfit’s auto-scoring algorithms. Preliminary analyses indicated these factors were likely to affect the quality of the Somfit outputs, so for all subsequent analyses, three subsets of data were created: Unfiltered subset (n = 26): contains Somfit records (and their matching PSG records) from all participants. Good-capture subset (n = 15): contains Somfit records (and their matching PSG records) from participants for whom > 80% of the 30-s epochs were captured/scored by Compumedics’ Profusion Nexus360 system. Excellent-capture subset (n = 7): contains Somfit records (and their matching PSG records) from participants for whom > 99.9% of the 30-s epochs were captured/scored by Compumedics’ Profusion Nexus360 system.

For the two-state categorisation of sleep/wake, the percent agreement and Cohen’s kappa were 84% and 0.45 for the unfiltered subset, 89% and 0.51 for the good-capture subset, and 94% and 0.64 for the excellent-capture subset (Table 3). In comparison, agreement for the two-state categorisation between pairs of sleep technologists scoring the PSG records was 93–96% for the unfiltered subset, 94–97% for the good-capture subset, and 94–97% for the excellent-capture subset (Table 3).

For the five-state categorisation of sleep/wake, percent agreement and Cohen’s kappa were 63% and 0.47 for the unfiltered subset, 66% and 0.52 for the good-capture subset, and 79% and 0.70 for the excellent-capture subset (Table 3). In comparison, agreement for five-state categorisation between pairs of sleep technologists scoring the PSG records was 77–87% for the unfiltered subset, 78–87% for the good-capture subset, and 78–87% for the excellent-capture subset (Table 3).

Error matrices for the five-state categorisation indicate that wake, N1, and REM were the most difficult states for Somfit to identify (Table 4, Table 5 and Table 6). When Somfit worked most effectively (see the excellent-capture subset, Table 6), its main sources of error were classifying wake as REM, classifying N1 as N2 or wake, classifying N2 as N3, classifying N3 as N2, and classifying REM as N2.

For the unfiltered subset, Somfit underestimated total sleep time (TST) by 144 min with an absolute bias of 152 min (Table 7, Figure 1A). For the good-capture subset, Somfit underestimated TST by 25 min with an absolute bias of 40 min (Table 8, Figure 1B). For the excellent-capture subset, Somfit overestimated TST by 10 min with an absolute bias of 14 min (Table 9, Figure 1C).

The modified Bland–Altman plots (Figure 1A–C) are presented in the standard fashion, with horizontal lines for mean bias and 95% confidence intervals. Regression analyses indicated that the three subsets did not have proportional bias, i.e., unfiltered subset—R² = .006, df = 24, p = .70; good-capture subset—R² = .046, df = 13, p = .44; and excellent-capture subset—R² = .130, df = 5, p = .43. Breusch–Pagan tests indicated the unfiltered subset had heteroscedasticity, but the good-capture and excellent-capture subsets did not have heteroscedasticity, i.e., unfiltered subset—BP = 5.1, df = 1, p = .02; good-capture subset—BP = 0.6, df = 1, p = .45; and excellent-capture subset—BP = 2.6, df = 1, p = .10.

The results obtained in this validation study of Somfit with athletes are similar to those obtained in a previous validation study with non-athletes [2]. In the previous study, records were only included if >80% of the 30-s epochs had scoreable Somfit data—equivalent to the threshold used to determine the good-capture subset in the current study. In that study, Somfit correctly identified 65% of all PSG epochs for the five-state categorisation of sleep/wake, with a kappa value of 0.52, which indicates a moderate level of agreement. In comparison, for the Good-capture subset in the current study, Somfit correctly identified 66% of all PSG epochs, with a kappa value of 0.52. Together, these two sets of results indicate that Somfit is as good at estimating sleep staging during a full night of sleep for athletes as it is for non-athletes.

The degree to which Somfit can be considered valid for the assessment of sleep in athletes depends on which subset of the current data is given the greatest weighting. Consider the critical outcomes for each of the subsets of data. First, agreement between Somfit and PSG for the five-state categorisation of sleep/wake was 63% for the unfiltered subset, 66% for the good-capture subset, and 79% for the excellent-capture subset. Second, compared with PSG, Somfit had a mean bias in TST, such that it was underestimated by 144 min for the unfiltered subset and 25 min for the good-capture subset, and overestimated by 10 min for the excellent-capture subset. Third, compared with PSG, Somfit had an absolute bias in TST such that it differed by 152 min for the unfiltered subset, 40 min for the good-capture subset, and 14 min for the excellent-capture subset. Finally, compared with the PSG-derived values for the summary sleep variables, based on means comparisons and effect size analyses, the Somfit-derived values differed for 6 of the 6 variables for the unfiltered subset, 1 of the 6 variables for the good-capture subset, and 0 of the 6 variables for the excellent-capture subset. If Somfit performed at a similar level with athletes outside the current study as it did for the unfiltered subset, it would not be considered a valid measure of sleep. In contrast, if Somfit performed at a similar level with athletes outside the current study as it did for the excellent-capture subset, then it would be considered a valid measure of sleep. Indeed, the five-state agreement between Somfit and PSG for the excellent-capture subset (79%) was similar to the inter-rater agreement between one pair of the trained sleep technicians scoring the gold standard PSG (78%). In addition, for the excellent-capture subset, the absolute bias in the estimate of TST by Somfit in comparison to PSG (14 min) was well under the threshold of 30 min and considered clinically satisfactory [22]. Furthermore, in a recent laboratory-based validation, albeit with a clinical sample rather than a sample of athletes, the five-state agreement between Somfit and PSG was 76% [5], which indicates that Somfit can perform at a similar level in other settings as it did for the excellent-capture subset in the current study.

If Somfit is used to assess sleep in athletes (and others), efforts should be made to ensure the quantity and quality of the data captured more closely resemble this study’s excellent-capture subset, rather than the unfiltered subset. Based on the authors’ experiences using Somfit in laboratory- and field-based settings, there are two major modifiable factors that influence the likelihood of maximising the quantity and quality of captured data. First, the quality of the signals received from the adhesive forehead patch that contains the Somfit electrodes seems to be highly dependent on the preparation of the forehead before the adhesive patch is applied. If users are given the instruction to “scrub the forehead” with an alcohol wipe prior to applying the adhesive patch, rather than to “clean the forehead” (as per the manufacturer’s user guides), the connection between the Somfit electrode and the forehead is superior, and a greater amount of high-quality data are likely to be collected. Therefore, it is recommended that users are advised to “scrub the forehead with an alcohol wipe as hard as possible without causing pain—then pause and repeat”. Scrubbing, rather than cleaning, more closely mimics the process that sleep technologists use when preparing skin prior to attaching PSG electrodes. Second, the quantity of the data captured by Somfit is affected by the degree to which the Bluetooth connection is maintained throughout a sleep period. Data are not captured during any time when the Somfit device adhered to the forehead is not connected to its paired mobile phone. To minimise loss of Bluetooth connection, and any associated loss of data, users should (a) position the mobile phone such that it has a direct line of sight with the Somfit device on the forehead, and (b) take the mobile phone with them if leaving the bedroom during a sleep period (e.g., for a bathroom visit).

There are two main circumstances in which it may be useful to assess an athlete’s sleep. First, the daily monitoring of sleep/wake behaviour—this typically involves tracking basic constructs such as the timing of sleep, total sleep time and sleep quality for weeks, months, or years to monitor for potential changes caused by training load, competition, travel, illness, injury, etc. Second, a one-off examination of sleep structure—this is typically undertaken if an athlete is feeling fatigued for no apparent reason, having difficulty obtaining a reasonable amount of good-quality sleep, or waking up tired after a full night of sleep. Given the anecdotal reports from some users regarding the potential discomfort associated with wearing Somfit’s adhesive forehead patch, particularly if it is used for successive sleeps, it is possible that Somfit may not be tolerated by some athletes for the daily monitoring of sleep. However, in situations where it is necessary to obtain data regarding the structure of an athlete’s sleep (i.e., amount of wake, light sleep, deep sleep, and REM within a sleep period) over a limited number of nights, then Somfit could be used if PSG is impractical. If Somfit is used with athletes as an alternative to PSG, steps should be taken to maximise the quantity and quality of data that are captured (see). In future, if a forehead patch is developed that provides sufficient adhesion but has a lower likelihood of discomfort for the wearer, then Somfit may become a more viable option for the daily monitoring of athletes’ sleep. Section 4.3

Any wearable device, including Somfit, that can capture valid sleep data may also be used to derive important measures regarding the strength of the circadian system, i.e., the internal body clock. For example, two such measures are sleep consistency, i.e., the day-to-day variability in the start/end times of sleep, and social jet lag, i.e., the difference in the timing of sleep on work days and free days. In non-athletes, low sleep consistency is associated with poor mental health [23], impaired cognitive function [24], and increased risk of mortality [25]; and high social jet lag is associated with depression [26], obesity [27], and poor academic performance [28]. In future, it may be possible to use data obtained from Somfit, or other sleep wearables, to examine the potential effects of circadian disruption on important outcomes for athletes, such as mental well-being, physical performance, and risk of illness or injury.