Does providing feedback and guidance on sleep perceptions using sleep wearables improve insomnia? Findings from “Novel Insomnia Treatment Experiment”: a randomized controlled trial

Inclusion criteria were: (1) individuals aged ≥18 years, (2) ability to communicate in English; (3) regular access to internet, smartphone, and email, and (4) score ≥10 on the Insomnia Severity Index (ISI) [61].

Exclusion criteria were: (1) shift work (fixed night shift working between 12am-5am and rotating work schedules involving night shift) during the study period, (2) significant symptoms of the following sleep disorders based on Duke Structured Clinical Interview for Sleep Disorders (DSISD) [62]: sleep apnea (apnea–hypopnea index > 15 and not or inadequately treated), periodic limb movement disorder (arousal index > 15), restless legs syndrome (3 times/week, at least 1 month), circadian rhythm sleep–wake disorders including irregular, non-24-hour, advanced (habitual bedtime earlier than 08:00 pm and habitual wake time earlier than 04:00 am, occasional deviation allowed), and delayed (habitual bedtime later than 03:00 am and habitual wake time later than 11:00 am, occasional deviation allowed) sleep phase types; (3) at risk determined by individuals who select either moderate or high on the suicidal ideation item on the baseline questionnaire, and (4) using Fitbit or Dreem in the month prior to participation.

Eligible participants were randomized 1:1 to Intervention or Control groups using permuted block randomization generated in advance, in block sizes of 4, 6, and 8. Random seeds were generated to assure allocation concealment and pre-guessing of the allocation sequence at the end of each block. Randomization was stratified by baseline ISI (≤14 and ≥ 15). The randomization scheme was generated and setup in REDCap [63] by a member of the research team (JFW) who (1) was not involved in recruitment or delivery of the study intervention and (2) was not one of the Principal Investigators. Follow-up study measures were either self-completed or conducted by research staff who were blinded to the group. Participants were not blinded but asked to withhold group allocation from research staff during the follow-up study measures.

This study was approved by Monash University Human Research Ethics Committee and was promoted as the “Novel Insomnia Treatment Experiment (NITE).” It was advertised at the Monash University Healthy Sleep Clinic (Victoria, Australia), online (e.g. relevant online forums, social media, websites, podcasts), and in public spaces (e.g. notice boards).

Interested participants provided informed consent online, and then completed a telephone screening interview to determine eligibility. Eligible participants were randomized at the end of the call. Online questionnaires were administered via REDCap (Baseline) and post-intervention (Post). All participants completed two meetings (in-person or video conference), at Baseline for orientation (~30 minutes), and 1 week later for the intervention content (~60 minutes for the Intervention and ~30 minutes for Control groups). All participants received four phone calls: (1) the screening call, (2) check-in calls 1 and 4 weeks after baseline (~10 minutes each) to encourage adherence and troubleshooting, (3) a final telephone call made by a researcher blinded to group allocation at post-intervention to administer the DSISD Insomnia module (~10 minutes).

Following the final assessment and return of research equipment, participants received up to 70 AUD in a gift voucher as a token of appreciation for participation. The gift voucher amount was calculated based on 10 AUD for each week the Fitbit was used for seven nights, and the sleep diary was completed for five nights, up to 50 AUD for 5 weeks of participation. All participants received 20 AUD for completing online questionnaires.

Interventions were fully manualized and scripted. All intervention sessions were recorded, and an independent rater (NQ) assessed fidelity on 10% of randomly selected participants, showing 100% fidelity for both intervention and control groups. Participants received the other group’s materials following the completion of the final assessment.

Participants in the Intervention group received all following components:

To control for nonspecific factors such as receiving sleep information and spending time with a researcher, the Control group was instructed to use Fitbit and Dreem without feedback about recorded sensor-based sleep throughout participation (e.g. sleep information was hidden on the Fitbit app). The Control group received a sleep education session conducted by the provisional psychologist (MAS) that covered: (1) psychoeducation about sleep (e.g. information about the function of sleep, sleep cycles, sleep stages, the homeostatic sleep drive, and circadian rhythms), and (2) sleep hygiene (e.g. healthy sleep habits regarding light, body temperature, physical activity, noise, food, caffeine, alcohol, and bedding).

Selected measures summarized below were well validated. Details and psychometric properties of each measure are in the Protocol [56].

Self-reported sleep (e.g. sleep duration, onset latency, WASO) was measured using a daily sleep diary, adapted from the Consensus Sleep Diary [64]. The sleep diary collected information about experiences over the previous day and night (e.g. sleep timing, number of nighttime awakenings, naps, caffeine consumption, and medication for sleep).

Participants wore the Fitbit Alta HR (Fitbit Inc., 2017 release) on their non-dominant hands. The Fitbit Alta HR has been found to provide equivalent estimates to research-grade actigraphy for sleep parameters such as TST, sleep latency, sleep efficiency and WASO, in and outside of the laboratory [65, 66]. However, like other actigraphy devices, compared to PSG, Fitbit has the limitations of low specificity and has been shown to overestimate TST and SE in individuals with insomnia [66, 67].

Participants were given a Dreem headband (Rhythm SAS., version 1) to wear on their head while sleeping. Compared to PSG, Dreem had high concordance for raw signal detection with 83.5% accuracy for overall scoring across five stages (i.e. all sleep stages and wake) and 74.0% accuracy with wake epochs alone (i.e. specificity) [68]. Due to discomfort and participant hesitancy associated with the headband reported by participants in the early phase of the trial, the protocol was revised, such that the use of Dreem was made an optional component after two participants were enrolled.

The primary outcome was scored in the 7-item ISI [61], with higher total scores indicating greater insomnia symptom severity. A score of 0–7 indicates no clinical insomnia symptoms; 8–14 sub-threshold insomnia; 15–21 moderate clinical insomnia; and 22–29 severe clinical insomnia [61]. Using 10 as a cut-off has demonstrated 86.1% sensitivity and 87.7% specificity for detecting clinically significant insomnia in a community sample [69]. In this study, Cronbach’s α for the ISI was 0.66 at Baseline (lower as only those ≥10 in the ISI distribution were included) and 0.75 at Post; Cronbach’s α for all other standardized questionnaire measures ranged 0.76–0.94 for baseline and 0.85–0.94 for Post.

Exploratory outcomes included (1) sleep–wake state discrepancy parameters: TST discrepancies were calculated for diary-Fitbit differences (sleep diary minus Fitbit) and diary-Dreem differences (sleep diary minus Dreem); SOL and WASO differences were calculated for diary-Dreem differences (sleep diary minus Dreem), (2) the 16-item Dysfunctional Beliefs and Attitudes about Sleep [74], and (3) PSA Scale [55].

The insomnia module of the DSISD [62] was used to identify rates of Diagnostic and Statistical Manual of Mental Disorders Fifth Edition (DSM-5) insomnia disorder. All telephone interviews were recorded and assessed for reliability by an independent rater (NQ) with discrepancies reconciled via discussion. At Baseline, demographics, mental and physical health history, and medication use were measured through questionnaires, and perceived credibility and expectancy of treatment were measured using the Credibility Expectancy Questionnaire [75]. The Credibility Expectancy Questionnaire was collected after randomization and after participants were explained what their intervention entails; the questionnaire included questions such as: “At this point, how successful do you think this program will be at enhancing your sleep?” and “By the end of the program, how much enhancement in your sleep quality do you think will occur?” Questions related to perceived helpfulness and feedback on each intervention component, as well as the sleep wearable devices were collected post-intervention. Due to ongoing data collection during the COVID-19 pandemic, questions related to personal experiences with coronavirus disease 2019 (COVID-19) and potential impact on participation were asked of participants who completed assessments after March 2020.

Adverse effects were monitored during each participant contact throughout the research project, at all check-in phone calls, and via post-intervention questionnaire. Participants were also instructed to contact the research team immediately if they experienced unwanted adverse effects during participation.

A priori power analyses were carried out for comparisons between the intervention and control groups. Assuming an attrition rate of 20%, randomizing 90 participants in total would provide power of 80% (two-tailed a = 0.05) to detect a medium effect size (d = 0.6). This effect size was based on previous behavioral interventions addressing discrepancies in self-report versus objective sleep information [44, 45].

All analyses were conducted on an intention-to-treat basis and carried out using R 4.2.0 [76]. Descriptive statistics were used to characterize demographic characteristics, rates of missing data, and all outcomes. Baseline group differences were tested using t-tests or χ² tests.

To examine group differences in primary and secondary outcomes, separate multiple regression models were carried out. Each outcome at Post was the dependent variable, the outcome at Baseline and stratification factors were included as covariates, and intervention group was entered as the focal predictor. Following the American Statistician Association’s recommendation [77], we did not base conclusions on whether a P-value passes a specific threshold, but present effect sizes (ES) and 95% CI along with P-values for group differences. ES between groups at each time point were adjusted, mean differences, standardized by the residual variance, akin to comparing within-person effects between groups. An effect size of ~0.2, ~0.5, and ~0.8 were considered small, medium, and large.

Missing data for regression analyses were addressed using multiple imputations through chained equations with a fully conditional specification [78, 79], and all findings from regression analyses presented below are based on multiple imputations. The multiple imputation model included data from all randomized participants and 53 variables. For each variable being imputed, included predictors were: (1) the same variable from the other time point, (2) related primary and secondary outcome variables at the same time point, (3) age and sex, and (4) group allocation. Multiple imputations was conducted using the mice package [80] with 50 maximum iterations, and 100 multiply imputed datasets were generated. Sensitivity analyses were conducted using only participants with complete data on each outcome, and all findings are consistent with those based on multiple imputations.

Participants (age M±SD = 47.96 ± 14.56 years) were mostly female (64.6%), white (82.9%), on average scored 17.24 (SD = 3.77; moderate insomnia severity) on the ISI, and 96.5% met DSM-5 diagnostic criteria for Insomnia Disorder. At Baseline, compared to Fitbit, participants self-reported lower average TST (by 38.79 minutes); compared to Dreem, participants self-reported lower TST (by 20.81 minutes), longer SOL (by 20.00 minutes), and greater WASO (by 18.46 minutes). Baseline characteristics, including credibility and expectancy of the intervention, were, in general, comparable between the Intervention and Control groups (d ranges 0.02–0.37 for continuous variables). Characteristics of participants are summarized in Table 1.

Findings from the multiple regression models examining differences in post-intervention outcomes between groups are summarized in Table 2 and visually displayed in Figure 2.

Adjusting for baseline ISI scores, the Intervention group showed lower ISI (p = .011, ES = 0.51) compared to the Control group at Post, suggesting a greater reduction in insomnia symptom severity from Baseline to Post in the Intervention group. Similarly, the Intervention group also had lower PROMIS Sleep Disturbance at Post, after controlling for baseline scores (p = .036, ES = 0.42). At Post, 58.8% and 78.4% of those allocated to Intervention and Control group met criteria for DSM-5 Insomnia Disorder, respectively (p = .054).

All other secondary and exploratory variables, including PROMIS SRI, Epworth Sleepiness Scale, Fatigue Severity Scale, PROMIS Anxiety, PROMIS Depression, DBAS, PSAS, and all sleep–wake state discrepancy variables, did not show notable differences between the Intervention and Control groups post-intervention (P-values range.23–.93, ES range 0.02–0.24).

To describe patterns of changes within either group, paired-sample t-tests were used to explore the differences between Baseline and Post within each group. These exploratory analyses showed that both Intervention and Control groups had similar patterns of change (see Supplementary Table S1 and S2 in the Supplement), including: (1) improvements in insomnia-related domains such as the ISI, PROMIS Sleep Disturbance and SRI, DBAS and cognitive PSA (d ranges 0.32–1.29), (2) reduced sleep duration discrepancy (d ranges 0.38-0.56), and (3) little changes in Fitbit or Dreem measured TST (d ranges 0.06–0.13), or diary-Dreem discrepancy on SOL or WASO (d ranges 0.04–0.2). The Control (but not the Intervention) group had a small reduction in symptoms of depression (d = 0.29), anxiety (d = 0.42), diary measured SOL (d = 0.37) and WASO (d = 0.28).

Of the 99 participants who completed the COVID-19 impact questionnaire, 68 (68.7%) reported that none of their participation was affected by the pandemic, with the remaining affected to some degree during their participation. No participant tested positive for COVID-19 during their participation. Participants reported that the pandemic had a negative impact on their mental health (49.5%), physical health (28.3%), relationships with others (44.4%), employment (29.3%), and finances (25.3%). Most participants (76.8%) reported that the pandemic had no impact on their ability to use strategies/ resources provided by the study.

Of the 51 participants allocated to the Intervention group and completed post-intervention questionnaires, most found the following helpful: the initial meeting to learn about sleep and sleep discrepancy (96.1%), check-in calls with the research team (94.1%), gaining greater insight into sleep through comparisons between own reported and recorded sensor-based sleep data in the weekly reports (92.2%), receiving daily information about sleep through the Fitbit (84.3%). Qualitative feedback from the Intervention group showed that participants found information provided in the study helpful in “reducing sleep anxiety/ worry,” and “sleep may not be as bad as is recalled,” but many expressed that “this knowledge did not help change sleep behaviors or quality.”

Of the 49 participants that completed the study feedback scale from the Control group at post-intervention, 44 (89.8%), and 39 (79.6%) of participants found the initial meeting to learn about sleep and the check-in calls with the research team to be helpful, respectively.

Throughout participation, 37 participants reported during study contact and check-in calls that they had discomfort in using the Dreem headband. Dreem discomfort was also reported by 67 participants in the post-intervention questionnaire, including Dreem impacting sleep (e.g. movement due to the loose fit, pressure due to sleep position) and/or causing tenderness or headaches. During the study, modifications were made to address some of these concerns (e.g. using elastic tighteners to improve fit and having breaks between uses); participants were free to discontinue using Dreem if they chose to.

Additionally, eight participants reported during study contact and check-in calls, that they had discomfort in using Fitbit. Fitbit discomfort was also reported by nine participants in the post-intervention questionnaire, including lighting on Fitbit disrupting sleep, loose fit, and/or skin irritation. During the study, modifications were made to address these concerns (e.g. providing a smaller Fitbit band, covering the device screen to reduce light); participants were free to discontinue using Fitbit if they chose to.

Exclusion criteria of the study means that findings may not generalize to individuals with comorbid insomnia and severe physiologically based sleep disorders or those with severe psychiatric or health conditions. The sample was mostly White, in a relationship, and working full-time, further limiting findings to more diverse samples. Missing data on sleep wearable devices, especially on the EEG Dreem headband, may have impacted the quality of sleep reports that participants received. This study occurred throughout the COVID-19 pandemic, and although participants reported relatively small impact of the pandemic on their participation and overall well-being, findings may not be generalized to different societal contexts and timeframes. This manuscript did not explore physiological mechanisms of sleep–wake state discrepancy (e.g. using EEG derived from Dreem), which needs to be explored in future research. We did not use a questionnaire to systematically assess itemized adverse events, but all participants were proactively asked about adverse events at each contact. Finally, this study did not explore individual differences in responses to the intervention; it is possible that some individuals with insomnia may benefit, whilst others do not benefit from receiving sleep feedback, and factors predicting these responses require future research.

De-identified data from this trial will be shared on reasonable request to the corresponding authors.