Assessment of sleep phase, quality, and quantity: the development and validation of the 3-dimensional sleep scale (3DSS)

This study used an online questionnaire administered from October 2021 to December 2022. The participants were 3609 day-shift workers employed by four companies in Tokyo who cooperated with the survey. Because the population of Tokyo is approximately 10 million, we considered a sample size of at least 1067 persons sufficient, assuming a 95% confidence level, a 3% margin of error, and a population ratio of 0.5. All workers at the four companies in this study worked a day shift.

The purpose and procedures of the study were explained to the employees. Participation was voluntary, and we explained that no disadvantages would arise from not participating in the study and that the data obtained would be used exclusively for the study. First, a survey form was created online, and the staff in charge of each company sent its URL to all employees via e-mail with a request for responses. Consent for participation was obtained through a web-based interface and received electronically; those who disagreed were barred from accessing and responding to any research items. Identifying information was anonymized using a substitute employee ID number, and the participants were informed that their responses would be deleted if they chose to withdraw consent after answering the study.

The 3DSS was designed for use by Japanese day workers [17]. It comprises three categories (sleep phase, sleep quality, and sleep quantity), each with 5 items (for a total of 15 items). All items were discussed with sleep specialists, psychiatrists, occupational physicians, and clinical psychologists to ensure sufficient content validity. The constitutive concepts of each subscale and scoring methods are described in Supplementary material 1.

The 15 questions of the 3DSS were tested for reliability and validity according to the COSMIN checklist, an international standard for scale development [19 –21]. We analyzed the ceiling effect, floor effect, skewness, and kurtosis to check for response bias. For construct validity, we performed exploratory factor analysis using the maximum likelihood method and promax rotation. Reliability was tested by calculating the intraclass correlation coefficient and McDonald's ω reliability coefficient [22]. To verify reproducibility, participants were asked to respond again within 2 weeks to the 15 items of the 3DSS, and the intraclass correlation coefficient between the first and second responses was calculated.

We tested convergent and discriminant validity using a multitrait-multimethod matrix that correlated with existing indicators. The existing indicators used were holiday wake-up time, social jetlag, the AIS [4, 5], and weekday sleep duration.

Holiday wake-up times and social jetlag were used as phase indicators. The sleep phase subscale of the 3DSS includes constructs related to chronotype and sleep rhythm regularity. The holiday wake-up time is a good indicator of chronotype because it reflects an individual's true wake-up time, unaffected by social constraints, and social jetlag is obtained from the difference between the mid-sleep time on weekdays and mid-sleep time on weekends; the extent of social jetlag is larger for those whose sleeping and wake-up times change between weekdays and holidays [23 –25].

The sleep quality subscale of the 3DSS includes constructs related to nocturnal symptoms associated with insomnia disorders, such as difficulty falling asleep, awakening in the middle of the night, and early morning awakening. Okajima et al. reported that the AIS has a two-factor structure with Q1–5 and Q6–8 [5]. Of these, Q1–5 of the AIS are good indicators of sleep quality, including difficulty falling asleep and awakening during the middle of the night. We therefore used the total AIS Q1–5 score to assess sleep quality.

The sleep quantity on the 3DSS subscale includes not only sleep duration but also daytime symptoms such as daytime sleepiness, fatigue, and dozing (unintentional sleep) caused by lack of sleep. Q6–8 of the AIS are good indicators of sleep quantity as they focus on daytime symptoms. The total score of weekday sleep duration and Q6–8 of the AIS were therefore used as measures of quantity.

First, we hypothesized convergent validity; we expected the sleep phase score to be strongly correlated (− 0.8 ≤ r ≤ − 0.5) with the time of waking on holidays because the sleep phase score includes items directly related to waking time. In contrast, since the sleep phase score does not include items directly asking about social jetlag, we expected a moderate correlation (− 0.5 ≤ r ≤ − 0.3) between social jetlag and the sleep phase score. Quality 1–3 in the sleep quality score of the 3DSS is quite close to Q1–3 of the AIS because both are based on nocturnal insomnia symptoms. Therefore, we expected a strong correlation (− 0.8 ≤ r ≤ − 0.5). Quantity 3–5 included in the sleep quantity score assessed insomnia symptoms upon awakening, as did Q6–8 in the AIS. Therefore, we expected a strong correlation (− 0.8 ≤ r ≤ − 0.5). Since only one item, quantity 1, contained a question about sleep duration on weekdays, we expected a moderate correlation (0.3 ≤ r ≤ 0.5).

Next, hypotheses were set up for discriminant validity: the sleep phase scores were predicted to be largely uncorrelated (− 0.3 ≤ r ≤ 0.3) with AIS Q1–5, Q6–8, and weekday sleep duration. The sleep quality scores predicted little correlation (− 0.3 ≤ r ≤ 0.3) with holiday wake-up time, social jetlag, and weekday sleep duration. Q6–8 of the AIS contains items related to mood, which may have some correlation with nocturnal symptoms of insomnia. Therefore, we predicted that a moderate correlation (− 0.5 ≤ r ≤ − 0.3) between the sleep quality score of the 3DSS and Q6–8 of the AIS would be observed, which would be weaker than the correlation with Q1–5 of the AIS. The sleep quantity score was predicted to show minimal correlation (− 0.3 ≤ r ≤ 0.3) with holiday waking time and social jetlag. For AIS Q1–5, Q4 contained items related to sleep duration. Therefore, we predicted that the sleep quantity score of the 3DSS would have a moderate correlation (− 0.5 ≤ r ≤ − 0.3) with Q1–5 of the AIS and a weaker correlation with Q6–8 of the AIS.

We decided to use social jetlag and AIS, which were used during convergent validation, as indicators to validate the cutoff values. Social jetlag is the reported risk of health problems when there is more than 1 h [12, 13]. Therefore, we used "social jetlag ≥ 1 h" as the outcome when validating the cutoff value for the phase score. An "AIS score of 6 points or more", the threshold for a suspected sleep disorder, was set as the outcome when validating the cutoff values of the 3DSS sleep quality score and sleep quantity score. The area under the curve (AUC) was determined from the receiver operating characteristic (ROC) curve, and the maximum Youden's index score was searched to confirm whether there were any major discrepancies with the current cutoff values.

We employed the maximum likelihood method with Promax rotation for exploratory factor analysis. Factor loading cutoff values are often set between 0.3 and 0.4 [17, 26 –28]. In the present study, we used a large sample size and employed the maximum likelihood method, which provides more precise estimates of factor loadings; therefore, we set the cutoff value at 0.3. Spearman's correlation coefficient was used for the correlation analysis. The scale's reliability was assessed using the McDonald's ω coefficient [22]. For the reliability value, 0.65 or higher was considered acceptable, based on criteria frequently used in previous research [29 –31]. All analyses were performed using IBM SPSS Version 28 for Windows. A two-sided p-value < 0.05 was considered statistically significant.

Responses were obtained from 2651 participants. Among these, 35 did not consent to participate in the study, and 11 had been on leave within the past month, leaving 2605 (1718 men and 887 women; mean age 42.2 years, standard deviation 11.7) individuals included in the analysis. The valid response rate was 72.2%. The basic characteristics of the analyzed participants are presented in Supplementary material 2.

In the response bias analysis of the 15 items, only Quality 3 had a ceiling effect. However, because it was slight and no kurtosis or skewness exceeded |1|, this result alone did not warrant exclusion. Regarding structural validity, exploratory factor analysis based on the maximum likelihood method indicated a three-factor structure with five items each, as expected. Quality 4, although factor loadings exceeding 0.3 were observed for both factors, was not excluded because the factor loadings for the first factor were clearly higher than those for the third factor. The decision not to exclude Quality 4 was also considered appropriate, considering the reliability coefficient when the item was excluded and the clinical importance of this item, which asks about sleep disturbance.

Reliability coefficients above or close to 0.7 were observed for all subscales, indicating that the scales were sufficiently reliable. The results of the intraclass correlation coefficient, which examined reproducibility, showed that most items had strong correlations exceeding 0.75, indicating high reproducibility. The results of Phases 1 and Quantity 3, which were approximately 0.7, were not low enough to be ruled out as not reproducible, and the results of other analyses were taken into consideration.

Regarding convergent validity, all subscales were correlated with relevant existing indicators to the assumed degree and direction, indicating their validity. Regarding discriminant validity, all subscales showed no more correlations than expected, with existing indicators measuring different factors, indicating their validity. In particular, the sleep phase score showed almost no correlation with existing indicators of sleep quality and quantity, suggesting that it is completely discriminative and that it showed a correlation with social jetlag, providing novel insights into factors not measurable by other sleep scales, such as the PSQI and AIS. The AIS evaluates the total score of Q1–8 because it is used to evaluate insomnia disorders. The diagnostic criteria for insomnia disorders include daytime symptoms in addition to nighttime symptoms [32]. Q1–5 in the AIS are items related to nighttime symptoms, while Q6–8 are items related to daytime symptoms. Consequently, although they were extracted separately as factors, both are found in insomnia; therefore, some correlation is possible.

The ROC curves, using social jetlag and AIS as outcomes, showed that the AUC area for all subscales exceeded 0.7. These results indicate that screening based on 3DSS is comparable to the accuracy in the case of screening based on social jetlag ≥ 1 h or AIS score ≥ 6. Regarding the cutoff value, the score that maximized Youden's index was one point lower than the current value, which itself was one point lower than the current sensitivity. Since Japan has a lower sleep condition compared to other countries [33], the current setting of "a cutoff value that captures many individuals (i.e., a cutoff value with high sensitivity)" would be acceptable. However, in situations where there is a shortage of personnel that limits the ability to respond to positive cases, or when the target population is expected to have a high level of health and a low pre-test probability, it may be more practical to lower the cutoff score by one point.

This study had some limitations. First, because all participants were workers, its reliability and validity cannot be guaranteed when applied to those who are not working. Second, the number of participants were not large in the study when verifying reproducibility. However, this number may be considered sufficient considering the practical challenge of conducting the same survey twice with busy workers. Third, there may be some degree of bias in the present study, as the sample included a slightly higher proportion of male participants and was drawn exclusively from Tokyo. Last, 3DSS created in Japanese for Japanese participants. In the future, it will be necessary to increase the proportion of female participants and include workers from regional cities in order to conduct confirmatory factor analysis under more diverse conditions. Furthermore, since sleep phase issues, as represented by social jetlag, are gaining attention not only in Japan but also in other countries, it is necessary to develop an English version of the 3DSS that can be easily adapted into various languages. In particular, as Japan is known to have shorter sleep duration compared to other countries, it will be necessary to rigorously re-evaluate the reliability and validity of the 3DSS when applying it in different cultural or national contexts.