Keeping Pace with Wearables: A Living Umbrella Review of Systematic Reviews Evaluating the Accuracy of Consumer Wearable Technologies in Health Measurement

Consumer wearable devices—such as watches, wristbands, pendants, glasses, armbands and other accessories—are rapidly permeating various aspects of daily life, becoming ubiquitous tools for monitoring, assessing and enhancing human behaviour and health [1]. These devices encapsulate a wide array of sensors and software, facilitating the continuous collection of individualised data including physical activity, heart rate, sleep patterns and even mood states [2]. The increasing pervasiveness of wearable technology is demonstrated by its global market size, which is expected to reach US$186.14 billion by 2030, expanding at a compound annual growth rate of 14.6% from 2023 to 2030 [3]. The American College of Sports Medicine (ACSM) also recently identified wearable technologies as the ‘#1 fitness trend’ for 2024 on the basis of a survey of > 4500 health and fitness professionals [4].

Wearables are now heralding a new epoch in several fields of research too; they are generally unobtrusive, cost-effective and comfortable to wear and yield a high level of acceptability among users [5–7]. This is reflected in the proliferation of research incorporating wearable technologies for remote data capture. For instance, the Datenspende study by the Robert Koch Institute deployed wearables to tackle the coronavirus disease 2019 (COVID-19) pandemic through anonymous data donations [8], while Perez et al. 2019 exhibited the capacity of the Apple Watch to detect atrial fibrillation [9], sparking discussions on the potential and limitations of these devices among healthcare providers, researchers and media. Kimura et al. investigated associations between steps and sleep patterns, collected with a wristband, and markers of Alzheimer’s disease in older adults [10], while Shilaih et al. utilised wristbands to correlate heart rate and wrist temperature with menstrual cycle phases [11]. Wearables have also started to be incorporated in clinical trials: at the time of writing, there are currently 58 active trials on clinicaltrials.gov using an Apple Watch, 323 using a Fitbit and 71 using a Garmin device [12].

This research demonstrates how, by allowing for long-term data collection in naturalistic environments, wearables facilitate ecological momentary assessments, and can generate insights into individual health patterns [13, 14]. But despite their potential, the relentless pace of technological advancements and the multifaceted nature of these devices raise pertinent questions about their validity and reliability [15, 16]. For researchers, accurate data are crucial for robust study methodologies, especially when these devices are used to infer health status or predict disease outcomes [16, 17]. For healthcare providers, the validity of data can directly affect clinical decisions, patient care and monitoring, and could ultimately impact health outcomes [18]. For individuals, accurate self-monitoring could potentially help shape health-related behaviours and lifestyle modifications, fostering a sense of empowerment and facilitating personal health management [19].

Numerous factors can impact the validity of data collected by wearable devices. These include variability in the algorithms used by different devices to estimate metrics such as heart rate, sleep or physical activity [20]; user-specific factors such as age, body size or skin tone [21]; and differences in device placement and wear time [22]. Environmental conditions, such as temperature and humidity, can also affect sensor performance [23]. Perhaps most importantly, the dynamic nature of the wearable technology industry—with new devices, software updates and algorithms being continually introduced—necessitates ongoing validation studies [24–27]. As such, ensuring the accuracy and validity of wearable devices is an ongoing endeavour, one that has been taken on in a number of research initiatives. For instance, the International Federation of Sports Medicine (FIMS) has advocated for a global standard for sport and fitness wearables amidst the rising concerns for quality assurance related to the products [28], while the INTERLIVE network is a joint European initiative focused on developing best-practice recommendations for evaluating the validity of consumer wearables to measure direct and derived metrics [25–27].

However, the pace of academic research struggles to keep up with the more agile commercial ecosystem: primary research studies are slowed by the need to secure funding, develop validation protocols, recruit and test participants and navigate the peer review process [16], while systematic reviews and meta-analyses are often out of date by the time they are published [28]. Commercial entities typically release new hardware annually and push software updates multiple times a year, vastly outpacing the academic validation and synthesis cycle [16]. This disparity underlines the need to make the research in this field ‘living’ and continually updated, leveraging methodologies such as living systematic reviews to maintain pace with the rapidly evolving technological landscape [29]. The vision of a ‘living’ body of research in the wearable technology field champions the principles of real-time evidence synthesis, capable of dynamically incorporating new findings as they emerge [30].

The aim of this study is to conduct a systematic review of systematic reviews evaluating the accuracy (i.e. validity and/or reliability) of consumer wearable technologies. This review focuses on consumer wearables rather than research wearables, as these devices have the largest user base and undergo regular updates in both software and hardware. The review is not constrained by setting, reflecting the widespread use of wearables among the general population, athletes and clinical populations. This review will be ‘living’, meaning it will be updated as new systematic reviews are published. Our objectives are as follows: (1) to evaluate what biometric outcomes consumer wearable technologies can measure, including metrics such as heart rate, physical activity, sleep and stress; (2) to synthesise the accuracy (including validity and reliability) of consumer wearable technologies for the outcomes in (1); and (3) to evaluate the methodological quality of the existing systematic reviews on this topic. This approach will allow us to identify potential gaps in the literature and provide recommendations for future research. By doing so, this review will bridge the gap between the rapid pace of wearable technology development and the slower tempo of academic research, ensuring that decision-making in this field is informed by the best available evidence.

This umbrella review aimed to synthesise evidence from systematic reviews evaluating the validity of consumer wearable technologies for measuring biometric outcomes such as heart rate, aerobic capacity, energy expenditure, sleep and surrogate measures of physical activity. A total of 24 systematic reviews were deemed eligible for inclusion. Upon removal of duplicated studies within reviews, the net unique studies was adjusted to 391. These studies collectively included 888,033 participants (approximately 42% female).

For biometrics related to the cardiovascular system, our review revealed a subfield of research where accuracy varied on the basis of the specific measure assessed and the context in which the wearables were utilised. Heart rate measurements for example were associated with error rates of approximately − 3.39 bpm [36], or ± 3% [15], contingent upon user characteristics (e.g. skin tone), intensities of exercise and types of activities and device models [15, 36]. In contrast, the available research evaluating heart rate variability showed a high level of accuracy in rest measurements, with strong agreement with criterion measures, but this deteriorated during periods of physical exercise or motion [34, 39]. For arrhythmia detection, research primarily on devices manufactured by Apple and Samsung demonstrated good sensitivity and specificity [51], although it is important to note that the participants included in these studies were generally derived from clinical populations who had a prior diagnosis of cardiac arrhythmias such as atrial fibrillation. For aerobic capacity, a powerful measure of overall mortality risk [56, 57], the review conducted by the INTERLIVE consortium noted a tendency for wearables to overestimate VO_2max during resting tests but found them to be more accurate during exercise tests [50]. Finally, when measuring blood oxygen saturation, the mean absolute differences in SpO₂ measurements taken from a wearable device (specifically, the Apple Watch series 6) and the criterion of pulse oximetry were up to 2.0% [54].

In the realm of physical activity, there was a trend for wearables to underestimate step counts, with disparities observed across various brands and models [15, 36]. Regarding the estimation of time spent in different physical activity intensity zones, there was marked inconsistency, with high measurement errors influenced by the type and intensity of activities [37]. Energy expenditure assessments by wearables seem to veer towards underestimation, with a significant range of error margins and device-specific accuracy nuances [36, 38, 48, 52].

Finally, in relation to sleep, wearable devices showed a consistent trend of overestimating total sleep time (TST) and sleep efficiency. For example, wearables typically overestimated TST by more than 10%, and sleep efficiency similarly showed overestimations in multiple studies. Additionally, wearables tended to underestimate sleep onset latency and wakefulness after sleep onset, with errors ranging from 12 to 180% [37, 38, 42]. These findings indicate significant disparities when benchmarked against polysomnography standards, highlighting the need for further refinement and validation of sleep tracking features in consumer wearables.

Taken at face value, our results would suggest that consumer wearables appear moderately proficient in capturing various health outcomes such as heart rate, heart rate variability, aerobic capacity and others. However, this does not capture the nuance of the current state of wearable technology research. To explore the current research landscape, we collated a list of 310 consumer wearables spanning various manufacturers. Of these devices, only 34 (11%) have been validated for at least one biometric outcome in the primary research studies included in our review. Given that most consumer wearables can measure multiple biometric outcomes, the potential number of discrete validation studies required is substantial. Specifically, if each device were validated for five outcomes (step count, heart rate, sleep, physical activity and energy expenditure), this list would necessitate 1550 individual validation studies. However, only 54 validation studies (3.5% of the potential total) have been conducted to date, highlighting a significant gap in the existing evidence landscape. This highlights the disparity between the number of wearable devices on the market and those that have undergone validation—an issue that is widely acknowledged among researchers [16]. This gap underscores the challenges of conducting validation research that keeps pace with the extremely agile commercial ecosystem. The field has attempted to stay current with a multitude of devices, which often have annual release cycles and use diverse methodologies, leading to significant heterogeneity in research outcomes [16].

Thus, our findings do not conclusively indicate that wearables consistently underestimate heart rate, overestimate sleep time or fail to accurately measure energy expenditure (for example). Instead, this umbrella review reveals the intricate variability across devices, outcomes, user contexts and reference standards, making a definitive assessment of wearables’ accuracy challenging. The pervasive heterogeneity in research methodologies and findings limits the practical applicability of these technologies, highlighting the urgent need for standardised validation protocols that are both rigorous and adaptable to the rapidly evolving wearable technology landscape. This need for standardisation is evident from the fact that only 71% of the primary research studies included in the reviews utilised an accepted gold standard criterion, and only 40% of these studies followed best practice statistical analysis [27] to determine device accuracy. Developing robust frameworks and fostering collaborative partnerships with industry are essential to enhance the reliability, consistency, and scientific integrity of wearable technology assessments [16]. Therefore, the question of wearables’ accuracy remains inherently indeterminate, influenced by device-specific, outcome-centric, user-related and contextual variables, making a decisive answer elusive based on the currently available literature.

One of the cardinal challenges encountered in reviewing the literature in this field is the swift and continuous evolution of the commercial landscape for wearable technologies. Given this rapid advancement, the research captured within our review inevitably serves as a historical snapshot, reflecting the accuracy and validity of devices as they existed approximately 2 years prior to the date the search was conducted. This temporal disconnect is exemplified by the chronology of our reference materials—only one of the included reviews was published in 2024 [53], with most being published in 2022 [36, 40, 44, 45, 47, 48, 50]—and the most recent primary study included therein was published in August 2022 [58]. Illustrative of the ongoing output of the commercial engine, consider that, as at the time of writing (June 2024), the market has seen the release of three new iterations of Apple Watch alone since the most recent primary research study included in our results. A universal observation across the wearables reviewed in this research synthesis is their transient market presence; each device analysed has since either been retired or superseded by a more recent model. Despite a semblance of hardware continuity in newer models, the frequent deployment of updated firmware and algorithms can profoundly impact device performance and measurement accuracy. This illustrates a fundamental tension: the measured and methodical pace of rigorous academic inquiry versus the agile, ever-fluxing dynamism inherent in the commercial technology ecosystem. Consequently, our findings, while insightful, may not fully mirror the current state of wearable technology capabilities.

This underscores the importance of making this a ‘living’ research synthesis that evolves concurrently with ongoing technological advancements and refinements. In the realm of wearable technology validation, there exists a kind of ‘validation economy’ marked by diverse, simultaneous efforts aimed at assessing and ensuring device accuracy and reliability. At one end of the spectrum, the popular media landscape is populated by vloggers and influencers who wield substantial reach and influence. Their ‘reviews’—often reactionary, anecdotal and based on single-subject analyses—command vast audiences, albeit with intrinsic methodological limitations. For instance, a review by Marques Brownlee amassed 3 million views within the first 3 weeks of the latest Apple Watch release, reflecting the potent sway of such platforms despite their often informal and experiential evaluation methods [59]. In parallel, more structured and formalised efforts are underway within academic and professional circles to cultivate rigorous validation frameworks. Initiatives such as INTERLIVE spearhead these endeavours by advocating for standardised protocols and best practices in evaluating wearable technologies [25–27]. Through its collaborative network, INTERLIVE strives to formalise and standardise many aspects of validity assessment for consumer-grade wearables, towards the development of recommendations and guidelines that bolster the utility and reliability of wearable-derived data in capturing physical activity indicators. In addition, organisations such as FIMS have instituted quality assurance standards to scrutinise and verify the marketing claims of wearable device companies [24]. Their approach envisions a proactive validation process, wherein manufacturers engage in a pre-market validation exercise, submitting their devices for rigorous evaluation against established research benchmarks or appropriate proxies [24]. Meanwhile, vast data repositories are being cultivated through national initiatives such as the All of Us program from the National Institutes of Health (NIH) [60] and the UK Biobank [61]. These databanks harvest extensive, longitudinal health data from wearable devices, fostering a richer, more nuanced understanding of various health conditions and the role of wearable technologies in managing them. Private enterprises, too, are making significant strides, specialising in harnessing and analysing wearable-derived data to bolster both individual and corporate wellness endeavours [62–65].

In this context, we hope that this living umbrella review will serve to synthesise and harmonise the disparate threads of research and evaluation emanating from various quarters. While its primary focus is to provide researchers with a consolidated and continuously updated synthesis of the latest evidence, we recognise that our readership will likely extend beyond the traditional academic audience. This includes healthcare professionals, policy makers, technology developers and an informed public interested in the accuracy and reliability of wearable technologies. Given the broad and diverse interest in wearable technology, our dissemination strategy aims to maximise the impact of our findings across multiple stakeholder groups. Our public engagement activities will include dissemination of these findings via personal and institutional social media, on our YouTube channel [66] and in specific undergraduate and postgraduate modules in digital health and medical devices. We will involve patient and public representatives in creating a plain language summary of findings to be distributed to the general population, informing policy makers across different countries with written communications [16]. By adopting this approach, our review seeks to become a resource and ally for a multitude of stakeholders. For the vloggers and influencers navigating the vibrant but volatile landscape of new device releases and software updates, it will offer a repository of up-to-date research findings. For formal validation bodies and research consortiums such as INTERLIVE and FIMS, it will provide a cohesive and continuous synthesis of global research efforts, bolstering their frameworks and recommendations with a broader perspective and the latest insights. End-users, too, stand to gain from this living umbrella review. With wearables becoming more ingrained in society [67], and as their user base expands and diversifies, the review will be a valuable resource to help users determine the accuracy and reliability of various devices. It is anticipated that the value of the review will grow as the pace of research accelerates and wearable devices permeate deeper into everyday health management and lifestyle practices.

Yet, despite its strengths and potential value, this review is not without limitations. As previously mentioned, an intrinsic challenge lies in the temporal incongruence between the fast-paced evolution of consumer wearable technologies and the more deliberate and methodical pace of academic research and publishing. Due to the procedural necessities of academic rigor—protocol development, study implementation, data analysis and the subsequent publication processes—our synthesis inherently lags behind the latest commercial innovations and releases [16]. Second, our review operates primarily as a tertiary research synthesis, grounding its insights in secondary research—systematic reviews of primary studies. This methodology, while offering a powerful and practical way of collating and synthesising large amounts of data, introduces vulnerability to potential biases or errors that might pervade the secondary research layers—conclusions drawn herein are reflections of the interpretations, methodologies and potential biases of the underlying reviews and their authors. Indeed, our effort to mitigate these limitations manifested in our risk of bias assessment, which showed that the 24 systematic reviews deemed eligible for inclusion rarely met all of the established criteria. This underscores the need for cautious interpretation and application of our findings. Ultimately, this umbrella review should first be considered a directory to research in the field and, second, a high-level overview of results. The variability that pervades various dimensions of the incorporated studies—including their chosen protocols, selected devices, criterion measures, demographic considerations and statistical methodologies—and the nested structure of conclusions necessitate a cautious approach in extracting coherent and reliable signals amidst the noise of disparate findings [68].

In conclusion, this umbrella review illuminates the varied landscape in consumer wearable technology research, showcasing the potential and complexity inherent in their validation and utilisation. Our findings also highlight the need to formalise and standardise device- and biometric-specific validation protocols, and the potential benefit of an agile research model according to which new devices can be evaluated. Furthermore, fostering collaborative synergies between formal certification bodies, academic research consortia, popular media influencers and industry could augment the depth, reach and inclusivity of wearable technology evaluations, enabling a richer, multifaceted dialogue that resonates with a broad spectrum of stakeholders. Opportunities also lie in the expansion of validity assessments into diverse and unexplored terrains of wearable utility, such as stress, training readiness and ‘body battery’ scores; wearable technology companies such as Fitbit, Garmin, Oura and Whoop have each created variants of these ‘bespoke biometrics’ which collate multiple biometric signals to give a user an idea of their current health status; however, none have undergone formal validation. Crucially, as wearable technologies burgeon, penetrating various facets of health and lifestyle, a continued commitment to ethical considerations, data privacy and user autonomy remains imperative. The pursuit of enhancing accuracy and reliability should unfold alongside endeavours to nurture an ecosystem of ethical technology use, marked by transparency, user empowerment and a conscientious alignment with overarching health and societal objectives.

Below is the link to the electronic supplementary material.

Open Access funding provided by the IReL Consortium.