Using mathematical models to predict doctors' performance and alertness on different work schedules

Medical errors have been a leading cause of death in the United States for over a decade [1, 2]. Physician sleep deprivation increases the risk of accidents and patient health through medical errors, as well as physician health through the risk of needle stick injuries and motor vehicle crashes [3–15]. In response both to emerging research documenting the hazards of resident physicians’ sleep deprivation and to public concerns with this risk, the U.S. Accreditation Council for Graduate Medical Education (ACGME) initiated limitations on the number of consecutive hours a physician trainee can work for all residency programs in 2003 [16]. These limits, however, continued to allow residents to work for up to 30 consecutive hours every other shift (Q3), work 88 h per week (averaged over 4 weeks, permitting much longer hours), and to work 24 days in a row. Following a subsequent year-long investigation and literature review that uncovered particular concerns with the duration of resident physicians’ traditional extended work shifts, the U.S. Institute of Medicine recommended that residents’ schedules be further redesigned, such that no resident works more than 16 consecutive hours without sleep [12]. The ACGME, however, implemented a 16 h consecutive work limit for first year interns only (PGY1) in July of 2011 [16], and continued to allow PGY2 and more senior residents to work for up to 28 consecutive hours, 88 h per week averaged over 4 weeks, and 24 consecutive days in a row. This recommendation therefore produced little change in overall schedule hours. Residency program directors, who are not trained in work schedule design or sleep and circadian biology, must design their interns’ work schedule to provide around-the-clock coverage, while ensuring that no individual exceeds the maximum allowable number of hours. Little guidance and few tools exist to aid them as they attempt to do this.

Both circadian misalignment and increased wake duration (i.e., sleep deprivation) are associated with decreases in objective performance and subjective alertness (e.g., [17]). Mathematical models have been used to predict their effects on accidents. Simulation predictions from two mathematical models of performance correlate with train accident rate and cost [18] and with driving accidents [19]. Our goal was to develop a tool to facilitate resident program directors’ efforts to design evidence-based schedules that would: (1) predict an increase over current schedules in the performance and alertness of residents and (2) adhere to the IOM recommendations for intern work hour limits. Both objective performance and subjective alertness were simulated because the time-course of their response to sleep deprivation and recovery is different and because while objective performance decrements may be correlated with errors and accidents [3, 6, 8], the decision to stop a task (e.g., driving) and seek help or a countermeasure (e.g., caffeine, nap) is based on subjective alertness assessments, even though these self-assessments are known to be unreliable (e.g., [20]). To accomplish our goal, we applied methods developed initially to design NASA-related mission schedules to quantify and compare the effects of different resident work schedules [21, 22] on both the predicted performance and alertness of each individual on each schedule, and on the overall performance of the group working a particular schedule (i.e., the net objective performance and subjective alertness of the individual group members).

Since performance decrements related to shift work are associated with a change in the distribution (which may be more dramatic than a change in the average value) of cognitive responses to more time spent at lower performance levels [23], we focused on the lowest percentiles of predicted performance rather than the mean or median level. Our previous simulation work demonstrated that the lowest quartile of predicted performance is sensitive to scheduling changes that cause circadian misalignment [21].

During the day work hours, there is almost no difference among group schedules. However, for all schedules, the performance and alertness is worse during the night - when both the circadian system and length of time awake factors both predict lower performance and alertness and when there is less attending and senior resident oversight - than during the day. The major difference in predicted performance and alertness between the schedules is during night work; all group schedules spend more time at lower values, but the three intervention schedules have much less time spent at these relatively low performance and alertness values: 10 % of time was spent at performance values less than 40 for the Q3 and Q4 schedules but ~0 % of the time was at performance values less than 40 for the three intervention schedules of RCR, XC1 and XC2. During night work hours, more time is spent with lower values (i.e., 75 or less) in predicted alertness than with lower predicted performance.

We found that three proposed intervention schedules that limited interns to no more than 16 consecutive hours were each predicted by our mathematical modeling tools to yield better performance and alertness than traditional Q3 or Q4 work schedules, especially during night work hours when there is less supervision. Safety, patient care, and learning of medical skills therefore would also be expected to be affected by these schedules. As residency program directors struggle to meet the changing work hour requirements for their trainees, tools such as that we have designed here can assist them in designing novel schedules that are likely to yield safer care, though as discussed below, further development of these models would help to ensure the precision and usefulness of predictions.

Per the Institute of Medicine’s 2009 recommendations, no resident should work for longer than 16 h without sleep. While the ACGME’s implementation of a 16-h limit for interns alone falls short of this recommendation, it nevertheless represents a substantive change in the structure of medical care delivery and training in the United States. In conjunction with work hour limits, the IOM’s recommendations address essential issues of supervision, workload, and handoffs that are essential to maintaining a high level of patient care. We have previously proposed that a successful transition to evidence-based work hour limits requires: (1) development of evidence based work-hour limits for physicians; (2) dissemination of best practices in safe scheduling that adhere with these limits and incorporate principles of sleep and circadian practices; and (3) development of infrastructure changes that support the implementation of shorter work hours, and promotion of a team culture [35, 36]. Examples of successful efforts to reduce resident physician work hours while implementing needed infrastructural changes are beginning to emerge [40]. Technological tools, both to help with transitions of care [41] and to aid in schedule design may help facilitate these needed changes.

The research presented in this paper is designed primarily to assist in identifying and implementing safe scheduling practices that adhere to the 2009 IOM recommendations, the 2011 ACGME limits, and principles of sleep and circadian science. Evaluating proposed schedules against the IOM’s recommendations is an essential step in our approach to ensure that collective experience of the safety community and practicing physicians accumulated by the IOM are adhered to. For example, the IOM guidelines were selected in part to reduce the exposure and implications of chronic sleep restriction (i.e., multiple days with insufficient sleep) on physician trainees [12]. The consequence of the pre-screening potential schedules for their predicted effects on performance and alertness is to reduce the number of schedule alternatives that need to be considered. The mathematical models provide a way to calculate the predicted performance by taking into account the input of the circadian and homeostatic system that includes this interaction. These nonlinear models have been developed from experimental data collected from hundreds of individuals from experiments designed to understand the relative contribution of the circadian and homeostatic systems in the prediction of performance. Thus, the mathematical models provide an efficient way for which to determine the effect of schedules on predicted circadian phase and therefore performance and alertness. The combined pre-screening of the schedules and the application of the mathematical models is designed to provide quantitative evaluation of the schedules while compensating for the limitations of mathematical models (as discussed in the Limitations section below) that in practice do not take into account all of the features of the operational environment.

An important outcome of the paper is an objective approach to evaluating schedules through the application of novel technology. Although simulation approaches cannot replace experimental approaches, the simulation results presented in this paper could provide a viable approach for determining, collecting and disseminating best scheduling practices. The efficacy of a simulation approach to evaluating resident schedules is supported by our results that demonstrate schedules consistent with the new guidelines are superior to traditional schedules and the quantitative information provides a way to compare schedule alternatives.

Our results suggest that implementation of any of the three novel schedules tested would improve performance over the Q3, or Q4 schedule. The results demonstrate that effective planning can ameliorate the exposure to circadian misalignment and extended work shifts. It also appears that, in practice, elimination of periods of poor performance and presumably periods of increased risk cannot be eliminated, but can be minimized. Field studies substantiate the notion that interventions to reduce resident work hours can improve resident alertness and patient safety. We found previously in a randomized controlled trial that implementation of a 16 h rapid cycle rotation schedule resulted in improved sleep and alertness, and decreased medical errors [3, 6]. Further high-quality studies comparing alternative scheduling interventions are needed, but preliminary evidence evaluating a range of scheduling interventions suggests that most interventions reducing or eliminating shifts over 16 h convey safety and quality of life benefits [5]. Of note, the novel schedules proposed here need not be restricted to interns; quite the contrary, the simulations would predict that such schedules would improve the performance of second-year and higher residents as well, consistent with the IOM’s 2009 recommendations.

Creating a dialog with the Boston Combined Residency Program (BCRP) in Pediatrics that included actual schedules under consideration provided realistic constraints for our residency scheduling research. Our techniques facilitate schedule simulations, although the rationale for generating different scheduling alternatives is not incorporated in our methods. Simulation results provide an objective way to assess the biological implications of schedules under consideration. Initially, our dialog with residency program officials involved defining schedule constraints and assessing scheduling alternatives. Following a meeting in which we shared our results, the BCRP selected schedules that were assessed to be best by our methods. Such a collaboration served not only to promote dissemination of best predicted schedules, but an ongoing relationship in which education in sleep and circadian biology will be provided to residents, and in which the door is open for future refinements of work schedules.

Mathematical models have been used to predict performance and alertness in experimental settings. However, comparison with actual real-work risk is rare. The results of applying the SAFTE model to train operator schedules and then relating those to the risk and cost of train accidents revealed a linear relationship with higher risk and cost associated with worse predicted performance [18]. Application of the SAFTE model to study the predicted performance of orthopedic resident physicians demonstrated that they often work in an impaired condition, but this study did not quantify the potential to address this risk through implementation of alternative schedules [42]. A different model applied to car accident risk also found a predicted relationship [19]. Our model was able to predict use of sleep medications in space [27], an indication of sleeping problems that may produce waketime performance and alertness deficits. A major difficulty however, is in applying the model to individuals, since the individual risk of most severe outcomes is sufficiently low that large numbers of people typically need to be studied to detect significant effects.

Mathematical modeling is a useful tool for evaluating residency schedules. The residency program we studied implemented a schedule we assessed and now educates residents on sleep deprivation, sleep hygiene, and the importance of napping before night shifts. Using the mathematical model, we can provide quantitative evidence to be used in schedule reform.

Nationally, there are approximately 100,000 interns and residents, distributed throughout thousands of residency programs in tertiary care and community hospitals nationwide. Resident physicians care for a substantial proportion of the 36 million patients seen at U.S. hospitals each year. Improved resident physician performance could therefore result in improved quality of care for millions of patients annually. In addition to the direct effect on residents and their patients, the methods developed in this project have the potential to provide objective assessment of current work schedule policies and to inform work hour policies in other occupations. A novel application of the tool could be a standard format for monitoring and evaluating residency work schedules. Schedules designed with a standard tool could form the basis for a national archive of approved schedules and provide an initial screening of residency program compliance with appropriate software tools. Such an approach might be pursued collaboratively with the ACGME, funding from the Agency for Healthcare Research and Quality (AHRQ), Residency Programs, and private scheduling software companies. Lastly, the ability to explicitly schedule specific types of work blocks (i.e. rotation, clinic and call) could provide a framework designed to assess different schedule components either though task specific models or comparison to data collected in the field. Using data collected during training rotations to generate operationally appropriate models could provide simulation estimates tailored to the residency environment.