Prediction models for depression after stroke: a review and combined analysis

For this systematic review and meta-analysis, we conducted a comprehensive and systematic search across four major databases: PubMed, Embase, Cochrane Library, and Web of Science (Publication year: 2000-present). The final search was conducted on January 5, 2025. Our search strategy combined subject headings and free-text keywords, ensuring no restrictions on region or language. The search terms were specifically designed to include keywords and medical subject headings (MeSH) related to post-stroke depression, clinical prediction models, and meta-analysis. The search results were refined and combined using the "AND" logical operator to create the final retrieval dataset. Detailed information on the search strategy of Pubmed was shown as example, and was provided in. 1

We used the PICOTS system for the systematic review, which allows framing of the review's aim, search strategy, and study inclusion and exclusion criteria, as described below.

P (Population): Patients with PSD.

I (Index models): Clinical prediction models including machine learning models and Logistic Regression.

C (Comparator): Other Clinical prediction models.

O (Outcome): Prediction accuracy for PSD.

T (Timing): The outcome was predicted after evaluating basic information upon admission, clinical scoring scale results, and laboratory indicators.

S (Setting): The intended use of the risk prediction models was to individualize the prediction of PSD.

To ensure the integrity of the data, H.W. and Y.X. independently conducted the data extraction process. Also, PROBAST+AI tool was applied independently by the two authors. Discrepancies were resolved through discussion or by consulting L.M. In detail, The retrieved articles were imported into EndNote X9, and duplicate records were removed. Titles and abstracts were reviewed to exclude studies that did not meet the eligibility criteria. Full-text articles were then downloaded and assessed to identify studies eligible for inclusion in the systematic review. If studies reported missing summary statistics (such as missing values for AUC, sensitivity, or specificity), the reviewers attempted to contact the authors to request the missing data. If contacting authors was not feasible, studies with missing essential information were excluded from the analysis. The key characteristics of the included studies are summarized in. 2

We assessed the risk of bias in predictive models using the Prediction model Risk of Bias Assessment Tool +AI (PROBAST+AI) (14). PROBAST+AI is an updated tool for evaluating the quality, risk of bias, and applicability of prediction models, including AI/machine learning models. It follows four steps: defining the model's purpose using PICOTS criteria, classifying the study type (model development, evaluation, or both), assessing quality or risk of bias for each domain (e.g., study design, predictors, outcomes), and making an overall judgment on the model's quality, risk of bias, and applicability. This tool improves upon PROBAST-2019 and can be applied to both traditional and AI-based models, helping in the critical appraisal of predictive modeling studies. The publication bias were evaluated by Begg test.

We conducted a meta-analysis to evaluate the AUC indicators of predictive models, including SVM, Decision Tree, Logistic Regression, Neural Network, K-nearest Neighbor. We also evaluate the AUC indictors of different data source including sociological data and clinical data, liver function test, Functional, physical and cognitive tests. A random-effects model was prioritized for the meta-analysis of AUC. The analysis was performed using R version 4.2.0 (R Development Core Team, Vienna, http://www.R-project.org↗). The R packages employed for this process included 'metafor,' 'meta,' and 'forestplot.' The possible causes of heterogeneity among study results will be explored by subgroup analysis. A sensitivity analysis was performed using the leave-one-out method.

Major data sources were PubMed, Embase, Cochrane, and Web of Science databases. The literature screening procedure is shown in Figure 1. There were 20 studies included in this meta-analysis, which incorporated both machine learning models and the Logistic Regression Model. Specifically, the machine learning models included Support Vector Machine (SVM) (9, 15, 16); Gradient Boosting (9), Light Gradient Boosting (LightGBM) (17, 18), Extreme Gradient Boosting (XGBoost) (15, 17 –19), CatBoost (15, 17, 18), Gradient Boosting Decision Tree (GBDT) (15, 17), Random Forest (15 –17, 20), AdaBoost (17), decision tree (17), Artificial Neural Network (ANN) (21), Log-Linearized Gaussian Mixture Network (22) and K-nearest neighbor (16); and linear regression methods and extensions such as Stepwise Multiple Linear Regression (22)and Partial Least Squares (PLS) regression (22). Logistic regression model included logistic regression analysis (12, 17, 18, 22 –30), binary logistic regression analysis (31) and multivariate logistic regression model (32).

In these included studies, multiple data sources were utilized to predict the occurrence of PSD. According to the data sources, we categorized the articles into five groups: sociological and clinical data (3, 9, 12, 17, 19, 23, 25 –27, 29, 32, 33), liver function tests (15), functional, physical, and cognitive assessments (16, 18, 21, 22), and biomarkers (17, 20, 24, 25, 28, 30, 31, 33). The detail of included studies were listed in Supplementary Table S2.

The risk of bias in prediction models was assessed using the PROBAST tool (34), with a focus on four key domains: predictors, participants, outcomes, and analysis. The detailed findings from the assessment are presented in Supplementary Table S3. Five studies (highlighted in red) were excluded due to unclear risk of bias and applicability concerns. The result of the Begg test (z = 0.87, p-value = 0.3847; Bias estimate = 77.0000, SE = 88.5818) indicates that there is no statistically significant evidence of publication bias.

We first compared the predictive performance of different models using a random-effects model. The analysis revealed the following AUC values with corresponding 95% confidence intervals (CI): the Support Vector Machine (SVM) group achieved an AUC of 0.68 (95% CI: [0.66, 0.70]), the Decision Tree group had an AUC of 0.83 (95% CI: [0.76, 0.88]), the Logistic Regression group showed an AUC of 0.79 (95% CI: [0.74, 0.82]), the Neural Network group achieved the highest AUC of 0.88 (95% CI: [0.45, 0.98]), and the K-Nearest Neighbor group demonstrated an AUC of 0.80 (95% CI: [0.76, 0.83]). Among these models, the Neural Network group appeared to demonstrate the highest predictive performance based on the AUC, although this finding was derived from only two studies and was accompanied by wide confidence intervals. Logistic Regression, Decision Tree, and K-Nearest Neighbor models demonstrated comparable performance, with AUC values ranging from 0.77 to 0.78. In contrast, the SVM group showed the lowest predictive performance with an AUC of 0.68. However, the SVM group have the lowest I². These findings suggest that Neural Network models may provide superior predictive accuracy, while traditional models such as Logistic Regression and Decision Trees also perform well in this context (Figure 2). The stability and generalizability of this result still need further verification due to the limited number of studies included and the significant heterogeneity observed.

We then a comprehensive analysis of the predictive performance of models for PSD based on different data sources, including sociological and clinical data, liver function tests, functional, physical, and cognitive assessments, and biomarkers. The predictive performance of each data source group is expressed through AUC values and their corresponding 95% confidence intervals (CI), with heterogeneity analysis conducted. A random-effects model is preferred for the subgroup analysis. The results indicate the following predictive performances: the sociological and clinical data group (AUC = 0.76, 95% CI: [0.68, 0.82]), the liver function test group (AUC = 0.73, 95% CI: [0.70, 0.75]), the functional, physical, and cognitive assessment group (AUC = 0.86, 95% CI: [0.85, 0.86]), and the biomarker group (AUC = 0.80, 95% CI: [0.71, 0.83]). Moreover, these findings highlight the variations in predictive performance across different data sources, with the functional, physical, and cognitive assessment group showing the highest AUC, indicating superior predictive ability (Figure 3).

This study has several limitations. The number of studies evaluating the use of machine learning for PSD prediction is relatively small, which may limit the generalizability and robustness of the findings. Additionally, differences in participant characteristics, such as age, comorbidities, and stroke severity, may have contributed to the observed heterogeneity in results. Variations in data quality, measurement techniques, and the timing of assessments could also influence the predictive accuracy of the models. Lastly, the use of diverse machine learning algorithms, with varying degrees of calibration and validation, might further explain the inconsistency in model performance across studies.

The predictive performance of the Support Vector Machine (SVM) was relatively low, indicating that it may not be the best choice for PSD prediction in the current dataset and model setup. However, SVM could demonstrate different performance in other studies or datasets, and future research should explore its applicability in various conditions. Similarly, the predictive performance of the biomarker group was relatively low, suggesting limited predictive ability in this study. Nevertheless, certain inflammatory and metabolism-related biomarkers showed predictive potential, and further research should validate their application value in PSD prediction. The risk of bias in several included studies is unclear, which may weaken the strength of the evidence.