Multi-scale convolutional transformer network for motor imagery brain-computer interface

Over the past decade, CNNs have achieved remarkable success in computer vision, largely due to their ability to autonomously learn both local and global features through convolution operations. When processing EEG sequence signals, convolution operations are also effective in capturing temporal and spatial features, which are critical for decoding brain signals. Consequently, CNNs have been widely applied in BCI^31,32. CNN-based MI-EEG decoding methods encompass both SSCNNs and MSCNNs. SSCNNs extract features from EEG signals using a single temporal and spatial convolution scale. However, MI-EEG signals inherently exhibit complex spatiotemporal patterns and multi-band frequency characteristics, which cannot be fully captured by a fixed-scale temporal convolution. Moreover, due to significant inter-subject variability in EEG signals, the optimal convolution scale often varies across individuals.

To address these limitations, MSCNNs perform convolutions across multiple scales, allowing for a more comprehensive capture of the complex characteristics in EEG signals and improving classification performance. For example, Amin et al.³³ proposed MCNN, a multi-layer CNN fusion method that integrates four parallel CNN streams with varying depths and kernel sizes. By leveraging transfer learning and feature fusion, MCNN enhances MI-EEG classification accuracy by capturing diverse spatiotemporal patterns in EEG signals. Dai et al.³⁴ proposed HS-CNN, which decomposes raw EEG signals into , , and bands to address inter-subject variability, and employs hybrid-scale convolution kernels (1 × 45, 1 × 65, 1 × 85) combined with a time-frequency data augmentation method to achieve SOTA classification accuracy. However, the large parameter count (> 420 K per filter band) restricts the system’s applicability. Jia et al.³⁵ developed a multi-branch multi-scale CNN (MMCNN) for MI-EEG classification, using five parallel EEG Inception Networks (EINs) with varying kernel scales to capture diverse frequency information, and the squeeze-and-excitation (SE) attention mechanism to enhance performance by reweighting channel features. Altuwaijri and Muhammad³⁶ proposed MBEEGNet, composed of multiple EEGNets with different configurations, and MBShallowConvNet, made up of multiple distinct ShallowConvNets. Their multi-branch structure allows for more comprehensive EEG signal feature extraction, overcoming the single-scale limitation of traditional methods and leading to superior classification performance across multiple datasets. Roy et al.³⁷ proposed a multi-scale CNN, which filters EEG signals into , , , and bands, applying multi-scale convolution blocks with varying kernel sizes to each band. The model also incorporates user-specific features such as differential entropy (DE) and neural power spectrum (NPS), further enhancing performance.

While MSCNNs capture more information than SSCNNs, their limited receptive field restricts the modeling of long-term dependencies, limiting further improvements in MI-EEG decoding. In contrast, the self-attention mechanism in the Transformer architecture, with its global receptive field, effectively captures global dependencies and enhances decoding performance. Song et al.³⁸ proposed the Spatial-Temporal Tiny Transformer (S3T) for EEG decoding, addressing the limitations of CNNs in capturing global dependencies. Their model applies spatial filtering before utilizing self-attention along the feature channel and temporal dimensions to enhance relevant features. Tao et al.³⁹ proposed the Gated Transformer, a family of Transformer models incorporating various gating mechanisms to enhance EEG classification. By replacing standard residual connections with different gating mechanisms, their approach stabilizes training and improves long-term dependency modeling in EEG sequences. Xie et al.²⁹ proposed a Transformer-based deep learning framework for MI-EEG classification, incorporating both spatial and temporal dependencies. Their study introduced five Transformer-based models, explored three types of positional embeddings, and achieved SOTA accuracy on the PhysioNet EEG Motor Imagery Dataset. Song et al.⁴⁰ proposed EEG Conformer, a Convolutional Transformer model for unified EEG decoding that integrates CNN-based local feature extraction with Transformer-based global feature learning. Inspired by the Conformer model, Zhao et al.⁴¹ proposed CTNet, which integrates a single-scale CNN module similar to EEGNet for local feature extraction and incorporates a Transformer for global feature modeling, leading to enhanced classification performance.

Additionally, some researchers have begun exploring EEG decoding methods that integrate multi-scale CNNs with the self-attention mechanism. Ahn et al.⁴² proposed MS-TSformer-DS, a hybrid EEG decoding model that combines multi-scale convolutional blocks, temporal-spatial Transformer encoders, and a dual-stream spatial learner to enhance spatial feature representation. Their model demonstrated strong performance across multiple EEG datasets, including a private dataset, BCI Competition IV-2a, and the Arizona State University (ASU) dataset. Tao et al.⁴³ proposed ADFCNN, an attention-based dual-scale fusion CNN for MI-EEG classification. It employs dual-scale temporal and spatial convolutions to extract spectral-spatial features, while a self-attention mechanism enhances feature fusion by capturing cross-scale dependencies.

Building on these previous studies, our research makes further contributions to the field. Inspired by these prior studies, we propose MSCFormer as an effective solution for MI-EEG decoding.

To evaluate the effectiveness of our proposed model, we used two publicly available benchmark datasets: BCI IV-2a and IV-2b. Detailed descriptions of these datasets, preprocessing, and data augmentation are provided below.

CNNs are effective at capturing local features, while Transformer networks excel at modeling global dependencies. In this paper, we introduce MSCFormer, an end-to-end MI-EEG classification framework that integrates CNNs and the Transformer encoder to leverage both architectures’ strengths. The overall framework of MSCFormer is depicted in Fig. 2. MSCFormer first employs a convolution module to extract spatiotemporal features at various scales from the EEG data. The Transformer module then applies a multi-head self-attention mechanism (MHA) to model global dependencies across these multi-scale features, dynamically emphasizing the most relevant features for classification. Finally, a fully connected layer classifies the extracted features into distinct categories, completing the MI-EEG classification process.

The convolution module includes three CNN branches, each similar to the Shallow ConvNet proposed in²⁰. However, each branch uses spatial depthwise convolution instead of spatial standard convolution, which has demonstrated better performance. Additionally, each branch employs different hyperparameters compared to those in²⁰. The distinction between the three branches lies in the different kernel sizes used for temporal convolution, enabling the learning of EEG features at various time scales.

Each branch in the convolution module comprises a temporal convolution layer, a spatial depthwise convolution layer, and an average pooling layer, as illustrated in Fig. 2. The temporal convolution layer utilizes F₁ temporal filters with a kernel size of (1, K_c). Different values of K_c can capture EEG temporal features at various scales. Following the setup in reference³⁴, K_c values for the three CNN branches are set to 85, 65, and 45, respectively. The spatial depthwise convolution layer independently applies spatial convolutions to each temporal feature map, effectively learning spatial filters associated with specific frequency bands. Additionally, it reduces the number of training parameters, lowering model complexity and computational resource requirements. The spatial convolution kernel size is (C, 1), producing F₁ feature maps that integrate both temporal and spatial features with a shape of (F₁, T). Following the spatial convolution, a batch normalization (BN) layer is applied, stabilizing the data distribution and facilitating smoother gradient flow, thereby improving training efficiency and effectiveness. Subsequently, an exponential linear unit (ELU) activation function is applied. An average pooling layer with a kernel size of (1, P) is then used. This pooling step not only reduces the feature map dimensions but also smooths the spatiotemporal feature maps, reducing local noise and aiding the model in learning global features more effectively. Each branch applies a dropout operation with a probability of 0.5 after the average pooling layer to reduce overfitting. Therefore, the feature dimension output by each CNN branch is (F₁, T_p), where T_p represents the length of the feature series and is given by T divided by P. Next, the feature maps obtained from the three CNN branches are transposed by swapping the convolution feature channel dimension with the time dimension. These feature maps are then concatenated along the feature channel dimension, resulting in a fused feature map X_F with dimensions (T_p, F₂), where F₂ is equal to three times F₁. In this way, the fused feature maps at each temporal point are fed as tokens into the subsequent Transformer module.

To further enhance the features extracted by the CNN module, we employ a Transformer module to model the global dependencies of the multi-scale MI-EEG features. The self-attention mechanism in the Transformer module provides a global receptive field, enabling the model to capture long-range dependencies. It dynamically prioritizes the most relevant features for classification, ensuring the model focuses on the most informative aspects. To quantify the attention allocated by the model to different feature channels, we introduce an additional learnable vector () as a class token, similar to the class token used in BERT⁴⁵, appended to the front of the feature maps. Subsequently, the learnable position embedding F_pos are added to the sequence of features to retain positional information. Therefore, the feature embedding F₀ serves as the input to the Transformer encoder, where F₀ is calculated as follows.

The features are then encoded using an L-layer deep Transformer encoder, where the class token () from the L-th layer represents the output of the Transformer module, serving as the MI-EEG’s feature representation.

The Transformer encoder consists of MHA and feed forward (FF) blocks. MHA comprises multiple self-attention layers, known as heads, which implement scaled dot-product attention, as illustrated in Fig. 3. Each self-attention layer consists of three main components: query , keys , and values . , , and are computed from the input features F through linear transformations. Specifically, , , and at the i-th head of the self-attention layer are calculated using the following formulas :

Where , and are learnable parameters in the linear transformation. The self-attention scores are calculated using the dot product of the and matrices, scaled by the square root of the dimension of the (d_k) to prevent the scores from becoming too large:

The softmax function is applied to normalize these scores into probabilities. Finally, these probabilities are used to perform a weighted sum of the value vectors. MHA enables the model to focus simultaneously on information from different representation subspaces at various locations, enhancing its ability to capture complex patterns and relationships among EEG features. It executes multiple self-attention operations in parallel and then projects its concatenated output.

where W^O is a learnable weight matrix. The output of the MHA block is typically followed by a residual connection and layer normalization (LN):

The FF block consists of two linear transformations with a Gaussian error linear unit (GELU) activation function in between:

where W₁ and W₂ are weight matrices, and b₁ and b₂ are bias terms. Similar to the MHA block, the output of the FF block is accompanied by a residual connection and followed by LN:

Therefore, the MI-EEG feature representation in F_L serves as the output of the Transformer module and also acts as the input to the classifier module.

The classifier module consists of a fully connected layer with a softmax function, where the number of neurons M is set to match the number of classes in the classification task. To reduce overfitting, dropout is applied to the input features before classification, with the dropout rate set at 0.25. The cross-entropy loss is employed as the loss function for model training, which is expressed as:

where y represents the actual labels, and represents the predicted labels.

We employ the most commonly used metrics, accuracy and kappa, for the evaluation of the MI-EEG classification method. Accuracy is defined as follows:

where TP and TN represent true positives and true negatives, respectively, while FP and FN represent false positives and false negatives. The kappa coefficient is a normalized measure that takes into account the chance level and is defined as follows:

wheredenotes the observed accuracy (the average accuracy across all the trials) anddenotes the expected accuracy (the accuracy of a random guess). Generally, the higher the accuracy and kappa, the better the model’s classification performance. The Wilcoxon signed-rank test is used to assess statistical significance. A p-value greater than 0.05 indicates the absence of a statistically significant difference. Conversely, a p-value less than 0.05 (denoted as ‘*’) signifies a significant difference, while a p-value less than 0.01 (denoted as ‘**’) indicates a highly significant difference.

Our method is implemented in PyTorch and utilizes an Intel Core i9-9820X CPU and an NVIDIA RTX 4090 GPU. We classify the BCI IV-2a and IV-2b datasets using only EEG channel data, entirely discarding the three EOG channels in our experiments. We conducted subject-specific classification experiments and adhered to the data division scheme outlined in the competition guidelines. For the BCI IV-2a dataset, we used the first session as the training set and the second session as the test set. For the BCI IV-2b dataset, we used the first three sessions as the training set and the last two sessions as the test set. To evaluate the stability and generalization ability of our model, we performed five-fold cross-validation (CV) on the original training set. We divided the original training set into five approximately equal subsets. Then, we used one subset as the validation set and combined the remaining four subsets with the S&R augmented dataset for training. We selected the model with the lowest loss on the validation set as the best model and tested it on the test set. This process was repeated for each of the five original training subsets, and the final performance metric was obtained by averaging the results on the test set.

During S&R data augmentation, each EEG trial is segmented into eight segments (N_s = 8). We use the Adam optimizer to train the model, with the learning rate, β₁, and β₂ set to 0.001, 0.5, and 0.999, respectively. The batch size and number of epochs for training are set to 288 and 1000, respectively. These training hyperparameter settings are adopted based on the guidelines in⁴¹. Given the fewer electrode channels in the BCI IV-2b dataset, we applied L2 regularization to reduce overfitting, with the weight decay parameter set to 0.001. The three convolutional kernel sizes in the convolution module were adopted from³⁴, while the remaining hyperparameters of the MSCFormer architecture were determined through extensive experiments. Unless specified otherwise, the hyperparameters of the MSCFormer architecture are detailed in Table 2.

To systematically examine the impact of the Transformer module, data augmentation, and temporal convolution kernel sizes within the MSCFormer framework, we conducted a series of rigorous ablation studies on the BCI IV-2a and IV-2b datasets. The ablation study was conducted using five distinct experimental configurations: (1) the fully integrated MSCFormer model, (2) the model without the Transformer module (w/o Trans), in which multi-scale CNN features are concatenated, flattened, and fed into the classifier module, (3) the model without data augmentation (w/o Aug), (4) the model lacking both the Transformer and data augmentation (w/o Trans & Aug), and (5) the fully integrated model with smaller convolutional kernels (w/ Small-K), which utilizes reduced temporal convolution kernel sizes (64, 32, 16), as adopted in^22,36.

Figure 4 illustrates the average decoding accuracies for each subject obtained through a five-fold CV across the specified configurations. MSCFormer consistently achieved the highest average decoding accuracies on both datasets, reaching 82.60% on the BCI IV-2a dataset and 88.00% on the BCI IV-2b dataset. Removing the Transformer module resulted in a 3.30% decrease in average accuracy on the BCI IV-2a dataset (p = 0.055), with particularly substantial drops observed in subjects A04, A05, A06, and A07, where accuracies decreased by 8.40%, 5.63%, 8.13%, and 5.76%, respectively. On the BCI IV-2b dataset, removing the Transformer led to a notable decrease in average accuracy of 2.38% (p < 0.05). Without data augmentation, removing the Transformer module reduced the average classification accuracy by 5.69% (p < 0.01) on the BCI IV-2a dataset and by 1.06% on the BCI IV-2b dataset. Removing data augmentation alone led to significant decreases in recognition accuracy on both the BCI IV-2a and BCI IV-2b datasets, with significant reductions of 9.57% (p < 0.01) and 4.12% (p < 0.01), respectively. Simultaneously removing the Transformer module and data augmentation led to significant decreases in average recognition accuracy: 15.26% (p < 0.01) on the BCI IV-2a dataset and 5.18% (p < 0.01) on the BCI IV-2b dataset. These results demonstrate the critical role of the Transformer module and data augmentation in enhancing the decoding accuracy of the MSCFormer model. Furthermore, using smaller convolutional kernels in the convolution module led to a significant decrease in average classification accuracy, with a drop of 2.21% (p < 0.05) on BCI IV-2a and 1.45% (p < 0.01) on BCI IV-2b.

Figure 5 provides a comparative exposition of kappa coefficients across the various experimental conditions for each subject. The comprehensive MSCFormer setup outperformed all other conditions across both datasets, achieving the highest cumulative kappa. This result highlights the essential roles of both the Transformer and data augmentation in enhancing the model’s robustness and consistency. Additionally, the use of larger convolutional kernels yielded superior performance.

Typically, within the Transformer encoder module, the depth L of the Transformer significantly influences model performance. Figure 6 depicts the evolution of recognition accuracy with increasing depths. On the BCI IV-2a dataset, the model achieves the highest average accuracy at a depth of 5, which is 2.94% higher than at depth 1 (p < 0.05). Beyond this, further increasing the Transformer depth results in a decline in accuracy. Similarly, on the BCI IV-2b dataset, the average accuracy at depth 5 is notably improved by 0.68% compared to depth 3 (p < 0.05). These findings suggest that while increasing depth improves the model’s ability to capture complexity, it also raises the risk of overfitting if the training dataset does not scale accordingly.

In Transformer models, the number of heads in the MHA mechanism is a key parameter that helps in learning different aspects of features. We assessed the impact of varying the number of heads on MSCFormer’s performance, with the results shown in Fig. 7. The MSCFormer model with a single head had the lowest average classification accuracy on both datasets. For the BCI IV-2a dataset, the eight-head model achieved 2.58% higher accuracy than the single-head model (p < 0.05). On the BCI IV-2b dataset, the eight-head model outperformed the single-head model by 0.96% (p < 0.05) and the 24-head model by 0.75% (p < 0.01). These results suggest that appropriately increasing the number of heads can significantly improve accuracy. Additionally, performance on the BCI IV-2a dataset showed greater fluctuation compared to BCI IV-2b, possibly indicating that the model’s sensitivity to the number of heads increases with task complexity.

In our study, we explored the impact of varying pooling sizes on the performance of MSCFormer by adjusting the pooling size from 12 to 72, with increments of 4, resulting in corresponding token lengths ranging from 84 to 14. As depicted in Fig. 8, MSCFormer exhibited an increasing trend in accuracy followed by a decline across the BCI IV-2a and IV-2b datasets. Notably, on the BCI IV-2a dataset, two distinct peaks in classification accuracy were observed at pooling sizes of 28 and 44, both reaching an optimum of 82.95%. In contrast, on the BCI IV-2b dataset, the highest average classification accuracy of 88.00% was achieved at a pooling size of 52. Compared to the lowest performance observed at a pooling size of 12, these peaks represent improvements of 3.87% (p < 0.01) and 2.64% (p < 0.01) on the BCI IV-2a and IV-2b datasets, respectively. These findings suggest that pooling size should be carefully selected based on the specific characteristics of the dataset.

To comprehensively assess our model’s performance, we selected several SOTA methods, including three SSCNN-based models (Shallow ConvNet²⁰, Deep ConvNet²⁰, EEGNet²⁴), two MSCNN-based models (MMCNN³⁵, MBEEGNet³⁶), and two hybrid CNN-Transformer models (Conformer⁴⁰, ADFCNN⁴³). Below is a brief introduction to these methods:

The optimal results for MSCFormer on the BCI IV-2a and IV-2b datasets were compared to the SOTA methods. To further validate the effectiveness of our multi-scale convolution approach, we included MSNet, an ablation model without the Transformer module (w/o Trans), in the comparison.

To ensure a relatively fair comparison, we re-evaluated four representative models, including Shallow ConvNet, Deep ConvNet, EEGNet, and MBEEGNet, whose original experimental conditions in the literature differed significantly from those in our study. In these reimplemented experiments, we applied identical experimental conditions, including the same data preprocessing, data augmentation strategies, CV methods, and training hyperparameters (batch size, learning rate, and epochs). Table 3 presents comparisons of accuracy and kappa between MSNet, MSCFormer, and SOTA methods on the BCI IV-2a. Table 4 presents comparisons for the BCI IV-2b. The data for MMCNN, Conformer, and ADFCNN were obtained from their respective references.

A comparison of these CNN models reveals that our proposed MSCNN-based model (MSNet) achieved the second-highest average classification accuracy on the BCI IV-2a dataset, while also exhibiting the smallest accuracy standard deviation and the highest kappa value. Specifically, its average accuracy increased by 4.08% (p < 0.01), 1.84%, 1.47%, and 0.54% compared to Shallow ConvNet, EEGNet, Deep ConvNet, and MMCNN, respectively, but was 2.2% lower than that of MBEEGNet. On the BCI IV-2b dataset, MSNet ranked second in both average accuracy and kappa value, with a relatively small standard deviation. Specifically, its average accuracy was 2.36% (p < 0.01), 1.24%, 1.16%, and 0.34% higher than those of Deep ConvNet, MMCNN, MBEEGNet, and Shallow ConvNet, respectively, but 1.77% lower than that of EEGNet. These comparative results strongly demonstrate the effectiveness of MSNet’s multi-scale design in addressing the challenge of individual variability in EEG signals.

Tables 3 and 4 show that MSCFormer achieved the best average classification accuracy and kappa values across both datasets, with relatively small standard deviations compared to the other models. Specifically, on the BCI IV-2a dataset, MSCFormer’s average accuracy was 7.80% (p < 0.01), 5.56% (p < 0.05), and 5.20% (p < 0.05) higher than the SSCNN-based models Shallow ConvNet, EEGNet, and Deep ConvNet, respectively. It also outperformed the MSCNN-based models MBEEGNet, MSNet, and MMCNN by 4.26% (p < 0.05), 3.72%, and 1.52%, and exceeded the CNN-Transformer hybrid models Conformer and ADFCNN by 4.29% and 3.56% (p < 0.05). On the BCI IV-2b dataset, MSCFormer’s average accuracy was 4.72% (p < 0.01), 3.60%, 3.52% (p < 0.01), 3.37%, 2.70% (p < 0.01), 2.36% (p < 0.05), 0.59%, and 0.19% higher than Deep ConvNet, MMCNN, MBEEGNet, Conformer, Shallow ConvNet, MSNet, EEGNet, and ADFCNN, respectively. These comparative results highlight MSCFormer’s superior classification accuracy, consistency, and robustness.

Figure 9 presents the average confusion matrices from five-fold CV across nine subjects on the BCI IV-2a dataset. MSCFormer excelled in decoding the imagined left-hand, right-hand, and feet tasks, achieving accuracies of 83.80%, 85.96%, and 83.95%, respectively. In contrast, the results demonstrate that MSNet achieved the highest accuracy in decoding the imagined tongue task, reaching 82.96%, which is at least 4.8% higher than any other model. These results suggest that while MSCFormer demonstrates overall superiority in most tasks, MSNet may be more effective for certain specific tasks. Figure 10 illustrates the receiver operating characteristic (ROC) curves for these comparison models on the BCI IV-2b dataset, plotted based on their true positive rate (TPR) and false positive rate (FPR) data. Notably, MSCFormer achieves the highest area under the curve (AUC) value of 0.955, surpassing other models.

To elucidate the discriminatory capacity of the features extracted and enhanced by our MSCFormer model, we employed t-distributed stochastic neighbor embedding (t-SNE) for visualization. This method transforms the high-dimensional features from EEG sequences into a two-dimensional embedding. We visualized the raw EEG data and the transformation at three critical stages of the MSCFormer model: features learned by the CNN module, features enhanced by the first layer of the Transformer module, and features fully enhanced by the complete Transformer module, as illustrated in Fig. 11. The visualization data were derived from five-fold CV models for subject A03. Figure 11(a) presents the raw EEG signal features, where the four class labels are intermingled, making distinctions challenging. Figure 11(b) illustrates that after processing through the MSCFormer’s CNN module, the four categories become discernible, although inter-class boundaries remain blurred, and intra-class distances are still substantial. Figure 11(c) depicts the features enhanced after the first layer of the Transformer module, where inter-class boundaries are more defined, and intra-class distances are notably reduced, underscoring the efficacy of the MHA mechanism in global dependency modeling. Figure 11(d) illustrates the features after full enhancement by the complete Transformer module, where class labels are distinctly segregated, inter-class distances are further enlarged, and intra-class distances are significantly decreased. This demonstrates that increasing Transformer depth significantly enhances MSCFormer’s expressive capability. However, some misclassifications persist, indicating the need for further model optimization to improve accuracy and achieve clearer label separation.

This study systematically investigates the impact of the Transformer module, data augmentation, and temporal convolutional kernel sizes in MSCFormer through a series of ablation experiments, where each component is either removed or modified to assess its contribution to model performance. Removing the Transformer module led to varying degrees of performance degradation. Without data augmentation, accuracy on the BCI IV-2a dataset dropped significantly by 5.69% (p < 0.01). With augmentation, the BCI IV-2b dataset showed a notable accuracy decrease of 2.36% (p < 0.05). This demonstrates the crucial role of the Transformer’s global receptive field in capturing global dependencies and complex patterns in MI-EEG signals, enhancing decoding performance regardless of data augmentation.

Removing data augmentation alone resulted in a 9.57% (p < 0.01) and 4.12% (p < 0.01) drop in accuracy on the BCI IV-2a and IV-2b datasets, respectively, indicating the importance of data augmentation in increasing the model’s adaptability and coverage across feature space. This effect is especially pronounced for the BCI IV-2a dataset, suggesting that data augmentation significantly enhances model performance in more complex classification tasks.

Removing both the Transformer and data augmentation had the most substantial negative impact, reducing average accuracy by 15.26% (p < 0.01) and 5.18% (p < 0.01) on the BCI IV-2a and IV-2b datasets, respectively. This substantial drop underscores the combined importance of the Transformer’s global dependency modeling and the diversity introduced by data augmentation.

The selection of kernel sizes (85, 65, 45) in our multi-branch CNN module was inspired by³⁴, aiming to capture diverse temporal features in MI-EEG signals. While previous studies, such as^{22] and [36}, have used smaller kernels (64, 32, 16), we conducted an ablation study to assess their impact. The results show that replacing our original kernel sizes with smaller ones led to a significant drop in classification accuracy, with decreases of 2.21% (p < 0.05) on BCI IV-2a and 1.45% (p < 0.01) on BCI IV-2b. This suggests that larger temporal kernels contribute to more effective feature extraction, enhancing MI-EEG decoding performance.

The impact of removing components on the kappa coefficient was similar to that on accuracy. These experimental results reveal that the Transformer module performs global modeling to the multi-scale features extracted from different temporal scales, enhancing the model’s representational capacity and improving decoding performance, while data augmentation ensures robust training in scenarios with limited data.

To investigate the impact of hyperparameters on the performance of the MSCFormer model, we analyzed three key hyperparameters under data augmentation conditions: the depth of the Transformer module, the number of heads in the MHA mechanism, and the pooling size of the CNN module.

The depth of the Transformer encoder plays a crucial role in the model’s ability to capture complex temporal dependencies. Increasing the depth up to a certain point (depth 5) led to an improvement in accuracy, particularly for the BCI IV-2a dataset, where accuracy increased by 2.94% (p < 0.05) compared to depth 1. However, further increases in depth introduce the risk of overfitting.

The number of heads in the MHA mechanism also proved to be a critical factor. The eight-head configuration yielded the best performance on both datasets, with notable improvements of 2.58% (p < 0.05) on BCI IV-2a and 0.96% (p < 0.05) on BCI IV-2b, compared to the single-head configuration. However, adding too many heads (e.g., 24 heads) resulted in diminishing returns, suggesting that excessive splitting of attention may fragment the feature space.

The impact of pooling size showed that an intermediate value optimally balances the retention of temporal features and noise reduction. Pooling sizes of 28 and 44 achieved peak performance on the BCI IV-2a dataset, while a pooling size of 52 was optimal for BCI IV-2b. These results indicate that pooling size should be carefully tuned based on the characteristics of the dataset to avoid information loss or over-smoothing.

To comprehensively compare the performance of our proposed MSCFormer with SOTA methods, we analyzed key aspects such as data preprocessing, data augmentation techniques, model architecture, and parameter count, as summarized in Table 5.

Across these models, standardization was the most common preprocessing method. The Conformer model applied band-pass filtering before standardization, while ADFCNN incorporated both BPF and electrode-wise exponential moving standardization (EEMS). Most models, except ADFCNN, employed the S&R data augmentation technique, while MMCNN used a combination of sliding window (SW) and Gaussian noise (GN). A notable difference was in the selection of electrode channels: on the BCI IV-2a dataset, MMCNN utilized data from only three channels (C3, Cz, and C4), while other methods utilized data from all 22 channels. Roy et al.³⁷ previously investigated the impact of various data augmentation techniques on the BCI IV-2b dataset and demonstrated that the S&R method significantly outperformed GN, SW, and window warping. Additionally, combining these augmentation techniques further improved recognition accuracy. If all models had adopted the hybrid augmentation techniques proposed by Roy et al., their overall performance could have been further enhanced.

The results in Table 5 show that MSCFormer achieved the highest average accuracy across both datasets, demonstrating strong performance in MI-EEG decoding. However, the margin of superiority may have been narrower, or perhaps not the highest, if all models had used the same data preprocessing and augmentation techniques as MSCFormer.

Additionally, although MSCFormer outperforms SOTA models, its high performance comes at the cost of a significantly larger parameter count. This increased complexity enables MSCFormer to capture intricate patterns and long-range dependencies in the data, likely contributing to its superior accuracy. However, the larger parameter count also leads to longer training times and higher computational demands, which may present challenges in resource-limited environments. In practical applications, it is essential to balance MSCFormer’s parameter count with the available computational resources and the potential risk of overfitting. While its complexity improves accuracy, this may not always be practical in scenarios requiring real-time processing or environments with limited computational capacity. Future research could focus on optimizing the model architecture to maintain high accuracy while reducing computational demands, making MSCFormer more adaptable to real-world BCI applications.

While the MSCFormer model proposed in our study outperforms several SOTA methods in subject-specific classification tasks in terms of average recognition accuracy and kappa scores, there are still areas for improvement. First, the MSCFormer model contains numerous hyperparameters, and optimizing them is time-consuming. Key hyperparameters, such as the number of branches in the CNN module and the size and number of temporal convolution kernels within these branches, have not been fully optimized. As a result, the current experimental results may not yet reflect the model’s optimal performance. Future work will focus on automating hyperparameter optimization, integrating it with neural architecture search (NAS) to enable the model to autonomously identify the most effective parameter settings during training, thereby further improving performance. Second, the large number of parameters in the MSCFormer model may limit its deployment on devices with constrained hardware resources. Future research will explore strategies such as model pruning, quantization, and knowledge distillation to reduce the model’s size and computational demands while preserving its high performance. Third, this study primarily addresses subject-specific classification of MI-EEG signals, relying solely on EEG data from individual subjects throughout the training, validation, and testing phases, and excluding cross-subject scenarios. This approach limits the assessment of the model’s generalization capabilities. To address this, future work will focus on applying MSCFormer to cross-subject classification tasks, which will help to evaluate its effectiveness in more diverse application contexts. These efforts could lay a stronger foundation for advancing MI-EEG decoding technology and its practical applications.

The authors declare no competing interests.

The BCI IV-2a and IV-2b datasets analyzed during the current study are available in the BCI Competition IV repository [https://www.bbci.de/competition/iv/#datasets]↗.