A combined convolution and transformer network for classifying imagined movements from EEG signals

CNN has shown effectiveness in automatically extracting spatial features and classifying EEG signals, and it has gradually led to superior performance in MI-EEG data analysis tasks. Schirrmeister et al. introduced ShallowConvNet and DeepConvNet architectures inspired by the FBCSP transformation²⁷. The ShallowConvNet comprises two convolutional layers, an average pooling layer, and a fully connected layer with a softmax classifier. The first convolution operates across time, while the second convolution functions across space, serving as an analogue to the CSP spatial filter. Compared to ShallowConvNet, DeepConvNet enhances its feature network by incorporating three additional convolutional-pooling blocks. Each of these blocks consists of a convolutional layer followed by a pooling layer, with all pooling operations in DeepConvNet, utilizing max pooling for optimal feature extraction. Lawhern et al. developed the notable EEGNet architecture, an advancement over the ConvNet structure²⁸. EEGNet is composed of three types of convolution operations: temporal convolution, channel depth-wise convolution, and separable convolution. EEGNet stands out as a versatile architecture suitable for various BCI paradigms, achieving impressive results even when trained on limited data sets. Chowdhury et al. conducted an in-depth study on EEGNet and introduced a five-branch CNN architecture (EEGNet fusion V2) aimed at enhancing cross-subject motor imagery classification³¹. Each branch of this network incorporates EEGNet with varied hyperparameters to optimize feature fusion and classification performance. Their innovative approach demonstrated promising results across multiple datasets including eegmmidb, BCI IV-2a, and BCI IV-2b, significantly outperforming established models such as EEGNet, ShallowConvNet, and DeepConvNet in cross-subject scenarios. Ingolfsson introduced EEG-TCNet, a novel network that integrates EEGNet and TCN in a sequential manner³⁶. This network is characterized by its low memory usage and reduced computational complexity, making it particularly well-suited for embedded classification in resource-constrained environments, such as edge devices. When evaluated on the BCI IV-2a dataset, EEG-TCNet demonstrated a notable classification accuracy of 77.35%.

Beyond EEGNet, researchers have introduced several other impactful methods for CNN-based classification of MI-EEG. Jia et al. introduce an innovative CNN-based framework that integrates spatial and temporal information processing to enhance the classification of motor imagery EEG signals³⁷. The proposed model, featuring a time-contained spatial filtering (TSF) and a spatial–emporal analysis network (STAN), achieved average accuracies of 83.0% on the BCI IV-2a dataset and 88.0% on the BCI IV-2b dataset. The TSF-STAN model significantly surpasses existing approaches by effectively managing both spatial and temporal features. Amin et al. introduced MCNN, a fusion of multiple CNN models designed to harness diverse convolutional features for capturing both spatial and temporal elements from raw EEG data²⁹. The MCNN achieved a classification accuracy of 75.72% on the BCI IV-2a dataset, and an impressive 95.4% on the High Gamma Dataset. Zhao et al. introduced a multi-branch 3D convolutional neural network (M3DCNN) tailored for MI classification, employing a novel approach to transform EEG signals into sequences of 2D arrays that retain the spatial distribution of sampling electrodes³⁰. This innovative design allows the M3DCNN model to effectively capture both spatial and temporal dynamics of EEG data. For subject-specific analysis, the M3DCNN model demonstrated a commendable performance, achieving an accuracy of 71.02% and a kappa coefficient of 0.644 on the BCI IV-2a dataset. In the more challenging cross-subject evaluation, the model managed to attain an accuracy of 52.17% and a kappa coefficient of 0.453. Sakhavi et al. developed a classification framework for MI data that involves a novel temporal representation, created by modifying the FBCSP method, and a tailored CNN for classification³². This innovative approach significantly improved performance, outshining the existing method in the literature on the BCI competition IV-2a dataset with a 7% increase in average subject accuracy.

The attention mechanism is an effort to emulate the human brain behavior of selectively focusing on a few significant elements while ignoring others. The self-attention mechanism has the intrinsic ability to evaluate global dependencies on very long sequences. A few studies attempted to adopt Transformer models for MI-EEG classification. Tao et al.⁴⁶ implemented a modified version of the Transformer, known as the gated Transformer, for EEG signal analysis. This model employs a gating mechanism instead of traditional residual connections to learn feature representations along a sequence of embeddings. The gated Transformer demonstrated improved results in the PhysioNet dataset. Song et al. introduced an EEG decoding method focusing on self-attention mechanisms⁴⁷. It starts with preprocessing and spatial filtering of EEG data, followed by applying attention transformations on the feature-channel dimension to emphasize key spatial features. The critical step involves segmenting the data temporally for attention processing, yielding distinct representations. Xie et al. crafted five Transformer-based models for MI-EEG classification, leveraging the Transformer model's capabilities and the spatial–temporal attributes of EEG signals⁴⁸. These models achieved top classification accuracies of 83.31%, 74.44%, and 64.22% in two-, three-, and four-class motor-imagery tasks respectively during cross-individual validation on the PhysioNet dataset. They also highlighted that integrating positional embedding modules into the Transformer could further enhance EEG classification performance. However, these models bypass the importance of learning local features, which are crucial for EEG signal decoding.

Zhang et al. ingeniously integrated the Transformer with domain adaptation (DA) to address the variability in EEG signal distribution across different subjects. They introduced a cross-attention Transformer domain adaptive network, named MI-CAT. It achieves an average classification accuracy of 76.81% on the BCI IV-2a dataset and 85.26% on the BCI IV-2b dataset²⁴. Drawing inspiration from ShallowConvNet and previously mentioned Transformer models, Song et al. introduced a compact convolutional Transformer named EEG Conformer, designed to capture both local and global features within a unified EEG classification framework⁵³. The model utilizes ShallowConvNet for learning local features through one-dimensional temporal and spatial convolution layers. Additionally, a self-attention module is directly integrated to extract global correlations from the local temporal features. The EEG Conformer showcased notable accuracies of 78.66% on the BCI IV-2a dataset, 84.63% on the BCI IV-2b dataset, and 95.30% on the SEED dataset. Altaheri et al. developed an attention-based TCN (ATCNet) to enhance the accuracy of EEG-based motor imagery classification⁵⁴. ATCNet utilizes a convolutional module, similar to EEGNet, for extracting local features from EEG trials, multi-head self-attention for emphasizing the most significant features in MI-EEG data, a temporal convolutional network for high-level temporal feature extraction, and a convolutional-based sliding window for efficient MI-EEG data augmentation. Through 10 runs executed on all subjects in the BCI IV-2a dataset, ATCNet achieved an average accuracy of 81.98% across all runs.

For cross-subject motor imagery BCIs, Keutayeva and Abibullaev explore the efficacy of attention mechanism-based models⁵⁵. The study compares three models (ViT, Hybrid Spatial CNN + ViT, and Hybrid Temporal CNN + ViT) using the Leave-One-Subject-Out (LOSO) cross-validation method. Results highlight the effectiveness of the Hybrid Temporal CNN + ViT (HTCV) model on the BCI IV-2a dataset, though it underperforms on the BCI IV-2b dataset. This emphasizes the data size limitations affecting the performance variability of transformer models. Building upon reference⁵⁵, Keutayeva and Abibullaev delve deeper into attention mechanisms, proposing the Spatio-Temporal CNN + ViT (st-CViT) model⁵⁶. This study offers a comprehensive comparison of attention-based models, utilizing data augmentation techniques and employing nested LOSO for robust model selection. Their findings demonstrate that combining attention mechanisms with deep learning models significantly enhances the robustness and accuracy of subject-independent BCIs, particularly when leveraging enriched data environments.

Our research is an ongoing contribution to these works. Therefore, inspired by the works above, we propose the CTNet as an efficient backbone for MI-EEG decoding.

Two publicly available benchmark datasets, BCI IV-2aand IV-2bare used to evaluate the validity of our proposed end-to-end model. These two datasets are provided by the Graz University of Technology. The details of these two datasets, preprocessing and data augmentation are described below. ^{57 58} BCI IV-2a dataset: The IV-2a dataset contains four MI tasks: left-hand, right-hand, both feet and tongue, which is available for download at the following link: []. The dataset encompasses recordings from nine subjects (namely A01-A09), each participant contributed EEG data obtained through 22 Ag/AgCl electrodes with inter-electrode distances of 3.5 cm. The signals were sampled at 250 Hz, and the data underwent bandpass filtering from 0.5 to 100 Hz, with an additional 50 Hz notch filter activated to suppress line noise. Subjects participated in two sessions on separate days, with the first session allocated for training and the subsequent one for testing. Within each session, there were 288 trials, encompassing 72 trials per task. We used the temporal segment ofseconds in our experiments. Each trial had the shape of a matrix of dimension (22, 1000). https://www.bbci.de/competition/download/competition_iv/BCICIV_2a_gdf.zip↗ , ^{2 6} BCI IV-2b dataset: The IV-2b dataset contains the recording of nine subjects (namely B01-B09), involving left-hand and right-hand MI activities, which is available for download at the following link: []. Three bipolar recordings (C3, Cz, and C4) were recorded with a sampling frequency of 250 Hz. The recorded EEG signals were band-pass filtered from 0.5 to 100 Hz with a notch filter at 50 Hz set at the time of recording using signal acquisition hardware. Each subject participated in five sessions, where the first three sessions were for calibrating an EEG decoder and the rest sessions were for test purposes. There are about 400 trials and 320 trials in the training and test sets, respectively. We used theseconds of each trial in our experiments. Each trial was represented by a matrix with dimensions (3, 1000). https://www.bbci.de/competition/download/competition_iv/BCICIV_2b_gdf.zip↗ , ^{3 7} Preprocessing and Input Representation: Standardization, band-pass filtering and artifact removal are commonly employed methods to preprocess the raw EEG signals. In this study, we only applied the Z-score normalization to preprocess the EEG signals, ensuring a consistent data scale and enhancing the efficiency of model training. An MI-EEG raw trialconsists ofelectrode channels andtime samples. The standardization equation operation can be represented by the following equation:wheredenotes the output of normalization,andrepresent the mean value and the standard deviation of raw trial, respectively. The model's purpose is to associate each MI trial input, denoted as, with its corresponding category. This process involves utilizing a set ofMI trials labeled as, where, andis the total number of defined categories for set. For the BCI IV-2a dataset,= 22 EEG channels,= 1000 time samples,= 4 MI categories, and= 5184 MI trials. For the BCI IV-2b dataset,= 3 EEG channels,= 1000 time samples,= 2 MI categories, and= 6520 MI trials. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{i} \in {\text{R}}^{C \times T}$$\end{document} X i ∈ R C T × \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{i}{\prime} \in {\text{R}}^{C \times T}$$\end{document} X i ′ ∈ R C T × \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu$$\end{document} μ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta$$\end{document} δ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{i}$$\end{document} X i \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{i}{\prime}$$\end{document} X i ′ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S = \left\{ {X_{i}{\prime} ,{ }y_{i} } \right\}_{i = 1}^{m}$$\end{document} S = X i y i ′ , i = 1 m \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${ }y_{i} \in \left\{ {1,{ }...,{ }N} \right\}$$\end{document} y i ∈ 1 , . . . , N C T y m N S C T N m C T N m 1 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{i} ^{\prime} = \frac{{X_{i} - \mu }}{\delta }$$\end{document} X i ′ = X i - μ δ i

The convolutional module in this study draws inspiration from EEGNet. We have designed the convolutional module by decomposing the two-dimensional convolution operator into two one-dimensional components: temporal and spatial convolution. Additionally, we also introduce depth-wise convolution. In contrast to EEGNet, we opt for one-dimensional convolution instead of separable convolution, resulting in improved performance.

The convolutional module consists of three convolutional layers, as shown in Fig. 2. The initial layer conducts a temporal convolution utilizing F₁ filters with a size of (1, K_C1), where K_C1 represents the filter length along the time axis. This operation outputs F₁ feature maps containing the EEG signal at different band-pass frequencies. The value of K_C1 was configured to be one-fourth of the sampling rate (64). This choice enables the filters to capture temporal information associated with frequencies exceeding 4Hz. Subsequently, we employ a depth-wise convolution with F₂ filters of size (C, 1) for spatial filtering, where C represents the number of EEG trial electrode channels (22 for the BCI IV-2a dataset and 3 for the BCI IV-2b dataset). This approach facilitates the learning of spatial filters corresponding to each temporal feature map, thereby efficiently extracting frequency-specific spatial filters. The depth parameter D, empirically set to 2, determines the number of spatial filters learned for each feature map. Hence, the output of the channel depth-wise convolution comprises F₁ × D feature maps. Subsequent to the depth-wise convolution, an average pooling layer with a size of (1, P₁) is employed to extract temporal information, down-sampling the signal's sampling rate by a factor of 8. This results in a reduced signal sampling rate of approximately 32 Hz. The third convolutional layer, a spatial convolution, comprises F₂ filters of size (1, K_C2). We set K_C2 to 16 to decode motor imagery (MI) activities within a 500 ms window for data sampled at approximately 32 Hz. This is followed by a second average pooling layer of size (1, P₂) for dimensionality reduction. The parameter P₂ regulates the EEG sequential features sequence length (token size), which is then fed into the Transformer encoder.

All convolutional layers are succeeded by batch normalization (BN), which enhances the training process and alleviates overfitting. Following the second and third BN layers, exponential linear units (ELUs) are employed as the activation function to introduce non-linearity. The two average pooling layers are then followed by a dropout operation. For subject-specific classification, we set the dropout probability at 0.5 to help prevent overfitting when training with small sample sizes. For cross-subject classification, where the training sets are considerably larger, we reduced the dropout probability to 0.25. In the end, the convolution module generates feature maps \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$S \in {\text{R}}^{{T_{C} \times d}}$$\end{document}S∈RTC×d, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{C}$$\end{document}TC represents the length of the EEG trial's high-level feature representation, calculated as follows.2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{C} = \frac{T}{{8 \times P_{2} }}$$\end{document}TC=T8×P2where T is the time samples of the raw EEG trial. Here, d denotes the number of feature channels, equivalent to F₂, which is set to 16. Subsequently, all feature channels at each time point are fed as a token into the Transformer encoder.

The Transformer network was initially introduced for machine translation, employing an encoder–decoder architecture with stacked MHA and position-wise fully connected feed forward block. Layer normalization (LN) and residual connections are incorporated to enhance the training efficiency and robustness of the model. In our study, focused on classification, we exclusively utilize the encoder block within the Transformer, comprising L layers in depth. Each layer contains two sub-layers: MHA mechanism and feed forward network. The overall architecture of the Transformer encoder is depicted in Fig. 2.

The MHA enables the model to concurrently process and integrate information from various representation sub-spaces at different positions. By executing several SA operations in parallel, with each operation targeting distinct facets of the input, the MHA provides a comprehensive analysis of the input data. The outputs from these individual SA units are then coalesced through a linear transformation. This integration enables the model to effectively discern and encapsulate a broad spectrum of data dependencies, enhancing its representational capability.\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{MHA}}\left( {Q,{ }K,{ }V} \right) = Concat\left( {Z_{1} ,{ }Z_{2} ,{ }...,{ }Z_{h} } \right)W^{O} \in {\text{R}}^{{T_{C} \times d_{k} }} , W^{O} \in {\text{R}}^{{hd_{k} \times d}}$$\end{document}MHAQ,K,V=ConcatZ1,Z2,...,ZhWO∈RTC×dk,WO∈Rhdk×dwhere h denotes the number of heads. As a result of this mechanism within the MHA block, every node in the network acquires a global receptive field, enabling the model to capture and integrate information from across the entire input sequence. After processing through the MHA block, the output features are combined with the original input features S through a residual connection. Subsequently, the model employs LN to standardize each feature. The output of the MHA mechanism can be expressed as:5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O = {\text{LN}}\left( {{\text{MHA}}\left( {Q,{ }K,{ }V} \right) + S} \right)$$\end{document}O=LNMHAQ,K,V+S

The following position-wise fully connected feed forward network is also conducted in a residual network form. It is applied to each position separately and identically. This sub-layer consists of two linear transformations with a Gaussian Error Linear Unit (GELU) activation and a dropout operation in between. The GELU activation function's formula is given by:6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{GELU}}\left( x \right) = x{\Phi }\left( x \right)$$\end{document}GELUx=xΦxwhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\Phi }$$\end{document}Φ(x) is the cumulative distribution function of the standard Gaussian (normal) distribution, which can be expressed as:7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\Phi }\left( x \right) = \frac{1}{2}\left[ {1 + {\text{erf}}\left( {\frac{x}{\sqrt 2 }} \right)} \right]$$\end{document}Φx=121+erfx2where erf(x) denotes the error function, which is a special function integral of the Gaussian distribution. Subsequently, LN is performed. Finally, the sum of the input feature and output feature is used as the output in a residual operation:8\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E = {\text{LN}}\left( {PF\left( O \right) + O} \right)$$\end{document}E=LNPFO+Owhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$PF$$\end{document}PF denotes the position-wise feed forward operation.

In the classification block, the convolutional module and Transformer encoder's output features are added, enabling direct transmission of features extracted by the CNN to the classifier, and then flattened. Following this, a dropout operation is applied to mitigate overfitting and enhance generalization, with a dropout probability of 0.5. Finally, these processed features are fed into a fully connected layer comprising N units, where N represents the number of categories in the MI EEG classification task. Cross-entropy is used as the loss function for the entire model, as follows:9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal{L}} = - \frac{1}{M}\mathop \sum \limits_{i}^{M} \mathop \sum \limits_{j}^{N} y_{ij} {\text{log}}\left( {\hat{y}_{ij} } \right)$$\end{document}L=-1M∑iM∑jNyijlogy^ijwhere M is the number of the EEG trials, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y_{ij}$$\end{document}yij is the true label for the j-th class in the i-th sample, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}_{ij}$$\end{document}y^ij is the predicted probability for the j-th class in the i-th sample.

To ensure a thorough evaluation, four widely recognized metrics are employed: accuracy, Cohen's Kappa, Hedges’ g and Wilcoxon p-value. Accuracy is calculated as:10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Accuracy\left( {Acc} \right) = { }\frac{TP + TN}{{TP + TN + FP + FN}}$$\end{document}AccuracyAcc=TP+TNTP+TN+FP+FNwhere TP and TN represent the correct positive sample number and the correct negative sample number predicted by the model, respectively, and FP and FN denote the false positive sample number and the false negative sample number predicted by the model, respectively. Cohen’s Kappa is calculated as follows:11\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\kappa { } = { }\frac{{P_{o} - P_{e} }}{{1 - P_{e} }}$$\end{document}κ=Po-Pe1-Pewhere \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{o}$$\end{document}Po denotes the accuracy of the model, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{e}$$\end{document}Pe represents the probability or accuracy of a random guess.

To assess the effect size of the Transformer module and data augmentation operation on model performance, we use Hedges' g. This metric is particularly suitable for small sample sizes and provides a more accurate estimate of effect size by correcting Cohen's d for sample size. Hedges' g is calculated as follows ⁶⁰:12\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g = d \times \left( {1 - \frac{3}{{4\left( {n_{1} + n_{2} } \right) - 9}}} \right)$$\end{document}g=d×1-34n1+n2-9where d (Cohen's d) is calculated as:\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d = \frac{{\overline{X}_{1} - \overline{X}_{2} }}{{\sqrt {\frac{{\left( {n_{1} - 1} \right)s_{1}^{2} + \left( {n_{2} - 1} \right)s_{2}^{2} }}{{n_{1} + n_{2} - 2}}} }}$$\end{document}d=X¯1-X¯2n1-1s12+n2-1s22n1+n2-2where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{X}_{1}$$\end{document}X¯1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{X}_{2}$$\end{document}X¯2 are the means of the two groups being compared (e.g., with and without the Transformer module), s₁ and s₂ are the standard deviations of the two groups, and n₁ and n₂ are the sample sizes of the two groups, respectively. In this study, n₁ and n₂ are both equal to 9. Specifically, g around 0.2 suggests a small but potentially meaningful impact. g around 0.5 denotes a moderate impact that is likely to be of practical significance. g around 0.8 or higher indicates a large impact, which is very likely to be of substantial practical significance. These benchmarks help in interpreting the magnitude and importance of the effects in the context of experimental findings.

Additionally, we utilize the p-value derived from the paired Wilcoxon Signed-Rank Test to assess the statistical significance of differences between the proposed model and other state-of-the-art approaches. In this context, a p-value > 0.05 indicates the absence of a statistically significant difference. Conversely, a p-value < 0.05 (denoted as ‘*’) signifies a notable statistical difference, and a p-value < 0.01 (denoted as ‘**’) indicates a highly significant statistical difference.

The training of our models was conducted on an Nvidia RTX3090 with 24 GB memory GPU. We utilized PyTorch, an open-source deep learning framework, on a workstation equipped with the Debian 11 operating system and an Intel Core i9-9820X CPU. Our analysis focused exclusively on EEG-channel data, and we chose to directly discard the three electrooculography (EOG) channels without engaging in any artifact removal procedures. The proposed model is evaluated using subject-specific and cross-subject. We have reserved 30% of the initial training set to function as a validation set. The model that demonstrated the minimum loss on the validation set was selected.

For cross-subject evaluation, we employ the LOSO method. In this approach, we sequentially select one subject from the total of nine as the test subject, while aggregating the EEG data from the remaining subjects to form the training dataset. This process is repeated for each subject, thereby ensuring that the model is trained and evaluated on diverse subsets of data. The learning rate, batch size, and number of training epochs have been configured to 0.001, 512, and 600, respectively. The dropout rate p₁ is set to 0.25.

In our study, we performed comprehensive subject-specific and cross-subject experiments and compared our method against several state-of-the-art approaches on the BCI IV-2a and IV-2b datasets. To ensure a relatively fair comparison, we reimplemented 4 prominent models (ShallowConvNet, DeepConvNet, EEGNet, and Conformer) based on their open-source code, maintaining uniform experimental conditions including data preprocessing, training and validation set splits, and data augmentation strategies. For these models, hyperparameters were configured to align with the specifics provided in their respective papers, except for the learning rate, batch size, and training epochs. These parameters were standardized across all models, including CTNet, to facilitate a fairer comparison.

For the BCI IV-2a dataset, ablation experiments are depicted in the upper subplot of Fig. 5. It is evident from Fig. 5 that for the majority of subjects, removing either the Transformer module, data augmentation, or both, significantly diminishes the recognition accuracy. Specifically, excluding the Transformer alone resulted in a noticeable decrease in recognition accuracy for most subjects (A01-A05 and A09), with a particularly pronounced drop of 9.37% for subject A04; however, participants A07 and A08 did not experience any change in accuracy, and subject A06 actually showed a slight increase of 0.69%. Overall, the removal of the Transformer module led to an average notable reduction in model accuracy by 1.77% (p < 0.05, g = 0.179). The effect size for the Transformer module, as measured by Hedges' g, is 0.179. This indicates a small effect size, suggesting that the inclusion of the Transformer module has a positive impact on model performance.

When data augmentation was removed alone, all subjects except A06 exhibited a marked decrease in accuracy, averaging a significant decline of 7.21% (p < 0.01, g = 0.758). Given that Transformer models typically require substantial amounts of training data, this result indicates a medium to large effect size. This underscores the critical role of data augmentation in maximizing the performance benefits of the Transformer module, which typically requires substantial training data. Simultaneously removing both the Transformer and data augmentation resulted in an average accuracy reduction of 6.55% across all participants (p < 0.01, g = 0.595), highlighting the significant contributions of both components to recognition accuracy.

Additionally, we observed that in the absence of data augmentation, due to the limited training data, incorporating the Transformer module could actually decrease the recognition accuracy for four subjects (especially a 7.29% drop for A07), with an average decline of 0.66%.

For the BCI IV-2b dataset, as shown in the lower subplot of Fig. 5, the impact of removing the Transformer module mirrors that observed in the BCI IV-2a dataset, with a notable decrease in recognition accuracy across most participants, averaging a reduction of 1.79% (p < 0.05, g = 0.184). Removal of either data augmentation or both modules did not reduce accuracy as drastically as in the BCI IV-2a dataset, with average decreases of 1.33% (g = 0.143) and 1.32% (p < 0.05, g = 0.139) respectively. Furthermore, without data augmentation, the inclusion of the Transformer does not enhance the model's average recognition accuracy.

In this section, we meticulously assess the influence of various critical hyper-parameters on the performance of the model in subject-specific classification. These parameters encompass the length of the high-level EEG features (token size) inputted into the Transformer module, the number of self-attention heads, and the depth of the Transformer encoder architecture.

Conformer and MI-CAT are exemplary models for decoding MI-EEG, utilizing a hybrid architecture that combines CNN and Transformer technologies. In 2023, combining the local feature extraction capabilities of ShallowConvNet with the global modeling strength of the Transformer, Song et al. proposed the Conformer model. On the BCI IV-2a and IV-2b datasets, the average recognition accuracies of the Conformer model were 1.97% and 0.74% higher than those of ShallowConvNet. This also demonstrates that incorporating a Transformer to globally model the high-level features extracted by CNNs can enhance the model's ability to recognize MI-EEG signals. Correspondingly, the model's trainable parameter number also increased by approximately 0.12 million. Our CTNet model, inspired by both Conformer and EEGNet, is designed to achieve high recognition accuracy while maintaining a smaller trainable parameter number, thus reducing overfitting and enhancing the model's generalization capability. In 2023, Zhang and colleagues proposed the MI-CAT model to address the inter-subject variability of EEG signals. MI-CAT employs a temporal-spatial CNN to learn feature representations from paired EEG data, followed by two domain-related attention blocks that preserve domain-dependent information. It then utilizes the Transformer’s self-attention and cross-attention mechanisms to facilitate feature interaction and resolve differential distributions across different domains. Additionally, MI-CAT uses bandpass filtering (BF) and exponential moving standardization (EMS) for data preprocessing. Without data augmentation, MI-CAT achieved remarkable average recognition accuracies of 76.81% and 85.28% on the BCI IV-2a and IV-2b datasets, respectively, with Kappa values of 0.692 and 0.706. In comparison with the CTNet model, which did not use data augmentation, MI-CAT exhibited a 1.50% higher average recognition accuracy on the BCI IV-2a dataset, while CTNet performed 1.88% better on the BCI IV-2b dataset. This shows that the recognition accuracy of CTNet and MI-CAT models is comparable. However, MI-CAT has over 55,000 more trainable parameters than CTNet.

In summary, compared to state-of-the-art methods such as ShallowConvNet, DeepConvNet, EEGNet, TSF-STAN, Conformer, and MI-CAT, the CTNet model is relatively small yet achieves comparable decoding accuracy to the TSF-STAN method on both the BCI IV-2a and IV-2b datasets. Specifically, CTNet's accuracy is higher than other state-of-the-art methods by 4.74% to 6.83% on the BCI IV-2a dataset and by 0.78% to 3.36% on the BCI IV-2b dataset. Notably, while TSF-STAN utilizes a complex TSF data preprocessing method, CTNet employs a simple standardization process, greatly simplifying the preprocessing pipeline. TSF-STAN's complex TSF preprocessing requires substantial computation, whereas CTNet's straightforward standardization process reduces computational complexity and resource demands. Achieving high accuracy without sacrificing performance highlights the practical advantages of our approach. In practical applications, reducing computational complexity can lead to shorter processing times, lower power consumption, and easier deployment, particularly in resource-constrained environments. Therefore, achieving accuracy comparable to the state-of-the-art TSF-STAN, combined with our simpler preprocessing pipeline, underscores the practical significance of our method.

In cross-subject MI-EEG decoding comparative analysis, we utilized the LOSO cross-validation method under consistent experimental conditions to compare CTNet with ShallowConvNet, DeepConvNet, EEGNet, and Conformer. On the BCI IV-2a dataset, the recognition accuracies of Conformer, ShallowConvNet, EEGNet, CTNet, and DeepConvNet progressively improved. DeepConvNet, the best-performing model, achieved a recognition accuracy 1.51% higher than the second-best, CTNet. On the BCI IV-2b dataset, accuracies improved sequentially for Conformer, ShallowConvNet, EEGNet, DeepConvNet, and CTNet, with the top-performing CTNet surpassing the second-best best, DeepConvNet, by 1.09%. These results clearly demonstrate CTNet's effectiveness in solving cross-subject MI-EEG decoding challenges. Additionally, Chowdhury et al. introduced EEGNet Fusion V2³¹, which integrates five distinct branches of EEGNet with varying hyperparameters, achieving average recognition accuracies of 74.3% and 84.1% in cross-subject MI-EEG decoding on the BCI IV-2a and IV-2b datasets, respectively. The fusion strategy of EEGNet Fusion V2 potentially offers richer feature representations and decision boundaries, mitigating the risk of overfitting or underfitting through an ensemble learning approach. Unlike CTNet, which employs a LOSO cross-validation strategy widely used in BCI research, EEGNet Fusion V2 utilizes a session-based division strategy: one session for training and another for testing on the BCI IV-2a dataset, and three sessions for training with the remaining two for testing on the BCI IV-2b dataset. This strategy likely reduces differences between training and testing data, as data from the same subject across different sessions tend to be more similar. This session-based approach might better adapt to individual subject characteristics, thereby enhancing accuracy during testing. Consequently, EEGNet Fusion V2 significantly outperforms CTNet in terms of accuracy, drawing attention to the merits of utilizing multiple branches with unique hyperparameters for broader feature extraction from EEG data, given the significant variability among different subjects.

For cross-subject MI-EEG decoding, Keutayeva and Abibullaev proposed the HTCV model⁵⁵ and later the st-CViT model⁵⁶. Both models employed S&R data augmentation strategies and LOSO cross-validation methods. HTCV is tested on the BCI IV-2a and BCI IV-2b datasets for decoding left and right hand movements, while st-CViT is evaluated on the BCI IV-2a, BCI IV-2b, Weibo, and Physionet datasets. Additionally, reference⁵⁶ also tests the performance of st-CViT using the nested LOSO strategy, which enhances the reliability of their model evaluation.

Ablation studies demonstrate that in the task of decoding subject-specific MI-EEG, the S&R data augmentation method contributes to improved accuracy on both datasets, with a significant impact observed on the BCI IV-2a dataset. Furthermore, the removal of the Transformer module in the presence of data augmentation resulted in a notable decrease in recognition accuracy by 1.77% (p < 0.05) on the BCI IV-2a dataset and 1.79% (p < 0.05) on the BCI IV-2b dataset. This suggests that the Transformer module plays a notable role in harnessing the enriched data environment provided by augmentation techniques. The Transformer’s self-attention mechanism has the unique capability to process the entire input features sequence simultaneously, allowing it to capture global dependencies that are vital for understanding complex EEG patterns. This capability allows for a more nuanced understanding of EEG signals, particularly beneficial for tasks like MI where temporal dynamics are crucial. Furthermore, the global processing is particularly beneficial in datasets enriched through data augmentation, as it helps the model to generalize better across varied yet synthetically expanded data.

The effect size analysis provides further insights into the contributions of the Transformer module and data augmentation. For the BCI IV-2a and IV-2b datasets, when data augmentation was applied, adding the Transformer module resulted in effect sizes (Hedges' g) of 0.179 and 0.184, respectively. This suggests a small but positive impact of the Transformer module when data augmentation is utilized, highlighting its ability to enhance model performance by capturing global dependencies in the data. Conversely, when the model did not use data augmentation, adding the Transformer module resulted in effect sizes of − 0.063 and − 0.002 for the BCI IV-2a and BCI IV-2b datasets, respectively. These negative or near-zero effect sizes indicate that the Transformer module alone does not improve, and may even slightly detract from, model performance without data augmentation. These findings also indicate that combining CNN with a Transformer, especially without data augmentation, leads to a decrease in recognition accuracy. This decline is likely due to the introduction of the Transformer module, which increases the model's trainable parameters more than fourfold, thereby exacerbating issues of overfitting. Transformers are equipped with a large number of parameters and layers that are advantageous for capturing complex patterns in extensive datasets but can lead to overfitting when the training data is scarce. Under these conditions, the model may start memorizing noise and specific details of the training set instead of generalizing from it. Lacking sufficient data, the Transformer’s advanced mechanisms, such as multi-head attention, are not fully leveraged. This scenario results in a model that is overly complex for the available data volume, consequently underperforming. These findings align with the results from Keutayeva's research^55,56.

For the BCI IV-2a and IV-2b datasets, when the Transformer module was removed, using data augmentation resulted in effect sizes of 0.441 and -0.049 for the BCI IV-2a and BCI IV-2b datasets, respectively. This shows that data augmentation alone can have a substantial positive effect in the BCI IV-2a dataset but may have a slightly negative impact in the BCI IV-2b dataset without the Transformer. When the Transformer module was used, adding data augmentation resulted in effect sizes of 0.758 and 0.143, respectively. These results underscore the importance of the Transformer module in improving the model's performance. Its ability to capture global dependencies in the data, especially when combined with data augmentation, significantly boosts the model's effectiveness, particularly in datasets where data augmentation alone may not be sufficient.

For the BCI IV-2a and IV-2b datasets, when the model simultaneously uses both the Transformer and data augmentation, the effect sizes are 0.595 and 0.139, respectively. These results indicate that the combined use of the Transformer and data augmentation significantly enhances model performance on the BCI IV-2a dataset, while the effect is more modest on the BCI IV-2b dataset.

Overall, these results underscore the importance of the Transformer module in enhancing model performance. While data augmentation provides substantial benefits, the Transformer's advanced mechanisms, such as self-attention, are essential for fully leveraging the enriched data environment and capturing complex temporal dependencies in EEG signals. The Transformer's effectiveness is particularly pronounced when combined with data augmentation, as it significantly boosts the model's ability to generalize from enriched data.

We investigated three critical hyperparameters of the CTNet model: token size, the number of heads in the MHA, and the depth of the Transformer module. The CTNet model is sensitive to the settings of these parameters. Our findings suggest that smaller tokens effectively reduce local noise, which facilitates the learning of global features. When decoding EEG signals, capturing the spatial distribution of brainwaves is crucial. Each "head" in a Transformer can be viewed as an independent feature detector, focusing on different dimensions of information. CTNet performs best with a two-head attention mechanism. Two heads represent an optimal balance, sufficient to capture the essential spectral characteristics ofrhythm (8-13Hz) and(13-30Hz), while a higher number of heads could exceed the processing needs required for the complexity of MI-EEG signals, potentially reducing overall efficiency and effectiveness. Furthermore, CTNet achieves optimal recognition performance with a Transformer encoder of depth 6, mirroring findings from Conformer studies. Feature visualization further affirmed that the Transformer encoder facilitates learning more discriminative features than those extracted without the Transformer. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\upmu$$\end{document} μ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\upbeta$$\end{document} β \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text{rhythm}$$\end{document} rhythm

Although CTNet has demonstrated superior performance in both subject-specific and cross-subject MI-EEG decoding across two datasets, outperforming several advanced methods in terms of recognition accuracy, it still faces certain limitations. Firstly, there is significant room for improvement in CTNet’s recognition accuracy, especially in cross-subject MI-EEG decoding tasks. Secondly, CTNet appears sensitive to specific hyperparameters such as token size, the number of heads in the MHA, and the depth of the Transformer module. This sensitivity might necessitate extensive hyperparameter tuning to achieve optimal performance, which can be time-consuming and computationally demanding. Additionally, the S&R data augmentation strategy does not significantly contribute to the recognition accuracy of subject-specific MI-EEG decoding on the BCI IV-2b dataset. Moving forward, we plan to explore regularization strategies specifically aimed at addressing cross-subject variability, which may enhance the model’s recognition performance in cross-subject MI-EEG decoding. To address the issue of hyperparameter sensitivity, we will attempt to automate the search for optimal hyperparameter combinations using reinforcement learning-based methods. Furthermore, we also intend to explore the use of generative adversarial networks (GANs) to enhance the training dataset.