Using virtual electrodes and neural networks to identify features in brain signals during imagined movement

The Wadsworth research group in the United States pioneered MI research with the release of BCI2000, an open-source software platform that allows the design, implementation, evaluation, and verification of BCI systems, which has greatly contributed to the development of BCI technology [25,26]. The BCI research team of Graz University of Technology in Austria was the first to discover the ERD/ERS phenomenon in EEG and conduct BCI research to enable patients with cervical spinal cord injuries to control artificial neural prostheses and restore upper limb motor functions such as independent contact and grasping, and then, based on this, developed an MI-BCI system based on the left and right hands with a recognition accuracy of 82.5~90% [27,28].

In recent years, BCI groups have also carried out research on the identification of complex MI. In 2013, Garcia et al. combined MI and myoelectricity to control a robotic arm with six degrees of freedom [29]. In 2015, He et al. mapped the sensorimotor rhythm EEG backwards to the cortical source domain to improve the performance of the BCI system, allowing it to communicate with the outside world as an alternative to physiological neural pathways [30]. In 2015, Leeb et al. developed a BCI system to improve the independence of patients with dyskinesia by using a network shared control technique, and verified the feasibility of the system through the comparative performance experiment of nine patients and ten healthy individuals [31]. In 2019, Bartur G et al. studied the EEG of 14 patients with their first cerebral pawn, and found that the ERD amplitude of EEG was significantly correlated with the residual motor function of the upper limb in patients with mild paralysis [32]. Domestically, the group of Shangkai Gao and Xiaorong Gao from Tsinghua University is one of the earlier research teams of MI-BCI. They have achieved good results in the international BCI Competition many times, and have also developed several BCI systems. They designed a hybrid BCI spelling system that combines visual evoked potentials and MI potentials at the moment of motion onset. Subjects were trained to better master the manipulation of the character input system, and input accuracy could be as high as 93.3% [33].

In recent years, research on MI-EEG based on the network structure has also become a hot topic. Ma et al. proposed a novel temporal dependency learning convolutional neural network with an attention mechanism. The experimental results on the BCI Competition IV-2a (BCIC-IV-2a) and OpenBMI datasets show that the network outperforms the state-of-the-art algorithms and achieves the average accuracy of 79.48%, improved by 2.30% on the BCIC-IV-2a dataset. The network demonstrates that learning temporal dependencies effectively improves the MI EEG decoding performance [34]. Wang et al. proposed an MI-EEG classification method designed to improve the classification accuracy by combining Shannon complex wavelets and convolutional neural networks. BCI competition IV dataset 2b as a public motor imagination dataset was tested to prove the validation of this proposed method. The classification accuracy and kappa value were adopted to prove the superiority of the proposed method by comparing it with the state-of-the-art classification methods [35]. Through the review paper by Hamdi Altaheri et al., a lot of relevant information can also be learned, including discussions on three issues: Is preprocessing required for DL-based techniques? What input formulations are best for DL-based techniques? What are the current trends in DL-based techniques? And then, the current challenges and future directions are discussed [36].

MI-BCI technology has a wide range of applications in many fields. In rehabilitation medicine, MI-BCIs can be used to help restore motor function to patients with impaired function [37], such as stroke, spinal cord injury, or prosthetic subjects. By training them to use motor imagery to control external devices, patients can regain some degree of mobility and improve their quality of life [38]. In addition, MI-BCI can also be applied to fields such as virtual reality, human–computer interactions, and brain-controlled interfaces, providing subjects with a more natural and intuitive way of interacting [39,40]. Subjects can use imagination to control the actions of virtual characters, interact with computer interfaces, or participate in immersive gaming experiences. With the continuous advancement of technology, MI-BCI is expected to play an important role in medical rehabilitation, human–computer interactions, and other fields [41,42,43].

At present, many excellent MI-EEG feature extraction and classification theories and methods have emerged and achieved good results every year. However, from the practical application of MI-BCI, there are still many areas that need improvement. The main problems are as follows:

(i) Large individual differences. The MI-BCI system involves placing EEG electrodes on the cerebral scalp to obtain a signal. The differences in the brain structure and conductivity, electrode position errors, noise, and electromagnetic interference during the collection process often lead to significant differences between individual subjects. Even the same subject may have significant differences in their physiological and psychological states at different times. How to improve the generalization performance of the MI-EEG signal feature extraction and classification algorithm, enhance the adaptability and robustness to individual differences, and solve the problem of the poor performance of the cross-subject classification model has become one of the most challenging problems in current MI-BCI research.

(ii) Low classification accuracy. The MI-EEG signal is weak and has a low signal noise ratio (SNR). The collected original signal contains many noise-induced interference signals, that is, artifact signals, which are mainly noise interference generated by the external environment and other physiological activities of the human body. The low SNR of MI-EEG signals can lead to a low classification accuracy. In current studies, MI-EEG decoding paradigms are mostly limited to 2 to 4 classifications of hands and feet. For the two-classification problem of MI-EEG, many methods have achieved very good classification results, but for the multi-classification problem, the effect is not ideal. The more classification types, the lower the classification accuracy. How to improve the classification accuracy of multi-task MI-EEG remains to be studied further.

(iii) Poor online performance of the MI-BCI system. In addition to the low classification accuracy and large individual differences, the efficiency of the signal processing, data volume, and complexity of the algorithm are related to the poor online performance of the MI-BCI system. In order to achieve effective control, it is urgent to improve the decoding ability and fast response of MI-EEG. However, few existing algorithms can solve this problem.

An MI-EEG signal is a combined representation on the scalp surface of the electric potential and magnetic field generated by the electrophysiological activity of nerve cells inside the brain. It is usually measured using electrodes as sensors, which are fixed on the scalp surface and has a non-Euclidean spatial structure [44,45]. Currently, most MI-EEG classification algorithms are studied based on the data collected from various channels and identify and classify data according to one-dimensional or two-dimensional Euclidean structures, which may not fully reflect the spatial information of all MI-EEG channels [46]. To solve problems such as the large individual differences between subjects, low classification accuracy, and poor online performance of the MI-BCI system, it is crucial to fully utilize the spatial information of all MI-EEG channels and efficiently extract the motor intention from the MI-EEG signals [47].

In this paper, we take the spatial information of MI-EEG channels as a breakthrough and propose a combined virtual electrode-based ESA and CNN method for MI-EEG signal feature extraction and classification, which is used to explore the information about the electrophysiological activities of nerve cells inside the brain. The rest of the paper is organized as follows:introduces data acquisition and preprocessing, ESA, feature extraction, and CNN classification. The results of the feature extraction, classification, and MI-BCI system experiments are presented in detail in. In, the impact of noise cancellation, training duration, and individual differences on the MI-BCI system was discussed, and the article was concluded with a summary. Section 2 Section 3 Section 4

The main contributions are summarized as follows.

(i) The Boundary Elements Method (BEM) was used to solve the forward problem; the Low-Resolution Electromagnetic Tomography algorithm (LORETA) was used to constrain the inverse problem and construct the source model. The MI-EEG signals collected by electrodes were projected onto the cortex through EEG Source Analysis (ESA), and then virtual electrodes were constructed to obtain the source signals. Then, a Morlet wavelet was used for Joint Time Frequency Analysis (JTFA) to extract two-dimensional joint time frequency characteristics. In this way, the non-Euclidean spatial information of virtual electrodes can be converted into two-dimensional planar Euclidean information.

(ii) In the recognition and classification of the two-dimensional Euclidean structure data, CNN has the advantage of extracting multi-scale local features and combining them to improve the classification performance. In this paper, we proposed a combined virtual electrode-based ESA and CNN method for MI-EEG signal feature extraction and classification. The results indicate that the online MI-BCI system developed based on this method can improve the decoding ability of multi-task MI-EEG after training. At the same time, in interdisciplinary experiments, it can learn the generalized features of multiple subjects and has certain adaptability to the individual differences of new subjects, providing a new approach for the research of online MI-BCI systems.

The overall design framework of MI-BCI system is shown in Figure 1. The system mainly consists of EEG acquisition system, preprocessing, ESA, feature extraction, CNN classification, control strategy, wireless Bluetooth transmission, and a smart car system. Its working flow is as follows:

The EEG acquisition system mainly consists of the Emotiv Epoc+ wireless portable electroencephalograph and supporting software, which adopts non-invasive EEG technology with built-in amplifier, filter, and analog-to-digital converter modules. It has 16 electrodes, placed according to the 10–20 international standard lead system, including 2 reference electrodes (P3, P4) and 14 measurement electrodes (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4). By default, the sampling frequency is set to 128 Hz and can be changed as needed. MI-EEG signal acquisition interface is shown in Figure 2, which can realize real-time display of EEG data stream. There are two data-receiving modes, wireless Bluetooth and USB, and the acquired MI-EEG signal can be converted into digital signal form and stored in the computer.

When preprocessing MI-EEG signals, using an 8–30 Hz bandpass filter can preserve information closely related to MI. This can remove artifacts caused by external noise and internal biological noise while extracting effective frequency band information. During signal acquisition, the inevitable blinking of the subject can cause interference from electro-oculographic artifacts, which are evident in the prefrontal region of the brain and can be removed using independent component analysis (ICA). Figure 3 shows 2D (2-dimensional) topographic map of the MI-EEG signals under ICA. The color change from blue to yellow to red in the figure is the process of energy transition from weak to strong. Due to the small number of electrodes in this device and the small number of fractions formed, the 2D brain topography has certain limitations, which can be further analyzed in combination with the 3D (3-dimensional) brain topography again, as shown in Figure 4. Combining the 2D and 3D brain topography, it can be determined that components 5 and 12 are higher in energy in the prefrontal region only, with significant electrical ophthalmic interference, and these components can be excluded from the MI-EEG.

Due to the accumulation of bioelectric signals generated by the interactions between neurons within the brain in the scalp layer, and the fact that source activity within the brain can only be obtained through derivation, the spatial characteristics of EEG signals are relatively limited [48]. The conduction relationship between the source activity inside the brain and EEG signals is extremely complex. The EEG source analysis (ESA) mainly includes the Forward problem and the Inverse problem. The Inverse problem is a process of deriving the source signal inside the brain using EEG signals collected on the scalp surface. To solve the Inverse problem, it is necessary to first solve the Forward problem, which is the process in which the source signals are attenuated by layers of cortex, skull, and scalp, and finally transmitted to the limited electrodes on the scalp surface. Solving the Forward problem relies on the creation of the head model and it also constrains the location of the source when solving the Inverse problem. In general, the relationship between source activity inside the brain and EEG signals collected from the scalp layer can be analyzed using a linear model [49].

Electrodes of non-Euclidean spatial structure record the combination of electrical signals transmitted to the scalp from a source composed of numerous clusters of neurons inside the brain [50]. To precisely decode the source activity inside the brain, MI-EEG signals needs to be traced back into the brain via ESA. The specific steps can be described as follows:

Firstly, the data of the head model are imported, and the three-layer (scalp, skull, and cortex) head volume conduction model is constructed to determine the direction, position, and intensity of the dipole inside the brain. In this paper, the direction of the dipole is set perpendicular to the surface of the cortex. Then, BEM algorithm is used to solve the forward problem, the mathematical model of the brain electric field is established, and the expression for the scalp potential of the dipole in the distributed source model is constructed. The parameters of the dipole are adjusted to fit the scalp potential calculated by the forward problem to the MI-EEG data. Based on the fitting results, a source model is generated and the LORETA algorithm is used to constrain the EEG inverse problem. Figure 5 shows the schematic diagram of head model and source model. Finally, 18 virtual electrodes closely related to the MI task are identified in the cortex region of the source model. The range of each virtual electrode is extended to 20 vertices, each with a constrained dipole, which is considered as the source. Figure 6 shows the position of the selected virtual electrodes and the extracted time waveform information from the two virtual electrodes.

The virtual electrodes of the left and right hemispheres are LP1–LP9 and RP1–RP9, respectively, forming 9 virtual electrode pairs P1–P9. JTFA is performed on the MI signal of the virtual electrode to extract the joint time–frequency feature information of 5 s. The transverse coordinate of 2D data is time, and the transverse dimension is 640 at the sampling frequency of 128 Hz. The longitudinal coordinate is frequency, band of interest of 8–30 Hz extracted by band-pass filter, and the longitudinal dimension is 23. Therefore, the MI time–frequency feature size of a virtual electrode is 640 × 23, and then the data of the symmetric virtual electrodes of the left and right hemispheres are stitched longitudinally with a size of 640 × 46. The time–frequency map size of 640 × 46 × 9 for 9 pairs of virtual electrodes is used as the input to the CNN model.

CNN uses a series of filtering, normalization processes, and nonlinear activation functions to extract features from large training datasets, which is more suitable for complex MI-EEG signal processing [56,57,58,59]. The basic components of CNN include convolution layer, pooling layer, and fully connected layer.

The function of the convolution layer is to extract signal features, and the convolution operation mainly adopts Sparse Connectivity and Shared Weights, which can greatly reduce the number of CNN parameters and accelerate the training speed [60,61,62]. Pooling layer can reduce the data dimension and the complexity of network computing, accelerating the computing speed [63,64]. After feature extraction, FC layer is used to perceive global information and complete classification. Adaptive Moment Estimation (Adam) is adopted as the optimizer to minimize the loss function. In addition, weight and bias are updated through the back propagation algorithm, and the learning rate of the optimizer is 1 × 10⁻⁵ [65].

In this paper, a deep CNN classification model is constructed based on the self-collected 6-classification MI-EEG datasets, and the parameters are shown in Table 1. A 4-layer CNN is used to learn MI-EEG feature information, the activation function selects Leaky ReLU function, the 4-layer Pool uses max pooling to reduce dimensions, then the data are flattened by Flatten layer to get 1D data, and finally the FC layer uses the softmax function to achieve classification prediction. Each layer is convolved with Batch Normalization (BN) and 50% dropout to reduce the risk of network overfitting, and Adam with learning rate 1 × 10⁻⁵ is used as the optimizer to minimize the loss value.

An MI-EEG signal is a very weak bioelectrical signal that is susceptible to external interference (such as power frequency interference, electrostatic interference, and electromagnetic interference). It will also mix with other physiological signals of the subject, such as electromyography and electrocardiogram signals, and even contain many components with unknown physical meanings. In this study, the raw MI-EEG signals collected by the Emotiv Epoc+ wireless portable electroencephalograph also contained interference from external and subject-specific factors. Before the signal feature extraction and classification, 8–30 Hz band-pass filtering and 2D and 3D brain topographic map analysis are required for the raw MI-EEG signals.

Figure 9 is a set of MI-EEG signal waveforms of subject S1 during task T2 (right fist). Figure 9a is the raw MI-EEG signal waveform, and Figure 9b is the preprocessed MI-EEG signal waveform. Where the vertical axis represents 14 collecting electrodes and the horizontal axis represents time. The comparison shows that the preprocessed MI-EEG signal waveform image is clearer and more concise, and the interference of high-frequency noise is reduced, which improves the signal-to-noise ratio to a certain extent.

After the ESA calculation of 14 channels of MI-EEG, nine pairs of virtual electrodes were constructed on the cortex to obtain the source signals. A Morlet wavelet was performed to extract the joint time–frequency feature information. A set of time–frequency feature maps of subject S1 in task T2 (right fist) is shown in Figure 10, including the time–frequency data of nine pairs of symmetrical virtual electrodes in the left and right brain. The horizontal axis of each time–frequency map is the time range of 0~5 s, and the vertical axis is the frequency range of 8~30 Hz. The graph shows the intensity of the MI-EEG signal at different times and frequencies, with the color tending to blue representing lower energy and the color tending to red representing higher energy. The two areas of the cortex most associated with MI (somatomotor and somatosensory areas) are controlled and sensed by the contralateral torso, i.e., the left side of the brain corresponds to the right half of the torso, while the right side of the brain corresponds to the left half of the torso. The comparison of the left and right brain time–frequency maps shows that the energy of the left brain is stronger than that of the right brain when subject S1 performs the right fist MI.

In this paper, 10 groups of single-subject experiments were conducted, and 900 samples were collected from each subject and divided into five equal parts. The training set (540 samples), validation set (180 samples), and test set (180 samples) were divided in the ratio of 3:1:1. All samples in the training and validation sets were sent to the CNN network through uniform normalization. Afterwards, the model was trained based on the training set and the optimal parameters were selected on the validation set to achieve stable convergence and obtain the optimal model. Then the test set was fed into the optimal model to obtain the test results, and the average of the five results was used as the classification result. This can reduce the randomness caused by data partitioning and help improve the stability of the model.

Figure 11 shows the loss curves of 10 individual subjects obtained by the CNN model during training. Their loss values decrease continuously with the increase of the number of iterations, and reach the minimum value and remain basically stable at about 800 iterations, so as to obtain the optimal training effect. It can be seen from the figure that the CNN model trained on the data of 10 individual subjects is convergent.

In this paper, the classification performance of the BCI system on the self-collected data set of individual subjects is measured by the common evaluation metrics of accuracy, precision, recall, and the F1-score, as shown in Table 2. As can be seen from the table, subject S1 has the best test performance and S8 has the worst. The mean values of accuracy, precision, recall, and the F1-score for the 10 subjects are 81.08%, 82.77%, 92.07%, and 87.13%, respectively. The results verified that the BCI system achieved a good classification effect on the self-collected data sets of 10 subjects. Subjects with classification accuracy, from high to low, were S1, S7, S10, S3, S6, S4, S9, S2, S5, and S8, and the top six subjects all achieved more than 80% accuracy, and they were selected to participate in the next session of the testing experiment.

Figure 12 shows the accuracy histograms of six MI tasks for 10 subjects, with the highest accuracy of 90.75% (S1) and the lowest of 71.25% (S8) for T1; the highest accuracy of 93.23% (S1) and the lowest of 71.58% (S8) for T2; the highest accuracy of 87.60% (S7) and the lowest of 71.14% (S8) for T3; the accuracy ranged from a high of 92.91% (S1) to a low of 71.45% (S8) for T4; T5 had a high of 83.29% (S1) and a low of 66.85% (S8); and T6 had a high of 84.70% (S1) and a low of 66.92% (S8). The average accuracy of the six MI tasks was 84.41% (T1), 85.59% (T2), 82.67% (T3), 83.75% (T4), 74.01% (T5), and 76.05% (T6), ranked as T2 (right fist) > T1 (left fist) > T4 (right foot) > T3 (left foot) > T6 (right thumb) > T5 (left thumb). The best MI task for classification is the right fist, and the worst is the left thumb.

Figure 13 shows the ROC curve and AUC value of a single subject obtained when the CNN model was tested, from which it can be seen that the average AUC value for the 10 subjects was 0.945; S1 had the largest AUC value of 0.963 and S8 had the smallest AUC value of 0.920. Then, the CNN model achieved a good classification performance on the self-collected dataset of 10 subjects, S1 had the best classification effect after training, and S8 had the worst effect.

In summary, the six-classification CNN model constructed and trained by the method proposed in this paper achieved high classification accuracy on the self-collected datasets of 10 subjects, indicating that the method has a good generalization performance and can effectively complete the task of EEG intention classification.

After system debugging, the hardware and software of the MI-BCI are connected properly and the communication is in good condition. The control strategy module will continuously accumulate the classification results output by the CNN model, and convert the results into corresponding control instructions only when they reach the set threshold, which will be transmitted wirelessly to the smart car through the Bluetooth module. Table 3 shows the corresponding control functions and instructions of the six MI tasks in the BCI system.

In this paper, the intelligent car developed by our research group is selected as the external controlled device, which is mainly composed of a Microcontroller Unit (MCU) module, a drive motor control module, a steering gear control module, a speed detection module, a path information acquisition module, a power module, and a wireless Bluetooth transmission module.

The online test experiment of the MI-BCI system is carried out based on the trained optimal CNN model. During the experiment, the smart car was placed on the designed track, which was a square of 1 m in length and width surrounded by black tape on the white background floor. The car was required to run the whole course at a uniform speed according to the subjects’ brain intention. The online test environment of the MI-BCI system was shown in Figure 14.

Before the BCI system experiment, the steering gear angle and speed and threshold parameters of the intelligent car need to be set. The track is square with four 90-degree turns and the rest is straight. In order to turn the car smoothly and reduce the impact of the car turning left and right in a straight line by misoperation, the maximum angle of the steering gear is set at 23 degrees. Since it takes 5s for the BCI system to collect the MI-EEG signal, the car speed and threshold should not be set too high. Choose two values for the speed, 2 cm/s and 4 cm/s, and three values for the threshold, 1, 2, and 3. Six modes are then determined:

Mode 1: Speed is 2 cm/s and threshold is 1;

Mode 2: Speed is 2 cm/s and threshold is 2;

Mode 3: Speed is 2 cm/s and threshold is 3;

Mode 4: Speed is 4 cm/s and threshold is 1;

Mode 5: Speed is 4 cm/s and threshold is 2;

Mode 6: Speed is 4 cm/s and threshold is 3.

Subjects S1, S3, S4, S6, S7, and S10 were selected to participate in the BCI system experiment. Each subject carried out six modes of training experiments, respectively, and in each mode, the car completed five laps clockwise and five laps counterclockwise.

Table 4 shows the statistics of the number of laps completed by the car under the control of six subjects in different modes. The number of complete clockwise laps of the car was 22, 26, 21, 25, 10 and 3, totaling 107, while the corresponding counterclockwise laps were 18, 20, 18, 20, 8 and 1, totaling 85. In different modes, six subjects performed better in clockwise than counterclockwise direction, which also verified the experimental results of MI-EEG signal classification, where T2 (right fist) was classified more accurately than T1 (left fist), and so the right-turn command was more accurate than the left-turn command when translated into car commands. The total number of laps in each mode is as follows: 40 laps in mode 1, 46 laps in mode 2, 39 laps in mode 3, 45 laps in mode 4, 18 laps in mode 5, and 4 laps in mode 6. It can be seen that mode 2 and mode 4 were completed well, with the total number of laps being 46 and 45, respectively. These two parameter settings can be selected for the online testing of the BCI system.

After the training, the relevant parameters were selected, and the adaptive coordination between the subjects and the intelligent car was completed. The online test experiment of the BCI system can be carried out. The experiment consisted of six subjects running three sets of tests in two selected modes, each consisting of a clockwise and counterclockwise lap.

Table 5 shows the test results in mode 2. The mean time and the mean total number of instructions for subject S1 to control the car to complete a set of tests were 561.8 s and 58, respectively, 655.0 s and 64 for subject S3, 696.7 s and 69 for subject S4, 685.1 s and 66 for subject S6, 672.8 s and 65 for subject S7, and 708.8 s and 70 for subject S10. In this mode, it can be seen that the average of the total number of instructions is proportional to the average of the time spent, with subject S1 taking the shortest time, having the least total number of instructions, and the best test result.

Table 6 shows the test results in mode 4. The mean time and mean total number of instructions for subject S1 to control the car to complete a set of tests were 484.0 s and 68, respectively, 544.1 s and 80 for subject S3, 581.8 s and 82 for subject S4, 525.4 s and 78 for subject S6, 520.2 s and 75 for subject S7, and 569.0 s and 81 for subject S10. It can be seen that the test effect of subject S1 is also the best in mode 4.

The speed of Table 5 is 2 cm/s and the speed of Table 6 is 4 cm/s. To complete the same test, the time used in the former should be theoretically twice as long as the time used in the latter. However, from the data listed in the two tables, it can be seen that the time used in Table 5 is obviously less than the two times of Table 6. In addition, the number of executed instructions listed in Table 5 is less than that in Table 6, because the threshold value of Table 6 is 1, and the car will execute the instruction immediately after receiving it once. While the threshold value of Table 5 is 2, the car will execute the instruction only when it receives the same instruction twice in a row, so the response will be slow, but it can better eliminate some wrong operations.

Figure 15 shows the car motion track recorded by subject S1 when he controlled the car to run the whole course under two modes. The orange dots in the figure is the starting position of the car and the blue dots in the figure is the motion track of car. It can be seen that, in the same mode, the track curve of the clockwise direction has less fluctuation than that of the counterclockwise direction, and it fits the track of black line more closely. This is because T2 (right fist) has higher classification accuracy than T1 (left fist), so the corresponding right turn instruction is more accurate than the left command, and the motion track of the clockwise fist is better than the counterclockwise fist. In mode 2, although the setting speed of the car is relatively low and the time used is longer, the threshold is set to two, which filters out some misoperations. The motion track of the car in mode 2 is better and has less deviation than that in mode 4.

To sum up, the parameters have different effects on the system. When setting a high threshold value, the flexibility of the car will also be reduced although it can reduce some misoperation. When setting a high speed, although the overall running time will be reduced, the deviation caused by the wrong instruction is greater and the effect of the motion trajectory is not good. The test effect cannot be compared directly with the different parameters set and the parameters should be selected according to the actual situation. When the parameters were the same, T2 (right fist) had a higher classification accuracy than T1 (left fist), and the right turn instruction was more accurate than the left turn instruction. The time used, the total number of instructions, and the track curve all verified that the subject performed better in the clockwise direction than the counterclockwise direction. In general, the experiment time is proportional to the total number of corresponding instructions. The fewer instructions received by the subjects, the shorter the time spent, and the better the test effect.

MI-EEG is a kind of spontaneous signal that does not require external stimulation and reinforces brain compensatory functions by imagining limb movements without involving actual muscle movements. After a period of training, subjects can significantly activate their brain regions. In this paper, 10 subjects were selected to participate in the BCI system training for 7 days, and subjects trained for five cycles per day, performing six MI tasks per cycle, and 10 experiments for each task. The experimental data of subject S1 with the best training effect and subject S8 with the worst training effect were selected to analyze the influence of the factor of the training duration on the classification performance of the BCI system.

Figure 16 presents the daily training results for subjects S1 and S8. Subject S1 was trained daily, and accuracy was improved by 7.96%, 5.91%, 2.69%, 1.78%, −0.41%, and 0.46%; precision was improved by 9.75%, 7.44%, 1.37%, 0.38%, 0.32%, and 0.05%; recall was improved by 6.47%, 6.81% 4.08%, 2.75%, 0.60%, and −0.06%; and the F1-score was improved by 8.18%, 7.13%, 2.69%, 1.50%, 0.46%, and −0.01%. Subject S8 was trained daily, and accuracy was improved by 3.18%, 2.19%, 0.74%, −0.07%, 0.35%, and −0.08%; precision was improved by 3.41%, 1.89%, 2.16%, 0.93%, −0.18%, and 0.02%; recall was improved by −0.71%, 4.32%, 6.73%, 3.90%, 0.70%, and −0.20%; and the F1-score was improved by 1.63%, 2.97%, 4.11%, 2.14%, 0.17%, and −0.07%. After 7 days of training, the indicators for S1 of accuracy, precision, recall, and the F1-score were improved by a total of 18.39%, 19.31%, 20.65%, and 19.95%. S8 improved by a total of 6.31%, 8.23%, 14.74%, and 10.95%. From the experimental results, it can be seen that the performance of each indicator of the subjects after training was significantly improved, and only the improvement effect was different. For subjects in the first 5 days of training, the effect was obvious and the curve of each indicator tended to level off in the last 2 days, which indicates that the training had reached the optimal state.

In this study, three sets of cross-subject experiments were conducted to analyze the influence of individual differences on the classification performance of the BCI system. In the experiment, the dataset (9000 samples) of 10 subjects (S1–S10) was mixed first, and the CNN network training was carried out to obtain the optimal model. Then, three new subjects (S11, S12, and S13), including two males and one female, aged between 21 and 28 years old, were selected and 900 samples were collected for each subject. Finally, the CNN optimal model was tested for its classification performance to evaluate the adaptability of the BCI system to the individual differences of the new subjects.

Table 8 lists the classification performance evaluation indicators of the BCI system for cross-subject experiments, and it can be seen that subject S13 had the best performance with an accuracy of 82.11%, subject S12 had the worst performance with an accuracy of 74.11%, and the average value of accuracy reached 78.07%. The mean values of the other indicators of precision, recall, and the F1-score also reached 77.87%, 78.64%, and 78.23%.

Figure 17 shows the confusion matrix of the classification accuracy of six MI tasks for the three subjects, and it can be seen that the highest classification accuracy was T2 (85.67%) for S13 and the lowest accuracy was T5 (71.87%) for S12. The average accuracy of the six MI tasks were (Hou Y) 79.33% (T1), 82.06% (T2), 77.81% (T3), 78.56% (T4), 74.96% (T5), and 75.70% (T6), respectively. The average classification errors for T1 were 4.94% (T2), 4.20% (T3), 4.35% (T4), 3.48% (T5), and 3.70% (T6), respectively. The average classification errors for T2 were 4.96% (T1), 3.67% (T3), 3.77% (T4), 2.74% (T5), and 2.80% (T6), respectively. The average classification errors for T3 were 3.69% (T1), 4.09% (T2), 5.25% (T4), 4.36% (T5), and 4.80% (T6), respectively. The average classification errors for T4 were 4.03% (T1), 3.28% (T2), 5.30% (T3), 4.06% (T5), and 4.77% (T6), respectively. The average classification errors for T5 were 4.44% (T1), 4.13% (T2), 3.69% (T3), 4.55% (T4), and 8.23% (T6), respectively. The average classification errors for T6 were 3.55% (T1), 1.49% (T2), 5.33% (T3), 3.53% (T4), and 10.40% (T5), respectively.

The experimental results showed that the BCI system achieved good average accuracy and a low classification error on the six MI tasks, with the highest classification accuracy for the right fist (T2) and the lowest for the left thumb (T5). According to the confusion matrix, the kappa values of the three subjects were 0.732 (S11), 0.699 (S12), and 0.785 (S13), respectively, with an average value of 0.737. This indicates that the classification algorithm of the BCI system has good consistency in the new subject dataset and can effectively eliminate the effect of random classification.

In summary, the six-class CNN model of the BCI system learned generalized features from the dataset of 10 subjects, achieving a good classification and generalization performance on the dataset collected by new subjects, adapting to individual differences in subjects, and breaking through the limitations of the poor compatibility of other BCI systems with different subjects.

At present, the MI-BCI system has become a hot research topic for scholars both domestically and internationally, with new research results emerging every year.

The public PhysioNet 109-subject dataset was selected for experiments, and a total of 10 sets of experiments were conducted. The average value was taken as the classification result. The classification accuracy of the method proposed was compared with the other literature research results, as shown in Table 9.

Handiru V S et al. proposed an Iterative Multiobjective Optimization for the Channel Selection (IMOCS) algorithm. On the PhysioNet dataset, the SVM classifier achieved an average classification accuracy of 63.62% for the two MI tasks [69]. Youngjoo K et al. proposed the strong uncorrelating transform complex common spatial patterns (SUTCCSP) algorithm. The performance of multiple classifiers was evaluated based on PhysioNet’s two-MI dataset, in which the random forest (RF) classifier achieved the best classification accuracy of 80.05% [70]. Ma X et al. proposed a method based on Recurrent Neural Networks (RNNs) that can perform the parallel decoding of spatial and temporal information. An average classification accuracy of 68.20% was achieved on PhysioNet’s four-MI dataset [71]. In an end-to-end DL model constructed by Hauke Dose et al., the CNN model classifies the original MI-EEG signals without special feature extraction. On PhysioNet’s two-, 3-, and 4-MI datasets, the mean accuracy reached 86.49%, 79.25%, and 68.51%, respectively [13]. Hou Y et al. used a CNN model for classification based on Brain-source Imaging (ESI), and achieved an accuracy of 94.5% on the four-MI dataset of PhysioNet [72]. A hybrid optimization technique of the Flower Pollination algorithm and β-hill-climbing algorithm was proposed by Alyasseri Z et al. An accuracy of 96.05% was obtained using the SVM classifier [73].

In this paper, a combined virtual electrode-based ESA and CNN method is proposed, with an average classification accuracy of 97.83%, which is better than the classification performance of the other literatures in the Table 9, indicating that this method is effective in motion intention classification and can improve the decoding ability of MI-EEG signals.

Publicly available datasets were analyzed in this study. This data can be found at: https://github.com/KJHGVS/MI-EEG-dataset.git↗ (accessed on 29 October 2023).