ELEKTROTEHNIŠKI VESTNIK 90(3): 99-104, 2023 ORIGINAL SCIENTIFIC PAPER A System for Automatic Sleep Structure Recognition and Analysis Yuting Cao a, # , Baozhu Li b, # , Yuan Zhang a, * , Anton Kos c a College of Electronic and Information Engineering, Southwest University, China b Internet of Things Smart City Innovation Platform, Zhuhai Fudan Innovation Institute, Zhuhai 519031, China c University of Ljubljana, Faculty of Electrical Engineering, Ljubljana, Slovenia * Corresponding author e-mail: yuanzhang@swu.edu.cn # Yuting Cao and Baozhu Li contributed equally to this work. Abstract. Sleep is regarded as essential for maintaining optimal health and well-being; however, the existing sleep studies are homogeneous and subjectively intrusive. And, there is no golden standard for clinical sleep quality evaluation. Some researchers have explored sleep structure (sleep staging) as a crucial basis for sleep quality evaluation. Therefore, the paper proposes an automatic sleep structure recognition and analysis system to provide a novel approach to sleep quality evaluation. A deep convolutional neural network is applied to sleep structure recognition. It mainly includes data preprocessing, feature extraction, feature fusion and prediction. A sleep structure visualization platform with user-friendly interfaces and images is implemented to visualize and analyze the sleep structure. The method is evaluated on a both public and private dataset. The results show its good performance and a potential lore clinical application. Keywords: System, Sleep Structure Recognition, Visualization Platform Sistem za samodejno prepoznavanje in analizo strukture spanja Spanje je bistvenega pomena za ohranjanje zdravja, vendar so obstoječe študije spanja homogene in subjektivno vsiljive. Poleg tega ni zlatega standarda za klinične indekse za oceno kakovosti spanja. Nekateri raziskovalci so raziskali, da je struktura spanja (stopnjevanje spanja) ključna podlaga za oceno njegove kakovosti, zato je v tem delu razvit sistem za samodejno prepoznavanje in analizo strukture spanja, ki zagotavlja novo zamisel za ocenjevanje njegove kakovosti. Za prepoznavanje strukture spanja se uporablja globoka konvolucijska nevronska mreža, ki v glavnem vključuje predobdelavo podatkov, ekstrakcijo značilnosti, združevanje značilnosti in napovedovanje. Za vizualizacijo in analizo strukture spanja je zasnovana zlasti platforma za vizualizacijo strukture spanja z uporabniku prijaznimi vmesniki in slikami. Sistem je ovrednoten z javnimi in zasebnimi nabori podatkov, rezultat pa kaže, da je dobro deloval, in dokazuje učinkovitost potencialne vrednosti klinične uporabe. 1 INTRODUCTION Sleep not only plays a vital role in maintaining one’s mental and physical health but is closely related to one’s life quality [1]. Furthermore, a poor sleep quality increases the risk of many major diseases such as the heart disease, high blood pressure, high blood lipids, diabetes, and obesity. Global sleep trends indicate that sleep-related disorders are on the rise [2.3]. Considering these trends and the fact that sleep is a crucial biological process, sleep problems are considered a significant medical task. Polysomnography (PSG), as the gold standard for an objective standard sleep physiology evaluation [4], records distinct physiological signals including electroencephalogram (EEG), electrooculogram (EOG), electromyography (EMG), electrocardiogram (ECG), respiration, pulse oximetry, and other parameters [5]. Once PSG recordings are obtained, they are converted into 30-second (30s) successive epochs manually identified as different sleep stages by a technician depending on the American Academy of the Sleep Medicine (AASM) Manual [6]. In the AASM Manual, sleep recordings are divided into five stages: Wake (W), Rapid Eye Movement (REM), and Non-Rapid Eye Movement (NREM) including N1 (transition stage), N2 (light sleep), and N3 (deep sleep). Sleep in adults typically occurs in approximately four-six sleep cycles overnight, which are interconnected and circumscribed. The five stages in each sleep cycle usually switch to each other with a certain pattern. While there is a serious imbalance in the amount of the sleep data in the existing sleep datasets, where the N2 stage accounts for 45–55% of the whole sleep time, and the N3 stage accounts for ~20%, the Rapid Eye Movement (REM) stage accounts for ~25%, and the N1 stage accounts for only 2–5% [7]. Noteworthily, children’s brains are not yet fully developed and many of the waveforms are not yet out, therefore, the children’s sleep structure is slightly different from that of the adults and the analysis of the Received 17 February 2023 Accepted 26 April 2023 100 CAO, LI, ZHANG, KOS Figure 1. Flow chart of the automatic sleep structure recognition and analysis system. children's sleep structure is more complex. The discovery of the sleep structure pattern has facilitated the study of sleep by scholars and physicians. However, it is time-consuming, tedious, and labor- intensive for a sleep technician to delineate the sleep structure based on the physiological signals in PSG. Moreover, the quality of the process heavily relies on the technician's experience, professional level and rigorous work ethic. Some studies show that the level of the agreement between technicians is usually below 90% [8], which has a great impact on the study of the sleep structure. In recent years, several researchers exploit deep learning methods to implement an automatic classification of sleep structures. For example, Chriskos et al. [9] propose a novel classification framework based on convolutional neural networks. It tackles the difference between the N1 stages and other sleep stages. Zhang et al. [10] generate spectrograms from the EEG, EEG, and EMG data. Eventually, they obtain a better performance than sleep experts classifying sleep stages. Our previous studies also explore the automatic classification of sleep structures from the semi-supervision front of view [11.12]. Although these studies achieve satisfactory results in the sleep structure classification, they do not address the following challenges: (1) The classification results of the sleep structures are obscure to those who do not have theoretical studies of the sleep science, human cannot understand their sleep quality through the classification results. (2) The result of the sleep structure classification takes the form of disadvantageous characters that are non-interpretability. Therefore, the classification results do not directly present the overall situation of the sleep structure. To address these issues, the paper proposes an automatic sleep structure recognition and analysis system. Its main contributions can be summarized as follows: 1) The system delivers a novel solution to the assessment of sleep quality. It is deployed with a database, sleep structure visualization, and sleep structure statistical analysis, as well as an effective invocation of data. This means that the system’s stability is ensured, its complexity is reduced, and its cohesiveness is improved. 2) The deep learning algorithm and graphical interfaces are employed to attain the recognition and visualization of the sleep structure. Thus, the automatic analysis of the sleep structure can be visualized with the results presented clearly and easily. The paper is structured as follows. Section 2 describes the system in detail. Section 3 gives experimental analysis and results. Section 4 discusses the advantages and limitations of the previous work and the prospects for future work. Section 5 draws conclusions. 2 METHODOLOGY 2.1 Overview of the System The flow chart of the system is shown in Figure 1. It mainly consists of three modules: a database, sleep structure calculation algorithm and sleep structure visualization platform. The database completes the storage, recall, and deletion of the sleep structure data and people’s data. The sleep structure calculation algorithm based on deep learning is performed to accomplish data preprocessing, automatic recognition of the sleep structure, preservation of the predicted results and labeled data, and evaluation of classification results. The sleep structure visualization platform has four modules. The Create New Profile module is leveraged to acquire people’s data, the Algorithm Support Platform module monitors the program operation, the Sleep Structure Display module visualizes the sleep structure recognition, and the Statistical Analysis module displays the sleep final results data. 2.2 Database The database is used to store and monitor a fast access for a large amount of the data to be processed [13]. It achieves a reasonable deployment among the data, reduces the data redundancy, maintains the data consistency, and ensures the data security and reliability. Firstly, the people’s raw data which are stored in the database are imported into the sleep structure calculation algorithm for a series of operations through a connection between the database and sleep structure visualization platform. The results are then returned to the database for storage. The people’s data are presented in the sleep structure visualization platform by calling the data stored A SYSTEM FOR AUTOMATIC SLEEP STRUCTURE RECOGNITION AND ANALYSIS 101 Figure 3. Flowchart of the proposed sleep structure calculation algorithm. in the database. The database content is shown in Figure 2. Figure 2. Database content. 2.3 Sleep Structure Calculation Algorithm This method introduces a self-supervised deep learning algorithm [14]. The predicted labels are expanded as the output data. In this way, a data support is established for the visualization platform. For the private dataset, the algorithm reconstructs the data pre-processing part so that it corresponds to the access of the algorithm, and finally obtains an accurate prediction. The whole process of the computational method is as follows: (1) Data pre- processing is conducted on both the public and private dataset of EEG signals. (2) Feature extraction and classification recognition for the sleep structure are realized by applying a modified CNN network. The flow chart of the proposed method is shown in Figure 3. 2.3.1 Data Preprocessing Before accessing the module, the EEG data needs pre- processing, which involves: (1) The data is divided into 30s epochs. (2) Filtering the signal with a first- order Butterworth filter reduces the noise interference. (3) The high or low frequency noise is augmented to enhance the data. 2.3.2 The Overall Framework An improved CNN network (Figure 3) is designed to classify and recognize the sleep structure. The input signals are divided into a pretext (pre-training), train and test set. The final prediction is obtained after convolutional layers and fully connected layers. The results of the predicted labels are stored in a form of strings in text documents. Finally, the output on the probabilities of each classification is obtained by using the softmax activation function. 2.4 Sleep Structure Visualization Platform The sleep structure visualization platform, supported by Python, enables a human-computer interaction. With the “account login module”, administrators automatically access the system by entering a login data. With the “create a new file module”, the people’s data and sleep structure data are established and stored in the database. The module ensures the data security and reliability. The algorithm support platform module allows administrators to view the sleep structure algorithm operation. The visualization platform module transforms the sleep structure data from scratch to refined. Moreover, it provides an objective analysis of the sleep quality, develops sleep patterns and provides an early warning and intervention opportunity in the early stages of a sleep disorder. Finally, the statistical analysis module obtains the statistics and analysis of the collection of the people’s hours and other data. The implementation of the system is drawn in Figures 4- 9. Its capabilities are: 102 CAO, LI, ZHANG, KOS Figure 4. Account Login. Figure 5. Main Interface. Figure 6. Create a New File. Figure 7. List of All Persons. Figure 8. Sleep Profile-Sleep Structure Display. Figure 9. Statistical Analysis. Figures 4- 9. Implementation of the sleep structure visualization platform.  Account Login: The administrator signs into the visualization platform by inputting the account password.  Create a New File: The module saves and displays the newly collected data. After filling in the people’s data, importing the sleep profile data, and uploading the sleep profile data, the sleep parameter indicators are stored in the database.  Main Interface: Jumping the pages is accomplished in an efficient and convenient way.  Sleep Profile: The people’s data, original EDF waveform, sleep structure waveform and sleep structure percentage pie chart are presented.  Statistical Analysis: The people’s sleep profiles are analyzed and counted, and the sleep quality is obtained available. 3 EXPERIMENTAL RESULTS AND ANALYSIS 3.1 Dataset The public and private dataset are used to evaluate the system. The public dataset is the Sleep-EDF dataset [15.16]. It is expanded to contain 197 whole-night polysomnographic sleep recordings. Some records also contain the respiration and body temperature. There are 153 PSG recordings from Sleep Cassette Study (SC) and 44 PSG from Sleep Telemetry Study (ST). The PSG recordings contain two EEG channels: Fpz-Cz and Pz- Oz. The sampling frequency of the EEG signal is 100 Hz. The corresponding hypnograms (sleep patterns) are manually scored by well-trained technicians. The 153 A SYSTEM FOR AUTOMATIC SLEEP STRUCTURE RECOGNITION AND ANALYSIS 103 PSG records from the SC study are employed to evaluate the system. Our private dataset contains 34 PSG recordings of children, some of whom have OSA. The PSG recordings of these 34 persons contain six EEG channels: F3-M2, F4-M1, C3-M2, C4-M1, O1-M2 and O2-M1. The sampling frequency of the EEG signal is 256 Hz. For the PSG recordings, every 30s interval corresponds to a label provided by technicians according to the AASM Manual. 3.2 Implementation Details For each dataset, persons are randomly assigned to the pretext and train and test set in different proportions (90%:5%:5%), and then each person’s recording is then split into non-overlapping 30s epochs, which are the data samples in the experiment. The periods of the same segmentations are placed in the same set. The pretext set is used to pre-train the model to get better parameters. The train and test set command labels. The train set is used to train the model. The parameters are fine-tuned to get a better fit for the method. The test set is used to evaluate the trained method. The method is implemented by the PyTorch framework. 3.3 Result and Quantitative Analysis Different adopted metrics are used to evaluate the accuracy (ACC), precision (PR), recall (RE) and F1- score (F1). Since the sleep structure recognition is a multi-classification problem, each class is evaluated by treating it as a positive example and the others as a negative example. The indicators are denoted as follows: FN FP TP TN TN TP ACC      (1) FP TP TP PR   (2) FN TP TP RE   (3) RE PR RE PR F    2 1 (4) where TP is the true positive number of the positive sample, and FP is a false positive number of the negative sample. The same rule applies to TN and FN. Table 1, evaluation indicators, in the private and public dataset are conjoined. The accuracy, recall, precision and F1 are shown. The metrics are significantly lower in the private than in the public dataset due to the data imbalance in the private dataset. Table 1: Performance comparison of evaluation indicators in the private and public dataset N a m e ACC F1 RE PE Sleep-EDF Dataset 0 . 8 4 8 0.637 0.707 0.622 Private Dataset 0.622 0.550 0.530 0.711 Table 2: Performance comparison of evaluation indicators obtained from the method and two other methods Method ACC F1 RE PE Our Method 0 . 8 4 8 0.637 0.707 0.622 SimCLR [17] 0.831 0.608 0.665 0.610 MoCo [18] 0.821 0.319 0.395 0.286 The performance comparison about that our method outperforms the other two methods. 4 DISCUSSION An automatic sleep structure recognition and analysis system is proposed. The several advantages of the system are: (1) An increases user experience and a higher acceptability among the general public. (2) A large amount of the data to solve the deficiency of the data clutter and to reduce the workload of locating the sleep files. (3) The sleep structure pattern plays an important role in the study of the sleep quality [19]. Analyzing the sleep structure indirectly monitors a person's sleep quality, thus providing an objective way for the person’s sleep quality assessment. However, there is a gap between the accuracy of the private and public dataset because of the data imbalance. Also, the natural asymmetric distribution of the sleep stages results in fewer N1 frames and corresponding lower accuracy than the other sleep stages. In the sleep structure visualization platform, the deployment of each functional module and interconnection is a difficult issue. Therefore, the deployment of the various functional modules and the personalized presentation of the visualization platform are important for the system further updated. 5 CONCLUSION The paper proposes an automatic sleep structure recognition and analysis system based on the relationship between the sleep quality and sleep structure. The sleep structure is automatically recognized and the sleep structure data is analyzed and assessed. A sleep structure visualization platform is applied to assess sleep quality. For the sleep structure recognition, the system ensures good consistency with the results of manual segmentations, thus offering the technicians an intelligent way of automatic sleep staging. On the sleep structure visualization platform, a visual response to the sleep structure data is obtained, making the complex and obscure data intuitive and concrete. REFERENCES: [1] Y. Mitsukura, K. Fukunaga, M. Yasui, et al, 2020. Sleep stage detection using only heart rate. Health Informatics Journal, vol. 26, pp. 376-387, doi:10.1177/1460458219827349. [2] M. K. Pavlova, V. Latreille, 2019. Sleep Disorders. American Journal of Medicine, vol. 132, pp. 292-299, doi:10.1016/j.amjmed.2018.09.021. [3] K. Zitting, H. M. Lammers-van Der Holst, R. K. Yuan, et al, 2021. Google Trends reveals increases in internet searches for insomnia during the 2019 coronavirus disease (COVID- 19) global pandemic. Journal of Clinical Sleep Medicine, 104 CAO, LI, ZHANG, KOS vol. 17, pp. 177-184, doi:10.5664/jcsm.8810. [4] M. Sharma, D. Goyal, P. V. Achuth, et al, 2018. An accurate sleep stages classification system using a new class of optimally time-frequency localized three-band wavelet filter bank. Computers in Biology and Medicine, vol. 98, pp. 58- 75, doi:10.1016/j.compbiomed.2018.04.025. [5] C. Berthomier, X. Drouot, M. Herman-Stoieca, et al, 2007. Automatic analysis of single-channel sleep EEG: Validation in healthy individuals. Sleep, vol. 30, pp. 1587-1595, doi:10.1093/sleep/30.11.1587. [6] R. B. Berry, R. Budhiraja, D. J. Gottlieb, et al, 2012. Rules for Scoring Respiratory Events in Sleep: Update of the 2007 AASM Manual for the Scoring of Sleep and Associated Events. Journal of Clinical Sleep Medicine, vol. 8, pp. 597- 619, doi:10.5664/jcsm.2172. [7] J. H. Fan, C. L. Sun, C. Chen, et al, 2020. EEG data augmentation: towards class imbalance problem in sleep staging tasks. Journal of Neural Engineering, vol. 17, doi:10.1088/1741-2552/abb5be. [8] C. Stepnowsky, D. Levendowski, D. Popovic, et al, 2013. Scoring accuracy of automated sleep staging from a bipolar electroocular recording compared to manual scoring by multiple raters. Sleep Medicine, vol. 14, pp. 1199-1207, doi:10.1016/j.sleep.2013.04.022. [9] P. Chriskos, C. A. Frantzidis, P. T. Gkivogkli, et al, 2020. Automatic Sleep Staging Employing Convolutional Neural Networks and Cortical Connectivity Images. IEEE Transactions on Neural Networks and Learning Systems, vol. 31, pp. 113-123, doi:10.1109/TNNLS.2019.2899781. [10] L. D. Zhang, D. Fabbri, R. Upender, et al, 2019. Automated sleep stage scoring of the Sleep Heart Health Study using deep neural networks. Sleep, vol. 42, doi:10.1093/sleep/zsz159. [11] C. H. Zhang, W. W. Yu, Y. M. Li, et al, 2022. CMS2-Net: Semi-Supervised Sleep Staging for Diverse Obstructive Sleep Apnea Severity. IEEE Journal of Biomedical and Health Informatics, vol. 26, pp. 3447-3457, doi:10.1109/JBHI.2022.3156585. [12] Y. Li, S. Luo, H. Zhang, et al, 2022. MtCLSS: Multi-Task Contrastive Learning for Semi-Supervised Pediatric Sleep Staging. IEEE Journal of Biomedical and Health Informatics, vol. PP, doi:10.1109/JBHI.2022.3213171. [13] C. A. Gyorodi, D. V. Dumse-Burescu, R. S. Gyorodi, et al, 2021. Performance Impact of Optimization Methods on MySQL Document-Based and Relational Databases. Applied Sciences-Basel, vol. 11, doi:10.3390/app11156794. [14] C. Yang, C. Xiao, M. B. Westover, et al, 2021. Self- supervised EEG Representation Learning for Automatic Sleep Staging. ArXiv, vol. abs/2110.15278. [15] B. Kemp, A. H. Zwinderman, B. Tuk, et al, 2000. Analysis of a sleep-dependent neuronal feedback loop: The slow- wave micro continuity of the EEG. IEEE Transactions on Biomedical Engineering, vol. 47, pp. 1185-1194, doi:10.1109/10.867928. [16] A. L. Goldberger, L. Amaral, L. Glass, et al, 2000. PhysioBank, PhysioToolkit, and PhysioNet - Components of a new research resource for complex physiologic signals. Circulation, vol. 101, pp. E215-E220, doi:10.1161/01.CIR.101.23.e215. [17] T. Chen, S. Kornblith, M. Norouzi, et al, 2020. A Simple Framework for Contrastive Learning of Visual Representations, in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 1597-1607. [18] H. K., F. H., W. Y., et al, 2020. Momentum Contrast for Unsupervised Visual Representation Learning, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735. [19] L. Huang, L. Y. Yao, X. X. Li, et al, 2022. Staging study of single-channel sleep EEG signals based on data augmentation. FRONTIERS IN PUBLIC HEALTH, vol. 10, doi:10.3389/fpubh.2022.1038742. Yuting Cao is with the Chongqing Key Laboratory of Nonlinear Circuits and Intelligent Information Processing, College of Electronic and Information Engineering, Southwest University, Chongqing, China. Her current research interests include biomedical image/signal processing and deep learning. Baozhu Li received his Ph.D. degree in computer science and technology from the Beijing University of Posts and Telecommunications, Beijing, China, in 2017. He is currently with the Internet of Things & Smart City Innovation Platform, Zhuhai Fudan Innovation Institute. His research interests include machine learning and mobile edge computing. Yuan Zhang is with the Chongqing Key Laboratory of Nonlinear Circuits and Intelligent Information Processing, College of Electronic and Information Engineering, Southwest University, Chongqing, China. As the first author or corresponding author, he has published more than 60 peer- reviewed papers in international journals and conference proceedings, one book chapter and nine patents. He is a Senior Member of IEEE and ACM. Anton Kos received his Ph.D. degree in electrical engineering from the Faculty of Electrical Engineering, University of Ljubljana, Slovenia, in 2006. He is an assistant professor at the faculty. His teaching and research work includes signal processing, communication networks and dataflow computing. He is a Senior Member of IEEE. Since 2018 he has been on IEEE Slovenia ComSoc Chapter Chair.