› INFORMACIJSKA DRUZBA Zbornik 26. mednarodne multikonference Zvezek A INFORMATION SOCIETY Proceedings of the 26th International Multiconference Volume A Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Uredniki • Editors: Mitja Lustrek, Matjaz Gams, Rok Piltaver IS2023 Zbornik 26. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2023 Zvezek A Proceedings of the 26th International Multiconference INFORMATION SOCIETY – IS 2023 Volume A Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Uredniki / Editors Mitja Luštrek, Matjaž Gams, Rok Piltaver http://is.ijs.si 12. oktober 2023 / 12 October 2023 Ljubljana, Slovenia Uredniki: Mitja Luštrek Odsek za inteligentne sisteme Institut »Jožef Stefan«, Ljubljana Matjaž Gams Odsek za inteligentne sisteme Institut »Jožef Stefan«, Ljubljana Rok Piltaver Outfit7, Ljubljana Založnik: Institut »Jožef Stefan«, Ljubljana Priprava zbornika: Mitja Lasič, Vesna Lasič, Mateja Mavrič Oblikovanje naslovnice: Vesna Lasič Dostop do e-publikacije: http://library.ijs.si/Stacks/Proceedings/InformationSociety Ljubljana, oktober 2023 Informacijska družba ISSN 2630-371X Kataložni zapis o publikaciji (CIP) pripravili v Narodni in univerzitetni knjižnici v Ljubljani COBISS.SI-ID 170717955 ISBN 978-961-264-275-4 (PDF) PREDGOVOR MULTIKONFERENCI INFORMACIJSKA DRUŽBA 2023 Šestindvajseta multikonferenca Informacijska družba se odvija v obdobju izjemnega razvoja za umetno inteligenco, računalništvo in informatiko, za celotno informacijsko družbo. Generativna umetna inteligenca je s programi kot ChatGPT dosegla izjemen napredek na poti k superinteligenci, k singularnosti in razcvetu človeške civilizacije. Uresničujejo se napovedi strokovnjakov, da bodo omenjena področna ključna za obstoj in razvoj človeštva, zato moramo pozornost usmeriti na njih, jih hitro uvesti v osnovno in srednje šolstvo in vsakdan posameznika in skupnosti. Po drugi strani se poleg lažnih novic pojavljajo tudi lažne enciklopedije, lažne znanosti ter »ploščate Zemlje«, nadaljuje se zapostavljanje znanstvenih spoznanj, metod, zmanjševanje človekovih pravic in družbenih vrednot. Na vseh nas je, da izzive današnjice primerno obravnavamo, predvsem pa pomagamo pri uvajanju znanstvenih spoznanj in razčiščevanju zmot. Ena pogosto omenjanih v zadnjem letu je eksistencialna nevarnost umetne inteligence, ki naj bi ogrožala človeštvo tako kot jedrske vojne. Hkrati pa nihče ne poda vsaj za silo smiselnega scenarija, kako naj bi se to zgodilo – recimo, kako naj bi 100x pametnejši GPT ogrozil ljudi. Letošnja konferenca poleg čisto tehnoloških izpostavlja pomembne integralne teme, kot so okolje, zdravstvo, politika depopulacije, ter rešitve, ki jih za skoraj vse probleme prinaša umetna inteligenca. V takšnem okolju je ključnega pomena poglobljena analiza in diskurz, ki lahko oblikujeta najboljše pristope k upravljanju in izkoriščanju tehnologij. Imamo veliko srečo, da gostimo vrsto izjemnih mislecev, znanstvenikov in strokovnjakov, ki skupaj v delovnem in akademsko odprtem okolju prinašajo bogastvo znanja in dialoga. Verjamemo, da je njihova prisotnost in udeležba ključna za oblikovanje bolj inkluzivne, varne in trajnostne informacijske družbe. Za razcvet. Letos smo v multikonferenco povezali deset odličnih neodvisnih konferenc, med njimi »Legende računalništva«, s katero postavljamo nov mehanizem promocije informacijske družbe. IS 2023 zajema okoli 160 predstavitev, povzetkov in referatov v okviru samostojnih konferenc in delavnic, skupaj pa se je konference udeležilo okrog 500 udeležencev. Prireditev so spremljale okrogle mize in razprave ter posebni dogodki, kot je svečana podelitev nagrad. Izbrani prispevki bodo izšli tudi v posebni številki revije Informatica (http://www.informatica.si/), ki se ponaša s 46-letno tradicijo odlične znanstvene revije. Multikonferenco Informacijska družba 2023 sestavljajo naslednje samostojne konference: • Odkrivanje znanja in podatkovna središča • Demografske in družinske analize • Legende računalništva in informatike • Konferenca o zdravi dolgoživosti • Miti in resnice o varovanju okolja • Mednarodna konferenca o prenosu tehnologij • Digitalna vključenost v informacijski družbi – DIGIN 2023 • Slovenska konferenca o umetni inteligenci + DATASCIENCE • Kognitivna znanost • Vzgoja in izobraževanje v informacijski družbi • Zaključna svečana prireditev konference Soorganizatorji in podporniki konference so različne raziskovalne institucije in združenja, med njimi ACM Slovenija, SLAIS za umetno inteligenco, DKZ za kognitivno znanost in Inženirska akademija Slovenije (IAS). V imenu organizatorjev konference se zahvaljujemo združenjem in institucijam, še posebej pa udeležencem za njihove dragocene prispevke in priložnost, da z nami delijo svoje izkušnje o informacijski družbi. Zahvaljujemo se tudi recenzentom za njihovo pomoč pri recenziranju. S podelitvijo nagrad, še posebej z nagrado Michie-Turing, se avtonomna stroka s področja opredeli do najbolj izstopajočih dosežkov. Nagrado Michie-Turing za izjemen življenjski prispevek k razvoju in promociji informacijske družbe je prejel prof. dr. Andrej Brodnik. Priznanje za dosežek leta pripada Benjaminu Bajdu za zlato medaljo na računalniški olimpijadi. »Informacijsko limono« za najmanj primerno informacijsko tematiko je prejela nekompatibilnost zdravstvenih sistemov v Sloveniji, »informacijsko jagodo« kot najboljšo potezo pa dobi ekipa RTV za portal dostopno.si. Čestitke nagrajencem! Mojca Ciglarič, predsednica programskega odbora Matjaž Gams, predsednik organizacijskega odbora i FOREWORD - INFORMATION SOCIETY 2023 The twenty-sixth Information Society multi-conference is taking place during a period of exceptional development for artificial intelligence, computing, and informatics, encompassing the entire information society. Generative artificial intelligence has made significant progress towards superintelligence, towards singularity, and the flourishing of human civilization with programs like ChatGPT. Experts' predictions are coming true, asserting that the mentioned fields are crucial for humanity's existence and development. Hence, we must direct our attention to them, swiftly integrating them into primary, secondary education, and the daily lives of individuals and communities. On the other hand, alongside fake news, we witness the emergence of false encyclopaedias, pseudo-sciences, and flat Earth theories, along with the continuing neglect of scientific insights and methods, the diminishing of human rights, and societal values. It is upon all of us to appropriately address today's challenges, mainly assisting in the introduction of scientific knowledge and clearing up misconceptions. A frequently mentioned concern over the past year is the existential threat posed by artificial intelligence, supposedly endangering humanity as nuclear wars do. Yet, nobody provides a reasonably coherent scenario of how this might happen, say, how a 100x smarter GPT could endanger people. This year's conference, besides purely technological aspects, highlights important integral themes like the environment, healthcare, depopulation policies, and solutions brought by artificial intelligence to almost all problems. In such an environment, in-depth analysis and discourse are crucial, shaping the best approaches to managing and exploiting technologies. We are fortunate to host a series of exceptional thinkers, scientists, and experts who bring a wealth of knowledge and dialogue in a collaborative and academically open environment. We believe their presence and participation are key to shaping a more inclusive, safe, and sustainable information society. For flourishing. This year, we connected ten excellent independent conferences into the multi-conference, including "Legends of Computing", which introduces a new mechanism for promoting the information society. IS 2023 encompasses around 160 presentations, abstracts, and papers within standalone conferences and workshops. In total about 500 participants attended the conference. The event was accompanied by panel discussions, debates, and special events like the award ceremony. Selected contributions will also be published in a special issue of the journal Informatica (http://www.informatica.si/), boasting a 46-year tradition of being an excellent scientific journal. The Information Society 2023 multi-conference consists of the following independent conferences: • Data Mining and Data Warehouse - SIKDD • Demographic and Family Analysis • Legends of Computing and Informatics • Healthy Longevity Conference • Myths and Truths about Environmental Protection • International Conference on Technology Transfer • Digital Inclusion in the Information Society - DIGIN 2023 • Slovenian Conference on Artificial Intelligence + DATASCIENCE • Cognitive Science • Education and Training in the Information Society • Closing Conference Ceremony Co-organizers and supporters of the conference include various research institutions and associations, among them ACM Slovenia, SLAIS for Artificial Intelligence, DKZ for Cognitive Science, and the Engineering Academy of Slovenia (IAS). On behalf of the conference organizers, we thank the associations and institutions, and especially the participants for their valuable contributions and the opportunity to share their experiences about the information society with us. We also thank the reviewers for their assistance in reviewing. With the awarding of prizes, especially the Michie-Turing Award, the autonomous profession from the field identifies the most outstanding achievements. Prof. Dr. Andrej Brodnik received the Michie-Turing Award for his exceptional lifetime contribution to the development and promotion of the information society. The Achievement of the Year award goes to Benjamin Bajd, gold medal winner at the Computer Olympiad. The "Information Lemon" for the least appropriate information move was awarded to the incompatibility of information systems in the Slovenian healthcare, while the "Information Strawberry" for the best move goes to the RTV SLO team for portal dostopno.si. Congratulations to the winners! Mojca Ciglarič, Chair of the Program Committee Matjaž Gams, Chair of the Organizing Committee ii KONFERENČNI ODBORI CONFERENCE COMMITTEES International Programme Committee Organizing Committee Vladimir Bajic, South Africa Matjaž Gams, chair Heiner Benking, Germany Mitja Luštrek Se Woo Cheon, South Korea Lana Zemljak Howie Firth, UK Vesna Koricki Olga Fomichova, Russia Mitja Lasič Vladimir Fomichov, Russia Blaž Mahnič Vesna Hljuz Dobric, Croatia Mateja Mavrič Alfred Inselberg, Israel Jay Liebowitz, USA Huan Liu, Singapore Henz Martin, Germany Marcin Paprzycki, USA Claude Sammut, Australia Jiri Wiedermann, Czech Republic Xindong Wu, USA Yiming Ye, USA Ning Zhong, USA Wray Buntine, Australia Bezalel Gavish, USA Gal A. Kaminka, Israel Mike Bain, Australia Michela Milano, Italy Derong Liu, Chicago, USA Toby Walsh, Australia Sergio Campos-Cordobes, Spain Shabnam Farahmand, Finland Sergio Crovella, Italy Programme Committee Mojca Ciglarič, chair Marjan Heričko Baldomir Zajc Bojan Orel Borka Jerman Blažič Džonova Blaž Zupan Franc Solina Gorazd Kandus Boris Žemva Viljan Mahnič Urban Kordeš Leon Žlajpah Cene Bavec Marjan Krisper Niko Zimic Tomaž Kalin Andrej Kuščer Rok Piltaver Jozsef Györkös Jadran Lenarčič Toma Strle Tadej Bajd Borut Likar Tine Kolenik Jaroslav Berce Janez Malačič Franci Pivec Mojca Bernik Olga Markič Uroš Rajkovič Marko Bohanec Dunja Mladenič Borut Batagelj Ivan Bratko Franc Novak Tomaž Ogrin Andrej Brodnik Vladislav Rajkovič Aleš Ude Dušan Caf Grega Repovš Bojan Blažica Saša Divjak Ivan Rozman Matjaž Kljun Tomaž Erjavec Niko Schlamberger Robert Blatnik Bogdan Filipič Stanko Strmčnik Erik Dovgan Andrej Gams Jurij Šilc Špela Stres Matjaž Gams Jurij Tasič Anton Gradišek Mitja Luštrek Denis Trček Marko Grobelnik Andrej Ule Nikola Guid Boštjan Vilfan iii iv KAZALO / TABLE OF CONTENTS Slovenska konferenca o umetni inteligenci / Slovenian Conference on Artificial Intelligence ................ 1 PREDGOVOR / FOREWORD ............................................................................................................................... 3 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES ............................................................................... 5 Time-Series Cutmix Data Augmentation for Heart Sound Classification Using Neural Networks / Susič David, Gams Matjaž ...................................................................................................................................................... 7 Social Interaction Prediction from Smart-Phone Sensor Data / Martinšek Marcel Franse, Lukan Junoš, Bolliger Larissa, Clays Els, Šiško Primož, Luštrek Mitja .............................................................................................. 11 Comparison of Advanced Processing Methods for PPG Denoising using a Novel Signal Quality Metric / Kawai Kitoshi, Slapničar Gašper, Hirose Akira, Luštrek Mitja .................................................................................. 15 Machine-learning methods for analysis of gene expression data / Jordan Marko, Valič Jakob, Luštrek Mitja ... 19 Identifying Bumblebee Buzzes Using Neural Networks / Šket Tilen, Susič David, Galen Candace, Schul Johannes, Arbetman Marina, Campopiano Robinson Victoria, Villagra Gil Cristian Alfonso, Herrera Valentina, Gradišek Anton ............................................................................................................................... 20 Recognition of Bee Activity in the Hive Entrance Using Machine Vision and Other Methods / Rotar Oskar, Žnidaršič Maks, Vesel Tian, Silan Darja, Božič Janko .................................................................................... 24 Implementation of a Virtual Assistant ChatGPT into the Medical Platform / Zadobovšek Matic, Kocuvan Primož, Gams Matjaž ....................................................................................................................................... 28 Types of Democracy Defined: Keyword Extraction from Eleven Different Text Descriptions / Kolar Žiga, Lukšič Andrej A., Vozlič Andrej A., Gams Matjaž ......................................................................................... 32 Evaluation of the Effects of On-demand Dynamic Transportation of Employees to Their Workplaces in Ljubljana / Bohanec Marko, Guček Marko, Kontić Davor, Sirk Karina, Ženko Bernard, Žnidaršič Martin .. 36 Usability Test of a Modified WHCA* Algorithm for Multi-Agent Pathfinding in a Real-time Strategy Game / Antešić Ivan, Sadikov Aleksandar ................................................................................................................... 40 An Attempt at Predicting Algorithm Performance on Constrained Multiobjective Optimization Problems / Andova Andrejaana, Vodopija Aljoša, Cork Jordan, Tušar Tea, Filipič Bogdan ............................................ 44 A Multi-Step Evaluation Process in Electric Motor Design / Tušar Tea, Korošec Peter, Filipič Bogdan ........... 48 Indeks avtorjev / Author index ................................................................................................................... 53 v vi Zbornik 26. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2023 Zvezek A Proceedings of the 26th International Multiconference INFORMATION SOCIETY – IS 2023 Volume A Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Uredniki / Editors Mitja Luštrek, Matjaž Gams, Rok Piltaver http://is.ijs.si 12. oktober 2023 / 12 October 2023 Ljubljana, Slovenia 1 2 PREDGOVOR Kar se tiče umetne inteligence, živimo v vznemirljivih časih. V času Informacijske družbe 2022 ChatGPT še ni bil na voljo, ko se je pojavil, pa je bilo vse drugače – umetna inteligenca se je iz nečesa, o čemer so govorili predvsem strokovnjaki, prelevila v nekaj, o čemer govorijo vsi. Spremembe v dojemanju (generativne) umetne inteligence so še nekoliko večje kot napredek njenih dejanskih zmožnosti, a tudi slednji je velik: nesporno je postala zelo uporabno orodje za pisanje besedil in programske kode ter generiranje slik, uveljavlja se kot osebni pomočnik in še več. Če bo razvoj umetne inteligence še naprej tako hiter, kot je trenutno, si v prihodnosti zlahka predstavljamo dramatičen napredek v produktivnosti gospodarstva, raziskavah in vsakdanjem življenju. A marsikdo umetno inteligenco vidi tudi kot grožnjo: od tega, da bo generirala velike količine posamezniku prilagojenih škodljivih vsebin in ljudem prevzela delovna mesta, do tega, da nas bo vse pobila. Tehtanje, kaj od tega se res utegne zgoditi, je malo prehud zalogaj za tale predgovor, tako da si raje poglejmo, kaj tak razvoj dogodkov pomeni za naše delo. Dostopnost velikih jezikovnih modelov pomeni, da so naloge, ki zahtevajo razumevanje ali generiranje naravnega jezika, lažje izvedljive kot kdajkoli prej. To je odlično, nerodno pa je, da je gradnja tovrstnih modelov zaradi velike računske zahtevnosti izven dosega velike večine organizacij – tako raziskovalnih kot podjetij. A še vedno se lahko ukvarjamo z njihovim prilagajanjem, vključno z uporabo spodbujevanega učenja s človekovim povratnim odzivom (angl. reinforcement learning with human feedback), in tehnikami za tvorbo pozivov (angl. prompt engineering), ki dajo želena izhodna besedila. Veliko raziskovalnih izzivov nudijo metode za zagotavljanja zaupanja v umetno inteligenco in njeno uravnavanje s cilji snovalcev (angl. alignement), za kar v splošnem še ni dobrih rešitev. Lahko pa se ukvarjamo tudi z vprašanji, kako umetno inteligenco regulirati, kako bo preobrazila (informacijsko) družbo in kaj storiti, da jo bo na bolje, kaj inteligenca sploh je in kdaj bo njena umetna različica dobila zavest. Če se ozremo k Slovenski konferenci o umetni inteligenci, opazimo, da se velikih jezikovnih modelov dotika le en prispevek od 12, družbenih vidikov umetne inteligence pa nobeden. Da bi bolje naslovili vroče tematike zadnjega časa, smo povabili Mykolo Pechenizkiyja s Tehnične univerze v Eindhovnu, da nam predava o praktičnih problemih in metodah zaupanja vredne umetne inteligence. Po lanski uspešni izvedbi Data Science Meetupa v sklopu konference – dogodka, kjer imajo strokovnjaki iz industrije kratke predstavitve svojega dela – ga na podoben način organiziramo tudi letos. Povezovanje z industrijo je dandanes namreč bolj pomembno kot kdajkoli prej, saj se tam dogaja vedno več raziskav in razvoja umetne inteligence. Mitja Luštrek Matjaž Gams Rok Piltaver predsedniki Slovenske konference o umetni inteligenci 3 FOREWORD With regards to artificial intelligence, we live in exciting times. During Information Society 2022, ChatGPT was not yet available, but when it appeared, everything changed – artificial intelligence turned from a subject mainly discussed by experts to something everyone is talking about. The changes in the perception of (generative) artificial intelligence are somewhat greater than the changes in its actual capabilities, but the latter are also substantial: it has definitely become a highly useful tool for writing text and software code as well as for generating images, it is increasingly capable as a personal assistant, and more. If it continues to advance at the current pace, we can easily imagine the future to bring dramatic improvements in economic productivity, research and everyday life. But many also see it as a threat: it may generate personalised harmful content at scale and take away jobs from people, and some even believe it may kill us all. Considering which of these things are actually likely to happen is beyond the scope of this preface, so let us look instead at what these developments mean for our work. The accessibility of large language models means that tasks requiring understanding or generating natural language are easier than ever. This is excellent, but on the other hand building such models is beyond the reach of the vast majority of organisations – both research and business – due to its huge computational cost. However, we can still engage in fine-tuning of such models, including with reinforcement learning with human feedback, and develop techniques for engineering prompts that yield desired outputs. A lot of research challenges can be found in developing methods to ensure trust in artificial intelligence and align it with the goals of the creators, for which there are no generally good solutions yet. We can also tackle questions of how to regulate artificial intelligence, how it will transform (information) society and how to ensure it will transform it for the better, what intelligence is, and when its artificial version will gain consciousness. Looking at the Slovenian Conference on Artificial Intelligence, we notice that only one out of the 12 papers touches upon large language models, and none addresses the social aspects of artificial intelligence. To better address the hot topics of recent times, we invited Mykola Pechenizkiy from Eindhoven University of Technology to give a keynote on practical problems and methods of trustworthy artificial intelligence. Last year we successfully combined the conference with Data Science Meetup – an event where industry experts give short presentations of their work – so we are going to organise it in a similar manner this year. Connecting with the industry is more important than ever, as more and more research and development of artificial intelligence is happening there. Mitja Luštrek Matjaž Gams Rok Piltaver Slovenian Conference on Artificial Intelligence chairs 4 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Mitja Luštrek Matjaž Gams Rok Piltaver Cene Bavec Marko Bohanec Marko Bonač Ivan Bratko Bojan Cestnik Aleš Dobnikar Erik Dovgan Bogdan Filipič Borka Jerman Blažič Marjan Krisper Marjan Mernik Biljana Mileva Boshkoska Vladislav Rajkovič Niko Schlamberger Tomaž Seljak Peter Stanovnik Damjan Strnad Miha Štajdohar Vasja Vehovar Martin Žnidaršič 5 6 Time-Series Cutmix Data Augmentation for Heart Sound Classification Using Neural Networks David Susič Matjaž Gams Jožef Stefan Institute Jožef Stefan Institute Ljubljana, Slovenia Ljubljana, Slovenia david.susic@ijs.si matjaz.gams@ijs.si ABSTRACT In alignment with this research landscape, our study revolves around utilizing time-series PCGs as inputs for a NN. This par- In cutmix augmentation, a synthetic data instance is created ticular approach has been previously explored and reported in by mixing parts of a pair of original instances. In this paper, a [11, 3, 13, 16, 5, 6, 17, 10, 1, 18, 12]. domain-specific time-series cutmix approach was employed for When it comes to time-series augmentation, the basic ap- a heart sound classification task. The approach utilized neural proaches include augmentations in time, frequency and time- networks and was tested on a publicly available heart sound frequency domains such as jittering, flipping, scaling, warping, dataset. To assess the efficacy of generating realistic instances, we cutout, cutmix and others [15, 7, 19]. To the best of our knowledge, implemented three distinct constraints for pairing requirements. the only works that specifically analyse augmentation methods Our main focus of interest was evaluation of performance of the for heart sound classification deal with spectral imaged based approaches on datasets of varying sizes, thus we performed the approaches [8, 2, 21]. In this paper, we present a cutmix approach experiments using different fractions of the train set. Cutmix that has been specifically adapted to time-series PCG domain. In showed promising results as best improvements in accuracy over a cutmix approach, a synthetic instance is created from two orig- the no augmentation baseline reached 5.61% and 1.46% when 10% inal PCGs. To assess the efficacy of generating realistic instances, and 100% of the training data were used, respectively. we implemented three distinct constraints for pairing require- KEYWORDS ments. These constraints help limit the various ways in which the original PCGs can be distributed in pairs for the creation of cutmix, sythetic data generation, time-series augmentation, phono- the synthetic PCGs. cardiogram, heart sound classification, neural network 2 DATASET 1 INTRODUCTION In this study, we used heart sounds from the PhysioNet/CinC The accurate classification of heart sounds plays a crucial role Challange 2016 dataset [9, 4], which consists of six (a-f ) distinct when it comes to early detection of cardiovascular diseases (CVDs). PCG databases. Recordings were labeled as either normal or Although heart auscultation is a cost-effective and efficient diag- abnormal heart sounds and were split into train and validation set nostic method, the accurate identification of heart abnormalities by the competition organizers. For our experiments, we utilized requires proficient auscultation skills. Auscultation, typically the validation set as the test set, while adjusting the original performed with a stethoscope, involves listening to the various train set to achieve class balance. To ensure fair evaluation, both sounds produced by the heart, such as heartbeats and murmurs, the train and the test subsets were balanced with respect to the to detect any irregularities or issues. To reduce the burden on two classes. In total, we used 277 normal and abnormal train heathcare staff and to mitigate the medical costs of delayed CVDs recordings and 145 normal and 151 abnormal test recordings as detection, there is a growing demand for automated approaches depicted in Figure 1. The total lengths of train and test recordings for identification of heart sound abnormalities. The integration are 3h 18min and 1h 37min, respectively. of cutting-edge machine learning (ML) and signal processing technologies has led to the widespread adoption of automated computer methods for tackling this challenge. When it comes to heart sound classification from phonocar- diograms (PCGs) using ML technologies, a variety of approaches have been tried and established, most of which draw inspiration from the broader domains of general audio and image data analy- sis. These encompass strategies such as utilizing features derived from time-series signals as inputs for classical ML models or neural networks (NNs). Additionally, techniques involving spec- trograms as inputs for NNs, employing time-series directly as NN inputs, extracting deep features from NNs for use in ML models, and even applying wavelet analysis have all been experimented with. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this Figure 1: Data distribution used in our experiments. work must be honored. For all other uses, contact the owner /author(s). Information Society 2023, 9–13 October 2022, Ljubljana, Slovenia The dataset also includes segmentation annotations denot- © 2023 Copyright held by the owner/author(s). ing the locations of the fundamental heart states (S1, systole, 7 Information Society 2023, 9–13 October 2022, Ljubljana, Slovenia David Susič and Matjaž Gams S2, and diastole) for each of the PCGs. They were computed Note that only one synthetic instance was generated per input using Springer’s segmentation algorithm [14] and additionally instance. Cutmix scheme is shown in Figure 2. manually checked and corrected by the experts. Two cutting variations were considered, deterministic and random. In a deterministic variation, the cutting point was always 3 METHODOLOGY right after systole, as depicted with red dashed line in Figure 2, while in random variation, a cutting point was randomly chosen The oucome of interest used to evaluate the cutmix approaches to be either right after S1, systole, or S2 as depicted with dashed was a binary variable indicating whether a PCG represents a black lines. normal or abnormal heart. The methodology steps included pre- processing of the PCGs, selection of NN architecture and deter- mination of cutmix augmentations techniques. 3.1 Preprocessing The recordings were first down-sampled to 1000 Hz, filtered using Butterworth filter of order four to four frequency bands, 25- 45Hz, 45Hz-80Hz, 80-200Hz, 200-400Hz, and finally normalized using root mean square normalization with the target amplitude of -20 dBFS. The recordings were then split into separate heart cycles (segments) according to segmentation annotations and zero-padded to 2.5s. Each instance was thus a time-series 2-D array of shape (4, 2500), with the first dimension denoting the number of frequency band channels and the second dimension denoting the segment length. The goal of our study was to also find out how well the cutmix augmentation approaches perform on datasets of smaller sizes, therefore we split the training set into a variety of differently- sized smaller sets, specifically, 10%, 20%, 40%, 60%, and 80%. The selected smaller training sets were randomly selected and were Figure 2: Visualization of PCG cutmix. Inputs 1 and 2 have stratified with respect to the data subsets and class labels within the same class label. The black dashed lines denote possible each subset. First, the smallest (10%) subset was created. Then, cutting points, while the red dashed line denotes the se- new subjects were added without reselection of the smaller set, lected cutting point. Note that only one synthetic instance meaning that the first next larger set (20%) included all of the is generated per input pair. subjects from the smaller set. This process was repeated until all of the subsets were created. Data augmentation was performed during the model train- ing. If a batch underwent augmentation (this was determined by 3.2 Convolutional Neural Network the cutmix probability parameter for each batch separately at The neural network structure used in our experiments was in- random), then the whole batch was replaced with the synthetic spired by the one proposed in [11]. It consists of four convolu-batch. The synthetic batch was created in such a way, that each tional neural networks (CNN) where each takes a different fre- instance of the original batch played a role of input 1 and input quency band as the input. Besides the input layer, the CNNs con- 2 exactly once, which results in the synthetic batch having the sists of two additional layers, each including a convolution, ReLu same size as the original and no original instance being over- activation and max-pooling. The last CNN layer also includes or under-represented. In all of the cases, the synthetic instances 25% dropout. The output of the four CNNs are flattened and input were generated using PCGs of the same class label, thus elimi- to a dense network with 20 hidden neurons with dropout of 50% nating any uncertainties regarding the labeling of the generated and two output neurons. instance. Furthermore, we implemented some additional pairing requirements. In the Subset cutmix, the input pairs were con- 3.3 PCG Cutmix Augmentation strained to be drawn from the same data subset, while in the We tested different cutmix approaches of data augmentation for Subject cutmix, the input pairs had to belong to the same subject classification of the PCGs. A cutmix data augmentation technique (same PCG). In the Length cutmix, the instances in the batch were is originally an image data augmentation strategy that replaces a sorted into 10 bins based on their heart beat length, and each patch of pixels from one image with a patch from another image pair was drawn from the same bin. The unconstrained cutmix [20]. In our case, a section of one PCG was replaced by a section methodology is referred to as Basic cutmix. The idea behind the of another PCG. A synthetic instance is generated from a pair of limitations was to check if creating more realistic synthetic PCGs original instances, input 1 and input 2. by forcing the input PCGs of each synthetic instance to have the In the cutmix technique presented in this study, a synthetic same recording settings, come from the same subject and/or be instance was generated using the following steps. Firstly, a pair of of similar length improves the model performance. input instances with the same class label was selected. Secondly, 3.4 Hyperparameter Finetuning both input instances were cut at the same point within the four heart beat stages (S1, systole, S2, diastole). Subsequently, the first In our experiments, there were four parameters to be tuned. Three part of input 1 was mixed with the second part of input 2. This hyperparamers, e.g., batch size, epoch number and learning rate, process resulted in a new synthetic instance encompassing all as well as the augmentation probability parameter. The latter de- four heart beat stages that had the same class label as the inputs. notes the probability of applying augmentation to a batch during 8 Time-Series Cutmix for Heart Sounds Information Society 2023, 9–13 October 2022, Ljubljana, Slovenia training. To optimize these parameters, we employed a grid- check the statistical significance of the results, we included accu- search methodology, which involved evaluating performance on racy SD derived from a set of three experiment runs. In scenarios the training set through three-fold cross-validation. To reduce involving 10% and 20% of the training data, all cutmix approaches search time, batch size was fixed to 512, which demonstrated demonstrated statistically significant improvements compared effectiveness across all scenarios. Meanwhile, we explored epoch to the baseline approach. However, for higher percentages of numbers within the range of 50 to 100, learning rates ranging training data, not all of the approaches showed statistically sig- −4 −1 from 10 and 10 , and cutmix probabilities spanning 0.1 to 1.0. nificant improvements. The augmentation techniques seem to In order to minimize the number of experiments required for have insignificant effect when the whole train data is used. identifying the optimal configurations, we divided the fine-tuning When comparing deterministic and random cutting variations, process into stages. Initially, we determined the best combina- the outcomes show that in the majority of the cases there are no tion of epoch number and learning rate for the baseline scenario statistically significant differences between the two. without augmentation. Subsequently, utilizing these identified In terms of the pairing requirements constraints, the most settings, we fine-tuned the augmentation probability for each constrained method, Subject, performs the worst. Conversely, specific augmentation technique. Ultimately, the chosen parame- the least constrained method, Subject and unconstrained Basic ters were utilized to train the model using the complete training approach, perform the best, whereas the moderately constrained set, with final evaluation carried out on the test set. Length method shows intermediate performances. Consequently, It is important to note that additional improvement of both we deduce that the pairing diversity holds substantial signifi- baseline and the methods’ accuracies could be achieved by ex- cance, wheres the strategy of how the pair instances are mixed panding the hyperparameter search grid. Although we fixed the together holds lesser importance. Comparing the Dataset and Ba- batch sized at 512 due to computational constraints, our experi- sic approaches does not give a straightforward answer on which mentation revealed that smaller batch sizes tend to yield slightly one works better. improved outcomes. Interesting view of performance of the synthetic data gener- ation is also measuring how much the dataset size appears to 4 RESULTS expand due to synthetic data generation technique relative to the To explore the impact of generating synthetic data on datasets of initial dataset size. For this purpose, we first linearly extrapolated varying sizes, we assessed the effectiveness of cutmix augmenta-the baseline method above the 100% of the training data as well tion technique across various proportions of the training data: as interpolated the values below 100%. After that, the accuracy of 10% (56 PCGs), 20% (112 PCGs), 40% (222 PCGs), 60% (334 PCGs), each method at given fraction of training data was compared to 80% (444 PCGs) , and 100% (554 PCGs). Alongside the cutmix the baseline to see at which data fraction the (inter/extrapolated) approaches, we present results for the no augmentation base- baseline achieves the same performance. The expansion percent- line scenario as well. Accuracy served as the key performance ages were then calculated by dividing the two data fractions. For evaluation metric, with findings based on the mean and standard example, the Dataset approach at 10% of training data achieves deviation (SD) derived from three runs. The results are shown in the same accuracy as the (interpolated) baseline at 28% of the Figure 3 and are given in Table 1. training data, thus the apparent data size expansion due to syn- thetic data generation equals to 280%. Complete results are given in Figure 4. Figure 4: Apparent data size expansion vs. training data fraction. 5 DISCUSSION AND CONCLUSIONS Our analysis indicates that the proposed cutmix augmentations Figure 3: Accuracy vs. training data fraction of the cutmix substantially improve the no augmentation baseline when it approaches and the baseline. comes to heart sound classification tasks using variously-sized training datasets. The results also show that there is no statisti- We see that all of the methods work decently with some of cally significant difference between the random and determin- the methods providing big improvements in accuracy over the istic cutting variations of the approaches. In terms of the pair- baseline. At 10% and 100% of the train data, accuracy improve- ing requirements constraints, the methods with no or very lit- ments over the baseline are 5.61% and 1.46%, respectively. To tle constraints, Basic and Dataset, show superior performance. 9 Information Society 2023, 9–13 October 2022, Ljubljana, Slovenia David Susič and Matjaž Gams Table 1: Methods’ accuracies for different percentages of the training set. The numbers are given as mean (SD) derived from three runs. The best results for each considered percentage of training data are written in bold. Percentage of training data Method 10% (N=56) 20% (N=112) 40% (N=222) 60% (N=334) 80% (N=444) 100% (N=554) Basic 66.78±1.11 72.28±0.88 75.42±0.55 74.41±0.99 75.65±0.57 74.86±1.24 Basic(r) 68.8±1.41 71.72±0.73 75.42±0.55 75.53±0.84 75.53±0.32 75.87±0.57 Length 68.57±0.16 71.49±0.32 73.85±0.42 75.98±0.88 75.2±0.84 76.09±0.27 Length(r) 69.36±0.55 71.27±1.11 73.63±0.42 75.76±0.73 75.08±0.27 77.67±0.88 Dataset 67.34±0.27 71.27±1.27 73.74±0.55 77.1±1.1 74.86±0.63 76.32±0.16 Dataset(r) 70.37±0.48 71.38±0.82 73.51±0.69 76.54±1.38 75.87±0.84 75.65±0.57 Subject 67.34±1.26 71.16±0.88 72.95±1.24 76.09±0.99 75.53±0.69 76.43±0.99 Subject(r) 67.56±1.77 71.16±0.69 73.51±0.42 76.21±1.11 74.86±0.88 75.53±0.63 baseline 64.76±0.57 68.57±1.38 73.06±0.48 74.19±0.84 74.75±0.82 76.21±1.61 Conversely, the most constrained method, Subject, performs the International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 1408–1411. doi: 10.1109/EMBC.2018.8512578. worst, whereas the moderately constrained Length method shows [7] Brian Kenji Iwana and Seiichi Uchida. 2021. An empirical survey of data intermediate performances. As a result, we deduce that the vari-augmentation for time series classification with neural networks. PLOS ONE, ability in pairings carries considerable importance, whereas the 16, 7, (July 2021), 1–32. doi: 10.1371/journal.pone.0254841. [8] Tomoya Koike, Kun Qian, Björn W. Schuller, and Yoshiharu Yamamoto. cutting approach, whether deterministic or random, holds com- 2021. Transferring cross-corpus knowledge: an investigation on data aug-paratively less importance. mentation for heart sound classification. In 2021 43rd Annual International Although the results of the best augmentation methods look Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 1976–1979. doi: 10.1109/EMBC46164.2021.9629714. promising, further experimentation is essential. These experi- [9] Chengyu Liu et al. 2016. An open access database for the evaluation of heart ments should include various selections of fractions of the train-sound algorithms. Physiological Measurement, 37, 12, (Nov. 2016), 2181. doi: 10.1088/0967- 3334/37/12/2181. ing data. This is important due to intrinsically stochastic nature [10] Shu Lih Oh, V. Jahmunah, Chui Ping Ooi, Ru-San Tan, Edward J Ciaccio, of selecting a fraction of the training dataset. This influence of Toshitaka Yamakawa, Masayuki Tanabe, Makiko Kobayashi, and U Rajendra randomness becomes stronger as the fraction size decreases. For Acharya. 2020. Classification of heart sound signals using a novel deep wavenet model. Computer Methods and Programs in Biomedicine, 196, 105604. instance, if we repeatedly choose 10% of the training set at ran- doi: https://doi.org/10.1016/j.cmpb.2020.105604. dom, the resulting subsets will exhibit significantly less overlap [11] Cristhian Potes, Saman Parvaneh, Asif Rahman, and Bryan Conroy. 2016. compared to the scenario where we repeatedly select 80% of the Ensemble of feature-based and deep learning-based classifiers for detection of abnormal heart sounds. In (Dec. 2016). doi: 10.22489/CinC.2016.182- 399. training set randomly. [12] Ali Raza, Arif Mehmood, Saleem Ullah, Maqsood Ahmad, Gyu Sang Choi, The cutmix augmentation methods have demonstrated their and Byung-Won On. 2019. Heartbeat sound signal classification using deep learning. Sensors, 19, 21. doi: 10.3390/s19214819. ability to yield superior results compared to the baseline. Never- [13] Heechang Ryu, Jinkyoo Park, and Hayong Shin. 2016. Classification of heart theless, further experiments are needed to compare these meth-sound recordings using convolution neural network. In 2016 Computing in ods with standard time-series augmentations, including tech-Cardiology Conference (CinC), 1153–1156. [14] David B. Springer, Lionel Tarassenko, and Gari D. Clifford. 2016. Logistic niques such as cutout, manifold mixup, noise induction, polarity regression-hsmm-based heart sound segmentation. IEEE Transactions on flip, gain adjustment, and more. Biomedical Engineering, 63, 4, 822–832. doi: 10.1109/TBME.2015.2475278. [15] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue ACKNOWLEDGMENTS Wang, and Huan Xu. 2021. Time series data augmentation for deep learning: a survey. In Proceedings of the Thirtieth International Joint Conference on The authors acknowledge the funding from the Slovenian Re-Artificial Intelligence, IJCAI-21. Zhi-Hua Zhou, editor. Survey Track. International Joint Conferences on Artificial Intelligence Organization, (Aug. 2021), search and Innovation Agency, Grant (PR-10495) (DS) and Basic 4653–4660. doi: 10.24963/ijcai.2021/631. core funding P2-0209 (MG). [16] Bin Xiao, Yunqiu Xu, Xiuli Bi, Weisheng Li, Zhuo Ma, Junhui Zhang, and Xu Ma. 2020. Follow the sound of children’s heart: a deep-learning-based REFERENCES computer-aided pediatric chds diagnosis system. IEEE Internet of Things Journal, 7, 3, 1994–2004. doi: 10.1109/JIOT.2019.2961132. [1] Neeraj Baghel, Malay Kishore Dutta, and Radim Burget. 2020. Automatic [17] Bin Xiao, Yunqiu Xu, Xiuli Bi, Junhui Zhang, and Xu Ma. 2020. Heart sounds diagnosis of multiple cardiac diseases from pcg signals using convolutional classification using a novel 1-d convolutional neural network with extremely neural network. Computer Methods and Programs in Biomedicine, 197, 105750. low parameter consumption. Neurocomputing, 392, 153–159. doi: https://do doi: https://doi.org/10.1016/j.cmpb.2020.105750. i.org/10.1016/j.neucom.2018.09.101. [2] Neeraj Baghel, Malay Kishore Dutta, and Radim Burget. 2020. Automatic [18] Te-chung Issac Yang and Haowei Hsieh. 2016. Classification of acoustic diagnosis of multiple cardiac diseases from pcg signals using convolutional physiological signals based on deep learning neural networks with aug-neural network. Computer Methods and Programs in Biomedicine, 197, 105750. mented features. In 2016 Computing in Cardiology Conference (CinC), 569– doi: https://doi.org/10.1016/j.cmpb.2020.105750. 572. [3] Martin Gjoreski, Anton Gradišek, Borut Budna, Matjaž Gams, and Gregor [19] Hong Yang and Travis Desell. 2022. Robust augmentation for multivariate Poglajen. 2020. Machine learning and end-to-end deep learning for the time series classification. (2022). arXiv: 2201.11739 [cs.LG]. detection of chronic heart failure from heart sounds. IEEE Access, 8, 20313– [20] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, 20324. doi: 10.1109/ACCESS.2020.2968900. and Youngjoon Yoo. 2019. Cutmix: regularization strategy to train strong [4] Ary L. Goldberger et al. 2000. Physiobank, physiotoolkit, and physionet. classifiers with localizable features. (2019). arXiv: 1905.04899 [cs.CV]. Circulation, 101, 23, e215–e220. eprint: https://www.ahajournals.org/doi/pd [21] George Zhou, Yunchan Chen, and Candace Chien. 2022. On the analysis of f /10.1161/01.CIR.101.23.e215. doi: 10.1161/01.CIR.101.23.e215. data augmentation methods for spectral imaged based heart sound classifi- [5] Ahmed Imtiaz Humayun, Shabnam Ghaffarzadegan, Md. Istiaq Ansari, Zhe cation using convolutional neural networks. BMC Medical Informatics and Feng, and Taufiq Hasan. 2020. Towards domain invariant heart sound ab-Decision Making, 22, (Aug. 2022). doi: 10.1186/s12911-022-01942-2. normality detection using learnable filterbanks. IEEE Journal of Biomedical and Health Informatics, 24, 8, 2189–2198. doi: 10.1109/JBHI.2020.2970252. [6] Ahmed Imtiaz Humayun, Shabnam Ghaffarzadegan, Zhe Feng, and Taufiq Hasan. 2018. Learning front-end filter-bank parameters using convolutional neural networks for abnormal heart sound detection. In 2018 40th Annual 10 Social Interaction Prediction from Smart-Phone Sensor Data Marcel Franse Martinšek Junoš Lukan∗ Larissa Bolliger Jožef Stefan Institute Jožef Stefan Institute Department of Public Health and Department of Intelligent Systems Department of Intelligent Systems Primary Care Ljubljana, Slovenia Ljubljana, Slovenia Ghent University marcel.franse.martinsek@ijs.si junos.lukan@ijs.si Ghent, Belgium larissa.bolliger@ugent.be Els Clays Primož Šiško Mitja Luštrek∗ Department of Public Health and Jožef Stefan Institute Jožef Stefan Institute Primary Care Department of Intelligent Systems Department of Intelligent Systems Ghent University Ljubljana, Slovenia Ljubljana, Slovenia Ghent, Belgium sisko.primoz@gmail.com mitja.lustrek@ijs.si els.clays@ugent.be ABSTRACT sensor data labelled with the number of people a person inter- Occupational stress is often associated with social interactions. acted with in the last ten minutes. Importantly, unlike many We used data collected as a part of the larger study to predict studies that focus on remote social interactions (e.g., [2]), con-whether a person is interacting with another while at work and trolled settings (e.g., [13, 8]), or use dedicated hardware (e.g., [7, at home. The dataset consisted of three weeks of data of 55 par- 14]), we predict everyday face-to-face interactions using personal ticipants and included information on application and screen smartphone data. Our focus is solely on predicting the number of usage, calls, location, and Bluetooth and Wi-Fi data. We exploited people present during interactions. We intend to use this as a fea- a question about work activities to obtain approximate labels of ture to predict social support quality and aspects of occupational social interactions. Additionally, we derived a feature indicating stress. indoor location, which did not turn out to be useful in our case. In a binary classification problem tackled with a random forest 2 DATA COLLECTION model, we achieved an 𝐹1 score of about 0.57. We obtained this dataset as a part of a larger study called Stress at work (STRAW; [1]), where we developed an Android application KEYWORDS [12] based on the AWARE framework [6]. This app collected social interactions, mobile sensing, indoor localization, stress participants’ self-reports and smartphone data, including screen detection and application use, GPS location, calls, and Bluetooth and Wi- Fi access points. Participants completed short questionnaires 1 INTRODUCTION approximately every 90 minutes, which included questions from psychometrically validated scales. They also contained questions The relationship between stress and social interactions is a com- about work activities from the previous 10 minutes. plex one. Social interactions can both offer relief from psycho- Participants reported diverse work activities, including breaks, logical distress and be influenced by stressors [18]. For instance, transit, individual work, and working with others. In the case of distress can trigger support from empathetic individuals, but working with others, they specified in-person, telecommunication- persistent distress can erode such support over time. Interper- based, lecture, or other interactions, also indicating the number sonal encounters, especially instrumental support (e.g., help with of people involved as one, two, or more than two. tasks), can act as a protective factor against occupational stress In the primary study, we collected data from 55 participants [16] and burnout [4], but they can also be sources of conflict. employed at research organizations in Belgium and Slovenia, In the field of artificial intelligence, researchers have recog- with diverse genders (26 women) and ages (mean age 34.9 years, nized the significance of social interactions, often using smart- ranging from 24 to 63 years). Each participant provided data for phones for monitoring due to their widespread use. Various smart- 15 working days, responding to the questionnaires roughly every phone sensors, including GPS, Bluetooth, Wi-Fi, accelerometer, 90 min as designed [11]. microphone, and on-device analytics like call and message moni- toring, application usage, screen activity, and battery status, have 3 TARGET AND FEATURE EXTRACTION been employed for this purpose [20]. This paper outlines a method for detecting a person’s involve- To create a predictive model for social interactions, we first de- ment in a social interaction. We utilize a dataset of smartphone fined the target variable by processing answers to the question work activity questions and then selected the most informative ∗Also with Jožef Stefan International Postgraduate School. features. With these labelled data and the chosen features at hand, we trained a supervised random forest model, as elaborated in Permission to make digital or hard copies of part or all of this work for personal Section 5. or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this 3.1 Label Extraction work must be honored. For all other uses, contact the owner/author(s). We processed answers to the question about work activity in the Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia © 2023 Copyright held by the owner/author(s). last ten minutes and extracted the following attributes for these 10-minute segments of collected data: 11 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Martinšek et al. • n_others: Number of interactions with others in last 10- In cases where features were highly similar and had correlations minutes (−1: exact number unknown, 0: alone, 1: one addi- close to 1, the selection was arbitrary. We chose the feature based tional person, 2: two additional people, 3: more than two on its simplicity and ease of interpretation. Figures 1 and 2 depict additional people). correlation matrices before and after this feature selection step. • inperson: Interaction in person (True/False). 1.00 • formal: Formality of interaction (True/False) ecog. Due to question interdependence some attribute values were Activity r 0.75 impossible to determine, resulting in missing values. For instance, participants mentioned activities like “coffee, lunch or toilet ound 0.50 break” for which we didn’t collect the number of people present. egr These attributes served two purposes: data filtering (inperson App. for 0.25 and formal) and segment labelling (n_others), resulting in a labelled dataset of 3371 10-minute segments with features listed 0.00 in the next section. The target variable distribution was highly Bluetooth imbalanced (see Table 1). 0.25 Calls 0.50 Label Examples Count Binary Merged Count 3 479 Locations 0.75 2 179 een 1157 Messages Scr 1 390 Speech Wifi 1.00 −1 109 ecog. ound Calls een Wifi egr Scr Bluetooth LocationsMessages Speech 0 2214 2214 Activity r App. for Table 1: Class distributions of extracted and merged labels. Figure 1: Correlation matrix of all features grouped by sensor, excluding categorical features To simplify the problem, we merged labels into a binary rep- 1.00 resentation, indicating whether the number of people involved was greater than 0 (True) or not (False). ound 0.75 egr 3.2 Features App. for 0.50 3.2.1 Initial Feature Set. Following the literature referenced in Bluetooth Section 1, we selected sensors from the collected data that could 0.25 be informative in predicting interactions. We computed first and Calls 0.00 second-order features from these sensors during the 10-minute segments. Below are the sensors we utilized, along with brief 0.25 descriptions and feature counts in parentheses: Locations L • Activity_recognition (1): Physical activity (walking, run- 0.50 ning, cycling). Messages een Messages Scr • Applications_foreground (99): Use of various categories 0.75 of phone applications. Speech Wifi W 1.00 • Bluetooth (30): Count and variance of visible Bluetooth devices (own and foreign). . ound Calls een Wifi W Scr egr ocations Speech • Calls (29): Call duration, quantity, and entropy of duration. egr Bluetooth Locations L Messages • Locations (21): GPS location features like variance and App. for average speed. • Messages (10): Number of sent/received messages. Figure 2: Correlation matrix after manual elimination of • Screen (7): Screen details, such as unlock duration. highly correlated features. • Speech (5): Detection of human speech via microphone input. We utilized sklearn’s [15] random forest (RF) implementation • Wi-Fi (8): Visible Wi-Fi access point count, Wi-Fi localiza- to reduce the dimensionality of feature space, employing the tion (see Section 4). Gini impurity metric. We explored various RF hyperparameters (max_depth: 10, 20, 30 and 40; n_estimators: 100, 500 and 1000) Here, the speech sensor was a custom-implemented method to observe feature selection and their impact on Gini impurity. that was running on the device and classified audio data online Some features were infrequently chosen, and some of the selected (see our previous work, [9], for more details) ones had minimal impact (less than 0.02 reduction) on Gini impu- 3.2.2 Feature selection. In previous work [12, 11], we implerity. We performed 10-fold cross-validation to assess the selected mented numerous features from the sensors mentioned earlier. feature set’s predictive quality using the 𝐹1 metric. By evaluating To address high feature correlation, we reduced them to a smaller how features influenced the 𝐹1 score and retaining those signifi- subset before classification. We applied a threshold of 𝑟 = 0.8 cantly impacting it while removing others with minor effects on and retained only one feature from each highly correlated group. Gini impurity, we derived the final feature set: 12 Social Interaction Prediction from Smart-Phone Sensor Data Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia • phone_screen: Screen unlocks and total screen unlocked by visualizing the relationship between different combinations time. of principal components (see Fig. 3). • phone_speech: Mean and standard deviation of human voice proportion in audio. • phone_applications_foreground: Duration of communica- tion and tools app usage. • phone_locations: Average speed, time in significant loca- tions, location variance logarithm, and time at the most visited location. • phone_bluetooth: Number of unique devices, total scans of others’ devices, and their mean. • phone_wifi_visible: Number of unique sensed Wi-Fi access points. • phone_calls: Mean call duration (outgoing and incoming). Most of the features used were included in RAPIDS [19] with implementation of Doryab’s [5, 3] Bluetooth and location features. 4 WI-FI LOCALIZATION In addition to the simple features described in the previous sec- tion, we aimed to incorporate indoor location data into our mod- els, focusing on a subject’s home and work settings based on GPS location. We identified these settings by distinguishing between the two most frequented locations within each setting. 4.1 Rough GPS location For rough GPS location determination, we utilized GPS cluster Figure 3: Visualization of a subject’s Wi-Fi fingerprint clus- labels provided by Doryab [5]. We employed a simple heuristic, tering results using principal component pairs as returned considering the most common cluster label between 00:00 and by principle component analysis with colours representing 6:00 as home and between 6:00 and 20:00 as work. Subsequently, different clusters. we segregated each subject’s Wi-Fi scan data into home and work categories, excluding data recorded outside of these time After clustering individual-specific data, we assigned an indi- segments. This allowed us to analyze each setting independently. cator label unique to each person, which couldn’t be generalized. For each location, we determined whether it represented the most 4.2 Indoor Wi-Fi location or second-most frequent location in a given setting. We split the We adapted Wi-Fi localization from a supervised to an unsuper- data into home and work subsets, performing separate Wi-Fi data vised learning approach due to a lack of calibration steps during clustering for each, resulting in four localization features. data collection. This adjustment was made individually for each • Scope: General location classification as most common (1), subject’s data, given the differences in their environments. second most common (2), or neither (0), alongside personal The Wi-Fi sensor data included three key variables: timestamp, cluster labels. detected device’s media access control (MAC) address, and re- • Setting: Home or work (determined by rough GPS loca- ceived signal strength indicator (RSSI). We filtered out entries tion). If outside these settings, corresponding features were with uncommon MAC addresses (appearing fewer than 100 times) set to -1. and grouped the remaining entries into 1-minute segments to Ultimately, these features proved uninformative for our use create Wi-Fi fingerprints. Each fingerprint, specific to a subject case and were removed during feature selection in Section 3.2.2. and setting, comprised a combination of MAC addresses and This may be attributed to limitations in our modified clustering corresponding RSSI values recorded during that minute. This approach without initial calibration steps during data collection. often resulted in 70 or more unique values, presenting a high- dimensional clustering challenge. 5 INTERACTION CLASSIFICATION We selected the random forest model for its versatility in han- 4.3 Clustering dling both categorical and numerical data, capturing complex, We applied 𝑘-means clustering with the silhouette measure to non-linear feature-target relationships through ensemble tech- determine the optimal value of 𝑘. Given the uncertainty about niques. To tackle dataset imbalance, we initially used imblearn’s the number of significant locations per subject, we started with [10] random oversampling and undersampling, with undersam- 𝑘 = 12 and reduced it, ultimately stopping at a minimum of 𝑘 = 4, pling proving superior in our 𝐹1 score evaluation. Additionally, as we expected at least four significant locations (e.g., desk, lunch, we imputed missing values with zeros when it logically repre-meeting room, other). sented the intended feature content, e.g., missing duration values For each clustering result, we computed the silhouette mea- indicating non-usage of specific applications. sure [17] and selected the 𝑘 value yielding the highest silhouette Table 2 presents LOSO and 10-fold cross-validation results score. Additionally, we manually assessed the clustering results for models, comparing undersampling, oversampling, and the 13 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Martinšek et al. original class distribution. We explored various random forest [4] J. D. DeFreese and Jason P. Mihalik. 2016. Work-based social interactions, configurations by adjusting maximum tree depth (Max_d) and perceived stress, and workload incongruence as antecedents of athletic trainer burnout. Journal of Athletic Training, 51, 1, (Jan. 2016), 28–34. doi: the number of trees (N_estim). 10.4085/1062-6050-51.2.05. Dataset imbalance posed challenges; oversampling and the [5] Afsaneh Doryab, Prerna Chikarsel, Xinwen Liu, and Anind K. Dey. 2018. base approach led to numerous false classifications in the majority Extraction of behavioral features from smartphone and wearable data. abs/1812.10394, (Dec. 18, 2018). arXiv: 1812.10394 [cs.CY]. doi: 10.485 class, while undersampling misclassified more majority class 50/ARXIV.1812.10394. examples as the minority. A more balanced class distribution [6] Denzil Ferreira, Vassilis Kostakos, and Anind K. Dey. 2015. AWARE: Mobile yielded a less biased model and a slight increase in the average context instrumentation framework. Frontiers in ICT, 2, 6, 1–9. doi: 10.3389 /fict.2015.00006. 𝐹1 score with undersampling. LOSO was notably affected by [7] Michele Girolami, Fabio Mavilia, and Franca Delmastro. 2020. Sensing social sampling methods, with undersampling performing better due to interactions through BLE beacons and commercial mobile devices. Pervasive and Mobile Computing, 67, (Sept. 2020), 101198. doi: 10.1016/j.pmcj.2020.101 the problem’s subject-dependent nature and overlap in examples 198. between train and test splits in 10-fold CV. [8] Kleomenis Katevas, Katrin Hänsel, Richard Clegg, Ilias Leontiadis, Hamed Haddadi, and Laurissa Tokarchuk. 2019. Finding dory in the crowd. In Proceedings of the 1st Workshop on Machine Learning on Edge in Sensor Hyperparameters LOSO 10-Fold Systems. ACM, (Nov. 2019). doi: 10.1145/3362743.3362959. Max_d N_estim Under Over Base Under Over Base [9] Marko Katrašnik, Junoš Lukan, Mitja Luštrek, and Vitomir Štruc. 2019. Razvoj postopka diarizacije govorcev z algoritmi strojnega učenja. In Pro-40 100 0.573 0.504 0.471 0.553 0.530 0.471 ceedings of the 22nd International Multiconference INFORMATION SOCIETY – 40 500 0.562 0.508 0.482 0.549 0.551 0.476 IS 2019. Slovenian Conference on Artificial Intelligence (Ljubljana, Slovenia, 40 1000 0.557 0.512 0.481 0.556 0.556 0.479 Oct. 7–11, 2019). Mitja Luštrek, Rok Piltaver, and Matjaž Gams, editors. Vol. A, 57–60. http://library.ijs.si/Stacks/Proceedings/InformationSociety/20 30 1000 0.555 0.513 0.483 0.542 0.551 0.480 19/IS2019_Volume_A.pdf. 20 1000 0.561 0.519 0.477 0.547 0.556 0.482 [10] Guillaume Lemaître, Fernando Nogueira, and Christos K. Aridas. 2017. 10 1000 0.558 0.524 0.462 0.559 0.552 0.447 Imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research, 18, Table 2: Table displaying 𝐹1 scores for the model with vari-17, 1–5. http://jmlr.org/papers/v18/16-365.html. ous hyperparameter combinations. It presents results for [11] Junoš Lukan, Larissa Bolliger, Els Clays, Oscar Mayora, Venet Osmani, and Mitja Luštrek. 2021. Participants’ experience and adherence in repeated both CV and LOSO approaches using undersampling (Un-measurement studies among office-based workers. In Adjunct Proceedings der), oversampling (Over), and no class imbalance mitiga-of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous tion (Base). Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers. Afsaneh Doryab, Qin Lv, and Michael Beigl, editors. ACM, (Sept. 2021), 528–531. doi: 10.1145/3460418.3479367. [12] Junoš Lukan, Marko Katrašnik, Larissa Bolliger, Els Clays, and Mitja Luštrek. 2020. Straw application for collecting context data and ecological momen-tary assessment. In Proceedings of the 23rd International Multiconference 6 CONCLUSIONS INFORMATION SOCIETY – IS 2020. Slovenian Conference on Artificial Intelligence (Ljubljana, Slovenia). Mitja Luštrek, Rok Piltaver, and Matjaž Gams, In this paper, we developed an RF model to detect personal inter-editors. Vol. A. (Oct. 2020), 63–67. actions using various phone sensors, achieving an [13] Aleksandar Matic, Venet Osmani, and Oscar Mayora-Ibarra. 2012. Anal- 𝐹1 score of 0.57 ysis of social interactions through mobile phones. Mobile Networks and after feature filtering. The most informative features included Applications, 17, 6, (Aug. 2012), 808–819. doi: 10.1007/s11036-012-0400-4. the speech sensor and the subject’s application usage statistics. [14] Niklas Palaghias, Seyed Amir Hoseinitabatabaei, Michele Nati, Alexander Surprisingly, the indoor location did not improve results, possibly Gluhak, and Klaus Moessner. 2015. Accurate detection of real-world social interactions with smartphones. In 2015 IEEE International Conference on due to the COVID-19 pandemic’s remote work context. Despite Communications (ICC). IEEE, (June 2015). doi: 10.1109/icc.2015.7248384. social interactions not being the focus of the primary study, we [15] F. Pedregosa et al. 2011. Scikit-learn: machine learning in Python. Journal achieved a moderate of Machine Learning Research, 12, 2825–2830. 𝐹1 value. We anticipate that with a more [16] Maria C. W. Peeters, Bram P. Buunk, and Wilmar B. Schaufeli. 1995. Social focused data collection approach and a balanced dataset, fur-interactions, stressful events and negative affect at work: a micro-analytic ther improvements can be made, potentially enabling accurate approach. European Journal of Social Psychology, 25, 4, (July 1995), 391–401. doi: 10.1002/ejsp.2420250404. estimation of the number of people in social interactions. [17] Peter J. Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied ACKNOWLEDGEMENTS Mathematics, 20, (Nov. 1987), 53–65. doi: 10.1016/0377-0427(87)90125-7. [18] Marybeth Shinn, Stanley Lehmann, and Nora W. Wong. 1984. Social interThe stress at work (STRAW) study was supported by the Research action and social support. Journal of Social Issues, 40, 4, (Jan. 1984), 55–76. Foundation – Flanders, Belgium (FWO) under Grant (project no. doi: 10.1111/j.1540-4560.1984.tb01107.x. [19] Julio Vega, Meng Li, Kwesi Aguillera, Nikunj Goel, Echhit Joshi, Kirtiraj G.0318.18N); and the Slovenian Research and Innovation Agency Khandekar, Krina C. Durica, Abhineeth R. Kunta, and Carissa A. Low. 2021. (ARIS) under Grant (project ref. N2-0081). Reproducible analysis pipeline for data streams. Open-source software to process data collected with mobile devices. Frontiers in Digital Health, 3. REFERENCES doi: 10.3389/fdgth.2021.769823. [20] Heng Zhang, Ahmed Ibrahim, Bijan Parsia, Ellen Poliakoff, and Simon [1] Larissa Bolliger, Junoš Lukan, Mitja Luštrek, Dirk De Bacquer, and Els Clays. Harper. 2022. Passive social sensing with smartphones: a systematic review. 2020. Protocol of the STRess at work (STRAW) project: how to disentangle Computing, 105, 1, (Aug. 2022), 29–51. doi: 10.1007/s00607-022-01112-2. day-to-day occupational stress among academics based on EMA, physio- logical data, and smartphone sensor and usage data. International Journal of Environmental Research and Public Health, 17, 23, (Nov. 2020), 8835. doi: 10.3390/ijerph17238835. [2] Mehdi Boukhechba, Alexander R. Daros, Karl Fua, Philip I. Chow, Bethany A. Teachman, and Laura E. Barnes. 2018. DemonicSalmon: monitoring mental health and social interactions of college students using smartphones. Smart Health, 9-10, (Dec. 2018), 192–203. doi: 10.1016/j.smhl.2018.07.005. [3] Luca Canzian and Mirco Musolesi. 2015. Trajectories of depression: unob-trusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, Osaka, Japan, (Sept. 2015), 1293–1304. isbn: 9781450335744. doi: 10.1145/2750858.2805845. 14 Comparison of Advanced Processing Methods for PPG Denoising using a Novel Signal Quality Metric ∗ Kitoshi Kawai Gašper Slapničar kawai- kitoshi38@eis.t.u- tokyo.ac.jp gasper.slapnicar@ijs.si The University of Tokyo Jožef Stefan Institute Tokyo 113-8656, Japan Jožef Stefan International Postgraduate School Jamova cesta 39 Ljubljana, Slovenia Akira Hirose Mitja Luštrek ahirose@ee.t.u- tokyo.ac.jp mitja.lustrek@ijs.si The University of Tokyo Jožef Stefan Institute Tokyo 113-8656, Japan Jožef Stefan International Postgraduate School Jamova cesta 39 Ljubljana, Slovenia Figure 1: The top dashed box is baseline signal processing, the bottom right is advanced signal processing. The Upper left subplot is the raw recording of RGB channels. Both baseline and advanced signal processing were applied, and the lower subplot shows the denoised signal and its waveform evaluation. ABSTRACT denoising. Applying our state-of-the-art pipeline to our dataset demonstrated a 5-12% improvement in the resting state and a Photoplethysmography (PPG) is a non-invasive method measur- 27-28% improvement in the active state in terms of our proposed ing blood volume changes using light. By illuminating tissue and Signal Quality Index metric. This was compared to the baseline observing light variations, PPG captures blood flow fluctuations. denoising which employed only outlier removal and bandpass Vital physiological parameters can be obtained by analyzing the filtering. PPG signals, such as heart rate and oxygen saturation. In this study, we acquired PPG signals from the palm of a hand using KEYWORDS camera-based remote sensing. However, this approach is espe- cially sensitive to noise due to contact-free nature. We propose photoplethysmography, noise removal, empirical mode decom- novel metrics for waveform evaluation of PPG signals and a position, principal component analysis, independent component unique pipeline combining several advanced methods for PPG analysis, signal quality assessment Permission to make digital or hard copies of part or all of this work for personal 1 INTRODUCTION or classroom use is granted without fee provided that copies are not made or Non-invasive physiological monitoring has grown essential in distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this healthcare. Photoplethysmography (PPG) uses light to track blood work must be honored. For all other uses, contact the owner /author(s). volume changes. PPG often employs contact sensors, capturing Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia light variations through the skin, which unveil blood flow char- © 2023 Copyright held by the owner/author(s). acteristics and allow the extraction of metrics like heart rate (HR) 15 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Kitoshi Kawai, Gašper Slapničar, Akira Hirose, and Mitja Luštrek and oxygen saturation. However, obtaining such parameters is Lazaro et al. [4] and if the number of peaks did not correspond not straightforward due to PPG being susceptible to noise such to the HR specified previously, the data was discarded as too as motion artifacts. noisy during recording. For peak detection, we set a window (the Remote PPG (rPPG) acquires PPG signals without direct con- size of which was based on HR), identified the highest gradient tact, avoiding the discomfort of attached devices. However, rPPG within, and marked the subsequent local maxima as the systolic signals are more vulnerable to noise, especially from unintended peak. To isolate a PPG signal cycle, we also discerned the valley movements. This leads to degradation of the PPG signal. When points, taken as the minimum between two systolic peaks [11]. estimating blood pressure using PPG, accurate detection of sys- 3.1.1 Hampel Filter. The Hampel filter identifies outliers in time- tolic peaks is crucial [12]. Motion artifacts, however, can induce series data through the following procedures. problematic peak shifts in the PPG signal. The aim of this study was to refine the rPPG signal and as- • Take three points before and after the target data point sess its waveform with our proposed Signal Quality Index (SQI) and calculate the median within this window. detailed in Section 3.3. By examining different processing tech- • Compute the median absolute deviation (MAD) by deter- niques for noisy rPPG signals, we wanted to enhance PPG tech- mining the median of the absolute differences between nology and improve physiological monitoring reliability. each data point in the window and the window’s median. • Multiply the MAD by a constant to estimate the standard 2 RELATED WORK deviation under the assumption that the data follows a normal distribution. Several methods have been proposed to remove motion artifacts • If the absolute difference between a data point and the from PPG signals, with statistical approaches like Principal com- median of its window exceeds three times the MAD, the ponent analysis (PCA), Independent component analysis (ICA), data point is deemed an outlier and is replaced with the and Empirical mode decomposition (EMD) [3, 9, 6]. In estimating window’s median. the PPG-derived respiratory rate, separate utilization of PCA and EMD has demonstrated errors of 1.48 and 0.07 breaths/min, 3.1.2 Butterworth Filter. After processing the data using the respectively, suggesting their efficacy in noise removal. However, Hampel filter, we used the Butterworth filter to extract compo- the performance without any filter remained unassessed, so it is nents from 0.5Hz to 3.0Hz. By using this filter, we eliminated difficult to evaluate its performance. In another study, Motin et al. both low-frequency and high-frequency components. integrated EMD and PCA for PPG-based breath rate estimation. They reported absolute errors of 0.9 breaths and 9.9 breaths in 3.2 Advanced Signal Processing 5-minute recordings, depending on the dataset [6]. In contrast, Within the Butterworth filter’s cutoff frequency range, we furthe exclusive use of a bandpass filter resulted in errors of 5.4 and ther denoised using either EMD, ICA, PCA, or a combination 10.5 breaths in 5-minute respectively [8], indicating that combin-of these methods. In this study, we considered the following ing different processing methods might be effective in improving combinations: robustness. In these studies, the PPG signal was not evaluated directly but rather through the quality of variables extracted (i) BSP (iv) BSP + EMD from it. In contrast, Slapničar et al. focused on the waveform (ii) BSP + PCA (v) BSP + EMD + PCA of the PPG signal itself and evaluate the signal in a data-driven (iii) BSP + ICA (vi) BSP + EMD + ICA manner [11]. In their study, a metric called Signal Quality Indices (SQIs) was defined and a threshold was set to extract only good waveforms. 3.2.1 Empirical Mode Decomposition. EMD is a method of de- composing a signal into physically meaningful components and 3 METHODOLOGY is used to analyze nonlinear or non-stationary signals. Using EMD, signals can be decomposed into "intrinsic mode functions" Our methodology for processing PPG data was divided into two (IMFs), which correspond to components of different frequency. stages: baseline signal processing (BSP) and advanced signal We referred to the detailed algorithm in this paper [1]. For each processing. In the BSP stage, we first applied the Hampel filter component of the IMFs, only the first intrinsic mode function followed by the Butterworth filter to detect outliers and extract (IMF) consistently had peaks within the predefined HR range. specific frequency components. In the advanced signal processing Thus, we selected this first IMF as the PPG signal. stage, additional noise removal was accomplished using either EMD, PCA, ICA, or a combination of these methods. 3.2.2 Principal Component Analysis. PCA compresses multidi- We then assessed the performance of previously described mensional data and extracts essential features [10]. In this study, PPG processing methods using custom SQIs, which we describe each frame captured by the camera was divided into a 3x3 grid, in detail in Section 3.3. In this study, it was assumed that the HR giving 9 regions. Each region’s average pixel value formed a was between 50 and 140 beats per minute (50 ≤ HR ≤ 140) as nine-dimensional input. This was reduced to two dimensions, the HR of the subjects in our experiment ranged from about 60 representing the PPG signal and noise. By comparing the num- to 130. ber of peaks in the post-PCA data with the expected HR, the first component of the PCA was determined to be the PPG data. 3.1 Baseline Signal Processing However, it should be noted that PCA can occasionally produce We first used the Hampel filter to detect outliers. Afterwards, inverted outputs because the sign of eigenvectors, which deter- since the PPG data was expected to have some frequency range, mine the direction of principal components, is arbitrary and can a Butterworth filter was employed to extract specific frequency be positive or negative. To account for this, we checked the cor- components. Based on the data obtained from these processes, relation coefficient between the PCA output and the PPG signal we detected peaks using peak detection algorithm detailed by which was processed with BSP after averaging the entire frame. 16 Comparison of Advanced Processing Methods for PPG Denoising using a Novel Signal Quality Metric Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia If the correlation coefficient was negative, indicating inversion, 4 EXPERIMENTS the output was then multiplied by -1. 4.1 Recording Setup 3.2.3 Independent Component Analysis. ICA is a technique used In this study, we collected rPPG data from 11 subjects, both male to decompose multivariate signals into statistically independent and female, aged 22 to 45 years old. The iDS 3040SE-Q RGB cam- components [2]. Like PCA, each frame was divided into nine era set at 250 fps, equipped with the Sony IMX273 1/3" CMOS regions for input, and the output consisted of two dimensions: image sensor and the iDS-5M23-C1618 16 mm lens, was used for the PPG signal and noise. However, since ICA does not define the recordings. Each subject underwent four 30-second record- the order or sign of the separated sources, it is ambiguous to ings. For each recording, rPPG was obtained from red (R), green identify which outputs are related to PPG or noise, and also (G), and blue (B) channels. Of the four sessions, the initial two the components can be inverted. Therefore, considering both recordings were conducted in a "rest state" while the subsequent the output data sets and their inverses (making a total of four two were in an "active state". The rest state entailed subjects being potential sets), the data with the highest correlation coefficient in a relaxed condition for the recording, achieved by prompting to the PPG signal which was processed with BSP after averaging them to engage in meditation or deep breathing prior to the ses- the entire frame was selected as the PPG data. sion. Conversely, the active state referred to recordings taken after subjects performed physical activities like jumping or squats 3.3 PPG Waveform Quality Assessment to elevate their HR. Consequently, it was expected that the PPG For PPG waveform evaluation, a template was first created. This signal would be more stable in the rest state and the noise level template was then used to calculate the SQI by comparing the would be higher in the active state. denoised PPG data with the template. 4.2 Evaluation Pipeline 3.3.1 Template Wave Formation. The length of one cycle tem- Using the PPG data obtained by the signal processing described in plate waveform was computed using autocorrelation analysis. Section 3, the SQI of each channel (R, G, and B) was computed for Given the previously defined HR range, the potential cycle length each recording. The mean and standard deviation of the SQI were range was denoted by Eq. (1) computed across both rest and active states. Our analysis entailed fps × 60 fps × 60 comparing the BSP with advanced methods to identify the most ≤ 𝐿 ≤ (1) effective processing technique in terms of SQI. Additionally, we 140 50 evaluated the performance differences between the rest (stable where fps is the sampling frequency in Hertz (fps = 250 Hz) PPG) and active (potentially noisier PPG) states. and 𝐿 is the template length in samples. To compare channels, we also computed the mean and stan- We then shifted the signal by all the lengths within that range, dard deviation of SQI using the PPG data obtained only by the the correlation coefficients of the original and shifted signals best-performing processing method for both states. R, G, and B were compared and the shift length with the highest correlation channels were then evaluated based on these SQI values as given coefficient was selected as the template width. With the length 𝐿 in Table 1 (Best method). of the template determined, we extracted segments of width 𝐿 from each valley point of the PPG signal. The template waveform 5 RESULTS was then created by averaging these segments. The primary objective of this study was to investigate which 3.3.2 Signal Quality Indices. We defined SQIs based on method combination of methods was most effective in noise removal. We proposed by Slapničar et al. [11]. However, in our study, we made computed the SQI for each RGB channel in both rest and active some modifications to SQI3 to ensure normalization between -1 states, as detailed in Table 1. and 1. The template, created as detailed above, was compared and In the rest state with minimal motion artifacts, the highest SQI evaluated against each cycle of the PPG signal using the three was achieved using the BSP, EMD, and PCA filters. Conversely, following SQIs. post-motion data, which had pronounced motion artifacts, exhib- • SQI1: Data of length 𝐿 starting from each valley point ited maximal SQI when BSP, EMD, and ICA filters were applied. is directly compared with the template to calculate the Relative to the baseline SQI obtained by the BSP, enhancements Pearson’s correlation coefficient. were: 12% (R), 5% (G), and 10% (B) in the rest state, and 27% (R and • SQI2: Data between two adjacent valleys is considered as G) and 28% (B) in the active state. In the rest state with minimal one cycle. If the waveform length of one cycle is different original noise, filtering slightly enhanced the PPG signal. In con- from the template, the data is compared with the tem- trast, for active states with pronounced noise, the improvement plate by resampling to determine Pearson’s correlation was substantial. The SQI disparity between states was minimized coefficient. with EMD, highlighting its efficiency in motion artifact removal. • SQI3: Data between two adjacent valleys is considered as Besides, combining EMD with ICA and PCA further improved one cycle, dynamic time warping (DTW) is employed to robustness. Moreover, as highlighted by Jihyoung et al.[5], the find similar points with the template. DTW is a method green channel consistently provided superior PPG signals com- used to align two sequences by warping their time axis pared to the other channels. to best match each sequence to the other [7]. The corre-A specific view of the evaluation of the denoised PPG signal sponding points are then used to calculate the Pearson’s and waveform is shown in Fig. 2. The figure depicts the denoised correlation coefficient. PPG signal using the optimal pipeline (BSP + EMD + PCA) and Finally, the final SQI was computed by taking the average of the corresponding SQI assessment for the green channel during SQI1, SQI2, and SQI3, as expressed in Eq. (2). the rest state. For instance, during the collapse of the PPG wave- form (approximately at frame number 700), the SQI manifests a 1 𝑆𝑄 𝐼 = (𝑆𝑄𝐼 1 + 𝑆𝑄𝐼 2 + 𝑆𝑄𝐼 3) (2) diminished value. 3 17 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Kitoshi Kawai, Gašper Slapničar, Akira Hirose, and Mitja Luštrek R_SQI G_SQI B_SQI rest active rest active rest active BSP 0.73 ± 0.14 0.65 ± 0.13 0.80 ± 0.13 0.66 ± 0.12 0.74 ± 0.15 0.65 ± 0.13 BSP+PCA 0.74 ± 0.16 0.61 ± 0.12 0.80 ± 0.13 0.63 ± 0.11 0.75 ± 0.15 0.61 ± 0.12 BSP+ICA 0.71 ± 0.17 0.64 ± 0.13 0.77 ± 0.16 0.70 ± 0.14 0.72 ± 0.17 0.65 ± 0.13 BSP+EMD 0.79 ± 0.14 0.81 ± 0.14 0.83 ± 0.13 0.82 ± 0.14 0.80 ± 0.13 0.82 ± 0.13 BSP+EMD+PCA 0.82 ± 0.14 0.79 ± 0.14 0.84 ± 0.13 0.81 ± 0.15 0.82 ± 0.15 0.79 ± 0.14 BSP+EMD+ICA 0.79 ± 0.15 0.82 ± 0.14 0.83 ± 0.15 0.84 ± 0.13 0.80 ± 0.15 0.83 ± 0.14 Best method 0.82 ± 0.14 0.84 ± 0.13 0.83 ± 0.15 Table 1: The mean and standard deviation of SQI across different processing methods, scenarios, and color channels. The final "Best method" is computed only from best-performing methods (BSP+EMD+PCA and BSP+EMD+ICA). Figure 2: The top subplot shows the PPG signal (green channel) using basic signal processing and the best combination of advanced signal processing. The bottom subplot shows the corresponding waveform evaluation. SQI values, presented in Eq. (2) as the mean of SQI1, SQI2, and REFERENCES SQI3, should be adjusted based on the application. For applica- [1] Norden E Huang, Zheng Shen, Steven R Long, Manli C Wu, Hsing H Shih, Quanan Zheng, Nai-Chyuan Yen, Chi Chao Tung, and Henry H Liu. 1998. tions where precise waveform morphology are vital [12], SQI2 The empirical mode decomposition and the hilbert spectrum for nonlinear might be more crucial. Conversely, when precise morphology is and non-stationary time series analysis. Proceedings of the Royal Society of not as important, but only the dominant peak matters, SQI1 (with London. Series A: mathematical, physical and engineering sciences, 454, 1971. [2] Aapo Hyvärinen and Erkki Oja. 2000. Independent component analysis: its minimal computational demand) or SQI3 might be enough. algorithms and applications. Neural networks, 13, 4-5. Moreover, the template for SQI was created using the PPG signal [3] Byung S Kim and Sun K Yoo. 2006. Motion artifact reduction in photo- obtained from each processing method. This can affect the wave- plethysmography using independent component analysis. IEEE transactions on biomedical engineering, 53, 3. form, as some systolic peaks get shifted by signal processing, as [4] Jesús Lázaro, Eduardo Gil, José Marıa Vergara, and Pablo Laguna. 2013. Pulse observed in Fig. 2. This is an important limitation, that requires rate variability analysis for discrimination of sleep-apnea-related decreases in the amplitude fluctuations of pulse photoplethysmographic signal in further future investigation on how noise removal affects the children. IEEE journal of biomedical and health informatics, 18, 1. details of the PPG waveform. [5] Jihyoung Lee, Kenta Matsumura, Ken-ichi Yamakoshi, Peter Rolfe, Shinobu Tanaka, and Takehiro Yamakoshi. 2013. Comparison between red, green and 6 CONCLUSION blue light reflection photoplethysmography for heart rate monitoring during motion. In 2013 35th annual international conference of the IEEE engineering In this study, we assessed PPG signal processing techniques on in medicine and biology society (EMBC). IEEE. in-house rPPG dataset in terms of SQI. The BSP+EMD+PCA [6] Mohammod Abdul Motin, Chandan Kumar Karmakar, and Marimuthu Palaniswami. 2019. Selection of empirical mode decomposition techniques combination outperformed the BSP by 5-12% in the rest state, for extracting breathing rate from ppg. IEEE Signal Processing Letters, 26, 4. while the BSP+EMD+ICA combination improved by 27-28% in [7] Meinard Müller. 2007. Dynamic time warping. Information retrieval for music the active state. The superior quality and stability of the green and motion. [8] Lena Nilsson, Anders Johansson, and Sigga Kalman. 2000. Monitoring of channel in PPG reaffirms the findings of previous studies [5]. respiratory rate in postoperative care using a new photoplethysmographic The results highlight the potential for more advanced noise re-technique. Journal of clinical monitoring and computing, 16. [9] B Prathyusha, T Sreekanth Rao, and D Asha. 2012. Extraction of respiratory moval by integrating different signal processing, which is crucial rate from ppg signals using pca and emd. International Journal of Research for estimating HR and other physiological information from PPG in Engineering and Technology, 1, 2. signal. Future work will explore the impact of proposed noise [10] Jonathon Shlens. 2014. A tutorial on principal component analysis. arXiv preprint arXiv:1404.1100. removal techniques on precise morphology and subtle details of [11] Gasper Slapničar, Mitja Luštrek, and Matej Marinko. 2018. Continuous blood the PPG waveform. pressure estimation from ppg signal. Informatica, 42, 1. [12] Gašper Slapničar, Wenjin Wang, and Mitja Luštrek. 2022. Feasibility of ACKNOWLEDGEMENTS remote pulse transit time estimation using narrow-band multi-wavelength camera photoplethysmography. In Adjunct Proceedings of the 2022 ACM This work was supported by the Jožef Stefan Institute and by International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers. The University of Tokyo. 18 Machine-learning Methods for Analysis of Gene Expression Data Marko Jordan Jakob Valič Mitja Luštrek Department of Intelligent Systems, Department of Intelligent Systems, Department of Intelligent Systems, Jožef Stefan Institute Jožef Stefan Institute Jožef Stefan Institute Ljubljana, Slovenia Ljubljana, Slovenia Jožef Stefan International marko.jordan@ijs.si Postgraduate School Ljubljana, Slovenia mitja.lustrek@ijs.si ABSTRACT Gene expression and similar types of biological data are often studied because they provide rich information about the state of an organism, and machine-learning models can be built to predict the organism’s state from such data. A common challenge is that the number of genes, which correspond to features for machine learning, is typically large compared to the number of samples. This is tackled by feature-selection and dimensionality-reduction methods. The former have the advantage of providing information on important features, which may allow reducing the number of features that have to be collected prospectively. We present two feature-selection methods: an ensemble of established filter methods, and a custom bi-directional wrapper designed specifically for problems where the number of features is large compared to the number of instances and there may be interactions between the features. We compare the methods on a dataset consisting of multiple cohorts, by training models on some cohorts and testing on others, which best approximates real-life use. We find that some informative features can be identified, and while the wrapper does not outperform the filter ensemble, it does work better on the more challenging cases. KEYWORDS Gene expression, machine learning, feature selection Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia © 2023 Copyright held by the owner/author(s). 19 Identifying Bumblebee Buzzes Using Neural Networks Tilen Šket, David Susič Candace Galen, Johannes Marina Arbetman, Victoria Department of Intelligent Systems, Schul Campopiano Robinson Jožef Stefan Institute Division of Biological Sciences, Universisad Nacional del Comahue Ljubljana, Slovenia University of Missouri Bariloche, Argentina Columbia, Missouri, United States Cristian Alfonso Villagra Gil, Anton Gradišek Valentina Herrera anton.gradisek@ijs.si Universidad Metropolitana de Department of Intelligent Systems, Ciencias de la Educación Jožef Stefan Institute Santiago, Chile Ljubljana, Slovenia ABSTRACT Bumblebee monitoring can be carried out either manually, with people in the field observing target plants and writing down Bumblebees as important pollinators are keystone species and as notes, or with the use of technology. Clearly, even if manual such are crucial for functioning of the ecosystem. In Patagonia monitoring has the advantage of an expert observer being able to (in Argentina and Chile), the native species Bombus dahlbomii produce high-quality records, this approach is time and resource is under threat by the spread of invasive European species that consuming. Therefore, we explore the possibilities that smart sen- were introduced for agricultural purposes. An important aspect sors can offer, in particular sound recordings with microphones of conservation efforts is monitoring of the presence of native coupled with signal analysis with AI algorithms. and invasive species. Here we report on the analysis of sound When it comes to utilizing machine learning (ML) for insect recordings using neural networks, with the aim of detecting the detection and/or classification from sound recordings, a variety of presence of bumblebee buzzes in the recordings. approaches have been established, many of which are influenced KEYWORDS by the broader fields of general audio and image data analysis. These include strategies such as using features derived from time- bumblebees, neural networks, buzz detection, spectrograms series signals as inputs for classical ML models [11, 7], using spectrograms as inputs for neural networks (NNs) [14, 9, 8] and 1 INTRODUCTION employing time-series directly as NN inputs [13]. In the previous studies of bumblebee sounds, some of the co- Bumblebees (genus Bombus) are a group of social insects from authors of this paper have investigated whether it is possible to the bee family Apidae. They are important pollinators, often distinguish bumblebee species and type based on flight buzzing more efficient than honeybees. This comes in part due to their sound [4], where, in brief, a larger body size of a bumblebee will different morphology and lifestyle. They can forage in cold and likely result in a lower buzzing frequency. In another study, we rainy weather when honeybees will not even exit the hive, and monitored "bumblebee traffic" using microphones next to the due to a special technique called "buzz pollination" they can pol-nest-box entrance, where we used sound analysis to count the linate flowers where the pollen needs to be extracted - this is for workers flying in and out [5, 3]. In those studies, the approach example relevant for tomatoes, where bumblebee pollination has with microphones turned out to be highly efficient in detecting an extremely important commercial role as well. In ecosystems buzzes, as well as distinguishing between the arrivals and depar- where honeybees are absent, such as in the mountains, plants tures. In the present paper, we go one step further. We wanted to rely on bumblebees and other wild pollinators. develop an algorithm that would identify bumblebee buzzes from The largest bumblebee species, Bombus dahlbomii, lives in a long recording at several chosen plants during a field study temperate forests of South America, in southern Argentina and in South America. The task is more complex than the one with Chile[1]. It is an important pollinator of local plant species, how-bumblebee traffic as the bees do not necessarily pass close by ever, it is being threatened by the introduced species (B. terrestris the microphone, and the buzzes sounds are furthermore masked and B. ruderatus) that have been brought from Europe for agricul- by various ambient noises. Here, we present the initial results tural purposes in the past decades [12, 10, 2]. As these species are of an approach using convolutional neural networks for buzz expanding in range and increasing in numbers, the population of B. dahlbomii detection. We discuss the accuracy of the algorithm and outline is diminishing and is faced with possible extinction. the future steps. In order to boost the conservation efforts, careful monitoring of populations of local and introduced bumblebee species is needed as a starting point for policy makers and conservationists to plan 2 DATASET their actions. Recordings of bumble bees were obtained in November, 2022 at forest understory and rural field sites in Argentina and Chile. Permission to make digital or hard copies of part or all of this work for personal Audiomoth (Open Acoustic Devices) and DB-9 USB recording or classroom use is granted without fee provided that copies are not made or devices with a detection range of 1-2 m were used to collect distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this single channel audio recordings at 16 kHz. In this study, we used work must be honored. For all other uses, contact the owner /author(s). roughly 750 minutes of such recordings. This is a labeled dataset, Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia however, the size of all the recordings obtained during the field © 2023 Copyright held by the owner/author(s). study is substantially larger. It should be stressed that the labels 20 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Šket et al. Table 1: Test, validation, and test data set distributions. for buzzes were all approximate, as the expert noted when they saw the bumblebee, not necessarily when the bumblebee sound was picked by the microphone. Split Sound #Spectrograms As the recordings included various ambient sounds such as buzz 96 lawnmowers and passing cars (which produce somewhat similar Train no buzz 440 spectrograms to those of bumblebees, but more on this later), we machine 24 underwent a process of isolating highly noisy segments. These buzz 24 segments were then combined to create a supplementary dataset Validation no buzz 110 totaling 2 minutes in duration, referred to as the machine sounds machine 6 dataset. buzz 1984 3 METHODOLOGY Test no buzz 20564 machine 0 The objective of our study was to identify timestamps within the recordings that denote the occurrence of bumblebee buzzes. The methodology included initial recording preprocessing and data partitioning, selection of neural network architecture, and from the short recordings which either included buzz throughout determination of the model training settings. the whole length or did not include a buzz at all. Conversely, the test dataset was composed of spectrograms extracted from 3.1 Data Preprocessing the longer recordings. Spectrograms containing the bumblebee Bumblebees exhibit a natural flying frequency in the range of buzz sound were assigned with a label of 1, whereas the ones 200 Hz, alongside higher harmonics that can extend up to 1500 without buzzes were assigned a label of 0. The training data Hz (at least those we can detect). Similarly, when the bumblebee recordings were all manually checked and labeled by the experts. engages in sonication on a flower, the emitted sounds resonate at In contrast, the labeling process for the testing dataset involved a natural frequency of about 300 Hz, with corresponding higher experts identifying the presence of buzzing in the long recordings harmonics [4]. It is important to stress that bumblebees are much by providing a single time-stamp per bumblebee buzz. This time-larger and heavier than most of the other pollinating insects stamp indicated the approximate detection time, meaning some present in the area (such as honeybees or solitary bees), so their spectrograms might have been mislabeled. In addition, the cases buzzing frequencies will be lower than those of other pollinators. where the buzzing sound persisted for longer than 4 seconds, at For the initial preprocessing step, recordings were subjected least one spectrogram included a (part of a) buzz but was labeled to a frequency filtering process, limiting frequencies up to 1500 with 0. Hz. Next, the recordings were segmented into 4-second intervals The training data was split into train and validation sets with with a 50% overlap. These discrete intervals were then trans- the ratio of 4:1. The folds were stratified with respect to the formed into Mel spectrograms using the fast Fourier transform. dataset (bumblebee, machines) and with respect to the class label Consequently, each instance was represented as a 2-dimensional (buzz or no buzz). Our train, validation, and test set distributions array of dimensions 128x128. Unlike conventional spectrograms, are shown in Table 1. Mel spectrograms utilize the Mel frequency scale to mimic the As expected, we see that the test set is greatly unbalanced human ear’s perception of distinct frequencies (as an ear has a (most of the time, there are no bumblebees). In the approxi- logarithmic response, not a linear one). The examples of prepro- mately 710 minutes of the test recordings, the experts detected cessed spectrograms to be used in our model are given in Figure 1. and marked down 992 time-stamps of bumblebee buzzes, result- Note that the added machine sounds were preprocessed equally. ing in 1984 of the spectrograms being labeled with a buzz due to the 50% temporal overlap of the spectrograms (each time-stamp was covered by two spectrograms). 3.3 Neural Network Architecture From the machine learning perspective, our study was an image classification task, thus we implemented a convolutional neural network (CCN) as they are regarded as the state-of-the art in the image classification domain [6]. Our model architecture comprises three convolutional layers, followed by a fully connected network with two dense layers and a single output neuron. The convolutional layers included, starting from the initial layer, 128, 256, and 512 filters all size at Figure 1: Spectrograms of a bumblebee buzz (left) and no 3x3. Each convlotutional layer was accompanied by a 2x2 max buzz (right) pooling layer. The output of the last convolutional layer was flattened and input to a dense layer consisting of 8192 neurons, followed by a batch normalization, and a dense layer consisting of 512 neurons. Ultimately, the final output was consolidated 3.2 Data Partitioning into a single neuron, generating a numeric value between 0 and Our dataset comprises recordings of varying lengths: some are 1 that signified the degree of confidence in predicting a buzz. short (3-10 seconds), while others are longer (around 10 minutes). The activation function of all convolutional layers and the first The training dataset was composed of the spectrograms extracted dense layer was ReLu, while the last dense layer used a sigmoid 21 Identifying Bumblebee Buzzes Using Neural Networks Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Table 2: CNN model and the baseline performance results. activation function. The schematic of the the CNN used in our Baseline predicted randomly assuming train data class analysis is given in Figure 2. probabilites. As the train and validation data volume is fairly small, we tried implementing dropout regularization to the CNN layers. Surprisingly, our findings revealed that dropout did not yield Model Metric Result any performance improvement. Consequently, we decided not to Accuracy 76% incorporate any regularization into the final model architecture. Precision 60% CNN 3.4 Model Training Settings Recall 96% F1–score 74% The selected loss function to be optimized during the neural net- Accuracy 74% work training was the binary cross-entropy. The validation data Precision 9% was used to control and adjust the settings during the training. Baseline Recall 21% For the optimization algorithm, we employed the Adam opti- F1–score 12% mizer along with a learning rate scheduler. The initial learning −4 rate was set to 10 , which was dynamically adjusted as training progressed. Specifically, the learning rate was reduced by a factor of 10 after each epoch where there was no observed improvement in the validation loss. This adaptive approach helped the model model missed buzz-containing spectrograms entirely. This dis- navigate towards convergence. The batch size was 32 and the crepancy arises, same as before, due to the characteristic nature model trained for 50 epochs. of buzzes, which often span across multiple spectrogram dura- tions. Conversely, instances where the model correctly predicted 4 RESULTS a buzz, but with a slight delay compared to the expert-annotated Considering the significant imbalance inherent in our dataset, time-stamp, were marked as incorrect predictions due to minor we employed a heuristic approach to evaluate our model’s per- label misalignment. formance effectively. Because the test set spectrograms were While our model missed 298 (30%) buzz events, it often identi- extracted from 10 minute continuous intervals, there was a sig- fied events as buzz despite the label indicating otherwise. This nificant amount of temporal correlation. The buzz sounds some- discrepancy was attributed to label misalignment, absence of buzz times lasted continuoulsy for a few minutes, however during that labels alongside bumblebee presence, and primarily the model’s period, only two spectrograms ware labeled with a buzz, as only unfamiliarity with noise types not encountered in training (e.g., one time-stamp was assigned for that bumblebee and there is cars, lawn mowers, distant music). Figure 4 displays examples of 50% overlap between the spectrograms. a false negative (left) and a false positive (right) predictions. Hence, our devised methodology operated as follows: when In cases like this, where the event one is trying to detect is rare, our model detected a buzz across a series of consecutive spectro-it is crucial to catch as many occurrences as possible. Missing grams and at least one of these spectrograms has been expert- out on detections (false negatives) is more problematic than hav- annotated as a buzz, all the spectrograms within this contiguous ing a some incorrect ones (false positives). Mistaken predictions sequence were counted as true positives. can be manually reviewed by humans, which is quicker than re- We assess the effectiveness of the buzz detection model using screening the entire recordings. Therefore, although improving a set of four key metrics: F1-score, precision, recall, and accuracy. event detection rate and minimizing the false positive rate are How the metrics are calculated from true positive (TP), true crucial, the relatively high occurrence of false positive outcomes negative (TN), false positive (FP), and false negative (FN) values is of lesser concern. is given in Eqs. (1)–(4). We compare the results against a baseline model that predicted randomly assuming the class probabilities of 5 DISCUSSION AND CONCLUSIONS the training data. The models’ performance metrics are compared As demonstrated in the analysis of the algorithm performance, to the baseline in Table 2. We see that our model demonstrates the neural network does detect buzzes. Just looking at the com-significant improvements across all metrics, with the exception of parison with the validation set where (nominally) true positive accuracy, where it shows a modest advantage over the baseline. events are labeled, the performance is still not close to the values 𝑇 𝑃 + 𝑇 𝑁 that would be of strong use for actual field monitoring applica- 𝐴𝑐𝑐𝑢𝑟 𝑎𝑐𝑦 = (1) 𝑇 𝑃 + 𝑇 𝑁 + 𝐹 𝑃 + 𝐹 𝑁 tions. Nevertheless, we should stress that there are several improve- 𝑇 𝑃 𝑃 𝑟 𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = (2) ments possible. As outlined above, we are (i) dealing with a highly 𝑇 𝑃 + 𝐹 𝑃 unbalanced dataset, which is (ii) labeled following a protocol used 𝑇 𝑃 in biological field studies where the event labels only approxi- 𝑅𝑒𝑐𝑎𝑙𝑙 = (3) 𝑇 𝑃 + 𝐹 𝑁 mately corresponding to the buzzes picked up by microphones (or several segments contain buzzes and there is only one label). Af- 2 · 𝑃𝑟 𝑒𝑐𝑖𝑠𝑖𝑜𝑛 · 𝑅𝑒𝑐𝑎𝑙𝑙 𝐹 1–𝑠𝑐𝑜𝑟 𝑒 = (4) ter manually inspecting several miss-classified segments, the next 𝑃 𝑟 𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 step is to improve the training process of the CNN with larger The confusion matrix depicting the model’s predictions is pre- training sets, which will allow us to more accurately analyze the sented in Figure 3. It’s important to note that the false negative extensive dataset recorded during the field study. Ultimately, this rate of the predictions might be underestimated. This rate sig- approach will allow us to identify bumblebee buzzes in the area nifies the instances where the model failed to identify buzzes and, following an approach similar to the one in Gradišek et al. based on time-stamps, rather than indicating instances where the [4], distinguishing between the native and introduced species. 22 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Šket et al. Figure 2: Convolutional neural network architecture used in our study. DATA AND CODE AVAILABILITY STATEMENT The data and the code that support the findings of this study are available from the authors upon reasonable request. REFERENCES [1] A.H. Abrahamovich, Maria C. Telleria, and N.B. Diaz. 2001. Bombus species and their associated flora in argentina. Bee World, 82, 2, 76–87. [2] Benoit Geslin and Carolina L. Morales. 2015. New records reveal rapid geographic expansion of bombus terrestris linnaeus, 1758 (hymenoptera: apidae), an invasive species in argentina. Check List, 11, 1620–1620. [3] Anton Gradišek, Nicolas Cheron, David Heise, Candace Galen, and Janez Grad. 2018. Monitoring bumblebee daily activities using microphones. In Proceedings of the 21st Annual International Multiconference Information Society–IS, 5–8. [4] Anton Gradišek, Gašper Slapničar, Jure Šorn, Mitja Luštrek, Matjaž Gams, and Janez Grad. 2017. Predicting species identity of bumblebees through Figure 3: Confusion matrix of the model’s predictions. analysis of flight buzzing sounds. Bioacoustics, 26, 1, 63–76. [5] David Heise, Zachary Miller, Ellie Harrison, Anton Gradišek, Janez Grad, and Candace Galen. 2019. Acoustically tracking the comings and goings of bumblebees. In 2019 IEEE Sensors Applications Symposium (SAS). IEEE, 1–6. [6] Kavi B. Obaid, Subhi R. M. Zeebaree, and Omar M. Ahmed. 2020. Deep learning models based on image classification: a review. doi: 10.5281/ZEN ODO.4108433. [7] Satoshi Kawakita and Kotaro Ichikawa. 2019. Automated classification of bees and hornet using acoustic analysis of their flight sounds. Apidologie, 50, 1. doi: 10.1007/s13592- 018- 0619- 6. [8] Ali Khalighifar, Daniel Jiménez-García, Lindsay P Campbell, Koffi Men-sah Ahadji-Dabla, Fred Aboagye-Antwi, Luis Arturo Ibarra-Juárez, and A Townsend Peterson. 2021. Application of Deep Learning to Community-Science-Based Mosquito Monitoring and Detection of Novel Species. Journal of Medical Entomology, 59, 1, (Sept. 2021), 355–362. eprint: https://acade mic . oup . com / jme / article - pdf / 59 / 1 / 355 / 41963852 / tjab161 . pdf . doi: 10.1093/jme/tjab161. [9] Ivan Kiskin, Bernardo Pérez Orozco, Theo Windebank, Davide Zilli, Marianne Sinka, Kathy Willis, and Stephen Roberts. 2017. Mosquito detection with neural networks: the buzz of deep learning. (2017). arXiv: 1705.05180 Figure 4: False negative (left) and false positive (right) spec- [stat.ML]. [10] Josefin A. Madjidian, Carolina L. Morales, and Henrik G. Smith. 2008. Dis-trograms. The false negative spectrogram includes an audi- placement of a native by an alien bumblebee: lower pollinator efficiency ble buzz, however, the signal strength in the buzz frequency overcome by overwhelmingly higher visitation frequency. Oecologia, 156, range is overpowered by the prevailing background noise 835–845. [11] Quoc Viet Phung, Iftekhar Ahmad, Daryoush Habibi, and Steven Hinckley. in the lower parts of the spectrogram. The false positive 2017. Automated insect detection using acoustic features based on sound spectrogram, on the other hand, exhibits distinctive hori-generated from insect activities. Acoustics Australia, 45, 2. doi: 10.1007/s408 57- 017- 0095- 6. zontal lines that are similar to the patterns generated by the [12] Juliana Ordones Rego, Clemens Schlindwein, Ruben Garrido, and Victor H bumblebees, but were generated from unrelated sources Monzón. 2021. Low fruit set in an endangered tree: pollination by exotic that misled the model. bumblebees and pollen resource for relictual native bees. Arthropod-Plant Interactions, 15, 491–501. [13] Myat Su Yin, Peter Haddawy, Borvorntat Nirandmongkol, Tup Kongtha- worn, Chanaporn Chaisumritchoke, Akara Supratak, Chaitawat Sa-ngamuang, and Patchara Sriwichai. 2021. A lightweight deep learning approach to ACKNOWLEDGEMENTS mosquito classification from wingbeat sounds. In Proceedings of the Conference on Information Technology for Social Good (GoodIT ’21). Association The authors acknowledge the funding from the Slovenian Refor Computing Machinery, Roma, Italy, 37–42. isbn: 9781450384780. doi: search and Innovation Agency, Grant (PR-10495) and Basic core 10.1145/3462203.3475908. [14] Myat Su Yin et al. 2023. A deep learning-based pipeline for mosquito detec-funding P2-0209. The research was partially funded by the Na- tion and classification from wingbeat sounds. Multimedia Tools and Applica-tional Geographic Society. We also acknowledge the students tions, 82, 4. doi: 10.1007/s11042-022-13367-0. who assisted with field data collection. 23 Prepoznavanje aktivnosti čebel na panjskem žrelu s pomočjo strojnega vida in drugih metod Recognition of Bee Activity in the Hive Entrance Using Machine Vision and Other Methods Oskar Rotar Maks Žnidaršič Tian Vesel oskar.rotar@dijak.gjp.si maks.znidarsic@dijak.gjp.si tian.vesel@dijak.gjp.si Gimnazija Jožeta Plečnika, Ljubljana Gimnazija Jožeta Plečnika, Ljubljana Gimnazija Jožeta Plečnika, Ljubljana Šubičeva ulica 1 Šubičeva ulica 1 Šubičeva ulica 1 Ljubljana, Slovenija Ljubljana, Slovenija Ljubljana, Slovenija Mag. Darja Silan Dr. Janko Božič darja.silan@gjp.si janko.bozic@bf .uni- lj.si Gimnazija Jožeta Plečnika, Ljubljana Biotehniška fakulteta - Univerza v Šubičeva ulica 1 Ljubljani Ljubljana, Slovenija Jamnikarjeva ulica 101 Ljubljana, Slovenija Slika 1: Zaznave čebel narejene z uporabo YOLOv5 modela. POVZETEK to their small size and fast movements, it is challenging for re- searchers to study them. With the use of computer vision, a V tem delu smo raziskovali možnost uporabe najnaprednejših technology that interprets and understands visual information, tehnologij pri analizi gibanja čebel med vhodom v panj. Čebele as well as Multi Object Tracking (MOT), we were able to develop imajo ključno vlogo pri opraševanju in so bistvene za ohranja- a modern solution to the otherwise difficult task of bee tracking. nje uravnoteženega ekosistema, vendar je njihovo proučevanje With this, the study of bees becomes less constrained and far zaradi majhne velikosti in hitrega gibanja zahtevno. Z uporabo more effective. The research shows that the use of numerous strojnega vida, tehnologije, ki omogoča računalniško interpre- modern technologies offer a promising new approach to the col- tacijo in razumevanje vizualnih podatkov ter metod sledenja lection of a large number of accurate data on bee movement. večim objektom (MOT) nam je uspelo razviti moderno rešitev za njihovo proučevanje. Raziskava je pokazala, da uporaba teh teh- nologij ponuja nov, obetaven pristop k zbiranju velikega števila KLJUČNE BESEDE natančnih podatkov o gibanju čebel. zaznava čebel, računalniški vid, MOT, YOLO, Apis mellifera car- ABSTRACT nica In this article we researched the possible use of advanced tech- KEYWORDS nologies for the analysis of bee movement in front of a beehive. Bees play a key role in the pollination of plants, they have an bee recognition, computer vision, MOT, YOLO, Apis mellifera immense part in keeping our ecosystem balanced. However, due carnica Permission to make digital or hard copies of part or all of this work for personal 1 UVOD or classroom use is granted without fee provided that copies are not made or V nedavni zgodovini smo bili priča velikim tehnološkim dosež- distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this kom in inovacijam. Svet hitro napreduje, vendar nekatera podro-work must be honored. For all other uses, contact the owner /author(s). čja še zmeraj ostajajo nespremenjena. V Sloveniji je čebelarstvo Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia velik del naše bogate zgodovine in kulture. Zaradi podnebnih © 2023 Copyright held by the owner/author(s). sprememb, porasta parazitov in bolezni čebel ni bilo čebelarstvo 24 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Oskar Rotar, Maks Žnidaršič, Tian Vesel Tabela 1: Specifikacije naučenih modelov. Vse ločljivosti so nikoli tako zahtevno, zato želimo z uporabo sodobnih metod in kvadratne. Negativna velikost paketa označuje samodejno moderne tehnologije prispevati k uspešnejšemu uvajanju infor- izbiro macijske tehnologije za upravljanje čebeljih družin in uspešnejše delo čebelarjev in raziskovalcev. Želeli smo razviti učinkovit, stabilen in posplošen model strojnega učenja za zaznavo čebel, ime epohe velikost paketa model ločljivost ki vzdrži visoko uspešnost tudi v primerih, ko so algoritmične 1 BDM1-s 100 16 v5s 640 metode neuspešne. Model te podatke nato poda algoritmu za BDM2-s 100 16 v5s 416 sledenje večim objektom, ki natančno sledi gibanju čebel. Ves BDM3-s 100 16 v5s 640 program omogoča samodejno zbiranje velikih količin podatkov BDM4-s 174 -1 v5s 640 o gibanju na panjskem žrelu, kar bi lahko pomagalo pri preuče- BDM5-m 268 -1 v5m 640 vanju njihovega obnašanje in komuniciranja. 1 BDM6-s 174 -1 v5s 640 BDM7-x 27 -1 v5x 640 2 METODOLOGIJA BDM8-s 20 32 v5s 640 2.1 Izbira modela zaznavanja objektov BDM9-s 184 42 v5s 640 BDM10-7x 87 8 v7x 640 V nalogi smo za zaznavo čebel uporabili konvolucijske nevronske BDM11-7 166 10 v7 640 mreže. Pri učenju modela zaznave čebel smo uporabili modele BDM12-7 150 11 v7 640 dveh družin prednaučenih nevronskih mrež, namenjenih zaznavi BDM13-7x 150 11 v7x 640 objektov. To sta YOLOv5[1], ki je eden najbolj razširjenih in BDM14-m 135 16 v5m 640 najboljših modelov za zaznavanje objektov in YOLOv7[2], ki je BDM15-m 163 -1 v5m 640 novejša arhitektura. YOLOv7 domnevno opisuje boljše rezultate, 1Ta modela sta bila naučena na procesorju in ne grafični kartici. kot YOLOv5. Modele teh dveh družin modelov smo dodatno učili na lastnih podatkih. Učili smo prednaučene modele YOLOv5s, YOLOv5m, YOLOv5x in modela YOLOv7 ter YOLOv7x. Skupaj smo učili 15 modelov, štiri iz družine YOLOv7 in 11 iz 2.1.1 Poimenovanje modelov. Imena vseh modelov smo začeli družine YOLOv5 (Tabela 1). Pri izdelavi modelov smo imeli dva z akronimom BDM (model zaznave čebel - angl. Bee Detection Model glavna cilja - hitrost izvajanja in uspešnost zaznavanj. Večinoma ). Temu sledita identifikacijska številka modela. Za tem smo učili manjše, YOLOv5s modele. Za to smo imeli dva razloga: je oznaka za vrsto modela. Modeli YOLOv5s imajo oznako s, hitrejše učenje teh modelov in majhno število učnih podatkov. YOLOv5m oznako m, YOLOv5x oznako x, YOLOv7 oznako 7 in Po prvih testnih učenjih smo ugotovili, da so modeli že zelo hitro YOLOv7x oznako 7x. Primer poimenovanja modela je BDM1-s. začeli zaznavati velik delež čebel. Kljub temu smo učili tudi nekaj To je model z identifikacijsko številko 1, ki je naučen na predna- v5m modelov in en v5x model. Učili smo tudi YOLOv7 modele, saj učenem modelu YOLOv5s (Tabela 1, stolpec za ime). smo upali, da bodo nudili višjo hitrost in kvaliteto, kot napisano v članku[2]. 2.2 Obdelava video posnetkov Pri učenju model računa svojo uspešnost s funkcijo izgube. Pri Uporabljali smo 10 različnih video posnetkov panjskega žrela. YOLOv5 in YOLOv7 modelih je ta funkcija seštevek (pri YOLOv7 Izbrali smo zahtevne in raznolike posnetke (različna povečava, obtežen seštevek) funkcije izgube objektnosti, regresijske funk- osvetljenost, barva panja, gostota čebel), da bi dosegli čim bolj cije izgube omejevalnega okvira in funkcije izgube klasifikacije posplošene modele. Posnetki so bili posneti na Urbanem učnem (1). Slednji je v našem primeru 0, ker uporabljamo le en razred čebelnjaku botaničnega vrta v Ljubljani[3]. Za učenje smo iz objektov. video posnetkov na enakomernih časovnih intervalih vzeli 2100 slik. Te slike smo označili v orodju Roboflow[4]. Pri učenju se 𝑙𝑜𝑠𝑠 = 𝑙 + 𝑙 + 𝑙 (1) 𝑜𝑏 𝑗 𝑏𝑜 𝑥 𝑐𝑙 𝑠 je ločljivost slik samodejno zmanjšala na ločljivost modela. 70 % slik smo postavili v učno množico, 20 % v validacijsko in 10 % v 2.4 Sledenje gibanju čebel testno množico. Za sledenje smo uporabili knjižnici Norfair[5] ter ByteTrack[6]. 2.3 Učenje modelov Sledenje gibanja čebel smo izvedli s pomočjo podatkov, ki nam jih je vrnil model. Pridobljene vrednosti iz modela in sledilca Vse modele smo lahko učili na istih podatkih, saj YOLOv5 in smo primerjali ter vsakemu objektu določili identifikacijsko šte- YOLOv7 uporabljata isti format za označevanje slik. Da bi prišli vilko. Knjižnica Norfair[5] je to opravila sama, pri knjižnici Byte-do čim boljšega končnega modela, smo naučili več modelov. Pri Track[6] pa smo to dosegli z zunanjo knjižnico Onemetric[7]. učenju modelov smo spreminjali velikost paketa, število epoh, izbiro prednaučenega modela in število slik. Ostale hiperparame- 2.5 Analiza modelov tre smo ohranili na standardni nastavitvi. Pri optimizaciji modela je uporabljen stohastični gradientni spust. Isti algoritem je bil Hitrost delovanja modelov smo testirali na dveh posnetkih. Izbrali uporabljen pri učenju prednaučenih modelov [1]. smo posnetek, ki ima majhno število čebel in počasno gibanje Velikost paketa, pri kateri smo lahko učili, je neposredno ome- ter posnetek, ki ima več čebel ter bolj kaotično gibanje. Po uče- jena z velikostjo grafičnega pomnilnika na grafični kartici (8 nju nam je program vrnil dve verziji modela - tisto, ki je bila GB). Pri učenju manjših modelov (posebej YOLOv5s) se proces shranjena po zadnji epohi in tisto, ki je dosegla najvišjo metriko samodejno predčasno konča, ko se mAP@[0.5:0.95] modela ne mAP@[0.5:0.95] na validacijski množici. Vedno smo uporabili spreminja, kar omeji prekomerno prilagajanje modela. To temelji model z najvišjo mAP@[0.5:0.95], saj smo se s tem izognili pre- na validacijski množici. komernemu prileganju modela. 25 Prepoznavanje aktivnosti čebel Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia 2.5.1 Uspešnost zaznavanja modelov. Pri izračunu uspešnosti 1 modela se uporablja več metod in funkcij[8]. Modele smo primer-Natančnost Priklic jali po uspešnosti, priklicu, mAP@0.5 in mAP@[0.5:0.95]. Merila smo izračunali na testni množici, torej slikah na katerih modeli niso bili naučeni. 0.9 2.6 Analiza metod sledenja premikanja čebel Pri analizi premikanja čebel smo, kot pri analizi modelov, upo- rabili dva posnetka. Za zaznavo čebel smo uporabljali model 0.8 BDM15-m, saj se ta model izvaja hitro z dobro uspešnostjo. Za ocenjevanje natančnosti zaznave smo z obema knjižnicama pre- šteli čebele, ki so se približale vhodu oziroma oddaljile od vhoda v panj. Te podatke smo nato primerjali z ročno preštetimi količi- 0.7 nami. 3 REZULTATI BDM1-s BDM2-s BDM4-s BDM3-s BDM8-s BDM6-s BDM5-s BDM9-s BDM14-m BDM15-m BDM7-x BDM11-7 BDM12-7 BDM13-7x BDM10-7x 3.1 Hitrost izvajanja modelov Pri primerjavi modelov lahko vidimo, da gostota čebel na po- snetku ne vpliva na hitrost zaznav. Na hitrost je imela bistven Slika 3: Natančnost in priklic modelov strojnega učenja vpliv le velikost modela. Najhitrejši so bili vsi modeli prednau- čenega modela YOLOv5s, ki so tudi najmanjši, najpočasnejši pa modeli YOLOv7x (Slika 2). Skoraj vsi modeli so dovolj hitri, da z zmogljivo grafično kartico posnetke obdelajo sproti pri hitrosti 0.6 vsaj 40 slik na sekundo. Testiranje je potekalo na grafični kartici RTX 3070 Ti. model s 0.4 25 model m model x 20 model 7 mAP@[.5:.95] model s model 7x 0.2 model m [ms] 40 slik/s model x 15 60 slik/s model 7 izvajanja model 7x 10 0 čas BDM1-s BDM2-s BDM3-s BDM4-s BDM5-s BDM6-s BDM8-s BDM9-s BDM14-m BDM15-m BDM7-x BDM11-7 BDM12-7 BDM10-7x BDM13-7x 5 0 Slika 4: Kvaliteta zaznavanja na testni množici po metriki BDM1-s BDM2-s BDM3-s BDM4-s BDM5-s BDM6-s BDM8-s BDM9-sBDM14-m BDM15-m BDM7-xBDM11-7 BDM12-7 BDM10-7x BDM13-7x mAP@[.5:.95] času izvajanja (Slika 2), je kvaliteta zaznav YOLOv7 primerljiva z Slika 2: Povprečne hitrosti modelov. modeli YOLOv5. Pri slabših modelih je najpogostejša napaka združevanje več čebel v eno in zaznavanje grč. To lahko najlažje vidimo na pri- merjavi med najboljšim in najslabšim modelom (Slika 5). 3.2 Uspešnost detekcije modelov 3.3 Rezultati sledenja Pri primerjavi modelov po priklicu in natančnosti (Slika 3) vidimo, da bistveno izstopata modela BDM1-s in BDM3-s, saj več kot Pri primerjavi natančnosti štetja vidimo, da je Norfair bolj natan- tretjine čebel ne zaznata, petina zaznav je pa napačnih. Slabi čen od ByteTracka (Slika 6). Norfair[5] se je še posebej izkazal pri rezultati so očitni tudi po merilu mAP. Merilo mAP@[.5:.95] štetju čebel, ki vstopajo v panj, saj na posnetku z več čebelami od najbolje prikaže dejansko razliko med modeli (Slika 4). realnih rezultatov odstopa le za 2,5 % . Obe knjižnici zaznamuje Vsi slabši modeli imajo arhitekturo YOLOv5s, vendar imajo slabše štetje čebel, ko se oddaljujejo od panja. Sledenje je bilo v drugi YOLOv5s modeli (BDM4-s, BDM5-s in BDM6-s) dobro uspe- vseh primerih hitrejše od 50 slik/sekundo (20 ms na sliko). Norfair šnost. Ni neposredne povezave med arhitekturo in kvaliteto mo- je rahlo počasnejši na posnetku z več čebelami in rahlo hitrejši dela. Najbolje se je izkazal model BDM11-7. 8 modelov ima po- na posnetku z manj čebelami, vendar je ta razlika majhna, okoli doben mAP@[.5:.95], malo nad 0.6. Kljub močno počasnejšemu 0,8 milisekunde. 26 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Oskar Rotar, Maks Žnidaršič, Tian Vesel preprostejši, specializirani strojni opremi, kar bi omogočilo za- znavo na kraju, kjer so bili posnetki pridobljeni[10]. Uspešnost zaznave in sledenja bi lahko izboljšali tudi s sestavo poligona, ki omeji hitrost gibanja čebel na posnetem območju. Z vključi- tvijo termalne kamere v zaznavo, bi lahko model še izboljšali. Ne nazadnje, bi lahko natančnost algoritmičnega sledenja gibanja izboljšali z uporabo umetne inteligence, z učenjem nevronske mreže, ki bi zagotavljala boljšo časovno doslednost sledenja. 5 ZAKLJUČEK Štetje čebel, ki priletijo iz oziroma v panj lahko čebelarjem nudi vpogled v stanje panja. Avtomatizirano zbiranje podatkov o čebe- (a) Zaznave najboljšega modela BDM11-7 lah ni uporabno le za čebelarje, ampak lahko tudi podpira mnogo drugih raziskav o vedenju in delovanju teh čudovitih živali. Z uporabo računalniškega vida in metodami sledenja večim objek- tom (MOT) nam je uspelo dokazati, da je taka avtomatizacija mogoča, natančna in uporabna. ZAHVALA Zahvalili bi se radi naši mentorici Mag. Darji Silan, ki nam je nudila podporo in pomagala organizirati celotno izdelavo naloge. Radi bi se tudi zahvalili somentorju Dr. Janku Božiču, ki nam je pomagal s strokovno literaturo in splošnim znanjem na tem področju ter za posnetke čebel. LITERATURA (b) Zaznava najslabšega modela BDM1-s [1] Rač. prog. Glenn Jocher in sod., ultralytics/yolov5: v7.0 - YOLOv5 SOTA Realtime Instance Segmentation nov. 2022. url: https://github.com/ultralyt Slika 5: Primerjava boljšega in slabšega modela ics/yolov5Retrieved 26. feb. 2023 from. [2] Chien-Yao Wang, Alexey Bochkovskiy in Hong-Yuan Mark Liao. 2022. YO- LOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors, (jul. 2022). https://github.com/WongKinYiu/yolov7. Prešteto ročno Norfair ByteTrack [3] 2023. Urbani učni čebelnjak, botanični vrt ljubljana. (2023). Retrieved 28. feb. 2023 from http://www.botanicni- vrt.si/urbani- ucni- cebelnjak. [4] Rač. prog. Brad Dwyer, Joseph Nelson (2022), Jacob Solawetz in sod., ver. 1.0. url: https://robof low.comRetrieved 26. feb. 2023 from. [5] Rač. prog. Tryolabs, Norfair. url: https : / / github . com / tryolabs / norf air 100 Retrieved 26. feb. 2023 from. [6] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu in Xinggang Wang. 2022. Bytetrack: multi-object tracking by associating every detection box. [7] Rač. prog. Piotr Skalski, onemetric 2021. url: https://github.com/Skalski P/onemetric/Retrieved 26. feb. 2023 from. 50 [8] Mark Everingham in John Winn. 2011. The pascal visual object classes challenge 2011 (voc2011) development kit. Pattern Analysis, Statistical Modelling and Computational Learning, Tech. Rep, 8. [9] Rač. prog. Glenn Jocher, Ayush Chaurasia in Jing Qiu, YOLO by Ultralytics ver. 8.0.0, jan. 2023. url: https://github.com/ultralytics/ultralyticsRetrieved 26. feb. 2023 from. [10] Ratko Pilipović, Vladimir Risojević, Janko Božič, Patricio Bulić in Uroš 0 Pr Stran Lotrič. 2021. An approximate gemm unit for energy-efficient object detection. oti Sensors, 21, 12. https://www.mdpi.com/1424-8220/21/12/4195. od Slika 6: Število letov proti in stran od vhoda v panj, prešte- tih na posnetku z več čebelami. 4 DISKUSIJA Obstajajo različni načini opazovanja gibanja, od preprostega siste- matičnega opazovanja do uporabe modelov strojnega vida. Kljub sorazmerno visoki natančnosti modelov, ki smo jih naučili me- nimo, da bi se dalo naučiti še bolj natančne modele. Večje število podatkov in boljše eksperimentalno okolje (grafična kartica z več grafičnega pomnilnika), bi omogočila še boljše rezultate. Eno od možnosti predstavlja uporaba novejše arhitekture YOLOv8[9], ki je bila objavljena med izdelovanjem naloge in je zato še ni- smo uporabljali. Obstaja tudi možnost izvajanja programa na 27 Vpeljava virtualnega asistenta ChatGPT v medicinsko platformo Implementation of a Virtual Assistant ChatGPT into the Medical Platform Matic Zadobovšek Primož Kocuvan Matjaž Gams matic.zadobovsek@gmail.com primoz.kocuvan@ijs.si matjaz.gams@ijs.si Univerza v Ljubljani Institut "Jožef Stefan" Institut "Jožef Stefan" Fakulteta za računalništvo in Jamova cesta 39 Jamova cesta 39 informatiko Ljubljana, Slovenija Ljubljana, Slovenija Večna pot 113 Ljubljana, Slovenija POVZETEK način za pridobivanje zdravstvenih informacij. Na spletu in mobil- nih telefonih je vrsta medicinskih aplikacij, ki nudijo informacije V prispevku predstavimo vpeljavo ChatGPT v platformo za elek- in nasvete. Termin ’Dr. Google’ opisuje iskanje zdravniških in- tronsko in mobilno zdravje Insieme, ki uporabnikom omogoča formacij in diagnosticiranja s pomočjo iskalnika Google. učinkovito pridobivanje informacij s področja medicine, spletno Na drugi strani pa ChatGPT predstavlja zelo obetavno orodje, človeško pomoč s strani zdravstvenih izvedencev in uporabo ki je neprestano dostopno vsakomur, kar omogoča uporabnikom, virtualnega asistenta, ki je nastal z integracijo najnovejših tehno- da lahko dobijo zanesljive odgovore na svoja vprašanja v realnem logij na področju obdelave naravnega jezika. Opišemo delovanje času. S hitrim napredkom tehnologije in nenehnim izboljševa- platforme Insieme in podamo razlago ter opis implementacije vir- njem ChatGPT se zdi, da bo uporaba le še rasla. tualnega asistenta. Prototipna vpeljava ChatGPT služi testiranju V okviru projekta Interreg je bila razvita platforma za elektron- zmogljivosti z namenom revolucije slovenskega zdravstva. sko in mobilno zdravje Insieme, ki jo je razvilo nekaj partnerjev, ABSTRACT ključni del pa je prispeval Odsek za inteligentne sisteme na Insti- tutu ’Jožef Stefan’. Platforma ne le olajšuje iskanja zdravstvenih In this paper, we present the introduction of ChatGPT into the storitev, ampak hkrati ponuja spletno človeško pomoč s strani Insieme platform for electronic and mobile health, which enables zdravstvenih izvedencev, pridobitev dodatnih koristnih informa- users to efficiently acquire information in the field of medicine, re-cij in ogled video vsebin s področja medicine. Ključno vlogo na ceive online human assistance from healthcare professionals, and tej platformi pa ima virtualni asistent, ki temelji na najnovejših utilize a virtual assistant created using the latest natural language dosežkih na področju obdelave naravnega jezika. V platformi so processing technologies. We report on the functionalities of the virtualni asistenti starejše generacije, pred nastankom ChatGPT, Insieme platform and provide an explanation and description of tj. botov z generativno splošno inteligenco. the implementation of the virtual assistant. The prototype imple- V nadaljevanju je opisano delovanje platforme Insieme, vloga mentation of ChatGPT into a medical platform serves as a test virtualnega asistenta ChatGPT za medicino ter razlaga, kako smo for potential advancement of the Slovenian healthcare system. združili tehnološko znanje in domensko strokovnost, da bi zago- tovili natančne in uporabniku prijazne odgovore na vprašanja s KLJUČNE BESEDE področja zdravja. Predstavljene so besedne vložitve in razlogi za virtualni asistenti, vektorske podatkovne baze, besedne vložitve, uporabo vektorskih podatkovnih baz, vse skupaj pa je povezano GPT-4, obdelava naravnega jezika z orodjem LangChain in velikim jezikovnim modelom GPT-4. Vse omenjene enote omogočajo, da razvijamo zdravstvene aplikacije, KEYWORDS ki prej niso obstajale in močno presegajo npr. dr. Googla. Ta štu- dija je služila tudi kot prvi preizkus nove tehnologije ChatGPT virtual assistants, vector databases, word embeddings, GPT-4, za medicinske namene. natural language processing 2 CHATGPT 1 UVOD Po nekaj testih s konkurenčnimi produkti smo se odločili za upo- V današnjem svetu, kjer se količina podatkov in informacij ne- rabo velikega jezikovnega modela GPT-4, ki smo ga obogatili z nehno povečuje, je dostop do zanesljivih virov informacij in znanjem naše platforme Insieme. GPT-4 je bil razvit marca 2023 strokovnih nasvetov postal ključnega pomena. Še posebej na in predstavlja velik napredek na področju obdelave naravnega področju medicine, ki je eno izmed temeljih področij družbe, je jezika, predvsem pa je koristen za odgovarjanje na vprašanja, pomembno, da uporabnikom zagotovimo enostaven in učinkovit generiranje besedil in prevajanje v druge jezike. V primerjavi z modeli prejšnjih generacij se je povečala zanesljivost in pra- Permission to make digital or hard copies of part or all of this work for personal vilnost odgovorov, hkrati pa je izboljšano upravljanje glede na or classroom use is granted without fee provided that copies are not made or uporabnikove ukaze (naprimer, da povemo, v kakšnem slogu naj distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this bo generirani odgovor). Številna testiranja [6] (različni izpiti in work must be honored. For all other uses, contact the owner /author(s). preizkusi znanja z različnih področij) so pokazala, da GPT-4 do- Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia sega rezultate, ki so povsem primerljivi s tistimi, ki jih dosegamo © 2023 Copyright held by the owner/author(s). ljudje. V primerjavi z zdravniki je generiral daljše odgovore, ki 28 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Zadobovšek, Kocuvan, Gams so hkrati bili s strani ostalih zdravstvenih izvedencev označeni kot boljši tako po kvaliteti odgovorov kot tudi empatiji [2]. Posledično smo se odločili, da bo naš virtualni asistent temeljil na modelu GPT-4. Namesto spreminjanja obstoječega modela smo omogočili, da lahko uporabnik povprašuje po podatkih, ki so pridobljeni iz naše platforme in ostalih dokumentov, ki jih lahko poljubno mnogo podamo. Na ta način ločimo jezikovni model in našo bazo znanja, omogočimo uporabniku, da komunicira s podanimi dokumenti, pri generiranju odgovora pa so uporabljene zgolj informacije, ki se nahajajo znotraj naših podanih dokumen- tov, kar omogoča, da se uporabniku zagotovi najbolj relevanten odgovor. Z omenjenim pristopom lahko enostavno dodajamo nove vire informacij in prilagodimo model za specifične naloge; brez treniranja obstoječega modela, kar bi sicer seveda bilo ča- sovno in računsko zahtevno. Postopek je podrobneje opisan v nadaljevanju referata. Slika 2: Prikaz glavne strani ob obisku platforme Insieme. Uporabnikom je na voljo tudi spletna človeška pomoč. Na vstopni strani so nanizani klicni centri in aktivni uporabniki, ki jih lahko kontaktirate preko spletnega klepeta v živo, ki je vgrajen v platformo. Insieme platforma nudi več vgrajenih asistentov: čakalne vrste, IJS asistenta, iskanje po storitvah, virtualnega asistenta za medi- cino, hkrati pa so priložene še povezave do ostalih ne-vgrajenih Slika 1: V več testih je ChatGPT presegel človeške zdrav- asistentov. Npr. asistent za čakalne vrste omogoča, da vnesemo nike. Vir: John Ayers in sod. 2023. Comparing physician ime posega oziroma storitve, podamo približno nujnost, kdaj and artificial intelligence chatbot responses to patient que- izbrani poseg potrebujemo ter želeno regijo v Sloveniji za opra- stions posted to a public social media forum. JAMA internal vljanje posega. Asistent za medicino odgovarja na poljubna vpra- medicine, 183, (apr. 2023). šanja uporabnika s področja zdravstva, kot odgovor pa mu poda ustrezne napotke in nasvete. 3 INSIEME 4 VIRTUALNI ASISTENT ZA MEDICINO Za testno medicinsko platformo smo izbrali Insieme, ki je bila 4.1 Ozadje nedavno razvita v sodelovanju s slovenskimi in italijanskimi Virtualni asistent v platformi Insieme je namenjen odgovarja- partnerji v okviru čezmejnega projekta ISE-EMH [7]. Opremljena nju na vprašanja o zdravju. Obstoječe asistente smo dogradili je s prijaznim uporabniškim vmesnikom, ki omogoča, da lahko z asistentom tipa ChatGPT. Tak asistent ima ogromno svojega uporabnik na enem spletnem mestu na enostaven in eleganten splošnega znanja s spleta, na voljo pa ima tudi dodatne lokalne način pridobi koristne informacije s področja zdravstva. informacije, povezane s projektom Insieme. Prvi problem pri Glavne funkcionalnosti so možnost iskanja storitev z uporabo uporabi je, če bi želeli, da bi podali neko večjo količino besedila stranske menijske vrstice oziroma z uporabo iskalne funkcije, (morda kar celo knjigo), ali pa imamo več dokumentov, ki bi jih spletna človeška pomoč (klepet v živo s klicnim centrom ali zdra- radi uporabili. Veliki jezikovni modeli imajo običajno omejitev, vstvenim izvedencem), ogled video vsebin s področja zdravja koliko besedila lahko sprejmejo [4]. Zato je pomembno, da veli-ter uporaba virtualnega pomočnika, ki je v nadaljevanju članka kemu jezikovnemu modelu podamo le informacije, ki so bistvene. predstavljena kot osrednja tematika. Vsa omenjena vsebina je Pri tem so ključne besedne vložitve in vektorske podatkovne uporabniku na voljo v treh oz. štirih jezikih. baze. Na levi strani slike 2 se nahaja seznam storitev, ki jih ponuja platforma Insieme. Ta menijska vrstica omogoča izbiranje med 4.2 Besedne vložitve in vektorske podatkovne različnimi vejami medicine, zatem pa se nadaljnji izbor razširi na baze bolezni in bolezenska stanja, ki pripadajo izbrani veji medicine, poleg tega pa so prikazane še informacijske storitve, ki pripadajo Vložitve (angl. embeddings) so način, kako lahko predstavimo izbrani specializaciji medicine. S klikom na eno izmed bolezen- besede, povedi ali pa kar celotne dokumente. Za njihov izračun skih stanj je uporabnik preusmerjen na ustrezno podstran. Tu potrebujemo ustrezne modele, ki so bili trenirani na ogromni so mu na voljo bistveni podatki o poteku bolezni, simptomih, količini podatkov in znajo poiskati razmerja med besedami s morebitni preventivi in nadaljnjemu ukrepanju. Spodaj se nahaja pomočjo analiziranja vzorcev, ki se pojavljajo v podatkih [3]. V še več preusmeritev na zunanja spletna mesta, ki uporabniku našem primeru smo uporabili model, ki ga ponuja OpenAI — omogočijo, da pridobi ustrezno znanje o izbrani bolezni. text-embedding-ada-002. Na ta način, da pridobimo vektor za Poleg ročnega prehajanja med podstranmi platforme je upo- vsako izmed besed, lahko predstavimo pomen besedila. Besedne rabniku na voljo še funkcionalnost iskanja, ki uporabniku prikaže vložitve lahko predstavimo v večdimenzionalnih prostorih, kjer vse storitve na platformi, ki so ujemajoče glede na niz iskanja. so si besede oziroma povedi s podobnim pomenom blizu — med 29 Platforma Insieme in vpeljava virtualnega asistenta za medicino Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia vektorji lahko izračunamo razdalje in tako poiščemo pomensko predloge (angl. prompt templates), kjer točno definiramo sorodne besede. obliko vnosa. Vektorske podatkovne baze shranjujejo informacije v obliki • Moduli za nalaganje dokumentov: omogočajo pretvorbo vektorjev, kar pogosto imenujemo kar besedne (vektorske) vloži- različnih vrst podatkov (PDF dokumenti, HTML spletne tve. To omogoča, da lahko indeksiramo in preiščemo ogromno strani, slikovno gradivo) v besedilo, ki ga je mogoče pro- količino nestrukturiranih podatkov, kot so slike, surovo besedilo cesirati. ali pa senzorski podatki. Vektorska baza organizira podatke z • Agenti: za aplikacije, kjer zaporedje klicev ni določeno uporabo visoko-dimenzionalnih vektorjev, ki vsebujejo nešteto vnaprej, LangChain zagotavlja agente, ki lahko ukrepajo dimenzij, vsaka dimenzija pa opisuje točno določeno lastnost na podlagi vhodov, namesto da bi imeli vnaprej določeno podatkovnega objekta, ki ga predstavlja. Vektorske baze se to- zaporedje. rej od tradicionalnih baz, ki shranjujejo podatke v tabelarični obliki, razlikujejo v tem, da vrnejo rezultate na podlagi podobno- sti (tradicionalne baze vrnejo popolnoma ujemajoče se objekte) Naslednja bistvena komponenta sistema so vektorske podatkovne [9]. Za merjenje podobnosti med vektorji v vektorskem prostoru baze, ki so predstavljene v razdelku 4.2. Odločili smo se za upo-se lahko uporablja različne mere — pogosto uporabljamo kosi- rabo odprtokodne vektorske baze Milvus, ki omogoča učinkovito nusno podobnost. Te mere uporabljamo, da lahko primerjamo shranjevanje vektorjev, njihovo indeksiranje, hkrati pa ponuja vektorje, ki so shranjeni v naši vektorski bazi, in poiščemo ti- API (angl. application programming interface), ki omogoča eno- ste, ki so najbolj podobni vektorju, ki ustreza vnosu uporabnika. stavno integracijo z različnimi programskimi jeziki. Tudi sama Omogočajo torej delo s kompleksnimi podatki in hitro iskanje, povezava med vektorsko bazo Milvus in jezikovnim modelom kar bi sicer tradicionalnim bazam povzročalo težave. Recimo, GPT-4 je preprosta, saj ni potrebe, da bi podatke posebej označe- da imamo dokument, ki bi ga radi indeksirali. Uporabili bomo vali ali pa ponovno trenirali model. Podatke je potrebno pretvoriti model, ki omogoča ustvarjanje besednih vložitev (zgoraj smo v vektorsko obliko in jih shraniti v Milvus. Končni odgovor, ki ga omenili text-embedding-ada-002) [5]. Shranili jih bomo v izbrano model generira, je na takšen način ustvarjen z referenciranjem vektorsko bazo, pri tem pa se shrani referenca na dokument, iz vsebine v naši zbirki dokumentov, kar zagotavlja, da virtualni katerega je bila vložitev ustvarjena. Kadarkoli bo naš uporab- asistent pridobi prave podatke in posledično zmanjša verjetnost nik poslal poizvedbo, bo uporabljen enak model za ustvarjanje napak. vložitev — uporabili jih bomo, da v vektorski bazi poiščemo naj- Slika 3 grafično nakazuje postopek. Prvi korak v tem razvoju bolj podobne besedne vložitve, ki so zaradi omenjene reference je nalaganje podatkov v ’Dokumente’, ki so pravzaprav bese- povezane z originalnim dokumentom, kjer so bile ustvarjene. dilni kosi. Modul za nalaganje dokumentov v orodju LangChain Pridobljene dokumente lahko zatem podamo velikemu jezikov- poenostavi to nalogo in omogoča enostavno nalaganje in pred- nemu modelu — dokumenti bodo uporabljeni kot kontekst za obdelavo naših podatkov — uporabimo lahko DirectoryLoader, generiranje odgovora. Ravno zaradi vseh omenjenih lastnosti so ki omogoča, da shranimo vse uporabljene dokumente v skupen vektorske podatkovne baze odlična izbira, da obogatimo naše direktorij. Sledi razdeljevanje dokumentov na manjše kose — generativne modele. delilec besedila (angl. text splitter) omogoča razbijanje dolgih besedilnih delov na manjše, pomensko smiselne koščke [8]. Ta naloga se morda zdi preprosta, vendar vključuje nekaj komple-4.3 Implementacija ksnosti. Cilj je besedilo razdeliti na način, ki ohranja pomensko Eno izmed pomembnejših orodij za implementacijo ChatGPT povezane dele skupaj, pri čemer je ’pomenska povezanost’ od- v Insieme je knjižnica LangChain, ki omogoča delo z velikimi visna od vrste besedila, ki se obdeluje. Delilci besedila razdelijo jezikovni modeli (LLM). LLM lahko učinkovito opravljajo ogro- besedilo na majhne kose, pogosto na podlagi mej med stavki. Te mno število različnih opravil, vendar obstaja verjetnost, da ne majhne kose združijo v večje kose, dokler ne dosežejo določene bodo zmožni pravilno odgovarjati na vprašanja s specializiranih velikosti, ki jo določi predhodno določena funkcija za merjenje področij, kot je naprimer medicina. LangChain pomaga, da lahko velikosti kosa — ko doseže kos želeno velikost, postane samosto- naše modele nadgradimo z znanjem specifičnih področij in jim jen kos besedila. Zatem se ustvari nov kos z nekaj prekrivanja omogočimo zavedanje o podatkih ter kontekstu pogovora. Lan- (angl. chunk overlap), da se ohrani kontekst med posameznimi gChain predstavlja zmogljivo orodje, ki zapolnjuje praznino v kosi. povezavi med jezikovnimi modeli in domenskim znanjem, kar je Sledi generiranje besednih vložitev, ki igrajo ključno vlogo pri tudi razlog, zakaj se LangChain vse pogosteje uporablja v aplika- predstavitvi besedilnih informacij. Razred Embedding v orodju cijah, ki opravljajo naloge, povezane z obdelavo naravnega jezika. LangChain služi kot standardiziran vmesnik za različne ponu- LangChain vsebuje številne module, ki pomagajo pri razvoju [1]: dnike vložitev, vključno z OpenAI. S pomočjo generiranja be- sednih vložitev se besedilo pretvori v vektorsko predstavitev, • LLM: omogoča uporabo zmožnosti specifičnega velikega ki omogoča semantično analizo in opravljanje nalog, kot je se- jezikovnega modela. mantično iskanje. Vse to se s pomočjo vgrajenih metod shrani • Verige (angl. chains): glavna enota, kar je razvidno že iz v našo vektorsko podatkovno bazo kot nov indeks — nad tem imena LangChain, ki združuje več LLM klicev. Primer tega objektom lahko opravljamo semantično iskanje in pridobimo bi bil, da najprej preberemo uporabnikov vnos, ta vnos dokumente, ki so relevantni glede na uporabnikov vnos. Dobljeni pa uporabimo, da sestavimo nov vnos (angl. prompt), ki dokumenti se zatem posredujejo jezikovnemu modelu, ki doku- se poda velikemu jezikovnemu modelu, ki zatem generira mente obravnava kot kontekst za generiranje odgovora. Na sliki odgovor. 4 lahko vidimo prikaz odgovarjanja na uporabnikova vprašanja. • Vnosi, pozivi (angl. prompts): LangChain omogoča ve- ChatGPT tekoče odgovarja na zdravstvena vprašanja z upošte- liko različnih načinov, kako lahko spreminjamo vnos, ki vanjem splošnega znanja, znanja medicine s celotnega spleta in je posredovan jezikovnemu modelu. Lahko uporabljamo posebnih znanj s platforme Insieme. 30 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Zadobovšek, Kocuvan, Gams Preverjanje in zagotavljanje informativnih ter ustreznih od- govorov je potekalo s strani avtorjev prispevka. ChatGPT je bil testiran z vprašanji, ki so v celoti pokrili znanje, ki ga vsebuje platforma Insieme. Odgovore, ki so bili generirani, smo kritično ovrednotili in uvedli popravke ob morebitnemu odstopanju od pričakovanj. Točnost pridobljenih odgovorov je za uporabnika ključnega pomena, saj lahko zavajanje in netočni odgovori v pri- meru, da jih uporabnik upošteva, v hujših situacijah privedejo celo do poslabšanja zdravstvenega stanja. Vsebina platforme Insi- eme, ki virtualnemu asistentu predstavlja kontekst za generiranje odgovorov, je bila predhodno pripravljena s strani zdravstvenih izvedencev, kar uporabniku omogoča pridobitev preverjenih in koristnih informacij. Slika 4: Prikaz delovanja virtualnega asistenta ChatGPT nad zdravstveno platformo Insieme. nenehno dostopnega vira informacij, ki temelji na najsodobnejših pridobitvah raziskav. Eksperimenti v tej študiji kažejo, da je generativna umetna inteligenca dejansko uporabna in obeta radikalne izboljšave, če jo bomo uspeli vpeljati v slovensko zdravstvo. Slika 3: Prikaz integracije vseh komponent sistema. ZAHVALA Platforma Insieme je nastala v okviru čezmejnega projekta In- 5 ZAKLJUČEK terreg ISE-EMH, ki ga financira Program sodelovanja Italija- Slovenija iz Evropskega sklada za regionalni razvoj. Predstavljena je implementacija pogovornega virtualnega asi- stenta ChatGPT v platformo za elektronsko in mobilno zdravje, LITERATURA imenovano Insieme. Eksperimenti kažejo, da je sistem sposoben [1] Amogh Agastya. 2023. Harnessing retrieval augmented generation with nuditi tako običajne odgovore ChatGPT kot dodatne na osnovi langchain. (Sep. 2023). Retrieved September 5, 2023 from https://betterprogr informacij, dosegljivih s platforme. Vpeljava je bila uspešna. amming.pub/harnessing- retrieval- augmented- generation- with- langchain- Platforma ima še veliko možnosti za nadaljnje izboljšave, pred- 2eae65926e82. [2] John Ayers in sod. 2023. Comparing physician and artificial intelligence vsem bi bila koristna integracija z drugimi zdravstvenimi infor-chatbot responses to patient questions posted to a public social media forum. macijskimi sistemi in viri v Sloveniji, kar bi omogočilo, da bi JAMA internal medicine, 183, (apr. 2023). doi: 10.1001/jamainternmed.2023.1 838. platforma Insieme postala še bolj celovit vir informacij za upo- [3] Qilu Jiao in Shunyao Zhang. 2021. A brief survey of word embedding and rabnike — pri tem se bomo povezali z zdravstvenimi institucijami its recent development. V 2021 IEEE 5th Advanced Information Technology, po Sloveniji. V okviru vpeljave virtualnega asistenta je bil preiz-Electronic and Automation Control Conference (IAEAC). Zv. 5, 1697–1701. doi: 10.1109/IAEAC50856.2021.9390956. kušen tudi najšibkejši jezikovni model iz družine Llama 2, ki je [4] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, bil lokalno nameščen. Pojavila se je težava, saj je večina učnih Muhammad Usman, Naveed Akhtar, Nick Barnes in Ajmal Mian. 2023. A podatkov, ki so bili uporabljeni, bila v angleščini, zato je model comprehensive overview of large language models. (2023). arXiv: 2307.06435 [cs.CL]. kot tak neustrezen za uporabo v slovenskem jeziku. Tu se nam [5] Arvind Neelakantan in sod. 2022. Text and code embeddings by contrastive odpira možnost, da v prihodnje preizkusimo zmogljivejši model pre-training. (2022). arXiv: 2201.10005 [cs.CL]. [6] OpenAI. 2023. Gpt-4 technical report. (2023). arXiv: 2303.08774 [cs.CL]. iz družine Llama 2, ki bo prav tako lokalno nameščen. Pri tem se [7] Insieme Platform. 2023. Insieme platform. (Sep. 2023). Retrieved September 3, pojavlja nov vidik uporabe, saj bi na ta način vsi podatki bili do-2023 from https://ise- emh.eu. stopni lokalno in bi izpodrinili potrebo po zunanjemu dostopanju [8] Adith Sreeram A S in Pappuri Jithendra Sai. 2023. An effective query system using llms and langchain. INTERNATIONAL JOURNAL OF ENGINEERING do podatkov, kot je to sedaj nujno z uporabo ChatGPT. RESEARCH & TECHNOLOGY (IJERT), 12, 06, (jun. 2023). Naš glavni namen je, da bi platforma predstavila temeljne [9] Jianguo Wang in sod. 2021. Milvus: a purpose-built vector data management ideje, kako bi se lahko moderniziral zdravstveni sistem, pri tem system. V Proceedings of the 2021 International Conference on Management of Data (SIGMOD ’21). Association for Computing Machinery, Virtual Event, pa stremimo k razbremenitvi ljudi, ki delajo v tej stroki, hkrati pa China, 2614–2627. isbn: 9781450383431. doi: 10.1145/3448016.3457550. želimo vsem uporabnikom omogočiti dostop do učinkovitega in 31 Types of Democracy Defined: Keyword Extraction from Eleven Different Text Descriptions Žiga Kolar Andrej A. Lukšič Jožef Stefan Institute Faculty of Social Sciences Jamova cesta 39 Kardeljeva ploščad 5 Ljubljana, Slovenia Ljubljana, Slovenia ziga.kolar@ijs.si andrej.luksic@fdv.uni- lj.si Andrej Vozlič Matjaž Gams Independent Researcher Jožef Stefan Institute Kersnikova ulica 3 Jamova cesta 39 Ljubljana, Slovenia Ljubljana, Slovenia hommage.artists@gmail.com matjaz.gams@ijs.si ABSTRACT their existing systems in response to changing circumstances and needs [6]. The various forms of democracy worldwide display a range of However, given the vast amount of textual data describing distinct traits, influenced by regional, historical, and cultural these democratic forms, there arises a need for a systematic ap- backgrounds. This research undertook the challenge of extracting proach to identify their defining features. Keyword extraction, keywords from textual descriptions of eleven distinct democracy an established technique in the realm of Natural Language Pro- types to identify and categorize their core principles and nuances. cessing (NLP) [4], offers a solution. By extracting keywords from By employing Natural Language Processing (NLP) techniques, descriptions of different democratic models, we aim to distill specifically, the Text Rank, RAKE and TF-IDF (Term Frequency- their essence, thereby providing a concise yet comprehensive Inverse Document Frequency) methodologies, the study aimed to overview of each type’s foundational principles. extract meaningful keywords that encapsulate the foundational Keyword extraction serves as a fundamental tool for informa- principles governing each democracy type. The results provide tion retrieval. Researchers, academics, policymakers, and prac- a deeper understanding of how various democratic structures titioners often rely on keywords to quickly identify relevant operate and differentiate from one another. Such insights are documents amidst an overwhelming volume of textual data. By crucial for researchers, policymakers, and political enthusiasts extracting keywords from eleven different text descriptions of to navigate the intricate landscape of global democratic models. democracy, this research facilitates more effective and precise KEYWORDS information retrieval in the realm of democratic studies. Un ad- dition, by extracting keywords, this paper helps enhance the keywords, data mining, natural language processing, democracy accessibility and comprehension of your research findings. It en- ables readers to gain a rapid overview of the various democracy 1 INTRODUCTION types under study, making it easier for them to delve deeper into specific aspects of interest. Democracy, a term rooted in the Greek words "d¯ emos" (people) In this paper, we delve deep into the intricacies of eleven spe- and "kratos" (power), has been a guiding principle of governance cific democratic models, unraveling their characteristics through for centuries [18]. However, the interpretation and application the lens of extracted keywords. he structure of this paper un-of democracy have evolved and diversified across regions, taking folds in this manner: Section 2 delves into concise explanations shapes influenced by historical experiences, cultural nuances, of various democratic types. In Section 3, we describe and ap-socio-political challenges, and economic contexts. Today, the ply machine learning algorithms for keyword extraction to the world does not see a monolithic form of democracy but rather realm of text descriptions associated with types of democracy. an array of forms, each with its unique characteristics and mech- In Section 4, we present the results and provide commentary on anisms [1]. them. We wrap up the paper with a conclusion and suggestions Understanding these distinct democratic forms is crucial for for subsequent research. several reasons. Firstly, it offers a glimpse into the diverse ways in which societies prioritize and ensure the participation of their cit-izenry in governance. Secondly, it helps in identifying the checks 2 DEMOCRACY TYPES and balances incorporated in each system to protect against po- In this section, we explore various forms of democracy, highlight- tential excesses and abuses of power. Lastly, it provides a roadmap ing their unique characteristics. Democracy, which promotes for nations looking to either adopt a democratic model or refine individual freedoms and collective decisions, differs based on context. These descriptions, initially from a PhD study [19], are Permission to make digital or hard copies of part or all of this work for personal condensed here for clarity. The dataset is available at [10]. Each or classroom use is granted without fee provided that copies are not made or text description corresponds to one democratic type and is char-distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this acterized by an English text description containing a maximum of work must be honored. For all other uses, contact the owner /author(s). 10,000 words. It’s worth noting that while some democratic terms Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia discussed here are widely accepted, others still lack a uniform © 2023 Copyright held by the owner/author(s). definition. 32 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Kolar, et al. 2.1 Participative Democracy 2.10 Source democracy Participative democracy involves direct citizen participation in This democracy focuses on the foundational decision-making decision-making, using methods like town halls and online plat- structures. It advocates transparency and collective input. How- forms. While fostering inclusivity and informed decisions, it ever, inclusive participation and decision quality remain chal- might be resource-intensive and not always ensure equal repre- lenges [15]. sentation [17]. 2.11 Ideal typical democracy 2.2 Deliberative Democracy This normative concept outlines the optimal features of a democ- Deliberative democracy emphasizes informed discussion among racy, including universal suffrage, free elections, representation, citizens to make collective decisions. While it can promote nu- civil liberties, rule of law, power separation, and government anced decision-making and civic engagement, ensuring equal transparency. voice, especially for marginalized groups, remains a challenge [7]. 3 METHODOLOGY 2.3 Transdemocracy Our exploration delved deep into three contemporary keyword extraction methodologies, namely TextRank, Rake, and TF-IDF, Transdemocracy proposes expanding democratic principles to each offering its unique algorithmic underpinning for processing global governance, recognizing the need for collective action textual data. on global challenges. It seeks more direct citizen involvement The TextRank methodology stands out for its holistic approach in global decisions but faces challenges like reconciling diverse to text analysis, where the textual content undergoes a series of values and ensuring equal global representation [12]. rigorous preprocessing steps to distill its core meaning. These steps include tokenization, which breaks the text into individual 2.4 Guided Democracy words or phrases, and the removal of irrelevant or stop words that Guided democracy, also referred to as authoritarian democracy, carry little semantic value. Once the text is prepared, TextRank centralizes power with restricted participation from citizens. constructs a graph based on term co-occurrence. This graph elu- Though it can provide stability, it might compromise on authentic cidates the intricate relationships between words, uncovering the democratic liberties. There’s also an inherent risk of corruption underlying semantic structure of the text. In this graph, each term and potential power misuse [5]. becomes a node, and the strength and number of their connec- tions determine their significance. The top-scoring terms, which 2.5 Modern Direct Democracy act as representative keywords, find applications in various do- Modern direct democracy empowers citizens with immediate con- mains. These applications extend far beyond traditional keyword trol over political decisions, using platforms like electronic vot- extraction and encompass areas such as text classification, where ing. It encourages civic engagement and responsive governance keywords play a pivotal role in categorizing documents, senti- but may struggle with equal access and ensuring well-informed ment analysis, which relies on keywords to gauge the emotional choices [16]. tone of text, and document clustering, where keywords aid in grouping similar documents together [11]. 2.6 E-democracy Transitioning to RAKE (Rapid Automatic Keyword Extrac- tion), this method lives up to its name by offering an expedient E-democracy uses digital tools to enhance citizen participation approach to keyword extraction. Unlike the graph-based struc- in politics. This includes online voting, social media, digital pe- ture of TextRank, RAKE’s algorithmic focus lies in dissecting titions, and e-forums. It encourages political involvement, es- textual information into individual components, often referred pecially among the youth, and offers transparency. However, to as phrases or keyphrases. Each of these components is then concerns arise regarding online security, digital literacy, and scored based on its occurrence and relation to other words within accessibility [9]. the text. This nuanced approach to keyword extraction results in 2.7 Representative democracy the isolation of top-scoring keyphrases, providing a richer and more contextually meaningful representation of the underlying In representative democracy, citizens elect officials to act on their text’s content. The efficiency of RAKE makes it particularly use-behalf, as direct involvement in every decision isn’t feasible. This ful in scenarios where quick and accurate keyword extraction is system promotes informed decisions and offers stability. Yet, true essential for tasks such as summarization, information retrieval, representation and influence of special interests are concerns [3]. and content indexing [14]. Lastly, the TF-IDF (Term Frequency-Inverse Document Fre- 2.8 Liquid democracy quency) methodology delves into the granular essence of a term’s Liquid democracy merges direct and representative democracy. importance within a document and across a broader corpus. This Citizens can vote directly or delegate their vote. This system method combines two essential components: "TF" (Term Fre- promotes active engagement but faces challenges in equal repre- quency) and "IDF" (Inverse Document Frequency). The "TF" met-sentation and decision quality [8]. ric captures the frequency of a term within a particular document, highlighting its significance in that context. However, TF alone 2.9 Blockchain democracy may emphasize commonly occurring words that appear signifi- Blockchain democracy employs blockchain technology for a se- cant in isolation but lack uniqueness. cure, transparent voting process, allowing remote voting. Despite The "IDF" component evaluates the term’s rarity or uniqueness its security, potential system vulnerabilities and accessibility is-across a broader corpus of documents. This measure ensures that sues persist [13]. commonly occurring terms, which might be abundant within a 33 Keywords Extraction for Eleven Democracy Types Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia single document, are appropriately contextualized against their In summary, the Text Rank algorithm ranks highest with four prevalence across multiple documents. The multiplicative result unique keywords. Following closely is the TF-IDF algorithm with of TF and IDF offers a composite score that indicates the weight two distinct keywords. Lastly, the Rake algorithm doesn’t have or significance of the term. Terms with notable TF-IDF scores any unique keywords, placing it at the bottom of this comparison. emerge as the textual frontrunners, illuminating the primary Text Rank words best because dataset contains a lot of nuanced themes and subjects of the source material. TF-IDF is widely used contextual information and that’s why TextRank might be better in information retrieval, document ranking, and text mining tasks, suited to extract these nuances due to its graph-based approach. making it a foundational method in natural language processing and information science [2]. 5 CONCLUSION In this research, we delved into the intricate fabric of descriptions for eleven distinctive types of democracies. The aim was to eluci-4 RESULTS date and capture the most salient keywords that epitomize the The essential keywords for the Text Rank, Rake, and TF-IDF essence of each democracy type. By employing three different algorithms have been laid out in Tables 1, 2, and 3. A striking keyword extraction algorithms — Text Rank, RAKE, and TF-IDF observation is the omnipresence of the keyword "Democracy" — our study has not only shed light on the specific linguistic across all types of democracies and within all three algorithms. constructs and terminologies inherent to each democratic type Given its recurrent use throughout the texts, the prominence of but also the differential efficacy of the algorithms in context. this keyword was anticipated. Our findings reveal that the keyword "Democracy" consis- Furthermore, a pattern emerges wherein many keywords bear tently appears in all types of democracies across various keyword close resemblance or are derivatives of the nomenclature of extraction methods. This is expected since each text description their respective democracy types. For instance, for Deliberative provides definitions for its respective democracy type, leading democracy, we see "Deliberation" and "Deliberative" making ap-to the frequent use of the term "Democracy." In the analysis of pearances. Similarly, "Participation" and "Participatory" are high-democracy types, keywords often closely align with or derive lighted for Participative democracy, "Direct" and "Modern" for from the corresponding democracy nomenclatures. Notably, the Modern direct democracy, "Representative" for Representative term "Participation" consistently emerges across Participative democracy, and "Liquid" for Liquid democracy. The frequency of democracy, Deliberative democracy, and E-democracy in all al-these terms in the texts underscores their significance, thereby gorithms, indicating shared principles among these democracies. warranting their selection as keywords. However, the keyword "Citizens" varies in its occurrence across The keyword "Participation" is shared among Participative different algorithms and democracy types. These discrepancies democracy, Deliberative democracy, and E-democracy in all three in keyword results stem from the distinct methodologies of the keyword extraction algorithms. This common keyword suggests Text Rank, RAKE, and TF-IDF algorithms. Each approach leads to alignment in themes among these democracy forms, potentially unique sets of keywords, even when applied to the same content. reflecting shared core principles. In contrast, other democracy This research underscores the importance of algorithm selec- types exhibit more distinct keywords. Additionally, the keyword tion tailored to the specific requirements of the textual dataset in "Citizens" is present in Source democracy, Participative democ-hand. Moreover, the keywords extracted present an invaluable racy, and Blockchain democracy using the Text Rank algorithm. repository for political scientists, historians, and policymakers It appears only in Representative democracy with RAKE and in understanding the foundational pillars of varied democracy in both Blockchain democracy and Representative democracy forms. Future work might consider combining every three key- with TF-IDF. These variations in results across Text Rank, RAKE, word extraction algorithms into one single table, removing du- and TF-IDF stem from their distinct methodologies. Text Rank plicates and extracting only unique keywords that belong to considers relational context between terms, extracting keywords the corresponding democracy type. Additionally, by converting with conceptual relationships.RAKE focuses on local frequency keywords into numerical values, clustering algorithms could be and co-occurrence, sensitive to document structure. Finally, TF- applied. This would enable the identification of democracy types IDF emphasizes term uniqueness across a document set. These that fall within the same group or cluster. Furthermore, in future differences mean that each algorithm yields a somewhat different endeavors, experts possessing a background in political education set of keywords, even when analyzing the same content. will assess the accuracy and validity of the extracted keywords. In all three algorithms, a set of 12 keywords serves as unique identifiers for distinguishing each type of democracy. These key- ACKNOWLEDGEMENTS words are "Ancient", "Rule", "Laws", "Deliberation", "Deliberative", The authors acknowledge the funding from the Slovenian Re- "Modern", "Liquid", "Vote", "Information", "Blockchain", "New", search and Innovation Agency (ARRS), Grant (PR-10495) and and "Government". They are depicted by red color in Tables 1, 2, Basic core funding P2-0209. 3. Further delving into the specifics, the Text Rank algorithm pos- REFERENCES sesses its own exclusive set of four keywords that differentiates [1] Christopher H Achen and Larry M Bartels. 2017. Democracy for realists. In it from the others. These are "Theory", "Communication", "Sys-Democracy for Realists. Princeton University Press. [2] Akiko Aizawa. 2003. An information-theoretic perspective of tf–idf mea-tems", and "Power". They are represented by blue color in Table sures. Information Processing & Management, 39, 1, 45–65. 1. Conversely, the TF-IDF algorithm also has its unique identi- [3] Sonia Alonso, John Keane, and Wolfgang Merkel. 2011. The future of repre-fiers with two sole keywords: "Axis" and "Counter democracy". sentative democracy. Cambridge University Press. [4] KR1442 Chowdhary and KR Chowdhary. 2020. Natural language processing. In Table 3, they are represented using the color green. These Fundamentals of artificial intelligence, 603–649. individual sets of keywords not only underscore the differences [5] Peter Christoff. 2003. Ecological citizens and ecologically guided democracy. in each algorithm’s framework but also hint at their distinctive In Democracy and green political thought. Routledge, 159–176. analytical approaches. 34 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Kolar, et al. Table 1: 5 keywords for each democracy type extracted with Text Rank algorithm. Type of Democracy Keyword 1 Keyword 2 Keyword 3 Keyword 4 Keyword 5 Source democracy Democracy Society Direct Ancient Citizens Ideal-typical dem. Democracy Society Rule Executive Laws Participative dem. Democracy Political Society Participation Citizens Deliberative dem. Democracy Deliberation Deliberative Participation Process Modern direct dem. Democracy Direct Society Theory Modern Liquid democracy Democracy Liquid Vote Representative Representatives E-democracy Democracy Information Participation Public Communication Blockchain dem. Democracy Blockchain New Systems Citizens Representative dem. Democracy Representative Government Political Power Control democracy Democracy Civil Democratic Society Monitoring Transdemocracy Democracy Political System Concept Form Table 2: 5 keywords for each democracy type extracted with Rake algorithm. Type of Democracy Keyword 1 Keyword 2 Keyword 3 Keyword 4 Keyword 5 Source democracy Democracy Society Executive Ancient Direct Ideal-typical dem. Democracy Society Rule Executive Laws Participative dem. Democracy Participation Political Participatory Society Deliberative dem. Democracy Deliberation Deliberative Participation Participatory Modern direct dem. Democracy Direct Modern Referendum Instrumental Liquid democracy Democracy Liquid Vote Control Representative E-democracy Democracy Information Participation Public Decision-making Blockchain dem. Democracy Blockchain New Decentralized Political Representative dem. Democracy Representative Government Political Citizens Control democracy Democracy Civil Society Monitoring Representative Transdemocracy Democracy System Political Form Representative Table 3: 5 keywords for each democracy type extracted with TF-IDF algorithm. Type of Democracy Keyword 1 Keyword 2 Keyword 3 Keyword 4 Keyword 5 Source democracy Democracy Executive Society Ancient Direct Ideal-typical dem. Democracy Rule Society Executive Laws Participative dem. Democracy Participation Political Participatory Society Deliberative dem. Democracy Deliberation Deliberative Process Participation Modern direct dem. Democracy Direct Modern Instrumental Referendum Liquid democracy Democracy Liquid Vote Representative Control E-democracy Democracy Information Decision-making Participation Axis Blockchain dem. Democracy Blockchain Citizens New Decentralized Representative dem. Democracy Representative Government Political Citizens Control democracy Democracy Civil Counter democracy Democractic Society Transdemocracy Democracy System Political Concept Form [6] Klaus Gründler and Tommy Krieger. 2016. Democracy and growth: evidence [12] Marcelo Neves. 2017. From transconstitutionalism to transdemocracy. Euro-from a machine learning indicator. European Journal of Political Economy, pean Law Journal, 23, 5, 380–394. 45, 85–107. On Institutions and Well Being. doi: https://doi.org/10.1016/j.ej [13] Peter Racsko. 2019. Blockchain and democracy. Society and Economy, 41, 3, poleco.2016.05.005. 353–369. [7] Amy Gutmann and Dennis F Thompson. 2004. Why deliberative democracy? [14] Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic Princeton University Press. keyword extraction from individual documents. Text mining: applications [8] Anson Kahng, Simon Mackenzie, and Ariel Procaccia. 2021. Liquid democ-and theory, 1–20. racy: an algorithmic perspective. Journal of Artificial Intelligence Research, [15] Douglas Rushkoff. 2003. Open source democracy: How online communication 70, 1223–1252. is changing offline politics. Vol. 10753. Demos. [9] Marianne Kneuer. 2016. E-democracy: a new challenge for measuring democ- [16] Theo Schiller. 2011. Local direct democracy in Europe. Springer. racy. International Political Science Review, 37, 5, 666–678. [17] Marius H Smit and Izak J Oosthuizen. 2011. Improving school governance [10] Žiga Kolar and Matjaž Gams. 2023. Democracy dataset. Available at: https: through participative democracy and the law. South African Journal of //drive.google.com/drive/f olders/1W_5qa9scDXvQwR7TjBzbMan91o2- g Education, 31, 1, 55–73. Af e?usp=sharing. 11.9.2023. (2023). [18] Charles Tilly. 2007. Democracy. Cambridge University Press. [11] Rada Mihalcea and Paul Tarau. 2004. Textrank: bringing order into text. In [19] Andrej Vozlič. 2020. Transdemokracija kot rekonceptualizacija demokracije v Proceedings of the 2004 conference on empirical methods in natural language okolju digitalne države: doktorska disertacija. PhD thesis. Univerza v Ljubljani, processing, 404–411. Fakulteta za družbene vede. 35 Evaluation of the Effects of On-Demand Dynamic Transportation of Employees to Their Workplaces in Ljubljana Marko Bohanec Marko Guček Davor Kontić marko.bohanec@ijs.si marko@goopti.com davor.kontic@ijs.si Jožef Stefan Institute, Department of GoOpti d.o.o. Jožef Stefan Institute, Department of Knowledge Technologies Ljubljana, Slovenia Environmental Sciences and Centre Jamova cesta 39 for Participatory Research Ljubljana, Slovenia Karina Sirk Bernard Ženko Martin Žnidaršič karina.sirk@ipop.si bernard.zenko@ijs.si martin.znidarsic@ijs.si Institute for Spatial Policies Jožef Stefan Institute, Department of Jožef Stefan Institute, Department of Ljubljana, Slovenia Knowledge Technologies Knowledge Technologies ABSTRACT emissions, improve modal split, reduce traffic congestions and alleviate related problems. On-demand dynamic transportation is an innovative information- The focus of this paper are not the IT needed to implement technology supported service that enables passengers to book an on-demand dynamic transport (e.g., mobile apps, databases, and configure their rides. It is foreseen as a promising service to scheduling and optimization systems), but rather we present improve the sustainable mobility of citizens and alleviate traffic the results of a real life evaluation of such a system that has problems. In this paper, we present the results of a three-month been carried out in Ljubljana Urban Region, Slovenia, within the pilot implementation of dynamic transportation of employees SmartMOVE project [10]. SmartMOVE addresses the challenges from their homes to their workplaces. The transports were man-of sustainable mobility in the Ljubljana Urban Region with the aged by the company GoOpti, d.o.o., they were free and took place capital city of Ljubljana, which is the primary destination of daily using vans on the routes connecting the cities Kamnik and Kranj migration flows in Slovenia. with two areas (BTC and UKC) in Ljubljana. The project was The aim of this study is to answer two questions that are cru- very successful in terms of sustainable mobility: it attracted users cial if such a service is to be implemented in practice: (1) Can that normally drive passenger cars, travel times were comparable on-demand dynamic transport really help in reducing harmful to conventional modes of transportation and users were very emissions and traffic congestion? and (2) Under what conditions satisfied with the service, while substantially reducing (from 30% can such a service sustainably operate in a given economic envi- to 70%) the harmful emissions of CO , NO and solid particles. 2 x ronment? It is important to note that before this study no evalu- However, two challenges for the future still remain: improving ation of on-demand dynamic passenger transport in comparable the occupancy rate of vehicles and bridging the gap between the environments has been done, and its benefits and limitations economic price and users’ willingness to pay for the service. used to be assessed only theoretically through simulations [1]. KEYWORDS sustainable mobility, on-demand dynamic transport, employee 2 RELATED WORK transportation, Ljubljana Urban Region Ljubljana is home to over 220,000 jobs, which accounts for over 25% of all jobs in Slovenia. As a result, over 120,000 people flock 1 INTRODUCTION to Ljubljana daily from elsewhere. This means approximately 100,000 vehicles entering and exiting Ljubljana on a daily basis Within the last thirty years information technologies (IT) pro- (with an average occupancy of vehicles at 1.2 pers/car). Since foundly changed many aspects of our life and the society. IT the majority of this is associated to the personal car transport, provide us with tools that enable creation of new services, which the main goal is to transfer the car drivers/passengers to public have the potential to significantly improve our every day lives. transport. However, the road public transport is faced with the One of such services is on-demand dynamic passenger trans- same problems as car transport – congestions due to absence of port [11], which typically makes use of vans or mini busses dedicated lanes, low travel speeds, poor occupation of vehicles. that operate without fixed itineraries or fixed stops, enabling The problem of public transport in Ljubljana was tackled by passengers to book their ride and select their pick up and drop studies that focused on various aspects. Some of them outline the off locations. On-demand dynamic transport bridges the gap overall context of the problem and the related phenomena [4]. between the conventional public transport and private car trans- The current public transport organization in Ljubljana is reflected port, and promises to reduce green house gas and other pollutant well in a paper that describes the public transport plans from a decade ago [6]. There are studies about the state of the public Permission to make digital or hard copies of part or all of this work for personal transport in Ljubljana, such as accessibility [12], speed [7], studies or classroom use is granted without fee provided that copies are not made or about the effects of public transport management measures [13] distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this and about its potential future developments. work must be honored. For all other uses, contact the owner /author(s). On-demand dynamic transportation, which is in the focus of Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia this paper, shares several characteristics with carpooling [14]. © 2023 Copyright held by the owner/author(s). However, we could find no previous studies about on-demand 36 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Bohanec, et al. Table 1: Basic ride statistics. dynamic transportation in Ljubljana and even in general, the studies of on-demand dynamic transportation are to a large extent Route Passengers Distance Time Speed Occupancy dedicated to theoretical models [8, 5] and simulations [2, 1]. A [total] [km] [min] [km/h] [%] good showcase of dynamic transport practices are the on-demand Kamnik 5440 21.20 42.81 29.72 38.0 airport shuttles, which have indicated a potential solution to the Kranj 1742 33.66 45.62 44.27 25.9 downsides of regular public lines; they support the users’ needs and commodity as one of the most important factors for choosing Total 7182 25.18 43.70 34.57 34.1 the preferred means of transport. the Statistical Yearbook of the Republic of Slovenia [9] and EU 3 STUDY DESIGN 1 emission standards (EURO standards ). We considered the users’ distances from home to workplace and the number of journeys The main objective of this pilot study was to test using on-demand they would have made if dynamic transport had not been em- dynamic transportation of employees as a sustainable alternative ployed. As a weighting factor for the calculation, we considered to existing modes of transport, especially in comparison with us- whether the users mainly drive with personal vehicles (several ing passenger cars. The study took place in the trial period from times a week) or perhaps combine the drives with other modes of February to April 2023, when selected passengers were trans- transportation to work (public passenger transport, bicycle, etc.). ported free of charge by the company GoOpti, d.o.o., employing The emission factors for GoOpti vehicles were obtained from the vans that can carry up to eight passengers. Transportation costs manufacturer’s specifications. were covered from the SmartMOVE project. Two pilot routes were established that connected two nearby cities Kamnik (14,000 4 RESULTS inhabitants, about 23 km from Ljubljana) and Kranj (38,000, 28 km) with the areas of two large employer organisations located 4.1 Traffic Data Analysis in Ljubljana: UKC and BTC. UKC, the University Medical Centre Traffic statistics. In three months of the pilot study, there were of Ljubljana, is with 8,000 employees the largest employer in 2,629 rides of the total distance of 66,199 km and time of 1,915 Slovenia; daily, it is visited by additional 20,000 people. BTC, the hours (almost 32 days). Here, each "ride" means picking up the Business Trade Centre, is the largest shopping area in Slovenia. passengers at one or more origin locations and dropping them It does not have many direct public transportation links, but is down at one or more destination locations. Table 1 shows, grouped located close to a highway, inducing high volumes of car traffic. by the routes and in total, the total number of passengers and Before the study, the opportunity to join the experiment was average ride distances, times, speed and vehicle occupations. The advertised using different channels, particularly in the UKC and term "passenger" refers to one ride of a single person. BTC areas. Among more than 500 interested individuals, 131 were User statistics. The service was used by 131 individual users, eventually selected and invited to participate. All the operation, who used the service daily or less. The most active user used the including the IT solution, customer management and logistics, service 121 times, and the average was 54.88 times per user. Fe- was carried out by the project partner GoOpti. male users prevailed over males (73% vs. 27%). The prevailing age Two data sources were collected during the study: groups were 31–42 (43%) and 45–64 (46%), while the distribution • Traffic data: Collected by GoOpti while providing the ser- of users’ education levels was close to uniform. vice. This included detailed data about the travelled routes Vehicle occupancy. As large as possible vehicle occupancy is (distances, times and GPS locations) and provided services essential for the effectiveness of dynamic transportation. Figure 1 (anonymized individual user’s rides). displays the occupancy achieved in the study. • User survey: Collected using a survey questionnaire once per each individual user at the end of the study period. 4.2 User Survey Analysis The questions mainly addressed users’ current mobility The survey was completed by users at the end of the trial period, habits (with more detailed questions for users using cars) so that each user completed the survey at most once. Out of and their experience with the service. A full version of the the 131 users, the completed survey was submitted by 88: 30 questionnaire (in Slovene language) is available in [3]. travelling from/to Kranj and 58 in the Kamnik direction. Using this data, we carried out the following analyses: basic Current mobility habits. Figure 2 shows relative proportions traffic and demographical statistics, average occupancy of vans of transportation modes used by the survey respondents. The and individual users’ rides, users’ current mobility habits, and prevailing mode is using personal cars: 70% as sole drivers and user satisfaction with and willingness to pay for the service. 20% as fellow passengers. The train and city bus come next at By combining the data sources, we estimated the differences approximately 30% each, the bicycle at 10%, while the remaining between the current and dynamic means of transportation in modes are barely indicated. Both routes exhibit similar usage terms of travelling time and contribution to lower emissions of patterns. CO , NO (nitrogen oxides) and PM10 (particles with a diameter The respondents that use cars estimated their average occu- 2 x of 10 µm or less). pancy at 1.37 per ride. Most of the cars have diesel or gasoline In order to assess environmental burdens, we analyzed the engines (about 45% each). Hybrid and electric cars account only difference in air emissions between the "BaU - Business-as-Usual" for about 2%. The average age of cars is 9.15 years (7.28 on route scenario (i.e., trips without the introduction of dynamic trans- Kranj and 10.04 on route Kamnik). port) and the GoOpti service. We took into account the distances User satisfaction. The users were generally very satisfied with traveled by the types of transport that users used before the in- the service. They most appreciated the easiness and comfort of troduction of GoOpti: mainly driving passenger cars of different 1 Euro 1 (1992): 91/441/EEC, 93/59/EEC; Euro 2 (1996): 94/12/EC, 96/69/EC; Euro 3 types (gasoline, diesel), taking into account the age of the vehi- (2000): 98/69/EC; Euro 4 (2005): 98/69/EC (& 2002/80/EC); Euro 5 (2009): 715/2007/EC; cles and corresponding average emissions. Data sources included Euro 6 (2014) 37 Evaluation of the Effects of Dynamic Transportation Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Kamnik Kranj Total 60 Domžale Kamnik Komenda Kranj Mengeš Radomlje Total 50 Better 40 [%] 30 Occupancy Equal 20 10 orse W 01.02.2023 21.02.2023 13.03.2023 02.04.2023 22.04.2023 exit xibility Days and Reliability Comfort Fle y duration friendliness el entr of Trav Figure 1: Average vehicle occupancy, per days and routes. onmental oximity Pr Envir Transportation quality aspect 80 Kamnik Kranj Total Figure 3: Comparison of the respondents’ usual transporta- tion mode with the dynamic one. 60 Table 2: Comparison of average traveling times, in minutes. Entry Usual Dynamic Difference location transportation transportation users 40 Domžale 30.97 26.67 -4.30 of Kamnik 42.00 44.16 2.16 % Komenda 59.00 36.91 -22.09 Kranj 54.72 38.97 -15.75 20 Mengeš 36.00 27.80 -8.20 Radomlje 40.71 40.09 -0.63 Average 41.22 35.07 -6.15 0 er bus bus cle cle cle oter oter cy driv Train bicy – Bicy Sco sco We used both data sources. It should be noted that the comparison passenger City Motor Car – Inter-city ctric ctric Car Ele Ele includes only passengers who have completed the survey (other- wise we have no information about their normal time traveling Figure 2: Current modes of transportation, normalized by time). Additionally, we had to exclude two respondents because the number of respondents. Since multiple answers were of a mismatch with traffic data. Traffic data do not include any possible, the values shown are not true relative values and time of walking and waiting for a van. The survey data was col- do not add up to 100%. lected only once for each passenger (𝑛 = 86), while there was substantially more traffic data (5,476 passenger rides). Therefore, this mode of transport, its reliability, the proximity of stations, there is a substantial difference in the amount and quality of the the possibility to effectively use the time for themselves and data, and a cautious interpretation is advised. the fact that they did not need to search for the parking place. Results are shown in Table 2. Considering all entry locations, Complaints, on the other hand, were scarce and mainly addressed except Kamnik, the dynamic rides are faster (in average by 6.15 too long collection/drop-off times, variable collection times and minutes). This comparison is not entirely fair because the dy- the need to check the collection information every day. When namic transportation times do not include possible waiting and asked to compare their usual means of transportation with the walking times, but we can safely assume that the times are at dynamic one (Figure 3), they evaluated the latter better in all least comparable. points except the flexibility. It is very likely (on average 9 on the 0–10 scale) that the respondents will recommend the service to 4.4 Analysis of Environmental Burdens their friends and relatives. The analysis of the environmental burdens focused on CO and 2 Willingness to pay. When asked about their willingness to pay NO emissions and particulate matter (PM) for human health x for the service, the average response was about 70€ per month reasons. The results in Table 3 show a significant reduction in CO2 (64€ for Kamnik and 86€ for Kranj). emissions after the introduction of the dynamic transportation service: reduction of emissions by approximately 27%, or over 4.3 Comparison of Traveling Times 50% in case of higher occupancy/exclusive use of the dynamic We compared the time of traveling between the dynamic trans- service. Similar reductions are also expected with NO (26% to x portation and the usual modes of respondents’ transportation. 68%) and PM (27% to 79%). It is important to emphasize that 38 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Bohanec, et al. Table 3: Comparison of CO2, NOx and PM emissions. their expenses for commuting to work. Better awareness of users would be needed on the costs and environmental impacts of BaU Dynamic (weighted) Dynamic (total) different transportation modes. CO [t] 23.90 17.30 (-27%) 10.90 (-54%) 2 In summary, dynamic transportation seems a feasible and ef- NO [kg] 23.21 17.20 (-26%) 5.30 (-69%) x fective alternative to using cars for commuting to work. However, PM [kg] 1.44 1.05 (-27%) 0.30 (-79%) in Slovenia, it requires a careful consideration, at the levels of communities, cities, regions and the whole country, of how to the absolute values of PM particle emissions are low, since the attract the users and support the transition to this and other sus- average vehicle age is about 9 years, when the minimum emission tainable transportation means. In perspective, it will also require standards for solid particles have already been introduced. intelligent software for supporting the service. The tasks that are Generally, the results show a significant reduction in emissions particularly interesting for applying artificial intelligence meth- when implementing the dynamic transport compared to the BaU ods, include the prediction of customers’ requests, optimization scenario. However, it should be noted that the data refer to the of costs and dynamic planning of routes. average number of users and that these may vary depending on vehicle occupancy, driving dynamics (driving speed), road con- ACKNOWLEDGMENTS ditions (e.g., duration of traffic jams), etc. These are preliminary This work has been funded by the SmartMOVE project, which is estimates that do not yet take into account possible errors in data co-financed by Iceland, Lichtenstein and Norway funds from the collection and interpretation of uncertainty. The numbers/results EEA Financial Mechanism and corresponding Slovenian partic- correspond to the implementation of the pilot project only – the ipation within the Climate Change Mitigation and Adaptation potential up-scaling analysis would show the actual potential of programme in the amount of 1 609 167€. The contents of this doc- such measures for solving the urban mobility issues. ument are the sole responsibility of the authors and can in no way be taken to reflect the views of the Climate Change Mitigation 5 CONCLUSIONS and Adaption Program Authority. The results of this pilot study confirm that on-demand dynamic The authors also acknowledge the financial support from the transportation of employees to their jobs has a great potential Slovenian Research Agency for research core funding for the for improving the sustainable mobility in the Ljubljana Urban programme Knowledge Technologies (No. P2-0103). Region. As part of the pilot project, which was taking place from st th the 1 February 2023 to 28 April 2023, 2,629 van rides were REFERENCES carried out with a total length of 66,199 km and a total duration [1] Claudia Archetti, M. Grazia Speranza, and Dennis Weyland. 2017. A simula-of almost 32 days. The average ride was about 25 km long and tion study of an on-demand transportation system. International transactions in operational research, 25, 4, 1137–1161. doi: 10.1111/itor.12476. lasted 44 minutes. 131 individual users were involved, who made [2] Carlos Lima Azevedo et al. 2016. Microsimulation of demand and supply of a total of 7,182 individual commutes. Some used the service in autonomous mobility on demand. Transportation Research Record, 2564, 1, 21–30. doi: 10.3141/2564- 03. both directions on a daily basis, others less often. [3] Marko Bohanec, Marko Guček, Davor Kontić, Karina Sirk, Bernard Ženko, Most users of the service (about 70%) usually use a passenger and Martin Žnidaršič. 2023. Vrednotenje učinkov dinamičnih skupinskih car for their transport to work. The transition to dynamic trans-prevozov zaposlenih na delovna mesta. Tech. rep. Institut Jožef Stefan, Delovno poročilo DP-14427 (in Slovene), Ljubljana, Slovenija. portation puts all these cars "off the road", while only moderately [4] David Bole, Matej Gabrovec, Janez Nared, and Nika Razpotnik Visković. competes with other forms of public transportation. This also 2012. Integrated planning of public passenger transport between the city causes a significant reduction in harmful emissions of CO , NO and the region: the case of ljubljana. Acta geographica Slovenica, 52, 1, 141– 2 x 163. and solid particles into the environment: from 30% to 70%, de- [5] Malcolm Egan and Michal Jakob. 2016. Market mechanism design for prof-pending on the substance and transportation. Also, the dynamic itable on-demand transport services. Transportation Research Part B: Method-ological, 89, 178–195. doi: https://doi.org/10.1016/j.trb.2016.04.020. transportation is comparable with other modes of transportation [6] Cveto Gregorc and David Krivec. 2012. Networking of public passenger considering the speed and time spent. The satisfaction of users transport modes, a step towards sustainable mobility in ljubljana urban was also highly positive, as they liked that somebody else took region. Procedia-Social and Behavioral Sciences, 48, 3009–3017. [7] Simon Koblar and Luka Mladenovič. 2020. Calculating the speed of city bus care of their everyday commute to work, which was done without trips: the case of ljubljana, slovenia. Urbani izziv, 31, 1, 112–122. stress, accurately and reliably. They especially appreciated the [8] Wengen Li, Jiannong Cao, Jihong Guan, Shuigeng Zhou, Guanqing Liang, "door-to-door" aspect and avoiding the search for a parking lot. Winnie K. Y. So, and Michal Szczecinski. 2019. A general framework for un-met demand prediction in on-demand transport services. IEEE Transactions However, from the data collected we cannot actually determine on Intelligent Transportation Systems, 20, 8, 2820–2830. to which extent was the users’ enthusiasm caused by the fact [9] Statistical Office RS. 2023. Statistical yearbook of RS. accessed on 23.8.2023. (2023). https://www.stat.si/Pages/en/goals/goal- 12.- ensure- sustainable- co that the transport was free of charge during the pilot study. nsumption- and- production- patterns/12.5- average- co2- emissions- per- k Two problems were identified that might jeopardize the per- m- f rom- new- passenger- cars. manent operation of the dynamic transportation. The average [10] SmartMOVE. 2022–24. Smart mobility measures for sustainable mobility in slovenia. project financed by iceland, liechtenstein and norway funds occupancy of the vehicles was only around 2.73 passengers per from the eea financial mechanism and the norwegian financial mechanism. ride (34% of 8 seats). Of the two routes, Kamnik and Kranj, the (2022–24). https://kt.ijs.si/project/smartmove/. former were better occupied – by almost one passenger in aver- [11] Amirmahdi Tafreshian, Neda Masoud, and Yafeng Yin. 2020. Frontiers in service science: ride matching for peer-to-peer ride sharing: a review and age per ride. According to GoOpti [3], a higher occupancy (75%) future directions. Service Science, 12, 2-3, 44–60. doi: 10.1287/serv.2020.0258. could be achieved during the traffic peak times around the city [12] Jernej Tiran, Luka Mladenovič, and Simon Koblar. 2015. Accessibility to public transport using the ptal method: the case of ljubljana. Geodetski of Domžale, which lies halfway to Kamnik. Vestnik, 59, 4, 723–735. Another problem is the price that the users are willing to [13] G. Titos, H. Lyamani, L. Drinovec, F.J. Olmo, G. Močnik, and L. Alados-pay for the service, which amounts to around 70€ per month. Arboledas. 2015. Evaluation of the impact of transportation changes on air quality. Atmospheric Environment, 114, 19–31. This is significantly less than the economic price estimated at [14] Steve Wright, John D. Nelson, and Caitlin D Cottrill. 2020. Maas for the sub-250€ at 50% occupancy and 160€ at 75% [3]. It seems that car urban market: incorporating carpooling in the mix. Transportation Research users are not really aware of the actual costs and underestimate Part A: Policy and Practice, 131, 206–218. 39 Test uporabnosti prilagojenega WHCA* algoritma za iskanje poti za več agentov v strateški igri v realnem času Usability Test of a Modified WHCA* Algorithm for Multi-Agent Pathfinding in a Real-time Strategy Game Ivan Antešić Aleksander Sadikov ivan.v.antesic@gmail.com aleksander.sadikov@f ri- uni- lj.si Laboratorij za umetno inteligenco Laboratorij za umetno inteligenco Fakulteta za računalništvo in informatiko Fakulteta za računalništvo in informatiko Univerza v Ljubljani, Slovenia Univerza v Ljubljani, Slovenia POVZETEK 1 UVOD IN MOTIVACIJA V zadnjih letih je bilo predlaganih več algoritmov za iskanje poti NP-težek problem iskanja poti za več agentov (angl. multi-agent za več agentov (angl. kratica MAPF), ki naj bi bili primerni za pathfinding, MAPF) je definiran z grafom 𝐺 = (𝑉 , 𝐸) ter mno- vodenje enot v strateških igrah v realnem času. Toda algoritmi žico 𝑛 sodelovalnih agentov 𝑎 . Veljavna rešitev problema 1, ..., 𝑎𝑛 so predstavljeni in testirani brez upoštevanja ključnih lastnosti je množica 𝑛 poti, ki pripelje vsakega agenta 𝑎 od njegovega 𝑖 kompleksnega okolja iger, kot so dinamični zemljevidi, različne začetnega vozlišča 𝑠 ∈ 𝑉 do ciljnega vozlišča 𝑔 ∈ 𝑉 po pove- 𝑖 𝑖 lastnosti in hitrosti agentov, prisotnost sovražnikov. Da bi ugoto- zavah 𝑒 ∈ 𝐸, ne da bi dve poti prišli v konflikt. Konflikt oz. trk 𝑖 vili, ali je MAPF pristop res primeren za uporabo v igrah, smo nastane, ko se agenta 𝑎 in 𝑎 oba nahajata v vozlišču 𝑣 ∈ 𝑉 v 𝑖 𝑗 v obstoječem igralnem pogonu za strateške igre v realnem času istem časovnem koraku 𝑡 . Optimalna rešitev je po navadi tista, 𝑖 implementirali in prilagodili WHCA* algoritem ter ga primer- ki minimizira skupno vsoto cen vseh poti [2]. jali s standardnim LRA* pristopom iskanja poti za posameznega Več znanstvenih del predstavlja MAPF algoritme, za katere agenta. Eksperimentalni rezultati kažejo, da naša implementacija trdijo, da so primerni za uporabo v igrah, še posebno za strateške WHCA* algoritma znatno izboljša kakovost poti ter lahko reši igre v realnem času (angl. real-time strategy, RTS). Vendar predsta- težke primere, ki jih LRA* ne more. Čeprav je čas iskanja poti z vljeni algoritmi pri razvoju in testiranju ne upoštevajo ključne la- WHCA* veliko daljši, menimo, da MAPF ima potencial v razvoju stnosti kompleksnega okolja RTS iger. RTS igre vsebujejo agente iger. različnih hitrosti in značilnosti. Med igro se obstoječim agentom kadarkoli lahko dodeli nov ukaz, poleg tega pa se neprenehoma ABSTRACT ustvarjajo novi agenti. Značilni so tudi dinamični zemljevidi, ki jih igralci lahko tekom igre spreminjajo z gradnjo objektov, ter Over the years, several multi-agent pathfinding (MAPF) algo- sovražni agenti, s katerimi ne sodelujemo, ampak jih moramo še rithms have been proposed as suitable solutions for guiding vedno upoštevati pri iskanju poti, kot neprehodne, premikajoče units in real-time strategy (RTS) games. However, algorithms are ovire. tested without considering the crucial properties of a complex Izkaže se, da v praksi večina iger išče pot za vsakega agenta game environment, such as dynamic maps, different unit prop- posebej (angl. single-agent pathfinding, SAPF), po navadi z A* algo- erties and agent speeds, the presence of enemies. To determine ritmom [3], brez upoštevanja načrtov ostalih agentov. Konflikte, whether MAPF approach really is suitable for use in games, we ki nastanejo kasneje med premikanjem, poskusijo igre lokalno implemented and modified the seminal WHCA* algorithm in razrešiti z ad-hoc preverbami in pravili. Ta pristop je znan kot an existing RTS game engine and compared it to the common Lokalno popravljanje A* (angl. Local-Repair A*, LRA*) [10]. Ko LRA* single-agent pathfinding approach. Our experimental re- zahtevnost iger narašča lahko preprosti LRA* algoritem postane sults show that our WHCA* implementation greatly improves nezadosten. Zmogljivost je mogoče izboljšati na več načinov, na the path quality and can solve difficult scenarios that the single- primer s hierarhično abstrakcijo [9] ali z upoštevanjem simetrije agent approach cannot. WHCA*’s search times are much longer, preiskovalnega prostora [5]. Marsikateri SAPF algoritmi se lahko but we still think MAPF has potential in game development. danes uporabijo za vodenje velikega števila agentov v RTS igri. KLJUČNE BESEDE Ker pa pri iskanju ne upoštevajo načrtov drugih agentov, so opa- zni pogosti zastoji ter nasploh okorno in nenaravno premikanje, iskanje poti za več agentov, strateške igre v realnem času, hevri- še posebej v ozkih hodnikih. stično preiskovanje, WHCA* V želji, da bi premostili razkorak med znanstvenim razisko- vanjem in praktičnim razvojem iger, smo implementirali enega KEYWORDS ključnih MAPF algoritmov, Okvirjeni hierarhični kooperativni A* multi-agent pathfinding, real-time strategy games, heuristic search, (ang. Windowed Hierarchical Cooperative A*, WHCA*) v OpenRA WHCA* igralnem pogonu za RTS igre in ocenili, ali je MAPF pristop res uporaben za RTS igre. WHCA* namreč velja za enega temeljnih Permission to make digital or hard copies of part or all of this work for personal sub-optimalnih MAPF algoritmov, ki služi kot izhodišče mnogim or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and novejšim algoritmom in nadgradnjam [8, 1, 6]. the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner /author(s). Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia © 2023 Copyright held by the owner/author(s). 40 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Ivan Antešić and Aleksander Sadikov 2 METODE novo delno pot vsakih 𝑅 = 𝑤 /2 taktov (angl. ticks, osnovna 2.1 OpenRA časovna enota pogona in časovni korak našega algoritma). Ena sekunda ponavadi vsebuje okoli 25 taktov. Zaradi večje zveznosti OpenRA [4] je odprtokodni pogon za strateške igre v realnem premikov en časovni korak ne ustreza več enemu premiku celice času, ki med drugimi poganja na novo ustvarjeno igro Dune — agenti, glede na njihovo hitrost, zasedejo celico med premikom 2000. Le ta predstavlja dokaj tipičen primer RTS igre. Igralci iz za različno število taktov. To lahko povzroči nemirno obnašanje ptičje perspektive opazujejo mrežni, dvodimenzionalni zemljevid agentov, ki želijo zapolniti iskalno okno z nepotrebnimi premiki, in nadzirajo svoje enote. Cilj igre je nabirati surovine, razširiti kar odpravimo z znižanjem cene čakanja na mestu. bazni tabor z novimi zgradbami in proizvesti enote, s katerimi Zaradi kompleksnega premikanja agentov in arhitekture po- poskušamo napasti in uničiti nasprotnikov tabor. Enote imajo gona je preverjanje rezervacij oteženo. Potrebno je izračunati različne hitrosti, so zmožne več akcij in se gladko premikajo iz ene čas prihoda in odhoda agenta za sosednjo celico. Čas odhoda je celice v drugo. To predstavlja kompleksno okolje za navigacijo težko predvideti, saj ne vemo z gotovostjo v katero od naslednjih enot, še posebej, ker se vse odvija v realnem času. sosednjih celic se bo napotil, od tega pa je odvisno koliko časa bo preživel v celici. Netočni čas prihoda in odhoda lahko povzroči 2.2 LRA* sprejem konfliktnih poti ali zavračanje veljavnih rešitev. OpenRA za iskanje poti uporablja različico standardnega LRA* RRA* hevristika je bila prvotno namenjena za uporabo na algoritma. Ker se agenti lahko premikajo med celicami diago- mreži, kjer se agenti premikajo v štirih glavnih smereh. V našem nalno, LRA* za hevristično funkcijo uporablja oktilno razdaljo, primeru se agenti lahko premikajo še diagonalno, kar povzroči √ definirano kot 𝐷 = (|𝑥 − 𝑥 | + |𝑦 −𝑦 |) + ( 2 − 2) · min(|𝑥 − potrebo po dodatnem nadaljevanju iskanja RRA* z oktilno raz- 𝑜𝑐𝑡 2 1 2 1 2 𝑥 | − |), med dvema celicama ( daljo , tudi ko agent sledi prvotni poti. Da bi to optimizirali, v 1 , |𝑦2 𝑦1 𝑥 1, 𝑦 1) in (𝑥 2, 𝑦2). Oktilna razdalja je monotona in dopustna hevristika. glavnem kooperativnem iskanju filtriramo sosede in obdržimo Pred premikom agenta v novo celico algoritem požene zapo- le tiste, ki so že bili razviti z RRA*. To vpelje dodatne zaplete in redje preverb, s katerimi poskuša ujeti in preprečiti vse možne povzroči, da agent včasih ne uspe najti poti okoli dinamičnih trke s čakanjem ali iskanjem nove poti. Ta sistem je zapleten, ker ovir. med igranjem lahko pride do množice različnih robnih pogojev. WHCA* ima problem z vrstnim redom obdelave agentov, na To je ena glavnih slabosti LRA* pristopov. kar sta opozorila tudi Sturtevant [8] in Bnaya [1]. Lahko se zgodi, da agent z višjo prioriteto prvi rezervira pot in zasede vse celice 2.3 Osnovni WHCA* okoli naslednjega agenta z nižjo prioriteto, ki nato ne more najti poti. V takem primeru je konflikt neizogiben. Problem postane Osnovni WHCA* algoritem je bil v izvirnem članku [7] označen še bolj pogost zaradi heterogenih agentov v OpenRA, kjer lahko za nadvse primernega za RTS igre. Z A* preiskuje 3D prostor hitrejši agent rezervira več celic okoli počasnejšega agenta. Te- (2D mrežni zemljevid in diskretna časovna dimenzija) za vsakega žavo smo poskusili rešiti s prepisovanjem rezervacij, kar pa lahko agenta posebej ter rezervira najdene poti v skupno časovno- privede do kaskadne izgube sodelovanja med agenti. prostorsko rezervacijsko tabelo. Vnos vozlišča (𝑥, 𝑦, 𝑡 ) v tabeli 𝑖 Obdržali smo nekatere principe lokalnih preverb za izogiba- oznanja, da je celica (𝑥, 𝑦) že zasedena v časovnem koraku 𝑡 in 𝑖 nje trkom, ker so prisotne tudi enote s katerimi ne sodelujemo, zato ni na voljo ostalim agentom. Med razvijanjem vozlišča se na primer zgradbe in sovražniki. Dodali smo tudi hevristično preveri, katere sosednje celice so proste ovir in rezervacij, kar im-posodabljanje cilja. Agent zamenja za cilj prvo prosto celico v plicitno omogoči izogibanje trkom in kooperativno premikanje. radiju 8 celic okoli prvotnega cilja, če je le ta med iskanjem poti WHCA* za hevristično oceno razdalj uporablja Obratno nada- že zaseden ali pa za cilj vzame svojo trenuten položaj, če je že v ljevalno A* (angl. Reverse Resumable A*, RRA*). Pred začetkom bližini cilja (8 celic) in med premikanjem naleti na oviro. iskanja kooperativnih poti se najprej za vsakega agenta požene RRA* iskanje, ki z A* poišče najkrajšo pot od agentovega cilja do začetka. Zaradi monotonosti oktilne razdalje imamo sedaj na 3 EKSPERIMENTI voljo točno ceno do agentovega cilja za vsako razvito vozlišče. Časovno zmogljivost in kakovost poti prilagojenega WHCA*(R) Če med kooperativnim iskanjem agent zaide z najkrajše poti in algoritma smo testirali v igri OpenRA pogona za različne dolžine potrebuje hevristično oceno za novo vozlišče, nadaljujemo RRA* iskalnega okvirja 𝑅 = 𝑤 /2 in primerjali rezultate s prvotnim iskanje, dokler le-ta ne razvije potrebnega vozlišča in pridobi algoritmom pogona LRA*. točno razdaljo. WHCA* išče poti le za naslednjih 𝑤 korakov, kar Posamezni eksperiment sestoji iz 50 iteracij. Ena iteracija je pripomore k zmogljivosti. Ko agent izvede 𝑤 /2 premikov, še en- premik skupine 25 agentov iz začetnega območja na levi strani krat poženemo iskanje za naslednjih 𝑤 korakov. To imenujemo zemljevida do ciljnega območja na desni. Agentom se pripadajoča iskanje z okvirjem. začetna in ciljna celica znotraj območij določi naključno. Iteracija se konča, ko vsi agenti prispejo na cilj ali pa ko se izteče vnaprej 2.4 Prilagojeni WHCA* določena časovna limita testa. Agenti so naključno izbrani iz Zaradi kompleksnosti in narave OpenRA okolja je bilo potrebno nabora treh enot: hitro vozilo s kratkim časom obračanja, srednje prilagoditi osnovni WHCA* algoritem. Igralec lahko kadarkoli hitrim tankom z dolgim časom obračanja in počasni pešak s ustvari nove agente in ukazuje nove premike obstoječim agentom. takojšnjim obratom. Zaradi tega se kooperativno iskanje poti ne sme nikoli končati. Iskanje poti smo testirali na treh različnih zemljevidih vidnih Posledično mora biti rezervacijska tabela krožna. Agenti morajo na Sliki 1. Vsak od njih predstavlja določen scenarij, ki lahko po premiku v naslednjo celico izbrisati rezervacijo prejšnje celice, nastane tekom igre: da ne bi vplivala na iskanje poti v prihodnosti. Vsi agenti ne dobijo ukaza v istem trenutku zato je potrebno • SOVRAŽNIKI: prečkanje skupine premikajočih sovražnih sinhronizirati individualna iskanja in za vse agente izračunati agentov, časovna limita 1,150 taktov, velikost 64x64 celic 41 Test uporabnosti prilagojenega WHCA* algoritma v RTS igri Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Slika 1: Sheme zemljevidov eksperimentnih scenarijev: levo SOVRAŽNIKI, v sredini GRLO, desno IGRA. Modra barva označuje začetne celice, zelena pa ciljne. Skupini agentov v scenariju GRLO, si izmenjata začetni poziciji. Bele, temno sive in temno rjave celice predstavljajo neprehodne zgradbe ter naravne ovire. Ostale barve onačujejo prosto prehodna območja. Scenarij SOVRAŽNIKI vsebuje sovražnike, ki patruljirajo med zgornjimi in spodnjimi rdečimi celicami. IGRA vsebuje še v igrah pogosto, nabiralniško enoto — prijateljska patruljira ob vhodu spodnje baze, sovražna pa ob vhodu zgornje. • GRLO: premik dveh manjših skupin iz nasprotnih smeri WHCA*(50) 0.98 WHCA*(50) WHCA*(100) 34.54 35.08 WHCA*(100) SOVRA NIKI 0.98 WHCA*(200) SOVRA NIKI 35.63 WHCA*(200) skozi ozko grlo, časovna limita 1,150 taktov, velikost 64x64 LRA* 0.99 LRA* 0.99 43.01 celic 0.96 58.89 • 54.9 IGRA: simulacija povprečnega premika v nasprotnikovo GRLO 0.96 GRLO 1 51.52 0.01 BREZPREDMETNA MERITEV bazo med igranjem igre, časovna limita 2,350 taktov, veli- 63.19 kost 69x69 celic 0.94 IGRA 63.06 IGRA 0.95 63.26 0.96 80.99 0.96 Število agentov in dimenzije zemljevida smo izbrali skladno s 0 20 40 60 80 100 0.0 0.2 0.4 0.6 0.8 1.0 koraki povprečnim primerom igre. Rezultati eksperimentov so vidni na Slikah 2 in 3. Trajanje (a) povprečni delež uspešnosti (b) povprečna dolžina poti iskanja poti smo merili s številom razvitih vozlišč (zgornja vrstica 100 WHCA*(50) 100 WHCA*(50) 95 WHCA*(100) 95 WHCA*(100) 90 WHCA*(200) 90 WHCA*(200) Slike 3) in z milisekundami (spodnja vrstica Slike 3) . Da lažje ana-85 LRA* 85 LRA* 80 80 75 75 liziramo vpliv RRA* hevristike, smo, kot v ostalih člankih, ločili 70 70 65 65 60 60 časovne meritve na prvi inicializacijski takt in na vse naslednje 55 55 50 50 45 45 takte. Mera prvega takta je povprečje vseh prvih taktov iteracije. 40 40 35 35 30 30 Za mero naslednjih taktov smo vzeli povprečje maksimalnih vre- 25 25 20 20 15 15 dnosti iteracije po prvem taktu. Mera nam tako predstavlja kako 10 10 5 5 0 0 dolgo, v povprečju, algoritem išče pot v najtežjih situacijah. SOVRA NIKI GRLO IGRA SOVRA NIKI GRLO IGRA Dolžino poti smo merili s povprečnim številom korakov, ki jih (c) število trkov (d) število neuispešnih iskanj potrebuje agent, da prvič prispe do cilja (Slika 2b). poti Izmerili smo tudi delež agentov, ki uspešno prispejo do cilja (Slika 2a), kolikokrat algoritem ne uspe najti veljavne koopera-Slika 2: Rezultati eksperimentov za različne scenarije. tivne poti (Slika 2d) ter število trkov (Slika 2c). Agentov po trku ne odstranimo iz igre. (zaradi izjemno nizke uspešnosti LRA* za scenarij GRLO, dolžine 4 DISKUSIJA poti ni smotrno primerjati z meritvami za WHCA*). V preostalih Na Sliki 2b lahko hitro odčitamo, da ima WHCA* v primerjavi z primerih oba algoritma do cilja privedeta skoraj vse agente. LRA* manjšo povprečno dolžino poti. Boljša kakovost rešitve je Na žalost WHCA* potrebuje veliko več časa za pridobitev očitna tudi, ko s prostim očesom opazujemo premikanje agentov, kooperativne poti za vse agente. Čas je odvisen od širine iskalnega tudi za scenarij IGRA, ki predstavlja dober vpogled v delovanje okvirja, vendar je tudi najmanjši okvir počasnejši kot LRA* v vseh algoritma v OpenRA okolju. WHCA* gladko vodi agente skozi primerih. V povprečni OpenRA igri WHCA* s kompromisnim kompleksno okolje v bolj strnjenih skupinah. Agenti so zmožni okvirjem 𝑅 = 100 lahko porabi 700ms za iskanje poti, kar je v sodelovanja, se umikajo, da razrešijo konflikte in lahko obvozijo RTS igri moteče in opazno igralcem, tudi če ne prekine igro za počasnejše agente. Agenti v LRA* eksperimentih se premikajo toliko časa ob čisto vsakem ukazu. okorno, razpršeno in imajo navado slediti eden drugemu v dolgih, Če na Sliki 3 primerjamo rezultate razvitih vozlišč s časom v ozkih kolonah do cilja. milisekundah opazimo, da RRA* predstavlja le majhen delež časa Zaradi sodelovalnega iskanja poti je WHCA* zmožen rešiti iskanja poti z WHCA*, tudi v primerih, ko je število razvitih voz- tudi težek scenarij z ozkim grlom. Večji kot je iskalni okvir, bolje lišč z RRA* bistveno večje kot število vozlišč razvitih z WHCA*. bo WHCA* vodil agente skozi majhno odprtino. Pri LRA* agenti Računanje dodatnih korakov potrebnih za iskanje poti v komple- vedno obstanejo v gneči sredi prehoda in ne dosežejo svojih ciljev ksnem OpenRA okolju, povzroči, da razvijanje vozlišč traja dlje 42 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Ivan Antešić and Aleksander Sadikov 351 prvi takt iskanja 561 prvi takt iskanja 360 poznej i takti iskanja poznej i takti iskanja WHCA*(50) 580 1386 726 5237 RRA* prvi takt WHCA*(50) 3983 RRA* prvi takt WHCA*(50) 22846 3104 RRA* poznej i takti 1238 RRA* poznej i takti 9537 872 4189 1360 WHCA*(100) 1766 6548 2193 5447 WHCA*(100) 4494 WHCA*(100) 23709 1908 2682 8328 3296 18449 3218 WHCA*(200) 7457 20700 6861 5693 WHCA*(200) 5536 WHCA*(200) 24584 3580 4064 10107 prvi takt iskanja poznej i takti iskanja 1170 1171 15737 RRA* prvi takt LRA* 180 LRA* 11340 LRA* 2288 RRA* poznej i takti 0 5000 10000 15000 20000 25000 30000 0 5000 10000 15000 20000 25000 30000 0 5000 10000 15000 20000 25000 30000 razvita vozli a razvita vozli a razvita vozli a 208 prvi takt iskanja 243 prvi takt iskanja 425 prvi takt iskanja poznej i takti iskanja poznej i takti iskanja poznej i takti iskanja WHCA*(50) 212 391 279 78 RRA* prvi takt WHCA*(50) 56 RRA* prvi takt WHCA*(50) 306 RRA* prvi takt 42 RRA* poznej i takti 17 RRA* poznej i takti 120 RRA* poznej i takti 407 1414 751 WHCA*(100) 591 2029 691 83 WHCA*(100) 66 WHCA*(100) 305 27 35 99 1159 6435 1415 WHCA*(200) 2658 7757 2176 78 WHCA*(200) 79 WHCA*(200) 304 48 53 122 20 19 159 LRA* 4 LRA* 125 LRA* 27 0 2000 4000 6000 8000 10000 0 2000 4000 6000 8000 10000 0 2000 4000 6000 8000 10000 ms ms ms Slika 3: Rezultati dolžine iskanja poti v razvitih vozliščih (zgornje slike) in v milisekundah (spodnje slike) za različne scenarije: levi sliki SOVRAŽNIKI, sredinski GRLO, desni IGRA. za glavno WHCA* iskanje kot pa za bolj preprosto RRA* hevri- večini RTS iger. Lahko bi jo uporabili za igre v realnem času, ki stiko. Izkaže se, da WHCA* razvije največ vozlišč, ko računa drugi vsebujejo manj agentov in preprosto okolje, ali za potezne igre, okvir poti, saj se takrat kompleksnost primera poveča, medtem kjer je na voljo več časa za pridobitev kakovostne rešitve. ko RRA* razvije večino vozlišč že v prvem taktu. Posledično naša Kljub temu mislimo, da to ni dokončna zavrnitev MAPF pri- implementacija potrebuje manj časa v prvem inicializacijskem stopa za RTS igre. Z nadaljnjim praktičnim delom bi lahko postal taktu kot pa v naslednjih taktih. bolj primeren. WHCA* bi lahko nadgradili s hierarhično abstrak- Dinamični scenarij SOVRAŽNIKI povzroči težave WHCA* al- cijo [8] ali pa z dinamično postavitvijo okvirja okoli možnih goritmu, saj pokvari točnost RRA* hevristike ter oteži kooperacijo konfliktov [1]. Pomagalo bi tudi, če bi RTS igro razvili od začetka s prisotnostjo nepričakovanih sovražnih agentov, s katerimi ne z WHCA* v mislih, namesto da prilagajamo algoritem obstoječi sodelujemo. Rezultati na sliki 2c in 2d prikazujejo številne trke SAPF arhitekturi. in neuspela iskanja poti WHCA* algoritma, ki jih v nasprotju z Kolikor nam je znano, je to edina delujoča implementacija drugima scenarijema, znatno ne zniža niti največja širina iskal- MAPF algoritma v okolju realnočasovne igre. Izvorna koda je nega okvirja. V ostalih primerih večji iskalni okvirji preprečijo prosto dostopna na https://github.com/ia6382/OpenRA. dovolj neuspelih iskanj poti in trkov, da le-ti niso opazni med igro. Zanimivo je, da se med opazovanjem preprostejši LRA* v LITERATURA primeru sovražnih agentov izkaže za bolj uspešnega, saj le počaka [1] Zahy Bnaya in Ariel Felner. 2014. Conflict-oriented windowed hierarchical cooperative A*. V 2014 IEEE International Conference on Robotics and nekaj trenutkov, da se agent umakne, preden nadaljuje prvotno Automation (ICRA). IEEE, 3743–3748. pot. Nasprotno, WHCA* nemudoma poišče novo pot okoli ovire. [2] Ariel Felner, Roni Stern, Solomon Eyal Shimony, Eli Boyarski, Meir Gol-Kljub temu, dinamične ovire ne vplivajo občutno na časovno denberg, Guni Sharon, Nathan Sturtevant, Glenn Wagner in Pavel Surynek. 2017. Search-based optimal solvers for the multi-agent pathfinding problem: zmogljivost WHCA* algoritma. summary and challenges. V Tenth Annual Symposium on Combinatorial Search. [3] Peter E Hart, Nils J Nilsson in Bertram Raphael. 1968. A formal basis for 5 POVZETEK the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4, 2, 100–107. Implementirali smo prilagojeni WHCA* algoritem v obstoječem [4] [n. d.] OpenRA. V https://www.openra.net/. [5] Steve Rabin in Nathan Sturtevant. 2016. Combining bounding boxes and jps RTS igralnem pogonu OpenRA, da bi dognali, ali je MAPF pristop to prune grid pathfinding. V Proceedings of the AAAI Conference on Artificial res primeren za uporabo v RTS igrah. Intelligence številka 1. Zv. 30. [6] Devon Sigurdson, Vadim Bulitko, Sven Koenig, Carlos Hernandez in William Naš prilagojeni WHCA* smo testirali in primerjali s standar-Yeoh. 2019. Automatic algorithm selection in multi-agent pathfinding. arXiv dnim SAPF algoritmom LRA*, ki je že bil implementiran v pogonu. preprint arXiv:1906.03992. Izkaže se, da WHCA* najde veliko bolj kakovostne poti kot LRA* [7] David Silver. 2005. Cooperative pathfinding. AIIDE, 1, 117–122. [8] Nathan R Sturtevant in Michael Buro. 2006. Improving collaborative pathfin je zmožen gladkega, kooperativnega premikanja agentov v inding using map abstraction. V AIIDE. Marina del Rey, 80–85. zahtevnih scenarijih, kjer LRA* agenti obtičijo. Toda WHCA* je [9] Nathan R Sturtevant, Devon Sigurdson, Bjorn Taylor in Tim Gibson. 2019. dosti počasnejši od preprostega LRA* in v realnih primerih potre- Pathfinding and abstraction with dynamic terrain costs. V Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Enterta-buje nekaj sto milisekund za pridobitev poti, tako v prvem kot inment številka 1. Zv. 15, 80–86. tudi v naslednjih okvirjenih iskanjih. Zato menimo, da trenutna [10] A. Zelinsky. 1992. A mobile robot exploration algorithm. IEEE Transactions on Robotics and Automation, 8, 6, 707–717. doi: 10.1109/70.182671. WHCA* implementacija ni primerna za premikanje agentov v 43 An Attempt at Predicting Algorithm Performance on Constrained Multiobjective Optimization Problems Andrejaana Andova Aljoša Vodopija Jordan Cork Tea Tušar Bogdan Filipič andrejaana.andova@ijs.si Jožef Stefan Institute and Jožef Stefan International Postgraduate School Ljubljana, Slovenia ABSTRACT These features ideally characterize the problems so that similar When solving new optimization problems, it is crucial that al- problems have similar feature values. gorithms are selected capable of both finding the best solutions In this work, we propose a first step towards automatic algo- and computing them in reasonable amounts of time. However, rithm selection for CMOPs. This task is much harder for con- testing multiple algorithms is time-consuming and impractical. strained multiobjective optimization, because, in this area, there A solution to this would be to build a model that automatically are fewer benchmark problems available, and the ELA methods selects the algorithm that performs best on a new problem. In this are not as well developed as in single-objective optimization. work, we build machine learning models to automatically predict Although the ultimate goal of our work is automatic algorithm algorithm performance on constrained multiobjective optimiza- selection, we here focus on predicting the algorithm performance tion problems (CMOPs) using exploratory landscape analysis of three widely used algorithms. By proposing a method for pre- (ELA) features. The results showed a high mean absolute error, dicting algorithm performance for a few well-known algorithms, which indicates that, with the currently available benchmarks and researchers can easily extend the set of algorithms, in the future. ELA features, automatically predicting algorithm performance The paper is further organized as follows. In Section 2, we in-on CMOPs is a very hard task. troduce the theoretical background of constrained multi-objective optimization. In Section 3, we briefly describe the ELA features KEYWORDS used in this study. In Section 4, we describe the algorithm performance measure used as the prediction target value. In Section 5, constrained multiobjective optimization, evolutionary algorithms, we present the experimental setup and, in Section 6, the obtained exploratory landscape analysis, machine learning, algorithm per-results. Finally, in Section 7, we summarize the findings and formance prediction outline ideas for future work. 1 INTRODUCTION 2 THEORETICAL BACKGROUND The common way of solving black-box constrained multiobjec- A CMOP can be formulated as: tive optimization problems (CMOPs) is to use multiobjective opti- mization algorithms with constraint-handling techniques (CHTs). minimize 𝑓 (x), 𝑚 = 1, . . . , 𝑀 𝑚 However, deciding which specific algorithm to use, which CHT (1) subject to 𝑔 (x) ≤ 0, 𝑘 = 1, . . . , 𝐾, 𝑘 to include, and which setting of the algorithm parameters to apply is not trivial. where x = (𝑥1, . . . , 𝑥 ) is a solution vector of dimension 𝐷, 𝑓 (x) 𝐷 𝑚 In the last few years, several authors have tried to find ways are objective functions, 𝑔 (x) are constraint functions, and 𝑀 and 𝑘 of automatically selecting evolutionary algorithms for solving 𝐾 are the numbers of objectives and constraints, respectively. single-objective optimization problems [10, 13, 7]. The core con-In multiobjective optimization, we use the term search space cept behind their work is to extract features of benchmark single- 𝑆 , representing a 𝐷 dimensional space where all possible solution objective optimization problems and construct a model for pre-vectors x are located. Additionally, we can define the 𝑀 dimen- dicting which algorithm performs best for each individual prob- sional objective space 𝑃 = {𝑓 (x) | 𝑥 ∈ 𝑆 } which represents the lem. When dealing with a new problem then, the model is able space consisting of objective values for solutions. to automatically decide which algorithm to use for solving the A solution x is feasible, if it satisfies all constraints, 𝑔 (x) ≤ 0, 𝑘 problem. for 𝑘 = 1, . . . , 𝐾. A feasible solution 𝑥 is said to dominate another Extracting optimization problem features can be done using feasible solution y if 𝑓 (x) ≤ 𝑓 (y) for all 1 ≤ 𝑚 ≤ 𝑀, and 𝑚 𝑚 exploratory landscape analysis (ELA). This is a technique that 𝑓 (x) < 𝑓 (y) for at least one 1 ≤ 𝑚 ≤ 𝑀. A feasible solution 𝑚 𝑚 takes a sample of solutions and their fitness values as input and, x∗ is a Pareto-optimal solution if there exists no feasible solution based on this, extracts statistical features about the problem. x ∈ 𝑆 that dominates x∗. All feasible solutions constitute the feasible region 𝐹 . All nondominated feasible solutions form the Permission to make digital or hard copies of part or all of this work for personal Pareto set 𝑆o, and the image of the Pareto set in the objective or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and space is the Pareto front, 𝑃o = {𝑓 (x) | x ∈ 𝑆o}. the full citation on the first page. Copyrights for third-party components of this Nondomination ranking is a concept in multiobjective opti-work must be honored. For all other uses, contact the owner/author(s). mization that helps sort the solutions in a population into fronts, Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia based on their dominance. Thus, all nondominated solutions get © 2023 Copyright held by the owner/author(s). a nondomination rank of 1, solutions that are dominated only by 44 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Andova, et al. the nondominated solutions get a nondomination rank of 2, and other landscape characteristics. However, to calculate these fea- so on. tures one needs a larger sample size (a sample size of 250,000), The point in the objective space with the best objective values which makes these features computationally very demanding. is the ideal point 𝑧 = (min 𝑓 𝑓 (x)). In our study, we used both the features by Alsouly et al. and 𝐼 x∈𝑆o 1 (x), . . . , minx∈𝑆o 𝑀 The nadir point represents the point in the objective space Vodopija et al. with the worst fitness values across all solutions in the Pareto front 𝑧 = (max 𝑓 𝑓 (x)). 4 EMPIRICAL CUMULATIVE DISTRIBUTION 𝑁 x∈𝑆o 1 (x), . . . , maxx∈𝑆o 𝑀 The most widely used quality indicator in multiobjective op- FUNCTIONS timization is the hypervolume indicator [17]. It maps the set of One drawback of using hypervolume as the quality indicator in solutions found by an algorithm to a measure of the region dom- constrained multiobjective optimization is that it does not take inated by that set and bounded by a given reference point. into consideration infeasible solutions. For this reason, Vodopija et al. [15] proposed a new quality indicator designed specifically 3 ELA FEATURES FOR CONSTRAINED for constrained multiobjective optimization that generalizes the MULTIOBJECTIVE OPTIMIZATION hypervolume-based quality indicator 𝐼𝐻𝑉 + from [5] as follows: PROBLEMS (1) When there are no feasible solutions in the set, the quality indicator takes on the value of the smallest constraint ELA is a methodology that extracts the features of an optimiza- violation of all solutions in the set plus a threshold ∗ 𝜏 . tion problem from a sample of its solutions. These features are (2) When the set contains at least one feasible solution, the usually statistical relations between the solutions and are de- quality indicator equals the value of 𝐼 signed by experts. Many ELA feature sets were designed for 𝐻 𝑉 + bounded above by the threshold ∗ ∗ 𝜏 , i.e., it equals min{𝐼 }. single-objective optimization problems. However, only a few 𝐻 𝑉 +, 𝜏 ∗ feature sets exist for CMOPs. The threshold value 𝜏 ensures that any infeasible solution will State-of-the-art features for CMOPs were collected by Alsouly be deemed worse than any feasible one. et al. [1], who adopted all of the fast-computing features for To measure algorithm performance during the algorithm run, CMOPs from the related work, and also proposed some addi- we keep track of how many function evaluations, called run- tional features. The set of all features can be divided into three times, are needed to reach a particular quality indicator value, groups that describe: the multiobjective landscape, the violation called target. We do so for a number of targets and visualize these landscape, and the combination of the two landscapes – the mul- runtimes using the Empirical Cumulative Distribution Function tiobjective violation landscape. (ECDF) [5]. The ECDF measures the proportion of achieved tar-All three groups of features consist of global and random walk gets at a given runtime by the given algorithm. Whenever an features. The global features were calculated on a sample of size algorithm achieves a target, the value of the measure rises. Thus, 1000· the maximum value that can be achieved by an algorithm is equal 𝐷 . The random walk features are computed during a random walk, where statistics are derived from neighboring solutions to 1, meaning that the algorithm achieved all the targets. that form a sequence within the random walk. The random walk In our work, we want to express algorithm performance in a neighborhood is of size single value which will serve as the target of our machine learning 𝑁 = 2 · 𝐷 + 1, the length of the random walk is equal to ( (ML) problem. However, the ECDF is given for any number of 𝐷 /𝑁 ) · 103, and the step size is 2% of the range of the search space. function evaluations (up to a maximum value). To end up with In the multiobjective landscape group, the features are designed a single value, we use the area under the curve (AUC) of the to describe the objectives and the relations between them. Thus, ECDF, in short AUC-ECDF. This way, the ML method needs to the global features in this group include the proportion of un- predict a single target variable, which also includes information constrained Pareto optimal solutions, the hypervolume of the about the convergence of the algorithm over time. To normalize unconstrained Pareto front, the correlation between the objective the AUC-ECDF value, we divide it by the maximum number of values, statistics on the unconstrained ranks, etc. The random function evaluations. walk features in this group include statistics on the distance between random walk neighbors in the objective space. 5 EXPERIMENTAL EVALUATION In the violation landscape group, the features are designed to We focus on constrained bi-objective optimization problems describe the constraints of the problem. Thus, the global features with 2D, 3D, and 5D search spaces and, thus, use three widely in this group include statistics of the constraint violations, while used benchmarks for constrained multiobjective optimization the random walk features include statistics of the constraint – MW [9], CF [16], and C-DTLZ [6]. Because some benchmark violations between random walk neighbors. problems are only defined for more than 3D or more than three In the multiobjective violation landscape group, the features are objectives, the total number of problems per dimension differs. designed to describe the relations between the objectives and the Specifically, for 2D, we have 8 out of 14 MW problems, 0 out of 10 constraints. Thus, the global features in this group include the CF problems, and 5 out of 6 C-DTLZ problems. For 3D, we have proportion of feasible solutions, the proportion of Pareto optimal 14 out of 14 MW problems, 5 out of 10 CF problems, and 6 out of solutions, the hypervolume, statistics on the correlations between 6 C-DTLZ problems. For 5D, we have 14 out of 14 MW problems, objectives and constraints, statistics on the distance between 7 out of 10 CF problems, and 6 out of 6 C-DTLZ problems. solutions in the Pareto front, etc. The random walk features in The focus of this work is on predicting the algorithm perfor- this group include statistics on the dominance relations between mance of three multiobjective optimization algorithms – NSGA- random walk neighbors. III [6], MOEA/D-IEpsilon [4], and C-TAEA [8]. Each algorithm is Another state-of-the-art feature set for CMOPs is the one pro-equipped with a different constraint-handling technique. Due to posed by Vodopija et al. [14]. This feature set includes important the stochastic nature of the algorithms, we conduct 31 individual information about CMOPs, including their multimodality and runs of each algorithm on every given problem. This approach 45 An Attempt at Predicting Algorithm Performance on CMOPs Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia allows us to obtain more precise values for algorithm perfor- Table 1: Mean absolute error of the predicted AUC-ECDF mance. The target of the ML task for each problem is the mean values with respect to the true values for 2D, 3D, and 5D AUC-ECDF value over all 31 runs of the algorithm. To facilitate problems in leave-one-problem-out evaluation. the comparison of results, we use the same parameter settings for all algorithms – the population size 100 · 𝑀, and the number Dim ML method NSGA-III MOEA/D C-TAEA of generations 60 · 𝐷. Dummy 0.18 0.17 0.18 The ELA features are calculated stochastically; each time the Linear Regression 0.16 0.14 0.18 feature calculation is started, a different sample of solutions is 2D RF Regression 0.19 0.18 0.18 selected. To handle this, we created 30 samples using the Latin hy- SVR 0.20 0.21 0.19 percube sampling method, resulting in 30 sets of features (learn- ing instances) for each problem. Dummy 0.14 0.12 0.13 Predicting algorithm performance is a regression task and, Linear Regression 0.22 0.12 0.15 3D therefore, we use regression ML methods – Linear Regression, RF Regression 0.12 0.09 0.11 Random Forest Regression (RF Regression) [2], and Epsilon-SVR 0.14 0.12 0.13 Support Vector Regression (SVR) [3]. We also included a dummy Dummy 0.14 0.10 0.12 model in the comparison, which predicts the mean value of the Linear Regression 0.70 0.42 0.75 5D target variable in the training data. We utilized the scikit-learn RF Regression 0.13 0.09 0.12 implementations [11] of these methods with default parameter SVR 0.10 0.09 0.10 settings. We tested algorithm parameter tuning as well, but there was no significant improvement of the results. To evaluate the performance of the ML models, we use two rest in the training set it predicts very high target values, which evaluation methodologies – leave-one-sample-out and leave-one- increase the mean absolute error. problem-out. In the leave-one-sample-out evaluation, we use To better understand why the ML models performed poorly one instance as test data and the rest of the instances (including under the leave-one-problem-out evaluation, we used t-SNE [12] instances from other problems) as training data. We repeat this to reduce the dimensionality of the ELA features to 2D and vi- process for each instance in our dataset and take the average sualized the results, as shown in Figure 1. Here we notice that mean absolute error as an evaluation metric. Since all remaining samples from the same problem form clusters. This explains why instances of the problem are used during the training of the the results of the leave-one-sample-out evaluation are signifi- model, we expected the results from this evaluation methodology cantly better than the leave-one-problem-out evaluation – in the to be overly optimistic. former case, the ML task transforms into predicting the specific The leave-one-problem-out evaluation methodology is more problem to which a sample belongs. fairly designed. In the real world, we have no information about Analyzing the colors indicating the AUC-ECDF values of the the target problem available in the training data. Thus, in the three algorithms in Figure 1, we notice all algorithms perform sim-leave-one-problem-out evaluation, we use all instances of a prob- ilarly on almost all problems. This raises the question of whether lem as test data and the instances from the rest of the problems a different algorithm parameter setting should be considered, em- as training data. This process is repeated for each problem in phasising the differences in the performance of the algorithms. the dataset and the average mean absolute error is used as an For example, we could check the algorithm performance on a evaluation metric. smaller number of generations. When analyzing the colors showing the AUC-ECDF values of an algorithm in a single dimension, we notice there is no 6 RESULTS visible pattern. This holds for each problem dimension-algorithm The results showed a mean absolute error in the leave-one-sample- combination. Notably, we often find high and low AUC-ECDF out evaluation lower than 0 values appearing close to each other in the plot. .01. This result is overly optimistic and shows that same-problem instances are similar to each other. The results show that, with the current benchmarks and ELA The results obtained in the leave-one-problem-out evaluation features, predicting algorithm performance is very difficult. are presented in Table 1. These results show that none of the ML models performs significantly better than the dummy model. 7 CONCLUSION Moreover, because the target variable was normalized to [0,1], a In this work, we attempted to predict the algorithm performance mean absolute error between 0.09 and 0.22 is large. This indicates on CMOPs, using three well-known multiobjective optimization that the tested models trained on the current benchmarks with algorithms. For this purpose, we used ELA features specially the current ELA features cannot be used to predict algorithm designed for CMOPs as inputs to a ML model. To calculate the performance accurately. Also, we note that for each problem ELA features, we used 30 samples for each problem, resulting dimensionality, there is a different ML method that performs in 30 learning instances per problem. The target of prediction best. For 2D problems this is Linear Regression, for 3D problems was the algorithm’s AUC-ECDF value, computed using the qual- RF Regression, and for 5D problems SVR. ity indicator designed explicitly for constrained multiobjective A significantly worse performance is achieved by Linear Re- optimization [15]. gression on 5D problems. When attempting to understand the We tested three ML regression methods – Linear Regression, cause of this, we noticed that Linear Regression achieves simi- RF Regression, and SVR. To compare the results from these meth- lar results to the other models for all problems except for one, ods, we also used a dummy model, which always predicts the for which it performs very poorly. A possible explanation for mean value of the target variable in the training data. To evaluate this could be that Linear Regression is a simple and unbounded the results, we used two evaluation methodologies – leave-one- regression method, and when a problem is different from the sample-out and leave-one-problem-out. 46 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Andova, et al. Figure 1: t-SNE visualizations of 2D, 3D and 5D problems. The colors in the first row of the plots represent the problems included in the benchmark. In the remaining rows, the colors represent the algorithm performance measured by AUC-ECDF for each algorithm considered. In the leave-one-sample-out evaluation, very optimistic re- [4] Zhun Fan, Wenji Li, Xinye Cai, Han Huang, Yi Fang, Yugen You, Jiajie Mo, sults were found, with a mean absolute error lower than 0 Caimin Wei, and Erik Goodman. 2019. An improved epsilon constraint- .01. handling method in MOEA/D for CMOPs with large infeasible regions. Soft However, the results from the leave-one-problem-out evaluation Computing, 23, 12491–12510. were poor; none of the ML models significantly outperforms the [5] Nikolaus Hansen, Anne Auger, Dimo Brockhoff, and Tea Tušar. 2022. Any-dummy model. To explain why this occurs, we used the t-SNE time performance assessment in blackbox optimization benchmarking. IEEE Transactions on Evolutionary Computation, 26, 6, 1293–1305. method to reduce the dimensionality of the ELA features and [6] Himanshu Jain and Kalyanmoy Deb. 2013. An evolutionary many-objective plotted them in a color scheme indicating the performance of optimization algorithm using reference-point based nondominated sorting approach, part II: Handling constraints and extending to an adaptive the algorithms. These visualizations show no visible patterns in approach. IEEE Transactions on Evolutionary Computation, 18, 4, 602–622. the algorithm performance figures. Thus, we conclude that, with [7] Anja Jankovic, Tome Eftimov, and Carola Doerr. 2021. Towards feature-based the currently available ELA features and benchmark problems, performance regression using trajectory data. In Applications of Evolutionary Computation: 24th International Conference. Springer, 601–617. predicting algorithm performance is a hard task. [8] Ke Li, Renzhi Chen, Guangtao Fu, and Xin Yao. 2018. Two-archive evolution-In future work, we aim to address two distinct aspects of the ary algorithm for constrained multiobjective optimization. IEEE Transactions problem. The first is to improve the ELA features via automatic on Evolutionary Computation, 23, 2, 303–315. [9] Zhongwei Ma and Yong Wang. 2019. Evolutionary constrained multiobjec-construction using an end-to-end deep neural network. The sec- tive optimization: Test suite construction and performance comparisons. ond is to reduce the complexity of the ML task by simplifying the IEEE Transactions on Evolutionary Computation, 23, 6, 972–986. [10] Ana Nikolikj, Carola Doerr, and Tome Eftimov. 2023. Rf+clust for leave-target. This could be achieved by changing the task to a classifica- one-problem-out performance prediction. In Applications of Evolutionary tion task or by changing the target to the number of generations Computation: 26th International Conference. Springer, 285–301. required for an algorithm to reach a feasible solution. [11] F. Pedregosa et al. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. [12] Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using ACKNOWLEDGEMENTS t-SNE. Journal of Machine Learning Research, 9, 11. [13] Diederick Vermetten, Hao Wang, Thomas Bäck, and Carola Doerr. 2020. The authors acknowledge the financial support from the Slove- Towards dynamic algorithm selection for numerical black-box optimization: nian Research and Innovation Agency (young researcher pro-Investigating bbob as a use case. In Proceedings of the 2020 Genetic and gram, research core funding No. P2-0209, and project No. N2-Evolutionary Computation Conference, 654–662. [14] Aljoša Vodopija, Tea Tušar, and Bogdan Filipič. 2022. Characterization of 0254 “Constrained Multiobjective Optimization Based on Problem constrained continuous multiobjective optimization problems: A feature Landscape Analysis”). space perspective. Information Sciences, 607, 244–262. [15] Aljoša Vodopija, Tea Tušar, and Bogdan Filipič. 2023. Characterization of constrained continuous multiobjective optimization problems: A perfor-REFERENCES mance space perspective. arXiv preprint arXiv:2302.02170. [1] Hanan Alsouly, Michael Kirley, and Mario Andrés Muñoz. 2022. An instance [16] Qingfu Zhang, Aimin Zhou, Shizheng Zhao, Ponnuthurai Nagaratnam Sug- space analysis of constrained multi-objective optimization problems. IEEE anthan, Wudong Liu, Santosh Tiwari, et al. 2008. Multiobjective optimization Transactions on Evolutionary Computation. Early Access. https://ieeexplore test instances for the CEC 2009 special session and competition, 1–30. .ieee.org/abstract/document/9899751. [17] Eckart Zitzler, Lothar Thiele, Marco Laumanns, Carlos M Fonseca, and [2] Leo Breiman. 2001. Random forests. Machine Learning, 45, 5–32. Viviane Grunert Da Fonseca. 2003. Performance assessment of multiobjec- [3] Harris Drucker, Christopher J Burges, Linda Kaufman, Alex Smola, and tive optimizers: An analysis and review. IEEE Transactions on Evolutionary Vladimir Vapnik. 1996. Support vector regression machines. Advances in Computation, 7, 2, 117–132. Neural Information Processing Systems, 9. 47 Večstopenjski postopek vrednotenja rešitev pri načrtovanju elektromotorja A Multi-Step Evaluation Process in Electric Motor Design Tea Tušar Peter Korošec Bogdan Filipič tea.tusar@ijs.si peter.korosec@ijs.si bogdan.f ilipic@ijs.si Institut “Jožef Stefan” in Institut “Jožef Stefan” Institut “Jožef Stefan” in Mednarodna podiplomska šola Ljubljana, Slovenija Mednarodna podiplomska šola Jožefa Stefana Jožefa Stefana Ljubljana, Slovenija Ljubljana, Slovenija POVZETEK 1 UVOD Pri načrtovanju elektromotorja je potrebno poiskati vrednosti Podjetje MAHLE Electric Drives Slovenija d.o.o. proizvaja za- optimizacijskih spremenljivk tako, da izdelek izpolnjuje tehnične ganjalnike in alternatorje za motorje z notranjim izgorevanjem, zahteve in je njegova cena minimalna. V ta namen uporabljamo avtonomno napajane enosmerne električne pogonske sisteme in optimizacijske postopke z iterativnim vrednotenjem rešitev na druge zahtevnejše komponente za avtomobilsko industrijo. Eden osnovi numerične simulacije. Ti so računsko zahtevni, zato je izmed pomembnejših izdelkov podjetja je sinhronski elektromo- glavni izziv načrtovanja, kako najti kakovostne rešitve v spreje- tor s površinsko nameščenimi magneti, ki poganja avtomobilski mljivem času. V prispevku predstavljamo računalniško podprto servovolanski sistem (slika 1). načrtovanje sinhronskega elektromotorja za servovolanske sis- teme, s poudarkom na prijemih za pohitritev optimizacijskega postopka. Med njimi je tudi posebej za ta problem razvit večsto- penjski postopek vrednotenja rešitev, ki omogoča učinkovitost optimizacije in robustnost rešitev. S tem postopkom razviti elek- tromotor je boljši od prvotnega prototipa, dobljenega z enostav- nejšim optimizacijskim postopkom, tako po tehničnih lastnostih kot stroškovno. Slika 1: Elektromotor za servovolanske sisteme (Vir: arhiv ABSTRACT Mahle Electric Drives Slovenija d.o.o.). In the design of an electric motor, one has to find the values of the optimization variables such that the product satisfies the tech- Razvoj takšnega elektromotorja zahteva določitev geometrije nical requirements and its price is minimal. For this purpose, we in materialnih lastnosti njegovih komponent tako, da bo izpol- deploy optimization procedures with iterative evaluation of solu- njeval vse tehnične zahteve in bo njegova cena minimalna. Ker tions based on numerical simulation. These are time-consuming, se pri vrednotenju načrtov elektromotorja uporablja numerični hence the key challenge of the design is how to find high-quality simulator po metodi končnih elementov, v katerega nimamo vpo- solutions in an acceptable time. In this paper, we present the gleda (gre za t.i. problem črne škatle, ang. black box-problem), computer-aided design of a synchronous electric motor for power optimizacijski postopek terja iterativno vrednotenje številnih steering systems, with an emphasis on measures for speeding načrtov. Numerične simulacije so dolgotrajne, zato je glavni izziv up the optimization process. Among them, a multi-step solution razvoja elektromotorja optimizacijski postopek zastaviti tako, da evaluation procedure has been developed particularly for this bo lahko našel dobre rešitve v doglednem času. problem. It enables the efficiency of optimization and the robust- V nadaljevanju prispevka opisujemo, kako smo se načrtovanja ness of solutions. The resulting electric motor outperforms the elektromotorja (2. razdelek) lotili na Inštitutu “Jožef Stefan” v original prototype obtained by a simpler optimization procedure sodelovanju s podjetjem MAHLE. Optimizacijski postopek smo both in technical characteristics and cost efficiency. pohitrili s tremi prijemi: – uporabili smo optimizacijski algoritem s hitro izračunlji- KLJUČNE BESEDE vimi nadomestnimi modeli (3. razdelek); načrtovanje, elektromotor, numerična simulacija, optimizacija, – vrednotenje rešitev smo razdelili na več korakov in tako evolucijski algoritem, robustnost omogočili izločanje nedopustnih rešitev pred dolgotraj- nimi simulacijami (4. razdelek); KEYWORDS – računsko najzahtevnejši korak vrednotenja rešitev smo poenostavili in paralelizirali (4. razdelek). design, electric motor, numerical simulation, optimization, evo- lutionary algorithm, robustness Metodologijo smo preizkusili na konkretnem tipu elektromotorja (5. razdelek). Tako optimiran načrt elektromotorja je dosegel 10 % Permission to make digital or hard copies of part or all of this work for personal nižjo ceno komponent v primerjavi z različico, ki jo je podjetje or classroom use is granted without fee provided that copies are not made or predhodno razvilo s preprostejšim optimizacijskim postopkom. distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this V prihodnje nameravamo računalniško implementacijo postopka work must be honored. For all other uses, contact the owner /author(s). razširiti tako, da bo omogočala prosto izbiro optimizacijskega kri- Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia terija in omejitev ter bo uporabna za raznovrstne elektromotorje © 2023 Copyright held by the owner/author(s). (6. razdelek). 48 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Tea Tušar, Peter Korošec, and Bogdan Filipič 2 NAČRTOVANJE ELEKTROMOTORJA Poleg strogih omejitev imajo optimizacijski problemi v praksi pogosto tudi šibke omejitve. To so funkcije ℎ : 𝑋 → Pri načrtovanju elektromotorja moramo nastaviti vrednosti šte- 𝑖 R, 𝑖 = 1, . . . , 𝑙 , za katere želimo, da velja ℎ (𝑥 ) ≤ 0, ni pa to pogoj, da je rešitev vilnih parametrov, ki določajo njegovo geometrijo in materialne 𝑖 dopustna. V optimizacijskem problemu jih upoštevamo tako, da lastnosti njegovih komponent. To so na primer dimenzije zoba na jih vgradimo v kriterijsko funkcijo na naslednji način statorju, število navojev tuljave, dimenzije magnetov ipd. Vsak parameter ima podano spodnjo in zgornjo mejo ter najmanjši 𝑙 ∑︁ smiselni korak znotraj teh meja. Cilj optimizacije načrtovanja je 𝑓 (𝑥 ) = 𝑓 (𝑥 ) + max{ℎ (𝑥 ), 0}, (3) ℎ 𝑖 najti načrt elektromotorja, ki zadošča vsem tehničnim zahtevam 𝑖 =1 in je obenem najcenejši. kjer je 𝑓 prvotna kriterijska funkcija, max{ℎ (𝑥 ), 0} pa nenega- 𝑖 Načrt elektromotorja lahko ocenimo na več načinov, z različno tivna kazen za kršitev šibke omejitve ℎ . Ker seštevamo prvotno 𝑖 stopnjo zaupanja. Njegove lastnosti lahko najzanesljiveje preve- kriterijsko funkcijo in kazni za kršitev šibkih omejitev, moramo rimo, če na podlagi načrta izdelamo prototip elektromotorja in zagotoviti, da so njihove vrednosti primerljive. V ta namen jih je ga preizkusimo v praksi. Vendar to zahteva veliko dela, materiala potrebno normalizirati oz. primerno utežiti. in časa ter s tem povezane visoke stroške, ki ne dovoljujejo, da bi Pri problemu načrtovanja elektromotorja ne poznamo ana- podjetje v fazi razvoja izdelalo večje število prototipov. Zato si pri litične oblike kriterijske funkcije in omejitev, zato za njegovo reševanju tega problema pomagamo z računalniško podprtimi reševanje uporabimo algoritem, ki se dobro obnese na problemih numeričnimi simulacijami, na osnovi katerih lahko izpeljemo črne škatle. To je evolucijska strategija s prilagajanjem kovari- ključne lastnosti elektromotorja. Računalniški programi, kot je ančne matrike, oz. CMA-ES (ang. Covariance Matrix Adaptation Ansys Maxwell [1], omogočajo simulacijo elektromagnetnega po-Evolution Strategy) [5]. Natančneje, poslužujemo se različice al-lja elektromotorja z uporabo metode končnih elementov [2]. Ta goritma CMA-ES imenovane lq-CMA-ES [3], ki vrednotenja z deluje na podlagi mreže objekta; gostejša mreža omogoča večjo dolgotrajnimi simulacijami delno zamenja s hitro izračunljivimi točnost simulacije, a je ta dolgotrajnejša. Zanesljivost numeričnih linearno-kvadratičnimi nadomestnimi modeli. Uporaba nado- simulacij je tako deloma nastavljiva – odvisna je od računalniških mestnih modelov je pogosto uporabljen pristop pri reševanju zmogljivosti in časa, ki jih imamo na voljo. problemov z računsko zahtevnim vrednotenjem rešitev. Vendar zanesljivost simulacij znižujejo praktični vidiki izde- Optimizacijski algoritem z nadomestnimi modeli lq-CMA-ES lave elektromotorja, saj lahko ujemanje izdelanega elektromo- deluje v dveh fazah. Najprej na podlagi začetnih rešitev, ovredno- torja z načrtom zagotovimo samo v okviru določenih toleranc. tenih z numerično simulacijo, zgradi nadomestni model. Nato Na primer, če za velikost odprtine reže nastavimo vrednost 2 iterativno predlaga nove rešitve in jih, dokler so te dovolj po- mm, lahko v proizvodnji zagotovimo le, da bo ta na intervalu dobne obstoječim, vrednoti z nadomestnim modelom ter tako [1.95 mm, 2.05 mm]. To pomeni, da je za načrt elektromotorja prihrani na času. Ko pride do rešitev, ki jih nadomestni model zelo pomembno, da je robusten, to je, da ob majhnih spremembah ne opisuje več dovolj dobro, pa jih ovrednoti z numerično simu- vrednosti parametrov znotraj toleranc lastnosti elektromotorjev lacijo in z njihovim rezultatom posodobi nadomestni model. To ne odstopajo bistveno. Robustnost načrta je najlažje preveriti s ponavlja, dokler ne izpolni zaustavitvenega pogoja. simuliranjem številnih načrtov, ki se malo razlikujejo od izho- diščnega. Vendar to zahteva še več računsko zahtevnih simulacij 4 VREDNOTENJE POSAMEZNEGA NAČRTA in podaljšuje trajanje optimizacijskega postopka. ELEKTROMOTORJA Uporaba nadomestnih modelov zmanjša število izvedenih vre- 3 OPTIMIZACIJSKI POSTOPEK dnotenj s simulacijami, a so te pri načrtovanju elektromotorja V optimizacijskem postopku skušamo čim učinkoviteje rešiti vseeno potrebne. Da bi vrednotenje posameznega načrta pohitrili, dani optimizacijski problem. Formalno (in brez škode za splo- smo ga razdelili na pet korakov. Če se rešitev slabo izkaže po šnost) lahko optimizacijski problem načrtovanja elektromotorja katerem od korakov, jo takoj zavržemo in s tem prihranimo čas, zapišemo v obliki ki bi ga sicer porabili za izvedbo preostalih korakov. Vrednotenje je orisano na sliki 2 in opisano v nadaljevanju. minimiziraj 𝑓 (𝑥 ), (1) (1) V prvem koraku s hitro izračunljivo skripto, ki so jo pripra- ob pogojih 𝑔 (𝑥 ) ≤ 0, 𝑖 = 1, . . . , 𝑘, 𝑖 vili domenski eksperti, preverimo zadoščanje nekaterim 𝑛 kjer je 𝑥 = (𝑥 ) ∈ rešitev iz strogim omejitvam. To nam pomaga izločiti precejšnje šte- 1, . . . , 𝑥 𝑋 ⊆ 𝑛-dimenzionalnega 𝑛 R prostora rešitev 𝑋 , 𝑓 : 𝑋 → R kriterijska funkcija, funkcije 𝑔 : vilo nedopustnih rešitev, med njimi tudi take, ki bi lahko 𝑖 𝑋 → R, 𝑖 = 1, . . . , 𝑘, pa so stroge omejitve. Rešitev problema (načrt zaradi slabo zasnovane geometrije povzročale težave pri iz-elektromotorja) je dopustna, če zadošča vsem strogim omejitvam. vedbi simulacije. Samo dopustne rešitve gredo v naslednji Sicer pravimo, da je nedopustna. korak vrednotenja. Pri nedopustnih rešitvah ne moremo vedno izračunati vredno- (2) V drugem koraku izvedemo simulacijo, s katero pridobimo sti kriterija. Zato takrat kriterij 𝑓 nadomestimo s funkcijo informacije o določenih pomembnih lastnostih elektromo- torja. Ta uporablja poenostavljeno, simetrično formulacijo 𝑘 ∑︁ geometrije elektromotorja, zato je relativno hitra (za iz- 𝑓 (𝑥 ) = 𝑝 + max{𝑔 (𝑥 ), 0}, (2) 𝑔 𝑖 brani problem traja približno 1 minuto). Trajanje simula- 𝑖 =1 cije je sicer odvisno od posameznih rešitev. Pri načrtih, za kjer je 𝑝 konstanta in max{𝑔 (𝑥 ), 0} nenegativna kazen za krši- katere je geometrija zaradi neposrečene kombinacije vre- 𝑖 tev stroge omejitve 𝑔 (ko omejitev, ni kršena, kazen znaša 0). dnosti parametrov nemogoča, simulator lahko po dolgem 𝑖 Konstanta 𝑝 mora biti dovolj velika, da je vrednost kriterija 𝑓 za času ne vrne nobenega rezultata ali pa se (redko) celo zruši. 𝑔 katerokoli nedopustno rešitev vedno višja (slabša) od vrednosti Če rešitev ni izvedljiva ali ne zadošča strogim omejitvam, kriterija 𝑓 za katerokoli dopustno rešitev. jo zavržemo. Sicer nadaljujemo z naslednjim korakom. 49 Večstopenjski postopek vrednotenja rešitev pri načrtovanju elektromotorja Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Večstopenjsko vrednotenje rešitev robustnosti ene rešitve izvesti več simulacij, ki so med seboj neodvisne, to izkoristimo za njihovo paralelizacijo. Rešitev 𝑥 To izvedemo tako, da vse simulacije z asimetričnim mo- delom elektromotorja poženemo hkrati na računalniku z večjedrnim procesorjem in tako prihranimo veliko časa. Če preverjanje robustnosti mine brez težav, nadaljujemo z 1. korak Rešitev zadnjim korakom postopka. nedopustna Izračun dopustnosti (5) V zadnjem, petem koraku izračunamo še dodatne lastnosti elektromotorja in preverimo dopustnost preostalih strogih Python E omejitev. Rešitev dopustna Za vsako rešitev 𝑥 po izvedbi opisanega postopka izračunamo Rešitev vrednost optimizacijskega kriterija. Ta je odvisna od koraka, do 2. korak nedopustna katerega se je izvedlo vrednotenje rešitve, in njene kakovosti: (izračunljiva ali Simulacija lastnosti neizračunljiva) s simetričnim modelom 1 1 300 + Í𝑘 max{𝑔 (𝑥 ), 0} 𝑥 nedopustna po 1. kor.  𝑖 =1 𝑖    Ansys U 250 𝑥 neizračunljiva v 2. kor.    2  2 200 + Í𝑘 max{𝑔 (𝑥 ), 0} 𝑥 nedopustna po 2. kor. Rešitev   𝑖 =1 𝑖 𝑓 (𝑥 ) = dopustna 150 𝑥 neizračunljiva v 4. kor.    5  5 3. korak 100 + Í𝑘 max{𝑔 (𝑥 ), 0} 𝑥 nedopustna po 5. kor.   𝑖 =1 𝑖   Izračun cene 𝑐 (𝑥 ) + Í𝑙 max{ℎ (𝑥 ), 0} 𝑥 dopustna 𝑖  𝑖 =1 (4) Python E 1 2 5 Pri tem 𝑔 , 𝑔 in 𝑔 po vrsti predstavljajo stroge omejitve v 1., 2. in 5. koraku, 𝑐 ceno rešitve in ℎ njene šibke omejitve. Kazni za 𝑖 kršitev obeh vrst omejitev 𝑔 in ℎ so normalizirane tako, da nji- 𝑖 𝑖 hova vsota nikoli ne preseže vrednosti 50. Na ta način poskrbimo, 4. korak da so dopustne rešitve vedno ocenjene bolje od nedopustnih in Simulacija robustnosti je rešitev, ki je šla čez več korakov postopka vrednotenja, oce- z asimetričnim modelom Rešitev njena bolje od tiste, ki se je slabo izkazala v katerem od prejšnjih neizračunljiva korakov. Ansys Ansys U U Ansys U Opisani večstopenjski postopek vrednotenja rešitev s pohi- ... tritvami je glavna novost, ki smo jo v okviru sodelovanja med inštitutom in podjetjem uvedli v optimizacijo načrtovanja elektro- Ansys Ansys U U Ansys U motorja. Razvijalci v podjetju so že prej uporabljali optimizacijo Rešitev z nadomestnimi modeli v okviru programskega orodja Ansys [1], izračunljiva vendar pa so v njej lahko upoštevali samo 2. in 3. korak vrednote- 5. korak nja. Začetnega preverjanja dopustnosti (1. koraka) ni bilo, zadnjih dveh korakov pa se ni dalo enostavno vključiti v optimizacijski Izračun lastnosti postopek, zato sta se izvedla šele po zaključku optimizacije na Python E izbranih (najboljših) rešitvah optimizacijskega problema. Rešitev (dopustna ali 5 NUMERIČNI POSKUSI IN REZULTATI nedopustna) Optimizacijski postopek smo preizkusili na konkretnem primeru Izračun optimizacijskega kriterija elektromotorja za servovolanske sisteme. Ta ima 13 optimizacij- 1 skih spremenjivk, od katerih je 12 ‘zveznih’ in ena celoštevilska . Optimizacijski kriterij (za dopustne rešitve) sestavljata cena in vsota kršitev dveh šibkih omejitev, medtem ko je vseh strogih Vrednost 𝑓 (𝑥 ) omejitev 10. Pri simulaciji robustnosti z asimetričnim modelom (4. korak) smo hkrati poganjali 15 simulacij. V optimizacijskih izračunih smo uporabili knjižnico pycma, Slika 2: Vrednotenje rešitev v petih korakih. ki nudi implementacijo algoritma lq-CMA-ES v programskem jeziku Python [4]. Algoritmu smo posredovali informacijo o tem, (3) V tretjem koraku iz vseh dobljenih podatkov izračunamo da je ena od spremenljivk celoštevilska, ostale je obravnaval ceno elektromotorja. kot zvezne. Velikost začetnega koraka 𝜎 smo nastavili na 0.2, 0 (4) V četrtem koraku preverimo robustnost rešitve. Pri tem se dovoljeno število avtomatskih ponovnih zagonov algoritma pa simulacije izvajajo na celotni, asimetrični geometriji elek- na 5. Vrednosti preostalih parametrov algoritma so bile enake tromotorja, zato so časovno bolj potratne. Pohitrimo jih z privzetim. uporabo manj natančne mreže (predhodni poskusi so po- kazali, da lahko nekaj zanesljivosti žrtvujemo za precejšnji 1 Tudi ‘zvezne’ spremenljivke so zaradi najmanjšega smiselnega koraka v resnici prihranek časa; tako za izbrani problem ena simulacija diskretne, a jih obravnavamo kot zvezne s stališča algoritma (preden rešitev ovre-traja 7 namesto 15 minut). Ker moramo za preverjanje dnotimo, jo zaokrožimo na najbližjo diskretno vrednost). 50 Information Society 2023, 9–13 October 2023, Ljubljana, Slovenia Tea Tušar, Peter Korošec, and Bogdan Filipič Rezultati algoritma lq-CMA-ES Kvantitativnega ovrednotenja doprinosa posameznih pohitri- 300 Zagon algoritma tev nismo izvedli, vemo pa, da se je vrednotenje rešitev končalo 200 1 po 1. koraku v 17,5 % primerov (8 min prihranka na rešitev) in 23 po 2. koraku v 29,4 % primerov (7 min prihranka na rešitev). Ker smo analizo robustnosti, ki se izvede v 4. koraku, precej 100 poenostavili, da smo jo pohitrili in vključili v optimizacijo, smo preverili, če so rezultati 4. in 5. koraka skladni s tistimi, ki bi jih 60 dobili z prvotno analizo. Zato smo nekaj izbranih (najboljših) 40 rešitev podrobneje analizirali in njihove rezultate primerjali s tistimi, ki veljajo za prvotni optimirani načrt. Ugotovili smo, da Optimizacijski kriterij 30 dobimo kakovostne rešitve, ki niso le dopustne, ampak so za 10 % 20 cenejše od prvotnega najboljšega načrta. 6 ZAKLJUČKI 10 0 1000 2000 3000 4000 5000 Problem načrtovanja elektromotorja je zahteven praktičen opti- tevilo re itev mizacijski problem, saj je mnogo načrtov nedopustnih, njihovo vrednotenje pa temelji na numeričnih simulacijah in je zato dolgo- Slika 3: Najnižja vrednost optimizacijskega kriterija tekom trajno. Da bi ga lahko uspešno reševali z uporabo optimizacijskih treh zagonov algoritma lq-CMA-ES. algoritmov, smo vrednotenje rešitev razdelili na pet korakov, s katerimi želimo čim prej izločiti nedopustne rešitve, da na njih Rezultati algoritma lq-CMA-ES ne tratimo časa. Poleg tega smo optimizacijski postopek pohitrili z uporabo algoritma s hitro izračunljivimi nadomestnimi modeli Zagon algoritma ter paralelizacijo in poenostavitvijo računsko najzahtevnejšega Cena prvotnega na rta 12 koraka vrednotenja rešitev. 3 Predlagani postopek smo preizkusili na konkretnem primeru elektromotorja za servovolanske sisteme, za katerega smo želeli minimizirati ceno in obenem zagotoviti, da bo zadoščal vsem omotorja tehničnim zahtevam. Primerjava tako dobljenih načrtov z najbolj- šim, ki so ga v podjetju prvotno našli s pomočjo enostavnejšega elena cena optimizacijskega postopka, je pokazala, da dosežemo dopustne Cena elektr rešitve, ki so cenovno za 10 % ugodnejše od obstoječih. Ob dejstvu, da se v celotnem obdobju proizvodnje takšnega izdelka proizvede več milijonov kosov, to za podjetje predstavlja bistven prihranek in močno izboljšuje njegovo konkurenčnost na trgu. V prihodnje želimo implementacijo postopka razširiti tako, 0 1000 2000 3000 4000 5000 tevilo re itev da bo omogočala enostavno izbiro optimizacijskega kriterija in omejitev ter dodajanje skript za preverjanje dopustnosti rešitev. Cilj je izdelati računalniško orodje, ki ga bodo inženirji lahko Slika 4: Primerjava najnižje cene elektromotorja tekom samostojno uporabljali pri optimizaciji raznovrstnih elektromo- treh zagonov algoritma lq-CMA-ES s ceno prvotnega opti- torjev brez poseganja v sam postopek ali optimizacijski algoritem. miranega načrta in želeno ceno. (Dejanske cene niso nave- dene zaradi varovanja poslovne skrivnosti.) ZAHVALA Zahvaljujemo se podjetju MAHLE Electric Drives Slovenija d.o.o. Zaradi časovnih omejitev smo uspeli izvesti samo tri zagone za financiranje projekta razvoja elektromotorja in sodelovanje algoritma lq-CMA-ES, ki se razlikujejo v semenu generatorja na- pri njegovi izvedbi. Naše temeljne raziskave evolucijskega raču- ključnih števil. Njihovi rezultati so prikazani na slikah 3 in 4. Obe nanja in večkriterijske optimizacije sofinancira Javna agencija kažeta, kako se opazovana veličina zmanjšuje s časom (številom za znanstvenoraziskovalno in inovacijsko dejavnost Republike pregledanih rešitev – štejemo vsako vrednotenje rešitev, tudi če Slovenije z raziskovalnima programoma št. P2-0098 in P2-0209 se je končalo že s prvim korakom). Na sliki 3 vidimo zmanjševater projekti št. J2-4460, N2-0239 in N2-0254. nje optimizacijskega kriterija (vrednosti pod 100 pomenijo, da je algoritem našel dopustne rešitve), na sliki 4 pa pripadajoče (do LITERATURA takrat najboljše) cene elektromotorjev. [1] Ansys, Inc. 2023. Ansys Maxwell. https://www.ansys.com/products/electroni Vidimo, da so rezultati treh zagonov algoritma precej raznoliki. cs/ansys- maxwell. Dostopano 15. 8. 2023. (2023). [2] Klaus-Jürgen Bathe. 2007. Finite element method. Wiley encyclopedia of comPri tako zahtevnem problemu in majhnem številu vrednotenj je puter science and engineering, 1–12. bilo to pričakovano. Vseeno pa za vse tri zagone velja, da potre- [3] Nikolaus Hansen. 2019. A global surrogate assisted CMA-ES. V Proceedings of bujejo manj kot 1000 pregledanih rešitev, da najdejo dopustne the 2019 Genetic and Evolutionary Computation Conference. ACM, New York, NY, USA, 664–672. doi: 10.1145/3321707.3321842. rešitve in hkrati izboljšajo najboljši načrt, najden s prvotnim [4] Nikolaus Hansen, Youhei Akimoto in Petr Baudis. 2019. CMA-ES/pycma on optimizacijskim postopkom v podjetju. V vseh treh primerih je Github. (Feb. 2019). doi: 10.5281/zenodo.2559634. potrebnih manj kot 1500 vrednotenj, da algoritem najde rešitev [5] Nikolaus Hansen in Andreas Ostermeier. 1996. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adap-s ceno, ki je boljša od želene. Po približno 2500 vrednotenjih se tation. V Proceedings of 1996 IEEE International Conference on Evolutionary cena elektromotorja neha bistveno izboljševati. Computation. IEEE, 312–317. doi: 10.1109/ICEC.1996.542381. 51 52 Indeks avtorjev / Author index Andova Andrejaana ...................................................................................................................................................................... 44 Antešić Ivan ................................................................................................................................................................................. 40 Arbetman Marina ......................................................................................................................................................................... 20 Bohanec Marko ............................................................................................................................................................................ 36 Bolliger Larissa ............................................................................................................................................................................ 11 Božič Janko .................................................................................................................................................................................. 24 Campopiano Robinson Victoria ................................................................................................................................................... 20 Clays Els....................................................................................................................................................................................... 11 Cork Jordan .................................................................................................................................................................................. 44 Filipič Bogdan ........................................................................................................................................................................ 44, 48 Galen Candace ............................................................................................................................................................................. 20 Gams Matjaž ...................................................................................................................................................................... 7, 28, 32 Gradišek Anton ............................................................................................................................................................................ 20 Guček Marko ................................................................................................................................................................................ 36 Herrera Valentina ......................................................................................................................................................................... 20 Hirose Akira ................................................................................................................................................................................. 15 Jordan Marko ............................................................................................................................................................................... 19 Kawai Kitoshi ............................................................................................................................................................................... 15 Kocuvan Primož ........................................................................................................................................................................... 28 Kolar Žiga .................................................................................................................................................................................... 32 Kontić Davor ................................................................................................................................................................................ 36 Korošec Peter ............................................................................................................................................................................... 48 Lukan Junoš ................................................................................................................................................................................. 11 Lukšič Andrej A. .......................................................................................................................................................................... 32 Luštrek Mitja .................................................................................................................................................................... 11, 15, 19 Martinšek Marcel Franse .............................................................................................................................................................. 11 Rotar Oskar .................................................................................................................................................................................. 24 Sadikov Aleksandar...................................................................................................................................................................... 40 Schul Johannes ............................................................................................................................................................................. 20 Silan Darja .................................................................................................................................................................................... 24 Sirk Karina ................................................................................................................................................................................... 36 Šiško Primož ................................................................................................................................................................................ 11 Šket Tilen ..................................................................................................................................................................................... 20 Slapničar Gašper .......................................................................................................................................................................... 15 Susič David .............................................................................................................................................................................. 7, 20 Tušar Tea ................................................................................................................................................................................ 44, 48 Valič Jakob ................................................................................................................................................................................... 19 Vesel Tian .................................................................................................................................................................................... 24 Villagra Gil Cristian Alfonso ....................................................................................................................................................... 20 Vodopija Aljoša ........................................................................................................................................................................... 44 Vozlič Andrej A. .......................................................................................................................................................................... 32 Zadobovšek Matic ........................................................................................................................................................................ 28 Ženko Bernard .............................................................................................................................................................................. 36 Žnidaršič Maks ............................................................................................................................................................................. 24 Žnidaršič Martin ........................................................................................................................................................................... 36 53 54 Slovenska konferenca o umetni inteligenci Slovenian Conference on Artificial Intelligence Uredniki • Editors: Mitja Lustrek, Matjaz Gams, Rok Piltaver Document Outline 02 - Naslovnica - notranja - A - TEMP 03 - Kolofon - A - TEMP 04 - IS2023 - Predgovor 05 - IS2023 - Konferencni odbori 07 - Kazalo - A 08 - Naslovnica - notranja - A - TEMP 09 - Predgovor podkonference - A 10 - Programski odbor podkonference - A 11 - Prispevki - A 01_IS_2023_-_SCAI_paper_377 Abstract 1 Introduction 2 Dataset 3 Methodology 3.1 Preprocessing 3.2 Convolutional Neural Network 3.3 PCG Cutmix Augmentation 3.4 Hyperparameter Finetuning 4 Results 5 Discussion and Conclusions Acknowledgments 02_IS_2023_-_SCAI_paper_3966 Abstract 1 Introduction 2 Data Collection 3 Target and Feature Extraction 3.1 Label Extraction 3.2 Features 4 Wi-Fi Localization 4.1 Rough GPS location 4.2 Indoor Wi-Fi location 4.3 Clustering 5 Interaction Classification 6 Conclusions Acknowledgements 03_IS_2023_-_SCAI_paper_9873 Abstract 1 Introduction 2 Related work 3 Methodology 3.1 Baseline Signal Processing 3.2 Advanced Signal Processing 3.3 PPG Waveform Quality Assessment 4 Experiments 4.1 Recording Setup 4.2 Evaluation Pipeline 5 Results 6 Conclusion Acknowledgements 04_IS_2023_-_SCAI_paper_4981_abstract 05_IS_2023_-_SCAI_paper_3282 Abstract 1 Introduction 2 Dataset 3 Methodology 3.1 Data Preprocessing 3.2 Data Partitioning 3.3 Neural Network Architecture 3.4 Model Training Settings 4 Results 5 Discussion and Conclusions Acknowledgements 06_IS_2023_-_SCAI_paper_8213 Povzetek 1 Uvod 2 Metodologija 2.1 Izbira modela zaznavanja objektov 2.2 Obdelava video posnetkov 2.3 Učenje modelov 2.4 Sledenje gibanju čebel 2.5 Analiza modelov 2.6 Analiza metod sledenja premikanja čebel 3 Rezultati 3.1 Hitrost izvajanja modelov 3.2 Uspešnost detekcije modelov 3.3 Rezultati sledenja 4 Diskusija 5 Zaključek Zahvala 07_IS_2023_-_SCAI_paper_7211 Povzetek 1 Uvod 2 ChatGPT 3 Insieme 4 Virtualni asistent za medicino 4.1 Ozadje 4.2 Besedne vložitve in vektorske podatkovne baze 4.3 Implementacija 5 Zaključek Zahvala 08_IS_2023_-_SCAI_paper_425 Abstract 1 Introduction 2 Democracy types 2.1 Participative Democracy 2.2 Deliberative Democracy 2.3 Transdemocracy 2.4 Guided Democracy 2.5 Modern Direct Democracy 2.6 E-democracy 2.7 Representative democracy 2.8 Liquid democracy 2.9 Blockchain democracy 2.10 Source democracy 2.11 Ideal typical democracy 3 Methodology 4 Results 5 Conclusion Acknowledgements 09_IS_2023_-_SCAI_paper_1853 Abstract 1 Introduction 2 Related Work 3 Study Design 4 Results 4.1 Traffic Data Analysis 4.2 User Survey Analysis 4.3 Comparison of Traveling Times 4.4 Analysis of Environmental Burdens 5 Conclusions Acknowledgments 10_IS_2023_-_SCAI_paper_6707 Povzetek 1 Uvod in motivacija 2 Metode 2.1 OpenRA 2.2 LRA* 2.3 Osnovni WHCA* 2.4 Prilagojeni WHCA* 3 Eksperimenti 4 Diskusija 5 Povzetek 11_IS_2023_-_SCAI_paper_1347 Abstract 1 Introduction 2 Theoretical Background 3 ELA Features for Constrained Multiobjective Optimization Problems 4 Empirical Cumulative Distribution Functions 5 Experimental evaluation 6 Results 7 Conclusion Acknowledgements 12_IS_2023_-_SCAI_paper_1585 Povzetek 1 Uvod 2 Načrtovanje elektromotorja 3 Optimizacijski postopek 4 Vrednotenje posameznega načrta elektromotorja 5 Numerični poskusi in rezultati 6 Zaključki Zahvala 12 - Index - A Blank Page Blank Page Blank Page Blank Page Blank Page 03_IS_2023_-_SCAI_paper_9873 - POPRAVEK.pdf Abstract 1 Introduction 2 Related work 3 Methodology 3.1 Baseline Signal Processing 3.2 Advanced Signal Processing 3.3 PPG Waveform Quality Assessment 4 Experiments 4.1 Recording Setup 4.2 Evaluation Pipeline 5 Results 6 Conclusion Acknowledgements Blank Page