37th Bled eConference Resilience Through Digital Innovation: Enabling the Twin Transition June 9 – 12, 2024, Bled, Slovenia Conference Proceedings Editors Andreja Pucihar Mirjana Kljajić Borštnar Staša Blatnik Roger W. H. Bons Koen Smit Marikka Heikkilä May 2024 Title 37th Bled eConference – Resilience Through Digital Innovation: Enabling Naslov the Twin Transition Subtitle Podnaslov June 9 – 12, 2024, Bled, Slovenia, Conference Proceedings Editors Andreja Pucihar Uredniki (University of Maribor, Faculty of Organizational Sciences, Slovenia) Mirjana Kljajić Borštnar (University of Maribor, Faculty of Organizational Sciences, Slovenia) Staša Blatnik (University of Maribor, Faculty of Organizational Sciences, Slovenia) Roger W. H. Bons (Open University; Heerlen; Netherland) Koen Smit (HU University of Applied Sciences Utrecht, Utrecth, Netherland) Marikka Heikkilä (University of Turku, Turku, Finland) Review Sasan Adibi (La Trobe University), Tashfeen Ahmad (The University of the West Recenzija Indies), Satu Aksovaara (Jamk University of Applied Sciences), Art Alishani (Tartu University), Joschka Andreas Hül mann (University of Twente), Alenka Baggia (University of Maribor), Matthias Baldauf (Eastern Switzerland University of Applied Sciences, Campus St. Gallen), David Baxter (University of Southampton), Staša Blatnik (University of Maribor), Morten Boesen (KEA - Copenhagen School of Design and Technology), Roger Bons (Open University of the Netherlands), Christer Carlsson (Institute for Advanced Management Systems Research and Åbo Akademi University), César Casiano Flores (University of Twente), Helen Cripps (Edith Cowan University), Lukas Daniel Klausner (St. Pölten University of Applied Sciences), Mitja Dečman (University of Ljubljana), Stefan Dedovic (Tartu University), Aidan Duane (South East Technological University), Kristel Edelman (University of Turku), Taina Eriksson (University of Turku), Riikka Franzén (University of Turku), Leonid Freidovich (Umeå University), Christian Geiger (Bern University of Applied Sciences), Matt Glowatz (University College Dublin), Torsten Gollhardt (University of Muenster), Linda Grogorick (Technische Universität Braunschweig), Thu Ha Dang (Swinburne University of Technology), Nerko Hadziarapovic (HU University of Applied Sciences Utrecht), Raija Halonen (University of Oulu), Nathalie Haug (Universität Konstanz), Marikka Heikkilä (University of Turku), Andreas Hermann (University of Muenster), Ralf Höchenberger (Universität Ulm), Matilda Holkkola (University of Jyväskylä), Irene Jonkers (HU University of Applied Sciences Utrecht), Tina Jukić (University of Ljubljana), Fuad Khan (University of Turku), Mirjana Kljajic Borstnar (University of Maribor), Eva Krhač Andrašec (University of Maribor), Pavlina Kröckel (Friedrich-Alexander Universität Erlangen-Nürnberg), Paul Kruse (University of Muenster), Rob Kusters (Open University of the Netherlands), Sam Leewis (HU University of Applied Sciences Utrecht), Gregor Lenart (University of Maribor), Yin Leng Tan (University of Reading), Xander Lub (Hogeschool Utrecht), Pasi Luukka (LUT University), Mahinda Mailagaha Kumbure (LUT University), Melanie Mandl (BANDAS Center University of Graz), Venkata Marel a (Open University of the Netherlands), Marjeta Marolt (University of Maribor), Tilen Medved (University of Maribor), Thomas Mejtoft (Umeå University), Axel Minten (German Coworking Association and FOM University of Applied Sciences), Ondrej Mitas (Breda University of Applied Sciences), Domen Mongus (University of Maribor), Rasim Mujalović (JadroAgent-Bar), Pradeep Murukannaiah (Delft University), Anastasija Nikiforova (Tartu University), Ayesha Nilashini (La Trobe University), Omid Noroozi (Wageningen University and Research), Guido Ongena (HU University of Applied Sciences Utrecht), Tiina Paananen (University of Jyväskylä), Zdravko Paladin (Administration for Maritime Safety and Port Management), Mrityunjoy Panday (Cognizant), Gabe Piccoli (Louisiana State University and University of Pavia), Marijn Plomp (Vrije Universiteit Amsterdam), Andriana Prentza (University of Piraeus), Andreja Pucihar (University of Maribor), Uroš Rajkovič (University of Maribor), Pascal Ravesteijn (HU University of Applied Sciences Utrecht), Christopher Reichstein (Duale Hochschule Baden-Württemberg), Joaquin Rodriguez (Grenoble Ecole de Management), Carsten Schmidt (Tartu University), Juergen Seitz (Baden-Wuerttemberg Cooperative State University), Anna Sel (Åbo Akademi University), Navin Sewberath Misser (HU University of Applied Sciences Utrecht), Anand Sheombar (HU University of Applied Sciences Utrecht), Muhammad Shuakat (Deakin University), Luuk Simons (Delft University), Erik Slingerland (HU University of Applied Sciences Utrecht), Koen Smit (HU University of Applied Sciences Utrecht), Ulrik Söderström (Umeå University), Ronald Spanjers (Roessingh Rehabilitation), Zachary Spicer (York University), Lucy Temple (University for Continuing Education Krems), Nevena Toporova (Technical University of Munich), Nalika Ulapane (La Trobe University), Annemae van de Hoef (HU University of Applied Sciences Utrecht), Esther van der Stappen (Avans University of Applied Sciences), John van Meerten (HU University of Applied Sciences Utrecht),mMathijs van Dijk (HU University of Applied Sciences Utrecht), Rogier Van de Wetering (Open University of the Netherlands), Stan van Ginkel (HU University of Applied Sciences Utrecht), Willemijn Van Haeften (HU University of Applied Sciences Utrecht), Johan Versendaal (HU University of Applied Sciences Utrecht and Open University of the Netherlands), Gabriela Viale Pereira (University for Continuing Education Krems), Doroteja Vidmar (RLS, Slovenia), Mirko Vintar (University of Ljubljana), Doug Vogel (Harbin Institute of Technology), Alex Wahlroos (Umeå University), Xiaolu Wang (Åbo Akademi University), Valtteri Wanne (Azets, Åbo Akademi University), Thorsten Weber (FOM Hochschule), Nilmini Wickramasinghe (Swinburne University of Technology & Epworth HealthCare), Gary Wills (University of Southampton), Varinia Wittholz (Technische Universität Braunschweig), Adrian Woick (AKAD University), Rio Yusri Maulana (Department of Government Science, Universitas Jambi), Zaid Zekiria Sako (La Trobe University), John Zelcer (La Trobe University), Hans-Dieter Zimmermann (Eastern Switzerland University of Applied Sciences, Campus St. Gallen), Manuel Zwicker (RMIT University) Technical editors Aljaž Murko Tehnična urednika (University of Maribor, Faculty of Organizational Sciences) Jan Perša (University of Maribor, University Press) Cover designer Jan Perša Oblikovanje ovitka (University of Maribor, University Press) Graphics material Sources are own unless otherwise noted. Grafične priloge The authors and Pucihar, Borštnar Kljajić, Blatnik, Bons, Smit, Heikkila (editors), 2024 Conference 37th Bled eConference – Resilience Through Digital Innovation: Enabling the Konferenca Twin Transition Location and date Kraj in datum Bled, Slovenia, June 9 – 12, 2024 Programme committee Art Alishani (Tartu University), Matthias Baldauf (Eastern Switzerland University Programski odbor of Applied Sciences, Campus St. Gallen), Roger Bons (Open University of the Netherlands), Christer Carlsson (IAMSR/Abo Akademi University), Helen Cripps (Edith Cowan University), Matt Glowatz (University College Dublin), Nerko Hadziarapovic (HU University of Applied Sciences Utrecht), Marikka Heikkilä (University of Turku), Tina Jukić (University of Ljubljana), Mirjana Kljajić Borštnar (University of Maribor), Gregor Lenart (University of Maribor), Xander Lub (HU University of Applied Sciences Utrecht), Pasi Luukka (LUT School of Business and Management), Venkata Marella (Open University of the Netherlands), Marjeta Marolt (University of Maribor), Axel Minten (German Coworking Association and FOM University of Applied Sciences), Domen Mongus (University of Maribor), Guido Ongena (HU University of Applied Sciences Utrecht), Marijn Plomp (Vrije Universiteit Amsterdam), Andreja Pucihar (University of Maribor), Uroš Rajkovič (University of Maribor), Pascal Ravesteijn (HU University of Applied Sciences Utrecht), Juergen Seitz (Baden- Wuerttemberg Cooperative State University), Anand Sheombar (HU University of Applied Sciences Utrecht), Koen Smit (HU University of Applied Sciences Utrecht), Stan van Ginkel (HU University of Applied Sciences Utrecht), Esther van der Stappen (Avans University of Applied Sciences), Johan Versendaal (HU University of Applied Sciences Utrecht), Doroteja Vidmar (RLS, Slovenia), Doug Vogel (Harbin Institute of Technology), Nilmini Wickramasinghe (La Trobe University), Hans-Dieter Zimmermann (Eastern Switzerland University of Applied Sciences, Campus St. Gallen) Organizing committee Staša Blatnik (University of Maribor, Faculty of Organizational Sciences), Petra Organiazacijski odbor Gorjanc (University of Maribor, Faculty of Organizational Sciences), Mirjana Kljajić Borštnar (University of Maribor, Faculty of Organizational Sciences), Klara Knific (University of Maribor, Faculty of Organizational Sciences), Maruša Luštrik (University of Maribor, Faculty of Organizational Sciences), Aljaž Murko (University of Maribor, Faculty of Organizational Sciences), Andreja Pucihar (University of Maribor, Faculty of Organizational Sciences) Published by University of Maribor Založnik University Press Slomškov trg 15, 2000 Maribor, Slovenia https://press.um.si, zalozba@um.si Co-published by University of Maribor Izdajatelj Faculty of Organizational Sciences Kidričeva cesta 55a, 4000 Kranj, Slovenia http://www.fov.um.si, dekanat.fov@um.si Edition Izdaja 1ST Publication type Vrsta publikacija E-book Available at Dostopno na https://press.um.si/index.php/ump/catalog/book/860 Published Izdano Maribor, May 2024 Text / Besedilo © The authors & Pucihar, © University of Maribor, University Press Borštnar Kljajić, Blatnik, /Univerza v Mariboru, Univerzitetna založba Bons, Smit, Heikkila (editors), 2024 This book is published under a Creative Commons 4.0 International licence (CC BY 4.0). This license lets others remix, tweak, and build upon your work even for commercial purposes, as long as they credit you and license their new creations under the identical terms. This license is often compared to “copyleft” free and open source software licenses. Any third-party material in this book is published under the book’s Creative Commons licence unless indicated otherwise in the credit line to the material. If you would like to reuse any third-party material not covered by the book’s Creative Commons licence, you wil need to obtain permission directly from the copyright holder. https://creativecommons.org/licenses/by /4.0/ CIP - Kataložni zapis o publikaciji Univerzitetna knjižnica Maribor 005:004(082)(0.034.2) BLED eConference- Resilience Through Digital Innovation: Enabling the Twin Transition (37 ; 2024 ; Bled) 37th Bled eConference - Resilience Through Digital Innovation: Enabling the Twin Transition [Elektronski vir] : June 9 - 12. 2024, Bled, Slovenia : conference proceedings / editors Andreja Pucihar ... [et al.]. - 1st ed. - E-publikacija. - Maribor : University of Maribor, University Press, 2024 Način dostopa (URL): https://press.um.si/index.php/ump/catalog/book/860 ISBN 978-961-286-871-0 (PDF) doi: 10.18690/um.fov.4.2024 COBISS.SI-ID 196387075 ISBN 978-961-286-871-0 (pdf) DOI https://doi.org/10.18690/um.fov.4.2024 Price Cena Free copy For publisher Prof. Dr. Zdravko Kačič, Odgovorna oseba založnika Rector of University of Maribor Attribution Pucihar, A., Borštnar Kljajić, M., Blatnik, S., Bons, R. W. H., Smit, Citiranje K., Heikkilä, M. (eds.) (2024). 37th Bled eConference Resilience Through Digital Innovation: Enabling the Twin Transition: June 9 – 12, 2024, Bled, Slovenia, Conference Proceedings. University of Maribor, University Press. doi: 1 0.18690/um.fov.4.2024 37TH BLED ECONFERENCE RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION JUNE 9 – 12, 2024, BLED, SLOVENIA, CONFERENCE PROCEEDINGS A. Pucihar et al. (ed.) Table of Contents CONFERENCE PROCEEDINGS A Study to Measure the Potential Impact of Generative Artificial 1 Intelligence on Academic Integrity 1 Aidan Duane Digital Asset Creation and Orchestration: Empirical Evidence 2 from Industrial Manufacturing 17 Gabriele Piccoli, Joaquin Rodriguez Smart Governance Chal enges in Indonesian Local Government 3 Rio Yusri Maulana, Usman Marjamat, Dimas Subekti, Moh.Arief 33 Rakhman, Alva Beriansyah Metaverse in Higher Education – A Systematic Literature 4 Review Adrian Woick, Heidi Rinn, Linda Grogorick, Tamara Mühleisen, 57 Daniel Markgraf Higher Education Student's Self-Efficacy Beliefs During and 5 Post Pandemic: an Explorative Learning Analytics Study 77 Satu Aksovaara, Tommi Kärkkäinen Investigating the Benefits of 5g to Leverage the Digital 6 Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 89 Ayesha Nilashini, Nilmini wickramasinghe Data Innovation Explorer: Design of a Data-Driven Business 7 Model Tool Joschka Andreas Hül mann, Torsten Gol hardt, Andreas Hermann, 113 Paul kruse Unmasking Biases and Mapping the Landscape of AI Adoption 8 in Diverse Business Functions Through a Delphi Study Amogha Ammava Sudarshan, Isabel a Seeber 135 ii Table of Contents. Assessment Instrument of Technology Acceptance Amongst 9 People with Minor Intellectual Disabilities 161 Mathijs Petrus Antonius van Dijk, Arjen Maris Cultivating a Data-Driven Culture: Exploring The Role of Data 10 Analytic Capability, Employee Empowerment, and Transformational Leadership 193 Alinda Kokkinou, Ondrej Mitas, Albert Mandemakers A Maturity Model for Evaluating Data-Driven Sustainability Management 11 Anna Sel , Jerker Björkqvist, Janne Hakala, Fredrik Hellman, Xiang 211 Hu, Oskar Karlström, Anders Lindgren, Andreas Lundel , Amaia Martin-Granizo Improving User's Confidence to Act when Using Advice 12 Algorithms Through Interactive Use of Counterfactuals Jeroen van Grondel e, Marlies van Steenbergen, Aletta Smits, Marcel 227 Stalenhoef, Koen van Turnhout, Tijs Timmerman, Harald Pol Empowering Data Sovereignty: Strategies of Data 13 Intermediaries in Data Ecosystems 243 Ruben D’Hauwers, August Bourgeus Analysis of Development and User Aspects of ePrescription in 14 Slovenia 259 Živa Rant, Dalibor Stanimirović Digital Public Service Improvement in Cross-Border Use Cases 15 Helen Raamat, Innar Liiv, Silvia Lips, Rozha Kamal Ahmed, Rahul 269 Sharma, Dirk Draheim Designing and Evaluating an LLM-based Health AI Research 16 Assistant for Hypertension Self-Management; Using Health Claims Metadata Criteria 283 Luuk P.A. Simons, Pradeep K. Murukannaiah, Mark A. Neerincx Fuzzy Set Qualitative Comparative Analysis as a Tool for 17 Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 299 Mahinda Mailagaha Kumbure, Pasi Luukka Digital Twin Technology in the Rail Industry: A Dutch 18 Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 319 Koen Smit, Joris Mens, Chaim de Gelder, John van Meerten Table of Contents iii. Responsible Implementation of AI in Higher Education: 19 Ethical Factors Guiding Dutch IT Teachers John van Meerten, Koen Smit, Marlies van Steenbergen, Matthijs 337 Berkhout Ethnical Anthropomorphism in Human-Robot Interaction: Personalized Robot Tutors 20 Annemae van de Hoef, Sam Leewis, Koen Smit, Łukasz Miętkiewicz, 357 Emirhan Tala, Fábio Seiji Sakai Iwashima, Berk Tahir Kılıç, Mirjam de Haas 21 Digital Sustainability Practices: A Research Agenda Irene R. Jonkers, Xiao Peng, Wil emijn van Haeften 369 Bridging the AI Skil s Gap in Europe: A Detailed Analysis of AI 22 Skills and Roles Wil emijn van Haeften, Ran Zhang, Sabine Boesen - Mariani, Xander 385 Lub, Pascal Ravesteijn, Paul Aertsen Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: 23 ePIcenter Projects' Case 403 Zdravko Paladin, Mira Šorović, Sanja Bauk, Frank Knoors, Julian Stephens, Nexhat Kapidani, Žarko Lukšić, Rasim Mujalović Enabling Cardiac Rehabilitation at Home: A Web-Based 24 Solution 421 Nilmini Wickramasinghe, Nalika Ulapane, Sasan Adibi Decision Analytics—Less Expected Motivators for HEPA 25 Programs 435 Christer Carlsson Balancing Personalization and Privacy: Towards Personalized 26 Saving Experience in Banking Apps for Young Adults Alex Wahlroos, Leonid Freidovich, Ulrik Söderström, Thomas 453 Mejtoft Decision Mining in the Rail Industry: A Case Study in the 27 Context of an Industrial Wheelset Revision Process 463 Bart Hoornstra, Joris Mens, Iris Beerepoot, Inge van de Weerd Towards Social Digital Twins – An Integrated Sociotechnical 28 Approach for the Urban Energy Transition Steven P. Haveman, Xiao Peng, Ran Zhang, Tadeo-Baldiri Salcedo- 479 Rahola iv Table of Contents. The Factors That Influence the Adoption of Tools That 29 Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 497 Rene Wieringa, Paul Morsch, Pascal Ravesteijn, Wouter Bronsgeest Smart Cities on the Waterfront: Citizens’ Perspectives on Smart 30 City in the Context of a Port City Kristel Edelman, Riikka Franzen, Irina Wahlström, Anastasia 515 Tsvetkova, Yiran Chen Relaxing or Exciting? Mobile Emotion Recording in Natural 31 Settings Ondrej Mitas, Ben Wielenga, Kees Evers, Jelena Farkić, Peter Ward, 531 Jeroen Klijs, Marcel Bastiaansen It’s a Match! – Finding the Perfect Participants for Health 32 Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 547 Melanie Mandl, Christine Malin, Stefan Thalmann Evaluating Covid-19 Movement Restrictions: A Finnish Case 33 Study 563 Valtteri Wanne, Xiaolu Wang, Anna Sel Towards Techno-Psychological Immersion: A Narrative 34 Literature Review of Immersion and Its Related Concepts Ti na Paananen, Ti na Kemppainen, Markus Makkonen, Matilda 577 Holkkola, Olli Tyrväinen, Lauri Frank The Drivers of Showrooming Behavior: A Meta-Analysis 35 Matilda Holkkola, Olli Tyrväinen, Markus Makkonen, Heikki 597 Karjaluoto, Tiina Kemppainen, Tiina Paananen, Lauri Frank Exploring Sustainable Value Propositions in Logistics Services 36 – How Digitalization and Data Support Greener Services? 615 Fuad Khan, Taina Eriksson, Marikka Heikkilä, Titiana Ertiö European Companies' Readiness and Chal enges in Data 37 Economy 635 Marikka Heikkilä, Jukka Heikkilä, Farhan Ahmad 38 Customer Perceptions of In-Store Identification Ti na Kemppainen, Lauri Frank 655 Table of Contents v. Generative AI in Assisting Patients with Syringomyelia and 39 Hydrocephalus: A Preliminary Comparison of Chatgpt and Gemini 671 Pavlina Kröckel Repeat After Me: Dynamic Workouts for The Wellbeing of 40 Elderly Individuals with Dementia Annemae van de Hoef, Sam Leewis, Koen Smit, Mark Slingerland, 679 Larissa Kaiser, Katari na Saukko, Ruben van Gend, Mirjam de Haas The Use of Neutralisation Techniques in the Context of 41 Responsible Online Shopping: A Latent Profile Analysis 693 Markus Makkonen, Lauri Frank, Matilda Holkkola, Ti na Paananen Mapping the Landscape of Digital Health Usage in Information 42 Systems Research 711 Sara Arzideh, Jozsef Mezei Preparing for Disease X: Predicting ICU Admissions Using 43 Time Series Forecasting with Decoder-Only Transformer Neural Networks 729 Nejc Čelik, Andrej Škraba RESEARCH IN PROGRESS Strengthening Resilience in a Danish Health System – A 44 Longitudinal Study 745 Jarl Tuxen, Morten Boesen, Erik Leth Møller Unravelling the Use of Digital Twins to Assist Decision- and 45 Policy-Making in Smart Cities 755 Lucy Temple, Gabriela Viale Pereira, Lukas Daniel Klausner Overview of Asthma Related Smartphone Applications on 46 Google Play and Apple App Stores 765 Zaid Zekiria Sako, Nilmini Wickramasinghe A Value Sensitive Design Approach to Equitable Remuneration 47 of the Dutch Music Copyright System in the Digital Age 775 Nerko Hadžiarapović, Marlies van Steenbergen, Pascal Ravesteijn vi Table of Contents. DOCTORAL CONSORTIUM Designing a Process Model of Integrated Lifelong Provision of 48 Healthcare to Patients 783 Živa Rant, Tomaž Kern Measuring Smart Public Governance Maturity in Public 49 Administration Institutions: A Multi-Attribute Approach 799 Petra Vujković Developing Public Values Based AI Systems Using Value 50 Sensitive Design 831 Erik Slingerland, Guido Ongena, Marlies van Steenbergen How are Companies Strategizing with Digital Transformation 51 to Integrate Sustainability Objectives? 841 Fuad Khan Enhancing Process Model Visualisation to Facilitate the 52 Understanding of Stakeholders 847 Iris Mulder Methodology Development for Open Data Maturity Assessment 53 in Smal and Medium-Sized Enterprises – A Literature Review 859 Staša Blatnik Comparative Analysis of the Personal Bankruptcy Model and 54 Their Effects in the Legal Orders in Slovenia and Croatia 881 Rea Šaina Table of Contents vii. A STUDY TO MEASURE THE POTENTIAL IMPACT OF GENERATIVE ARTIFICIAL INTELLIGENCE ON ACADEMIC INTEGRITY AIDAN DUANE South East Technological University (SETU), Waterford City, Ireland aidan.duane@setu.ie Platform business models like Uber Ride or Airbnb Lodging enable innovative business models by operating digital platforms to connect providers and consumers of products and services in two-sided markets. A particular challenge with platform business models is designing an appropriate revenue model to capture value. This paper presents a taxonomy that classifies the different dimensions and characteristics of revenue models for platform business models. A proven taxonomy development method is used that includes a review of current literature related to Keywords: platform business models. The taxonomy provides a generative comprehensive classification of platform revenue models and is artificial intelligence, applied to a real-life case. The results of this paper include a UML GenAI, class model and a final taxonomy with 14 dimensions and 64 ChatGPT, characteristics. The paper contributes to the design process of academic integrity, novel platform business models and expands the understanding university of how digital platforms can generate revenues. assessment DOI https://doi.org/10.18690/um.fov.4.2024.1 ISBN 978-961-286-871-0 2 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Artificial Intelligence (AI) is now firmly rooted in our daily lives. It is in our intel igent home devices, healthcare diagnostic tools that scan our bodies, and the cars we drive or pass on the street. AI has enhanced the way we live, and it is doing the same where we work and study. Increasingly, organizations are adopting AI and using AI platforms and other technologies to develop, deploy, and maintain AI- powered products and services at enterprise scale (Anaconda, 2023). According to McKinsey (2023) the rate of enterprise AI adoption has exponential y increased, and one-third of organisations surveyed disclose using Generative Artificial Intelligence (GenAI) regularly in at least one business function. A form of GenAI known as ChatGPT developed by OpenAI, and released publicly in November 2022, has become the fastest growing consumer application in history (Marr, 2023). Since it’s release, the use of GenAI has rapidly expanded in the field of education, as additional tools have emerged including DALL-e, Midjourney, Claude, Jasper, Duet AI, Bearly AI, Gemini, Synthesia, FontyAI, Jenni, Quillbot, and a multitude of what are known as co-pilots integrated with existing software. Microsoft (2024) describe its co-pilot as “your everyday AI companion”. The ability of GenAI to perform complex tasks (Janakiram, 2022; Liu et al., 2023; McKinsey, 2023) in the field of education has caused mixed feelings among educators (García-Peñalvo, 2023; Halaweh, 2023; Hearn, 2022; Kohnke et al., 2023; Malinka et al., 2023; Rudolph and Tan, 2023; Sawahel, 2023). GenAI is capable of creating original essays, ideation, musical compositions, graphics, videos, slides, and software code, while also producing detailed quantitative and qualitative analysis. Thus, integrating GenAI with education raises concerns about assessment and evaluation, as traditional methods may become obsolete in the face of AI-generated solutions (Fowler, 2023; García-Peñalvo, 2023; Hockly, 2023; Lancaster, 2021; OpenAI 2023a). The most significant concern for GenAI in education, is its implications for academic integrity (Firat, 2023a & 2023b; Fowler, 2023; Hearn, 2022; Lancaster, 2021; Malinka et al., 2023; Roe and Perkins, 2022). In this study, students applied GenAI to complete past assessments, adapted as research tests, with the goal of achieving a pass grade when graded by a university academic, undetected by GenAI writing detection tools. The formal objective of this A. Duane: A Study to Measure the Potential Impact of Generative Artificial Intel igence on Academic Integrity 3 study is “to empirically test the impact of generative artificial intelligence on the academic integrity of university assessments”. Two research hypotheses were developed to test the research objective: • Hypothesis 1: Students can complete university assessments using GenAI undetected by AI writing detection tools. • Hypothesis 2:Students can achieve a passing grade (>=40%) for university assessment submissions generated using AI when graded by a university academic. The study was undertaken in the South East Technological University (SETU) in Ireland, between April and September 2023. This paper describes the design, deployment and results of the study. 2 Academic Integrity Many definitions and descriptions of academic integrity exist in the literature and there are as many disparities as there are correlations. Arguably, academic integrity and compliance with such, is contextual, national (where national agreement exists), and essential y localised dependent on integration of national agreements into university governance policies. Launched in November 2019, the National Academic Integrity Network (NAIN) is a peer-driven network of academic and professional staff, student representatives and representative agencies from across the higher education landscape in Ireland. NAIN (2021) supports Irish higher education institutions in developing a common, national understanding of academic integrity and fostering a culture of academic integrity through enhancement activities and sharing of good practice. The South East Technological University (SETU), where the researcher works and the research was conducted, has ful y adopted NAIN definitions and standards of academic integrity into its governance policies. This study adopts the definition and principles of academic integrity established by NAIN (2021) as “the commitment to, and demonstration of, honest and moral behaviour in an academic setting”. It requires that all interactions with higher education institutions in Ireland are approached with honesty including al documentation submitted to the institution for academic purposes. Furthermore, academic integrity involves “compliance with ethical and professional principles, standards and practices and a consistent system of values, that serves as guidance 4 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION for making decisions and taking actions in education, research and scholarship” (NAIN, 2021). Maintaining academic integrity is critical to the reputation of higher education, and to the recognition of a graduate’s academic learning and resulting qualifications (NAIN, 2021). 3 Generative Artificial Intel igence (GenAI) Generative Artificial Intel igence (GenAI) transformer architectural models are defined by their use of multi-layered neural networks (NNs), also known as multimodal large language models (LLMs), trained using unsupervised and semi- supervised statistical, machine and deep learning algorithms to perform a variety of natural language processing (NLP) tasks, to identify patterns and structures within massive petabyte-sized datasets to generate new and original artifacts. For example, Generative Pre-Trained Transformer 4 (GPT4), commonly known as ChatGPT4, is a multimodal large language AI model created by OpenAI, and it is the fourth in a series of GPT foundation models (OpenAI, 2023b). 3.1 The Adoption of ChatGPT Since November 2022, ChatGPT has become the fastest growing consumer software application in history. Ofcom United Kingdom (Ofcom, 2023) reveals that 80% of British teenagers aged from 13-17 years old, and 40% of children between the ages of 7-12 years old use GenAI tools and services for schoolwork or leisure. ChatGPT is the most widely used GenAI service among internet users aged 16 and over (Ofcom, 2023). Education featured strongly in the target audience interests of ChatGPT site visitors (Similarweb, 2023) during the period of the study . ChatGPT's journey from concept to influential GenAI model exemplifies this rapid evolution of GenAI. This model has driven progress in GenAI development and spurred transformation of work practices across a wide range of industries (Marr, 2023). Thus, given the disruptive impact of GenAI on business and indeed al facets of life, it is critical that this technology becomes part of the educational experience of students particularly as it heavily influences their future careers. A. Duane: A Study to Measure the Potential Impact of Generative Artificial Intel igence on Academic Integrity 5 3.2 Enhancing Educational Praxis with GenAI The ability of GenAI to perform complex tasks within the field of education has caused mixed feelings among educators as it disrupts existing educational praxis (Baidoo-Anu and Owusu Ansah, 2023). However, as studies are only emerging with respect to the uses and benefits of GenAI for education, it is not yet possible to assert a consensus among academics with specific reference to GenAI in education (Firat, 2023a). Nevertheless, emergent studies identify a number of benefits of GenAI which are applicable to ChatGPT: • ChatGPT can be used to create intel igent tutoring systems capable of providing personalized assistance to students (Marr, 2023; Zhai, 2022). • ChatGPT can be used in education, especial y for autodidactic learners, because it can provide personalized learning support, tailor appropriate learning programs, and provide timely feedback (Firat, 2023a; Firat, 2023b). • ChatGPT can be used to create chatbots and virtual language tutors simulating real-life conversations and providing instant feedback (Bédi et al., 2023; Božić and Poola, 2023; Firat, 2023a; Firat, 2023b; Hockly, 2023). • ChatGPT can be used to help students improve their reading and writing skills. By analyzing a student's writing style, ChatGPT can suggest improvements and provide feedback on grammar, punctuation, and spel ing errors (Bédi et al., 2023; Božić and Poola, 2023; Hockly, 2023). • ChatGPT can be used to create personalized learning experiences. By analyzing a student's learning patterns and preferences, ChatGPT can recommend specific learning resources, such as articles, videos, and textbooks, that are tailored to their needs (An et al., 2023; Baidoo-Anu and Owusu Ansah, 2023; Božić and Poola, 2023; Wang, 2023; Zhai, 2022). • ChatGPT can generate prompts for formative assessment activities providing ongoing feedback (Baidoo-Anu and Owusu Ansah., 2023). • ChatGPT can be used to grade essays and other written assignments automatically. This can save teachers a lot of time and provide students with immediate feedback on their work (Božić and Poola, 2023; Zhai, 2022). • ChatGPT can improve motivation, engagement, and learning outcomes (Baidoo-Anu and Owusu Ansah, 2023; Deng and Yu, 2023; Wang, 2022). • ChatGPT to enhance participation and success for students from disadvantaged backgrounds (Sullivan et al., 2023). • ChatGPT can be used to foster critical thinking (García-Peñalvo, 2023). • ChatGPT can empower students to learn complex concepts in plain language (Sullivan et al., 2023). 6 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION From a strategic educational perspective, the emergence of GenAI compels academics to adapt teaching, learning and assessment practices to incorporate the new reality of living, working, and studying in a world where GenAI is widely available as open source software (Liu et al., 2023; García-Peñalvo, 2023; Rudolph et al., 2023; Sullivan et al., 2023). However, a common concern in academic literature pertains to the negative impact of GenAI on academic integrity. 3.3 Challenges Posed by GenAI for Academic Integrity For academics, artificial intel igence (AI) has been a source of significant concern with respect to its impact on academic integrity for many years (Abd-Elaal et al, 2019; Amigud et al., 2016; Janakiram, 2022; Lancaster, 2021; Roe and Perkins, 2022). Outside of academia, the use, or indeed misuse, of AI has also raised concerns about ethics and integrity in other competitive settings (Roose, 2022). The threat of GenAI to academic integrity has also been raised in the media (Fowler, 2023; Quach, 2023; Weale, 2023). The emergence of publicly accessible GenAI, and ChatGPT in particular, has dramatical y increased these concerns which cannot be dismissed easily (An et al., 2023; Amini-Salehi, 2023; Baidoo-Anu and Owusu Ansah, 2023; Malinka et al., 2023; Rosenblatt, 2023; Sullivan et al., 2023; Zhai, 2022). Professor Charles Terwiesch of University of Pennsylvania’s Wharton School contends that ChatGPT has important implications for education – a conclusion arrived at when ChatGPT passed an MBA exam (Rosenblatt, 2023). Thus, GenAI poses a threat to academic integrity when used to bypass learning in what the European Network for Academic Integrity (ENAI, 2023) terms “unauthorised content generation”. While OpenAI, the creators of ChatGPT, developed a software tool in February 2023 to detect text generated by ChatGPT (Quach, 2023), it was withdrawn from the market as it was ineffective in July 2023 (Horwood, 2023). In April 2023, Turnitin embedded an AI writing detection tool in its Turnitin Feedback Studio (Caren, 2023). The Turnitin AI writing detection tool quickly established itself as a key tool for academics to detect ChatGPT generated essays, presentation slides, and narrative analyses. However, several reports in the media highlight that the Turnitin AI writing detection tool is fallible and can indeed generate false positives (Fowler, 2023). A. Duane: A Study to Measure the Potential Impact of Generative Artificial Intel igence on Academic Integrity 7 The ability of GenAI to paraphrase texts and reduce plagiarism detection (Amini- Salehi, 2023) also raises concerns as it conceals that the work is not original and results in a distorted perception of the student’s writing abilities. Wang (2023) found that students with higher metacognitive levels were better able to describe their goals and processes using GenAI prompt engineering and were better able to critical y adapt GenAI solutions. By contrast, students with lower metacognitive levels rely more heavily on GenAI, rather than using it as a support tool. Thus, GenAI can provide some students with an advantage over others particularly if it is used to conceal plagiarism or used surreptitiously in the generation of an assessment solution. GenAI also raises concerns that students may outsource assessments to those capable of producing higher quality outputs (Zhai, 2022). Despite this important debate, very little literature has been published on GenAI in education and the student’s voice is poorly represented in research (Sullivan et al., 2023). 4 Research Method The study empirically tests the impact of GenAI on the academic integrity of university assessments. The implementation of the study is quite simple and easily replicated. The Principal Investigator (PI) first invited university academics to submit a sample of past assessments from 2020-2022. Past assessments were received from seven academics for nine modules including Fund Reporting and Risk Management; Personal, Professional and Academic Skills; Business Research and Communication Skills; Business Strategy; Global Business Ethics; Organisational Behaviour; Management Skills; Professional Development; and Behavioural Finance. The assessments include essays; spreadsheet analysis; industry and sectoral analysis reports; critical analysis of theories; presentation slides; reflective diaries; and discursive analysis. These past assessments were then adapted as research tests removing cover sheets and submission details/dates but preserving the original questions/instructions. Twenty-six (26) tests were generated from the assessments. The PI then posted invitations on the university course management systems (CMS) for students to enroll in two GenAI research events in April and September 2023. The research events were hosted in the SETU digital business laboratory. Each student was assigned a research test on the CMS. Students were instructed that the goal of the GenAI research event was to chal enge students to apply any GenAI of their choosing to complete the past assessments adapted as research tests, with the 8 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION goal of achieving a pass grade (>=40%) when graded by an academic, undetected by GenAI writing detection tools. While the original assessments gave students several weeks to complete their submissions, students only had two hours to complete the research tests. Once completed, students uploaded their solutions to a research folder in the CMS. The students were al 1st, 2nd and 3rd year business students. None of the students have received any training in the use of AI in SETU. Two (2) of the twenty-six (26) research tests submitted were spoiled and inadmissible in the analysis. The valid research tests (24) were processed using GenAI writing detection software to determine an AI writing detection score. The PI then distributed the research tests for grading to the academics who volunteered the original past assessment samples. The grading schemes and the approaches to grading used by the academics were precisely the same as for the original assessments in 2022-2022. The academics were not provided with the GenAI writing detection scores prior to grading the tests to minimise bias when they were being graded. Of the 24 valid research tests, 3 were not graded by the academic who provided the original assessment due to time constraints. The results of the data analysis are confined to 21 valid tests from the GenAI research events as detailed in Table 1. Once the 21 research tests were graded by the academics, the PI held informal interviews lasting 40 minutes to 1 hour, with each academic to discuss their experiences of the research tests, the grades they awarded, and their thoughts on the chal enges of GenAI for academic integrity. The PI considers this research methodology an appropriate and easily replicated benchmarking test of GenAI and GenAI writing detection tools, so that as academics, we can establish a clear understanding of the chal enges GenAI poses for academic integrity. A. Duane: A Study to Measure the Potential Impact of Generative Artificial Intel igence on Academic Integrity 9 Table 1: Analysis of Valid Test Submissions Test Academic No. Year Event Gender AI Writing Detection Score Grade 1 3 Apr-23 M 73% Not Graded 2 3 Apr-23 M 77% 60% 3 3 Apr-23 M 88% 40% 4 3 Apr-23 M 88% 60% 5 3 Apr-23 M 89% 35% 6 3 Apr-23 M 92% 40% 7 3 Apr-23 M 99% Not Graded 8 3 Apr-23 M 100% 41% 9 3 Apr-23 M 100% 25% 10 1 Sep-23 M 0% 32% 11 2 Sep-23 M 0% 58% 12 1 Sep-23 F 0% 40% 13 1 Sep-23 M 0% 50% 14 2 Sep-23 M 10% 60% 15 2 Sep-23 F 14% 58% 16 1 Sep-23 F 30% Not Graded 17 1 Sep-23 F 34% 50% 18 1 Sep-23 M 43% 19% 19 3 Sep-23 M 46% 60% 20 3 Sep-23 M 54% 45% 21 1 Sep-23 M 56% 40% 22 1 Sep-23 M 89% 35% 23 3 Sep-23 M 91% 57% 24 3 Sep-23 M 46% 0% Source: Own 5 Findings The findings reveal that both research hypotheses were validated: (1) students can complete university assessments using GenAI undetected by AI writing detection tools, and (2) students can achieve a passing grade (>=40) for university assessment submissions generated using AI when graded by a university academic? 10 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The AI writing detection tool vendor advises that low AI writing detection scores (less than 20% GenAI writing detection) have a higher likelihood of being false positives and are thus insufficient grounds to raise an academic integrity enquiry. In this study, the AI writing detection tool was successful in detecting high AI writing detection scores (>20%) in 75% (18/24) of the research tests. Thus, these tests would have warranted investigation for possible breaches of academic integrity. However, the analysis also shows that 25% (6/24) of the research tests achieved an AI writing detection score of 14% or less, and would not have been flagged to the academic, and would not have been investigated for breaches of academic integrity. Analysis of these six (6) research tests, reveals that four (4) achieved a 0% AI writing detection score, while two research tests yielded 10% and 14% AI writing detection scores. Three (3) of these research tests were submitted by first year students and three (3) were submitted by second year students. None of the third year students submitted research tests that achieved no/low AI writing detection scores. Five (5) of the six (6) research tests with no/low AI writing detection scores received a passing grade (>-40%) from the academic who provided the past assessment. This equates to 23.8% of all tests graded (n=21). The average grade for tests achieving a no/low AI writing detection score and achieving a pass grade from an academic was 53%, with 60% the highest grade and 40% the lowest grade. It is notable that one other research test submitted by a first year student achieved a 0% detection rate but only achieved a 32% grade from the academic. Thus, although the test received a fail grade, GenAI content was not detected by the AI writing detection tool. It is important to highlight that the five (5) (23.8%) research tests with no/low AI writing detection scores that achieved a pass grade were not attributable to a specific academic, module or submission type as they were evenly dispersed. The AI writing detection rate decreased significantly from 90% to just 37% between GenAI Research Event 1 in April 2023 and GenAI Research Event 2 in September 2023. It is also important to highlight that the five (5) research tests (23.8%) that achieved no/low AI writing detection scores were submitted during AI Research Event 2. This may be an indication of how wel GenAI has been adopted and mastered by students since its launch in November 2022, and the PI holding AI Research Event 1 in April 2023 and AI Research Event 2 in September 2023. A. Duane: A Study to Measure the Potential Impact of Generative Artificial Intel igence on Academic Integrity 11 In this study, students were free to choose and use whatever GenAI tools they could find to complete the research tests. While most of the students only used ChatGPT, several of the students reveal they used multiple GenAI tools. In this study, GenAI struggled to work with spreadsheet analysis. None of the research tests passed when graded by the academic. Initial y trained on data to September 2021, ChatGPT’s training set is continuously upgraded and it is now integrated with Microsoft Bing. Initially, ChatGPT only handled text based prompts, but it now incorporates image- based prompts. Integration with MSOffice now enables spreadsheets to incorporate micro-app AI agents capable of extracting, cleaning and analysing tables of data (OpenAI, 2023b). Hence, GenAI poses an ever-greater chal enge to academic integrity if ongoing advancements are not considered as assessments are being set. 5.1 Insights from One to One Interviews with University Academics Fol owing the col ection and analysis of al AI writing detection scores and research test grades, the PI held informal meetings with each participant academic. Concern about AI is pervasive among academics, and some academics and academic managers appear to have adopted a head in the sand approach to the impact of AI on their teaching and learning strategies and indeed the impact of AI on the entire education system. The participant academics contend that as GenAI improves, and as students increasingly engage with it, academics wil have to be subject matter experts to detect GenAI content. However, a counter-argument could be that with the evolution of GenAI, it may in fact replace the need for any academic or indeed graduate to be a subject matter expert. Participant academics were concerned that students can use GenAI to create a bibliography to accompany GenAI text, and that some or indeed al the bibliography may not be authentic. Participant academics are concerned they do not have the resources or time to fol ow every reference to verify if it is authentic, or indeed relevant to the text. Participant academics believe that fundamental questions need to be asked about what we are teaching, why we are teaching it, how we are teaching it, and what careers we are preparing graduates to enter. Students commencing university in 2023-2024 will emerge from degree programmes to an employment landscape transformed by AI, and much of what they are learning could be redundant if roles are supplanted by AI. Thus, participant academics contend that a bigger conversation must occur about the impact of AI on education beyond that of its impact on academic integrity. 12 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 6 Limitations While the study objective was completed as per the research funding proposal, the study encountered several limitations which impacted its potential output including low levels of academic engagement and a lack of support for the study by some senior managers. Participant academics may have been exposed to bias when grading the research tests as they knew the tests were generated using AI. 7 Conclusions and Future Studies The study provides empirical evidence of the ease at which students can use GenAI to produce viable solutions for university assessments while circumventing AI writing detection tools and academic integrity rules. Twenty five percent (25%) of the research tests achieved an AI writing detection score of 14% or less, meaning they significantly fal below the minimum threshold (20%) to be investigated for breaches of academic integrity in SETU or indeed perhaps many universities. The study provides evidence that students ability to conceal unauthorised generation of content using GenAI rapidly improves with use and exposure, as AI writing detection fel significantly from 90% to just 37% between GenAI Research Event 1 in April 2023 and GenAI Research Event 2 in September 2023. While the study, did not distinguish between student use of the different GenAI tools at their disposal, it became evident that the ability to detect unauthorised AI generated content and artifacts are further compounded by the use of multiple GenAI tools and manual human editing of AI generated output. Unquestionably, the chal enges for academic integrity wil increase exponential y as GenAI tools become more sophisticated and as additional GenAI tools emerge. This needs further study as GenAI tools emerge. This study provides tangible evidence that AI and GenAI in particular, is disrupting education and potentially undermining academic integrity. It is clear than many forms of traditional assessments are no longer viable. Future studies, aligned with Bloom’s Taxonomy could further explore GenAI’s impact on teaching, learning and assessment. It is imperative that the global education system adapts to meet the disruptive challenges and opportunities of AI. It is essential that policy makers, researchers, educators, and technology experts work together to ensure AI can be A. Duane: A Study to Measure the Potential Impact of Generative Artificial Intel igence on Academic Integrity 13 used safely and constructively to improve education and support teaching, learning, and assessment while maintaining academic integrity. However, it is important to state that although students can use GenAI to circumvent academic integrity, it doesn’t automatically mean that they will. My experience is that students are predominantly honest, upstanding and protective of their own and their universities integrity. Acknowledgements The PI would like to thank the senior management, academics and students who supported and participated in this study, without whose engagement and teamwork, it could not have been completed. The PI and the participants would like to acknowledge the funding provided by the Strategic Alignment of Teaching and Learning Enhancement Funding in Higher Education 2022/2023 and the N-TUTORR National Project 2023 without whose support this research would not have been possible. References Abd-Elaal, E.S., Gamage, S.H. and Mills, J.E. (2019). Artificial intelligence is a tool for cheating academic integrity. In 30th Annual Conference for the Australasian Association for Engineering Education (AAEE): Educators Becoming Agents of Change: Innovate, Integrate, Motivate, Brisbane, Queensland: Engineers Australia, January, 397-403. https://www.researchgate.net/profile/Elsayed- Abdelaal/publication/339375213_Artificial_Intelligence_Is_a_Tool_for_Cheating_Academic _Integrity/links/5e6759d792851c7ce0578899/Artificial-Intel igence-Is-a-Tool-for-Cheating- Academic-Integrity.pdf. Amini-Salehi, E., Hassanipour, S., Bozorgi, A., Keivanlou, M.H., Dave, T., Alotaibi, A., Mel atdoust, P., Joukar, F., & Bakhshi, A. (2023). The ability of Chat-GPT to paraphrase texts and reduce plagiarism. JMIR Medical Education. JMIR Preprints #53308: https://preprints.jmir.org/preprint/53308. Amigud, A., Arnedo-Moreno, J., Daradoumis, T. and Guerrero-Roldan, A.E. (2016). A behavioral biometrics based and machine learning aided framework for academic integrity in e-assessment. In 2016 International Conference on Intelligent Networking and Col aborative Systems (INCoS). September, 255-262, IEEE. https://ieeexplore.ieee.org/abstract/document/7695181. An, Yingyi, Ouyang, W., & Zhu, F. (2023). ChatGPT in higher education: Design teaching model involving ChatGPT. Lecture Notes in Education Psychology and Public Media. 24. 47-56. http://dx.doi.org/. Anaconda (2023). AI use cases for the enterprise, 9 industries using AI to accelerate, disrupt and innovate. https://know.anaconda.com/rs/387-XNW- 688/images/Anaconda_AI%20Use%20Cases%20for%20the%20Enterprise.pdf Baidoo-Anu, D. & Owusu Ansah, L. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and Learning, January 25, http://dx.doi.org/10.2139/ssrn.4337484. 14 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Bédi, B., Chiera, B., Chua, C., Cucchiarini, C., Ní Chiaráin, N., Rayner, M., Simonsen, A., and Zviel-Girshin, R. (2023). ChatGPT based learning and reading assistant: Initial report. https://www.researchgate.net/publication/372526096_ChatGPT- Based_Learning_And_Reading_Assistant_Initial_Report. Božić, V. and Poola, I. (2023). Chat GPT and education. https://www.researchgate.net/publication/369926506_Chat_GPT_and_education Caren, C. (2023). The launch of Turnitin’s AI writing detector and the road ahead, 4th April. Turnitin. https://www.Turnitin.com/blog/the-launch-of-Turnitins-ai-writing-detector-and- the-road-ahead. Deng, X. and Yu, Z. (2023). A meta-analysis and systematic review of the effect of chatbot technology use in sustainable education. Sustainability, 15(4), 2940, URL: https://doi.org/10.3390/su15042940. ENAI, (2023). Addressing unauthorised content generation working group (AUCG WG), European Network for Academic Integrity, https://www.academicintegrity.eu/wp/addressing- unauthorised-content-generation-working-group/. Firat, M. (2023a). What ChatGPT means for universities: Perceptions of scholars and students, Journal of Applied Learning & Teaching (JALT), 6(1). http://journals.sfu.ca/jalt/index.php/jalt/index. Firat, M. (2023b). How ChatGPT can transform autodidactic experiences and open education? https://www.researchgate.net/publication/367613715_How_Chat_GPT_Can_Transform_ Autodidactic_Experiences_and_Open_Education. Fowler, G. (2023). We tested a new ChatGPT-detector for teachers. It flagged an innocent student, April 3, The Washington Post. https://www.washingtonpost.com/technology/2023/04/01/chatgpt-cheating-detection- Turnitin/. García-Peñalvo, F.J. (2023). The perception of artificial intelligence in educational contexts after the launch of ChatGPT: Disruption or panic? Education in the Knowledge Society, 24, 1-9. https://doi.org/10.14201/eks.31279. Halaweh, M. (2023). ChatGPT in education: Strategies for responsible implementation. Contemporary Educational Technology, 15(2). https://doi.org/10.30935/cedtech/13036. Hearn, A. (2022). AI-assisted plagiarism? ChatGPT bot says it has an answer for that. The Guardian, December 31. https://www.theguardian.com/technology/2022/dec/31/ai-assisted-plagiarism-chatgpt-bot-says-it-has-an-answer-for-that. Hockly N (2023). Artificial intelligence in English language teaching: The good, the bad and the ugly. RELC Journal 54(2): 445–451. https://doi.org/10.1177/00336882231168504. Horwood, P. (2023). OpenAI shuts down its AI detection tool, Computing, 26th July. https://www.computing.co.uk/news/4120927/open-ai-shuts-ai-detection-tool. Janakiram, S.V. (2022). 5 AI tools that can generate code to help programmers. Forbes, March 14. https://www.forbes.com/sites/janakirammsv/2022/03/14/5-ai-tools-that-can-generate- code-to-help-programmers/. Kohnke, L., Moorhouse, B.L., & Zou, D. (2023). ChatGPT for language teaching and learning, RELC Journal. https://doi.org/10.1177/00336882231162868. Lancaster, T. (2021). Academic dishonesty or academic integrity? Using Natural Language Processing (NLP) techniques to investigate positive integrity in academic integrity research. Journal of Academic Ethics, 19(3), 363-383. https://doi.org/10.1007/s10805-021-09422-4. Liu, A., Bridgeman, D., & Mil er, B. (2023). As uni goes back, here’s how teachers and students can use ChatGPT to save time and improve learning. The Conversation, February 28. https://theconversation.com/as-uni-goes-back-heres-how-teachers-and-students-can-use- chatgpt-to-save-time-and-improve-learning-199884. Malinka, K., Peresíni, M., Firc, A., Ondrej, H., & Janus, F., (2023). On the educational impact of ChatGPT: Is artificial intel igence ready to obtain a university degree? In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education, A. Duane: A Study to Measure the Potential Impact of Generative Artificial Intel igence on Academic Integrity 15 Association for Computing Machinery, NY, 47-53. https://doi.org/10.1145/3587102.3588827. Marr, B. (2023). A short history of ChatGPT: How we got to where we are today. Forbes, May 19. https://www.forbes.com/sites/bernardmarr/2023/05/19/a-short-history-of-chatgpt-how- we-got-to-where-we-are-today/. McKinsey (2023). The State of AI in 2023. McKinsey Group. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023- generative-ais-breakout-year. Microsoft (2024) What will you do with copilot. https://www.microsoft.com/en- us/bing?ep=0&form=MA13LV&es=31 NAIN (2021). Academic integrity: National principles and lexicon of common terms. National Academic Integrity Network, Quality and Qualifications Ireland (QQI), https://www.qqi.ie/sites/default/files/2021-11/academic-integrity-national-principles-and- lexicon-of-common-terms.pdf. Ofcom (2023). Gen Z driving early adoption of Gen AI. Ofcom. https://www.ofcom.org.uk/news- centre/2023/gen-z-driving-early-adoption-of-gen-ai. OpenAI (2023a). Due to high demand, we’ve temporarily paused upgrades. OpenAI Developer Forum, November 21st. https://community.openai.com/t/due-to-high-demand-weve- temporarily-paused-upgrades-when-will-chatgpt-4-be-available-for-purchasing- again/517248?page=4. OpenAI (2023b). Chat GPT for Excel: Use GPT3 inside Excel Sheets. OpenAI Developer Forum, https://community.openai.com/t/chat-gpt-for-excel-use-gpt3-inside-excel-sheets/86697/13. Quach, A. (2023). OpenAI is developing software to detect text generated by ChatGPT. The Register, January 9, https://www.theregister.com/2023/01/09/in_brief_ai/. Roe, J. & Perkins, M. (2022). What are automated paraphrasing tools and how do we address them? A review of a growing threat to academic integrity. International Journal for Educational Integrity, 18(1), 1-10. https://doi.org/10.1007/s40979-022-00109-w. Roose, K. (2022). An A.I. generated picture won an art prize. Artists aren’t happy, The New York Times, September 2. https://www.nytimes.com/2022/09/02/technology/ai-artificial- intelligence-artists.html Rosenblatt, K. (2023). ChatGPT passes MBA exam given by a Wharton professor, NBC News, January 23. https://www.nbcnews.com/tech/tech-news/chatgpt-passes-mba-exam-wharton- professor-rcna67036. Rudolph, J. & Tan, S. (2023). ChatGPT: Bul shit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342-363. https://doi.org/10.37074/jalt.2023.6.1.9. Sawahel, W. (2023). Embrace it or reject it? Academics disagree about ChatGPT. University World News, February 7th, https://www.universityworldnews.com/post.php?story=20230207160059558. Similarweb (2023). ChatGPT Demographics. Similarweb. https://www.similarweb.com/website/chatgpt.com/#overview. Sullivan, M., Kelly, A. & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning and Teaching, Vol. 6, No.1, March 20th. https://doi.org/10.37074/jalt.2023.6.1.17. Wang, L (2023). Metacognitive strategies used in ChatGPT among college students. Lecture Notes in Education Psychology and Public Media, 21, 283-288. https://doi.org/10.54254/2753- 7048/21/20230170. Weale, S. (2023). Lecturers urged to review assessments in UK amid concerns over new AI tool. The Guardian, January 13. https://www.theguardian.com/technology/2023/jan/13/end-of-the- essay-uk-lecturers-assessments-chatgpt-concerns-ai. Zhai, X. (2022). ChatGPT user experience: Implications for education. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4312418. 16 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION DIGITAL ASSET CREATION AND ORCHESTRATION: EMPIRICAL EVIDENCE FROM INDUSTRIAL MANUFACTURING GABRIELE PICCOLI,1,2 JOAQUIN RODRIGUEZ3 1 Louisiana State University, Louisisana, United states of America gpiccoli@lsu.edu 2 University of Pavia, Pavia, Italy gpiccoli@lsu.edu 3 Grenoble Ecole de Management, Grenoble, France joaquin.rodriguez@grenoble-em.com This paper addresses the chal enge faced by many industrial manufacturers engaged in digital transformation as they rearchitect their physical products into digital assets that can be orchestrated by the firm’s business customers as part of their own digital strategic initiatives. In the context of a century old firm that pioneered the professional coffee machine market, we find that architectural innovation occurs at multiple levels – within and outside the individual machines – driven by three design principles: programmatic bitstring encapsulation, hardware abstraction, and physical extensibility and decoupling. Each principle is enacted through a series of cohesive design moves that result in a design hierarchy inversion subverting the historical supremacy of the machine's mechanical architecture Keywords: over the software, and the resulting digital solutions. This digital inversion is an example of ontological reversal, pushing our resources, architectural understanding of the phenomenon in industrial settings beyond innovation, the current notion of a temporal reversal in design. Our digital observations suggests that ontological reversal has deeper roots transformation, platformization, and far-reaching implications than the above view implies, potential challenging the very foundation of firms’ value creation activities. impact DOI https://doi.org/10.18690/um.fov.4.2024.2 ISBN 978-961-286-871-0 18 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Driven by intensified competition and evolving customer expectations, industrial manufacturers are facing a growing imperative to recombine and reconfigure digital and physical components of their products and services into novel value propositions (Liu et al., 2024; Piccoli et al., 2022; Woodard et al., 2013; Yoo et al., 2010). A key chal enge for industrial organizations seeking to embrace the primacy of digital resources (DR) (Piccoli et al., 2024) is that their core products must be abstracted to become components of a digital service layer. From a design standpoint, successful y navigating the transition cal s for rearchitecting products that used to provide value as standalone physical assets into digital asset: modular components encapsulated in a bitstring programmatic interface (Piccoli et al., 2022). Note that most industrial products are already sophisticated hybrid digital objects (Faulkner & Runde, 2019) with significant digital components and functionalities. The chal enge is therefore one of redesigning them so that they can be orchestrated as part of a value creating generative digital service layer (Piccoli et al., 2022). From a strategic standpoint, abstracting products into modules must be done in a way that prevents their commoditization. This concern is particularly salient for mature industrial product subject to low-end market disruption and design principles1 must strike the balance between fostering generativity through the openness of the emerging digital service layer and integration of the firms proprietary digital assets. Strategic and design considerations are intertwined because the incumbents’ ability to successful y embrace the primacy of DR hinges on their capacity to “extend competitive advantages tied to their legacy physical core” (Liu et al., 2024, p. 6191) during the digital transformation. Digital services must enhance, rather than deplete, the value of the technologies and resources embedded in the firm’s physical core (Drechsler et al., 2020). Successfully navigating this transition is critical for industrial incumbents to ensure their long-term viability in the face of relentless digital disruption, a chal enge that goes beyond prototype development (Lyytinen et al., 2016). We are aware of no study directly investigating how an incumbent manufacturing firm reconfigures its products into digital assets to enable their value 1 Design principles are those fundamental rules “derived inductively from extensive experience and/or empirical evidence, which provide design process guidance to increase the chance of reaching a successful solution (Fu et al., 2016, p. 138). They provide guidance for a coherent set of design moves: “discrete strategic actions that enlarge, reduce, or modify a firm's stock of design” (Woodard et al., 2013, p. 539), thus shaping “both short-term opportunities for capturing economic value and the system’s long-term path of design evolution” (Woodard et al., 2013, p. 542). G. Piccoli, J. Rodriguez: Digital Asset Creation and Orchestration: Empirical Evidence from Industrial Manufacturing 19 adding orchestration in digital services. Specifically, we ask: How do organizations rearchitect physical assets into digital assets? What design principles and design moves do firms implement to enable digital asset orchestration? Based on a longitudinal case study of a century-old firm that pioneered the professional coffee machine market, we empirical y investigate the design and development of digital assets by mapping the evolution of an industrial product as it undergoes multiple architectural transitions (Sandberg et al., 2020). Our results show that the firm simultaneously attempts architectural innovation at multiple levels – within and outside the physical coffee machines. Driving the effort are three design principles: programmatic bitstring encapsulation, hardware abstraction, and physical extensibility and decoupling. Each is enacted through a series of cohesive design moves that result in a design hierarchy inversion that subverts the historical supremacy of the machine's mechanical architecture over the software, and the resulting digital solutions. Such inversion is an instance of ontological reversal, one with deeper roots and farther-reaching implications than the original conceptualization as a “temporal reversal in the way that products are manufactured” (Baskerville et al., 2020, p. 511). Ontological reversal challenges the very foundation of firms’ value creation activities, calling into question the design principles that underpin the historical success of incumbents. The paper proceeds as fol ows: First, we investigate related work, and define the conceptual framing for our study. We then introduce the methodology and provide a case description, followed by data analysis and a discussion of the findings. Finally, we derive the theoretical implications of our results and offer concluding remarks. 2 Related Literature Several literature streams inform our research, with digital transformation studies providing the context for our inquiry. This literature largely takes an organizational design perspective and documents how, while some incumbents have effectively navigated the transition (Piccoli et al., 2024), numerous instances of failures underscore the complexities associated with leveraging digital material (Liu et al., 2024; Lyytinen, 2022) as a differentiating component of value propositions traditional y associated with the physical characteristics of products or product components (Grover et al., 2024). This challenge is particularly pronounced for 20 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION firms in long-established industries, where sector dynamics and economic structures are deeply ingrained in organizing logics (Lyytinen, 2022). Related research focuses on the internal design of physical products as they are increasingly digitized. Several findings in this stream, focusing mainly on the automotive industry, demonstrate the decoupling of digital control system from the physical product hierarchy (Lee & Berente, 2012) and recognize the “strong tension between the two architectural frames at the point when [digital] patterns were instantiated and deployed to physical parts” (Henfridsson et al., 2014, p. 38). More recently they show the “incongruity between hierarchical and layered configuration of modules” (Hylving & Schultze, 2020, p. 21) that requires technological innovation to become intertwined with traditional and new forms of organizing. Because the focus is on “enhancing physical artifacts by means of digital capabilities” (Hylving & Schultze, 2020, p. 17), it remains unclear how tensions between digital and physical product architecture, change when the product is rearchitected as a module to be orchestrated as a component in “multiple value paths, offered through design recombination” (Henfridsson et al., 2018, p. 89). Final y, there is an emerging research stream on “platformization” – the “process whereby the structural arrangement of the firm’s technology resources transitions from tightly coupled to loosely coupled” (Kaganer et al., 2023, p. 1017). Studies on product platforms (Sandberg et al., 2020) and enterprise infrastructure (Henfridsson & Bygstad, 2013; Kaganer et al., 2023) serve as the foundation for this line of inquiry. They recognize that design principles and mechanisms required for forming successful platforms are “highly interactive,” but those interactions have yet to be “examined in detail” (Sandberg et al., 2020, p. 142). In sum, despite valuable insight from the three research streams mentioned above, we lack a theoretical understanding of the architectural changes and design principles guiding the conversion of physical products into digital assets, as they are rethought as modules to be orchestrated by a digital service layer. G. Piccoli, J. Rodriguez: Digital Asset Creation and Orchestration: Empirical Evidence from Industrial Manufacturing 21 2.1 Digital Assets and Digital Strategic Initiatives2 Digital resources are “a specific class of digital objects that a) are modular, b) encapsulate objects of value, assets and/or capabilities, c) and are accessible by way of a programmatic interface” (Piccoli et al., 2022, p. 2293). DR have unique structural characteristics that differentiate them from IT resources (Piccoli & Ives, 2005) and IT-enabled resources (Nevo & Wade, 2010) – thus necessitating the use of the term “digital” to connote them. Digital assets are DR that encapsulate either nonmaterial or hybrid digital objects. Value creation with DR occurs through two pathways: a) creation of novel DR and b) orchestration of DR (Piccoli et al., 2022). The former entails the design and implementation of a new digital resource for internal or external use. The latter is the purposeful assembly of DR, IT resources and complementary organizational resources resulting in a value proposition for the firm’s customers. No prior work has identified and demonstrated the specific design principles and design moves a firm implements when rearchitecting physical assets as digital assets to make them available for orchestration internally or by its commercial customers. 3 Methodology and Case Description We analyzed CoffeeCo, a pioneer of the professional coffee machine industry based in Northern Italy. The firm presents an ideal context for our study because the machines are a remarkably complex, yet CoffeeCo’s narrow focus on professional coffee machines limits potential confounds. We col ected data from multiple sources over multiple waves: technical manuals and product specification, architectural diagrams and use case description; two in-depth visits to the company’s industrial museum to review historical product evolution, resulting in detailed analyses of 41 products spanning 1970 to 2021; a first wave of interviews focused on the executive team and subsequent waves on design and production staff, for a total of 16 semi- structured interviews (Table 1). 2 See Piccoli et al. (2022) for a comprehensive treatment of digital resources and related concepts. 22 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 1: Interview schedule, informants, and interviews duration Month, Year Length Participant Details 60 min CEO; Chief Product and Technology Officer; Group June 2022 Electronic and IoT Solutions Director; CMO; Head of Services April 2023 90 min Cloud Solutions, IoT & Digital Officer; Chief Engineer – Fully Automatic Machines April 2023 60 min Senior Software Engineer, Digital Touchpoints System Manager; R&D Engineer; May 2023 60 min Cloud Solutions, IoT & Digital Officer*; Chief Engineer – Fully Automatic Machines* May 2023 30 min Chief Engineer - Traditional Coffee Machines June 2023 60 min Cloud Solutions, IoT & Digital Officer ** July 2023 60 min Group Electronic and IoT Solutions Director * Nov. 2023 60 min Group Electronic and IoT Solutions Director ** Dec. 2023 45 min Group Electronic and IoT Solutions Director *** * Second interview; ** Third interview; *** Fourth interview From product and maintenance manuals (1,104 pages of documentation), we cataloged product components, such as the type of Printed Circuit Board (PCB), or the presence of digital functionalities (e.g., telemetry, connectivity). Analysis proceeded from open and selective coding to theoretical coding (Glaser 1978, Urquhart 2013). Then, based on constant comparisons between what was emerging from our data and existing theory, selective coding al owed us to examine the centrality of digital assets in the digital transformation efforts. Informed by theoretical sensitivity (Glaser 1978), the research questions in this paper materialized as we iterated through the data analysis and focused our conceptualization of the findings. In the tradition of architectural innovation research, we highlight “actions that designers take when working on real systems” (Clements & Northrop, 1996, p. 6) to surface key design moves implemented by CoffeeCo, drawing a coherent set of design moves into design principles. 4 Discussion Our data shows that CoffeeCo, like many industrial incumbents engaging in digital transformation, was chal enged to morph from a manufacturer sel ing machines as finished products in arms-length relationship with its business customers, to a provider of digital solutions fostering ongoing relationship with its clients. Navigating the transition requires the firm to establish design principles and enact G. Piccoli, J. Rodriguez: Digital Asset Creation and Orchestration: Empirical Evidence from Industrial Manufacturing 23 design moves that achieve two objectives. First, to rearchitect its products (i.e., professional and superautomatic coffee machines) from hybrid digital objects (i.e., physical assets with digital features) to fully formed digital assets – modular components that are encapsulated in a programmatic bitstring interface (Piccoli et al., 2022). Second, to establish a digital service layer enabling rapid implementation of customer’s digital strategic initiatives through the reuse and recombination of those digital assets along multiple and often unexpected value paths (Henfridsson et al., 2018) and design hierarchies (Yoo et al., 2010). The chal enge is complicated by the risk of commoditization of the firm’s physical core (Liu et al., 2024) in an environment where coffee brewing technology has matured to the point that “everyone can make a good coffee.” Establishing design principles and enacting design moves at this juncture is fraught with ambiguity (Brusoni et al., 2001). Specifically, the firm must leverage the generativity (Zittrain, 2005) and value co-creation potential afforded by the digital service layer while establishing predictable interdependencies between the physical architecture of the machines and the digital architecture of the service layer. Thus, despite its roots as an industrial manufacturing organization, CoffeeCo is chal enged to “know more than it makes” by developing system integration knowledge (Brusoni et al., 2001) and reconfigure its physical core accordingly (Liu et al., 2024). Consequently, design principles that historically underpin the success of the company are chal enged, requiring a design hierarchy inversion that establishes the primacy of DR while leveraging the established components of professional coffee machines (e.g., boilers, electropumps, touchscreen). The result is a design hierarchy inversion whereby the digital service layer comes to dominate the physical architecture of the machines. As our case analysis shows, before becoming digital assets, the machines had digital functionalities (e.g., configuration, telemetry, customization of the touch screen) that could only be accessed physical y or manual y via dedicated web applications. Such digital functionalities were developed ad-hoc for each coffee machine model in support of its mechanical functionalities, leading to duplications of efforts and limited standardization. Conceptualized and designed as a finished product, machines accommodated emerging or unforeseen digital services with great difficulty. After rearchitecting machines as digital assets, those digital features could 24 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION be exposed programmaticaly and became subordinated in the design hierarchy through the addition of architectural modules. In the remainder of this section, we discuss a) programmatic bitstring encapsulation; b) hardware abstraction; and c) physical extensibility and decoupling. CoffeeCo introduced these design principles to realize its design hierarchy inversion and facilitate digital asset orchestration by its customers. We describe each design principle, and the associated design moves. 4.1 Programmatic Bitstring Encapsulation The implementation of this design principle required the encapsulation of the machines within a standardized programmatic bitstring interface by way of several design moves. Aside from the obvious addition of API functionality, each machine needed an abstraction layer that enforced information hiding principles (Parnas, 1972) from the inner design of the machine (i.e., its physical and digital architecture). This design principle was pursued through a series of non-trivial design moves (Table 2). The architectural innovation implemented by CoffeeCo involved developing a standardized, interface for machine orchestration. The new interfaces were centralized and standardized communication with al machines models and brands, a result achieved by shifting the interfaces to the cloud and exposing them through an API gateway. While many digital functionalities remained available for manual human interaction, such as changing recipes on location using the machine’s touch screen or USB interfaces, the cloud-based programmatic interfaces enabled remote management, making the machines visible as “software libraries” to the digital services layer. Importantly, interface specifications and parameters were the same across al models, in al CoffeeCo owned brands. In other words, CoffeeCo completed an inversion whereby previously hidden information that had to reside in the machine to enable digital services (e.g., payment) became an architectural module in the cloud-based digital service layer, enabling external entities (e.g., payment providers) to interact with the machines without any need for information about the machines’ internal specifications. As a result, digital services became loosely coupled with the G. Piccoli, J. Rodriguez: Digital Asset Creation and Orchestration: Empirical Evidence from Industrial Manufacturing 25 machine’s architecture as shown by the fact that services could evolve at the cloud layer by leveraging ful information hiding of the machine internal operations. Most importantly, digital services could develop without requiring physical updates to the software instal ed on the machines. Architectural y, the cloud service layer needed no awareness of the machine make or model. Each machine in the customer’s fleet would identify itself once connected and register the functionalities and data it could expose based on its electromechanical makeup (e.g., number of brew heads, type of drink dispensers, sensors). The machine was then visible to the customers and ready for orchestration within the specific strategy of the client. This segregation is important because, if the machines are module complete, 3 services can evolve in different directions and at different speed from the constraints of the physical design and manufacturing process. Moreover, this architectural shift required that all new machines be equipped with built-in connectivity and telemetry capabilities by default. The rationale was the prerequisite for a standardized machine hardware to effectively deliver digital services. Offering such digital services on a fleet of heterogeneous machine was impractical and inefficient, and often technical y impossible. Therefore, this transition was not just a technological upgrade but a fundamental change to align with the company’s vision for an integrated and streamlined digital service offering. 4.2 Hardware Abstraction As described above, a standardized programmatic bitstring interface requires information hiding. Thus, CoffeeCo had to introduce architectural innovations for the machines. While largely reusing the same electromechanical components, the firm introduced an abstraction, in software, of al the physical components of the machine to create a loosely coupled relationship between the digital services layer communicating through the programmatic bitstring interface of each machine, and its mechanical components executing tasks in physical space (e.g., brew a cappuccino with soy milk). 3 By module complete we mean that they are not lacking a physical component, input device, or sensor needed to support a given service. For example, without a microphone the machine would not support voice ordering, regardless of the software available. 26 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION CoffeeCo pursued this design principle through a series of non-trivial design moves (Table 2). Most notably, CoffeeCo transitioned away from developing business and functional logic specific to each coffee machine model. Rather it sought to maintain a universal codebase that incorporated business and functional logic common to al models and brands. This move reduced the need for custom development and streamlined the process of accommodating customer requests for custom features. The creation of a unified module at CoffeeCo necessitated an architectural shift, making previously hidden information visible to all machines. The hardware abstraction layer guarantees uniform functionality across machines lines. It is also a central element of CoffeeCo’s ability to offer digital services by exposing the machines to customer orchestration via standard programmatic interfaces. As with the design hierarchy inversion described above, the architectural innovation of the machine was possible because, since the first implementations of electromechanical components, CoffeeCo recognized the imperative for deterministic and timely responses in coffee machines under al conditions. Consequently, the company established an architecture predominantly based on real- time and embedded systems, integrating them into the hardware of the coffee machines. Thus, during phases 1 and 2, each machine model required the custom integration of hardware and software. However, while the components were largely ready, architectural innovation was needed to trade-off flexibility for backwards or future compatibility of the codebase of al machines. Without it CoffeeCo faced significant overhead for maintaining previous models and working on new ones – hampering execution of the solution strategy. Due to the monolithic architecture and substantial interdependencies of electromechanical components in previous architectures, any change required considerable effort. Such deterministic and pre- established architectures could not enable rapid response to the changing customer needs that the design hierarchy inversion had created. Thus, innovation also impacted the physical architecture of the machine. 4.3 Physical extensibility and Decoupling Once it committed to the design hierarchy inversion, with the design principles of standardized programmatic bitstring interfacing supported by hardware abstraction, the firm had to redesign the machine’s physical architecture to become mechanical G. Piccoli, J. Rodriguez: Digital Asset Creation and Orchestration: Empirical Evidence from Industrial Manufacturing 27 component agnostic. This design principle resulted in a series of design moves that increased the modularity of the physical architecture (Table 2). The change required architectural innovation to move away from bespoke electromechanical architecture to a more flexible and scalable one, anchored around the Controller Area Network (CAN) bus system. While in the previous phase physical components could be reused and recombined across various machine models, they required specific, ad- hoc integrations with their onboard PCB. The use of custom PCBs used up to that point offered enhanced control over electromechanical components, surpassing the capabilities of traditional direct one-to-one connections. Yet, links between components had to be hardwired in the PCB during manufacturing, preventing future flexibility and evolvability of the machine. These limitations were significant, ranging from insufficient space for additional wiring, lack of suitable connector types for extra components on the PCB, constraints in data processing and transmission capacity, overload of the PCB and potential for overheating. When a PCB ran out of available pins to connect external actuators or sensors, an entire PCB had to be redesigned, tested, certified, and manufactured to al ow for an extra connection. Data sharing between subsystems was severely limited because components communicated through signals unique to their connection and they did not share common communication protocols. Data generated by one subsystem typically remained within its point-to-point connection with the PCB. It was the microcontroller residing in the PCB that was responsible for receiving, processing, and sending out instructions with the appropriate signal to each component. Thus, the microcontroller acted as a bottleneck in data handling and communication across components. Importantly, the introduction of the CAN bus facilitated the modularization of the architecture by eliminating bespoke connections to a central PCB. This capability enhanced the physical adaptability of the machine, not only during its design and manufacturing phases but even after deployment in the field. 28 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 1: Design principles and design moves at CoffeeCo4 Principles Design Moves Programmatic Create programmatic interfaces to orchestrate machine fleets Bitstring Establish standardized interfaces to digital services Encapsulation Deprecate legacy access points to the machine Re-architect existing software features and solutions Modularize software functionality for each machine Hardware Standardize hardware specifics, unified computing architecture Abstraction Implement configurable features across different machines Standardize around a unified codebase across al machines Implement remote activation capabilities for digital services Physical Adopt the CAN bus architecture extensibility Develop proprietary specs for the CAN bus architecture Transitioning to a standard common communication bus required adjustments in the machine’s components. Physical changes were required as each subassembly must incorporate their own microcontroller to process data, execute commands, and send data to the shared network bus – as opposed to having a central microcontroller that directly controls the behavior of each component. Moreover, each subassembly must comply with the CAN bus interface, which includes compatible wiring for data communication and power. This transition required software changes, to incorporate logic to enable each subassembly to communicate via the CAN bus, share data and receive inputs. Digital services also required the machines to al ow for physical extensibility over time and after delivery to customers. To this end, CoffeCo departed from their established approach where most software development was done by component suppliers and external vendors, with CoffeCo’s primary role relegated to integrating these physical components and related software. 4 Theoretical Implications and Conclusions The dynamics described above, where digital service design drives the digital and physical architecture of industrial assets, is arguably the catalyst of the recent frenzy of digital transformation of industrial manufacturing firms (Piccoli et al., 2024). Conceptual y, such design hierarchy inversion is a manifestation of reverse ontology (Baskerville et al., 2020) whereby the requirements of the “digital world” that fol ow 4 We documented 33 design moves across the three historical architectural transitions. Due to space constraints only the 11 underlying the transition from physical products to digital assets are shown. G. Piccoli, J. Rodriguez: Digital Asset Creation and Orchestration: Empirical Evidence from Industrial Manufacturing 29 the logics of digital design assume primacy in industrial organizations. In the case of CoffeeCo, it is the digital services the firm seeks to offer, such as telemetry, remote maintenance, or re-programmability of drink recipes that ultimately determined the architecture of the machines and the requirements for its electromechanical components. This finding pushes our theoretical understanding of ontological reversal in industrial organizations beyond its original formulation as “a temporal reversal in the way that products are manufactured. The digital version is created first, the physical representation second” (Baskerville et al., 2020, p. 511). Our case hints to far-reaching implications and deeper roots of ontological reversal than the above view would imply. Ontological reversal appears to call into question the very foundation of the firm’s traditional value creation activities. Design principles that historically underpin the success of the company are challenged, requiring a design hierarchy inversion establishing the primacy of DR. More specifically, decisions about what functionalities to incorporate and about how to implement them, chal enge the firm to accrue new architectural knowledge not needed in any of the previous transitions (i.e., phase 1 and phase 2). At CoffeeCo, while the functioning and characteristics of electromechanical components were established over decades of R&D and market research, new physical requirements emerged, guided by digital service imperatives originating from customer needs. The rise to prominence of the digital architecture required a detailed understanding of the “optimal” implementation for each functionality. How the asset should be architected to optimal y respond to market needs is also driven by a digital strategy that is “created first” (Baskerville et al., 2020) and is subject to the digital logics of “software companies” (Lyytinen, 2022). Consequently, the physical and digital architecture of the machine must accommodate such ontological reversal, introducing a new design principle: physical extensibility. For a company like CoffeeCo, this physical extensibility of the machine can only be achieved by modularity in architecture and the reliance on an ecosystem of partners. Despite the inevitable limitations of a single case analysis, such as generalizability, we believe the CoffeeCo case yields interesting insights. Hybrid digital objects, like industrial machines and components, face unique constraints. Identifying the three design principles of programmatic bitstring encapsulation, hardware abstraction, and 30 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION physical extensibility and decoupling that underpin a design hierarchy inversion provides empirical support for scholars drawing attention to the need for a nuanced understanding of digital objects in industrial organizations (Grover et al., 2024). References Baskerville, R. L., Myers, M. D., & Yoo, Y. (2020). Digital First: The Ontological Reversal and New Chal enges for Is Research. MIS Quarterly, 44(2), 509–523. https://doi.org/10.25300/MISQ/2020/14418 Brusoni, S., Prencipe, A., & Pavitt, K. (2001). Knowledge Specialization, Organizational Coupling, and the Boundaries of the Firm: Why Do Firms Know More Than They Make? Administrative Science Quarterly, 46(4), 597–621. Clements, P. C., & Northrop, L. M. (1996). Software Architecture: An Executive Overview. Carnegie Mel on University, Software Engineering Institute. Drechsler, K., Gregory, R., Wagner, H.-T., & Tumbas, S. (2020). At the Crossroads Between Digital Innovation and Digital Transformation. Communications of the Association for Information Systems, 47(1), 23. Faulkner, P., & Runde, J. (2019). Theorizing the Digital Object. MIS Quarterly, 43(4), 1279–1302. https://doi.org/10.25300/MISQ/2019/13136 Fu, K. K., Yang, M. C., & Wood, K. L. (2016). Design Principles: Literature Review, Analysis, and Future Directions. Journal of Mechanical Design, 138(10), 101103. Grover, V., Lyytinen, K., & Piccoli, G. (2024). A Prolegomenon to Digital Strategy. Manuscript Under Review. Henfridsson, O., & Bygstad, B. (2013). The Generative Mechanisms of Digital Infrastructure Evolution. MIS Quarterly, 37(3), 907–931. https://doi.org/10.25300/MISQ/2013/37.3.11 Henfridsson, O., Mathiassen, L., & Svahn, F. (2014). Managing Technological Change in the Digital Age: The Role of Architectural Frames. Journal of Information Technology, 29, 27–43. Henfridsson, O., Nandhakumar, J., Scarbrough, H., & Panourgias, N. (2018). Recombination in the Open-Ended Value Landscape of Digital Innovation. Information and Organization, 28(2), 89– 100. https://doi.org/10.1016/j.infoandorg.2018.03.001 Hylving, L., & Schultze, U. (2020). Accomplishing the layered modular architecture in digital innovation: The case of the car’s driver information module. The Journal of Strategic Information Systems, 29(3), 101621. Kaganer, E., Gregory, R. W., & Sarker, S. (2023). A Process for Managing Digital Transformation: An Organizational Inertia Perspective. Journal of the Association for Information Systems, 24(4), 1005–1030. Lee, J., & Berente, N. (2012). Digital innovation and the division of innovative labor: Digital controls in the automotive industry. Organization Science, 23(5), 1428–1447. Liu, S., Essén, A., & Mähring, M. (2024). Going Digital First as an Incumbent: Reaching Outwards and Anchoring Inwards to Safeguard the Incumbent Physical Core. Proceedings of the 57th Hawai International Conference on System Sciences. Lyytinen, K. (2022). Innovation Logics in the Digital Era: A Systemic Review of the Emerging Digital Innovation Regime. Innovation, 24(1), 13–34. Lyytinen, K., Yoo, Y., & Boland Jr, R. J. (2016). Digital Product Innovation Within Four Classes of Innovation Networks. Information Systems Journal, 26(1), 47–75. Nevo, S., & Wade, M. R. (2010). The Formation and Value of It-Enabled Resources: Antecedents and Consequences of Synergistic Relationships. MIS Quarterly, 34(1), 163–183. https://doi.org/10.2307/20721419 G. Piccoli, J. Rodriguez: Digital Asset Creation and Orchestration: Empirical Evidence from Industrial Manufacturing 31 Parnas, D. L. (1972). On the Criteria to Be Used in Decomposing Systems into Modules. Communications of the ACM, 15(12), 1053–1058. https://doi.org/10.1145/361598.361623 Piccoli, G., Grover, V., & Rodriguez, J. (2024). Digital Transformation Requires Digital Resource Primacy: Clarification and Future Research Directions. Piccoli, G., & Ives, B. (2005). IT-Dependent Strategic Initiatives and Sustained Competitive Advantage: A Review and Synthesis of the Literature. MIS Quarterly, 29(4), 747–776. https://doi.org/10.2307/25148708 Piccoli, G., Rodriguez, J., & Grover, V. (2022). Digital Strategic Initiatives and Digital Resources: Construct Definition and Future Research Directions. MIS Quarterly, 46(4). https://doi.org/10.25300/MISQ/2022/17061 Sandberg, J., Holmström, J., & Lyytinen, K. (2020). Digitization and Phase Transitions in Platform Organizing Logics: Evidence from the Process Automation Industry. MIS Quarterly, 44(1), 129–153. Woodard, C. J., Ramasubbu, N., Tschang, F. T., & Sambamurthy, V. (2013). Design Capital and Design Moves: The Logic of Digital Business Strategy. MIS Quarterly, 37(2), 537–564. https://doi.org/10.25300/MISQ/2013/37.2.10 Yoo, Y., Henfridsson, O., & Lyytinen, K. (2010). Research Commentary—the New Organizing Logic of Digital Innovation: An Agenda for Information Systems Research. Information Systems Research, 21(4), 724–735. https://doi.org/10.1287/isre.1100.0322 Zittrain, J. L. (2005). The Generative Internet. Harvard Law Review, 119, 1975. 32 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION SMART GOVERNANCE CHALLENGES IN INDONESIAN LOCAL GOVERNMENT RIO YUSRI MAULANA,1 USMAN MARJAMAT,2 DIMAS SUBEKTI,3 MOH.ARIEF RAKHMAN,1 ALVA BERIANSYAH1 1 Department of Government Science, Universitas Jambi, Jambi, Indonesia rioyusrimaulana@unja.ac.id, moh.ariefrakhman@unja.ac.id, alvaberiansyah@unja.ac.id 2 Faculty of Law, Universitas Jambi, Jambi, Indonesia usman.marjamat@unja.ac.id 3 Department of Political Science, Universitas Jambi, Jambi, Indonesia dimassubekti05@unja.ac.id This research examines the critical role of digital government evolution in enhancing the performance of local governments through smart governance concepts. Focusing on Indonesia's transformative journey, the research highlights the government's adoption of the Electronic-based Government System (SPBE) to tackle inefficiencies, bureaucratic hurdles, and corruption. Jambi City, actively participating in the 100 Smart City Movement, is investigated as a specific case study on smart governance dimension. Employing qualitative methods, the research aims to uncover insights into the chal enges of digital Keywords: transformation at the local level. Despite facing regulatory smart barriers and resource constraints, the study identifies governance, Smart opportunities for success, particularly through stakeholder cities, collaboration and public-private partnerships. This research digital carries substantial implications for policymakers, practitioners, transformation, local and academics interested in the intersection of technology and government, government in developing nations. challenges DOI https://doi.org/10.18690/um.fov.4.2024.3 ISBN 978-961-286-871-0 34 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The Indonesian government has ful y embraced ICT integration through the Electronic-Based Government System (SPBE), commonly known as e-government. SPBE serves as a key driver of digital transformation across all government levels, enhancing administrative efficiency, transparency, and public service delivery (BP2D Jawa Barat, 2022). By leveraging electronic platforms, the government aims to streamline processes, al ocate resources effectively, and promote citizen engagement. SPBE marks a significant step towards digital y advanced governance, meeting public expectations for modern and efficient administration in Indonesia. Mergel et al., (2019) assert that to meet public expectations for high-value, real-time digital services, governments adapt operational standards to enhance efficiency and effectiveness, focusing on transparency and citizen satisfaction. SPBE embodies Indonesia's commitment to digital transformation, integrating ICT to revolutionize governance (kominfo.go.id, 2020). Digital transformation in the public sector, as advocated by Bolívar (2017), emphasizes organizational change rather than just technical solutions. This shift involves developing online services and policymaking, departing from analog methods, influenced by internal and external factors (Gano, 2013). Despite chal enges in strategy formulation and expertise management, success lies in forward-looking leadership, digital capabilities, and aligning digitalization with broader transformation goals (Carayannis & Hanna, 2016). The rise of the Smart City concept in academic discourse and global policies reflects a broader shift toward digital transformation in the past two decades. Cities worldwide face the imperative to innovate and tackle emerging chal enges, notably enhancing transportation connectivity, land-use diversity, and urban services to foster long-term economic growth. Governments encounter chal enges in formulating effective strategies and managing expertise, closely tied to the global trend of smart cities. Hence, adopting a forward-looking perspective and seamlessly integrating digital capabilities are crucial for navigating both public service digital transformation and the broader evolution toward smart cities. For example, the development of efficient public transportation systems tailored to economic needs is pivotal for urban development. Many innovative approaches to urban service enhancement rely on technology, particularly information and communication technology (ICT), shaping the concept of "smart cities." Nam and Pardo (2011) distinguish smart cities from related terms such as digital or intelligent cities based R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 35 on technology, people, and community aspects. Technologicaly, a smart city heavily incorporates ICT into critical infrastructure components and services (Washburn et al., 2010). According to the framework proposed by Nam and Pardo (2011), the fundamental constituents of a smart city encompass technology, people (encompassing creativity, diversity, and education), and institutions (encompassing governance and policy). A smart city undeniably offers interoperable, Internet-based government services that facilitate widespread connectivity and streamline essential government functions for citizens and businesses, as highlighted by Al-Hader et al. (2009). The research focuses on the transformation of smart governance within the local administration of Jambi City government. The research goals aim to present an empirical and practical y implementable theoretical framework on the smart governance dimension in juxtaposition with the evolution of digital government. Aligned with these research goals, the central research question is: What are the chal anges of smart governance implementation in Jambi City? To address the research question, we outline our methodologies before presenting the actual case studies on West Java Province, Indonesia. The section dedicated to results and discussion highlights significant contributions to the existing scholarly body of work. Initial y, we investigate the impact of smart governance transformation on local administration digitalization initiatives. Subsequently, we delve into the chal enges on the implementation of smart governance transformation to gain a deeper understanding of how to enhance key areas of public service delivery in the future, as an integrated part of a long-term approach to digital transformation. This research contributes to the understanding of the transformative potential of digital government evolution in enhancing local government performance through smart governance concepts, with a focus on Indonesia's chal ages on the journey. By examining the adoption of the Electronic-based Government System (SPBE) to address inefficiencies and bureaucratic chal enges, the study sheds light on the potential of technology to mitigate corruption and improve governance effectiveness. Specifical y, through a case study of Jambi City's participation in the 100 Smart City Movement, the research offers insights into the chal enges implementation of smart governance transformation at the local level. Overal , this research contributes valuable knowledge to policymakers, practitioners, and 36 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION academics interested in leveraging technology to enhance governance in developing nations. 1.1 Smart Governance The concept of smart cities emphasizes the importance of human capital rather than simply relying on the belief that ICT alone can automatical y transform a city into a smart one (Shapiro, 2006, Holland, 2008). Smart governance work to eliminate barriers related to language, culture, education, and disabilities. Smart governance entails engaging various stakeholders in the decision-making process and the provision of public services. ICT-mediated governance, often referred to as e- governance, plays a pivotal role in making smart city initiatives accessible to citizens while ensuring transparency in decision-making and implementation. However, the essence of e-governance in a smart city should revolve around being citizen-centric and driven by citizen participation. The increasing role of technology in the operation of urban systems is causing governments to reconsider their position in a knowledge-based society. This role has been termed "Smart governance" (Giffinger et al., 2007). However, there is no unanimous consensus on the definition of this concept. While some prior studies have highlighted aspects such as political involvement and the efficiency of administrative processes (Giffinger et al., 2007), others have focused on the gathering of various types of data and information related to public management (Schuchmann & Seufert, 2015) In the context of this research, the dimension of smart governance wil be analyzed together with digital government transformation, known by Janowski (2015) as digital government evolution. The concept of the evolutionary stages of digital governance is intriguing when comprehensively viewed within the framework of implementing smart city initiatives in local government, which is the focus of this research. 1.2 Challenges to digital transformation Digital transformation encounters various formidable challenges over the years, encompassing technological hurdles, organizational complexities, legal and ethical dilemmas, and financial constraints. According to Tangi et al. (2021), deficiencies in skills, organizational intricacies, and coordination issues emerge as primary obstacles. R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 37 Table 1: Identification of dimensions of smart governance in the literature No Component Categories Values Smart use of ICTs Smart collaboration and participation Use of technology Defining elements of Smart internal 1 Organizational processes smart governance coordination Smart decision-making Smart administration Outcomes Smart outcomes First-order outcomes: Efficient government changes to the government Readiness for disaster organization management Second-order outcomes: Citizen-centric services changes in the position of Aspired outcomes of Interaction with citizens 2 government vis-à -vis smart governance Strong city brand other urban actors Economic growth Third-order outcomes: Social inclusion improvements to the city Ecological performance Highly educated citizens Ideas Vision Legislation Implementation strategies 3 Policies for smart governance Actions Use of ICTs Collaboration Source: Bolívar & Meijer (2016) Moreover, Wirtz & Daiser (2015) highlight cognitive barriers rooted in individual perceptions, particularly hindering the implementation of open government data initiatives. Cultural barriers, such as risk aversion and bureaucratic norms, intricately intertwine with structural impediments, necessitating concerted efforts across governance, organizational, managerial, and technical domains. Savoldelli et al. (2014) suggest that solely addressing structural barriers may not substantially boost e-government adoption. Pittaway & Montazemi (2020) contend that the lack of expertise in managing both structural and cultural barriers impedes digital transformation. Additionally, Howes & Bishop (2018) emphasize the importance of articulating compel ing arguments regarding the value of transformation and 38 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION acknowledging the inherent uncertainty, thereby avoiding rigid and unrealistic timelines. Table 2: Chal enges and Problems of Digital Government Reforms in Developing Countries Category Chal enges or Problems Low ICT infrastructure, poor e-government Infrastructure and technical aspect quality, lacking privacy/security, and limited computer literacy hinder digital advancement. Unclear institutional approach, insufficient Human resource and Institutional funding, leadership gaps, resistance to change, lack aspect of policy guidelines, and shortage of skil ed personnel. Lack of ability to create new legal and regulatory Regulatory aspect framework for e-government to protect privacy and restrict online crime. Citizens' resistance to new technologies can stem Environmental aspect (historical and from cultural and social factors, while lack of cultural) inclusivity may arise from geographical and demographic chal enges. Source: Own elaboration adapted from (Falco & Kleinhans, 2018; Gil-García & Pardo, 2005; Knox & Janenova, 2019; Ndou, 2004; Odat, 2012; Rakhmanov, 2009; Shaheen & Tassabehji, 2007; UNPA and ASPA, 2001) 2 Method The empirical research employed qualitative methodologies, which included conducting in-depth interviews, detailed observations, and extensive data collection from government agencies at both central and local levels. The research was carried out in Jakarta and Jambi City (Jambi Province), Indonesia, spanning from April 2023 to September 2023. To identify key informants for the study, a combination of purposeful and snowbal sampling techniques was employed, in line with Creswel (2013), snowball sampling involved identifying additional relevant informants by seeking referrals from each interviewee, as described by Brayda & Boyce (2014). The process commenced by dispatching official letters and reaching out to individuals holding roles as public managers or heads of specific units within the administration responsible for digital transformation, whether at the central or local level. Interviews were undertaken with key stakeholders, including the Mayor of Jambi, representatives from the Jambi City Communication and Informatics Service, R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 39 officials from the Ministry of State Apparatus Utilization and Bureaucratic Reform, delegates from the Ministry of Communication and Informatics, and members of the Jakarta Smart City Department. These individuals were then requested to suggest other informants contributing to the digital transformation agenda. Al interviews were conducted by the researchers, recorded digital y, transcribed, and subjected to coding for consistency, fol owing the methodologies outlined by Saldana (2014). The research then advanced into a coding phase, utilizing key concepts such as "smart city," "smart governance," and "digital government evolution" as foundational elements (Bolívar & Meijer, 2016; Janowski, 2015; Meijer et al., 2015). This coding strategy, a common initial step in qualitative research, facilitated data analysis. Through open coding, additional sub-codes were generated to capture specific nuances observed in the cases (Strauss & Corbin, 1998). Subsequently, individual coding of the interviews produced findings extensively deliberated upon in multiple data analysis sessions involving the entire research team. Ongoing discussions focused on interpreting the significance of additional codes, their relationship to existing literature, and their alignment with other empirical categories under scrutiny (Creswell & Creswell, 2018; Eisenhardt, 1989; Hancock et al., 2021; Yin, 2017) 3 Result 3.1 Indonesian Digital Transformation Policy Design Governments at various levels in Indonesia are steering diverse paths toward digital transformation, influenced by their digital maturity stages. The national digital government policy, known as The Electronic Based Government System (SPBE), is outlined in Presidential Regulation Number 95 of 2018. Aligned with the Long-Term Development Plan (RPJP) and the Indonesian Bureaucratic Reform Grand Design (2010-2025), SPBE envisions an integrated electronic-based government system fostering a high-performance bureaucracy and enhanced public services (indonesia.go.id, 2022). To propel the digital transformation agenda, President Joko Widodo outlined five key steps during a limited cabinet meeting in August 2020: Expand access and enhance digital infrastructure; Develop a digital transformation roadmap for strategic sectors, spanning government, public services, social assistance, education, health, trade, industry, and broadcasting; Accelerate the integration of the National Data Centre; Prepare human resources with digital talent for effective digital transformation; Develop funding and financing schemes for 40 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION digital transformation regulations promptly. These strategic steps emphasize the commitment to advancing digital capabilities, fostering innovation, and ensuring a comprehensive and inclusive digital transformation across various sectors in Indonesia (indonesia.go.id, 2022). In Indonesia, as a unitary state, local governments at the provincial and city-district levels hold autonomous authority but are subordinate to the central government. To address complex service delivery issues, major cities in Indonesia adopt e- government as a strategic solution. Regulatory support and government policies encourage local governments to leverage digital technologies for effective collaboration with citizens, businesses, civil society, and other government entities, facilitating digital transformation. The implementation of digital governance in different local practices reveals dilemmas, paradoxes, and tensions (Bertot & Jaeger, 2008; Dias & Gomes, 2021; Savoldelli et al., 2014). Despite the uneven progress in Indonesia's digital transformation, with cities like Jakarta, Bandung, Surabaya, and Medan facing chal enges, the granting of broad autonomy to local governments has been a contributing factor. The chal enges of the digital age are particularly evident in less-developed regions, where central government support, appropriate infrastructure, and human resources are crucial for accelerating digital transformation through SPBE. Digital transformation presents both opportunities and challenges for local governments, impacting public policies such as equal access to education, infrastructure, public transport, and health services. The SPBE framework enables collaboration with various sectors, including NGOs, to address public demands. The concept of digital government is anticipated to assist local governments in formulating new and innovative policies using ICTs (Anderson et al., 2015). To gauge the progress of the Electronic-Based Government System (SPBE) in both central and local governments, the Ministry of Administrative and Bureaucratic Reform issued Ministerial Regulation No. 5 of 2018. This regulation outlines the monitoring and evaluation framework for electronic-based government systems, conducted annually to measure the maturity level of SPBE implementation in ministries, institutions, and local governments. The resulting SPBE Index serves as a key indicator in assessing the modernization of governance structures in R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 41 Indonesia1. Various Central and Regional Government Agencies have undertaken SPBE initiatives, enhancing the efficiency and effectiveness of governance. However, the outcomes and maturity levels of SPBE development vary significantly across these agencies. National challenges in SPBE development include: 1. Lack of Integrated National SPBE Governance: The primary chal enge lies in the absence of a national y integrated governance framework for SPBE. 2. Incomplete Implementation of SPBE in Administration and Public Services: SPBE has not been fully and optimally applied in the administration of governance and public services. 3. Suboptimal Reach of ICT Infrastructure: The reach of ICT infrastructure across al regions and layers of society is not yet optimal. ICT infrastructure, especial y telecommunication networks, serves as the foundation for connectivity between SPBE implementers and users. 4. Limited Number of Civil Servants with ICT Technical Competence: There is a shortage of civil servants (ASN) with technical ICT competencies. Table 3: Achievement of the SPBE Index Score in Multilevel Government in Indonesia Achievement of Number of SPBE Evaluation the average SPBE index score Results Predicate Description 2019 2020 Above "Good" Below "Good" Implementation of the 2.18 2.26 247 (40.96%) 356 (59.04%) National SPBE Application of SPBE 2.74 2.9 69 (75.82%) 22 (24.18%) Central Agencies Implementation of Local 2.07 2.14 178 (34.77%) 334 (65.23%) Government SPBE Source: setneg.go.id (2022) The maturity level of Indonesia's national digital transformation faces chalenges, notably in the collaboration and integration between Central and Regional Government Agencies. To address this, the Indonesian Government introduced the 1 Interview results, Deputy for Policy Formulation and Coordination of SPBE Implementation, Ministry of State Apparatus Utilization and Bureaucratic Reform, 2023. 42 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Electronic-based Government System Architecture (SPBE), an Enterprise Architecture tailored to Indonesian characteristics. Enacted through Presidential Regulation Number 132 of 2022, SPBE serves as a foundational framework, integrating business processes, data, SPBE services, applications, infrastructure, and security for streamlined government services2. Concurrently, Indonesia's digital transformation agenda progresses with the "Towards 100 Smart City Movement" program launched in 2017. Aligned with SPBE goals, this initiative leverages information technology in government administration for enhanced efficiency. The program guides 100 cities/districts in developing Smart City master plans, showcasing exemplary execution in their regions. A collaborative effort among government agencies, the Smart City concept utilizes information technology to optimize city/district management for the col ective welfare. Figure 1: Digitalization Iniciatives in Indonesia Source: YCP Solidiance, 2020, 2022 In its trajectory, SPBE has paved the way for numerous digital projects across various sectors, contributing to Indonesia's digital transformation agenda, notably through the establishment of smart cities nationwide. Within the 100 Smart Cities Movement framework, the government acts as a key catalyst for digital transformation, recognizing the imperative role of smart cities in delivering 2 Interview results, Deputy for Policy Formulation and Coordination of SPBE Implementation, Ministry of State Apparatus Utilization and Bureaucratic Reform, 2023. R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 43 exceptional services to communities. The surge in the popularity of the smart city concept in local governments aligns with global discussions on the future of urban development. Urbanization in Indonesia has skyrocketed, with an estimated 68% of the population living in urban areas by 2025 and a projected 83% by 2045 (YCP Solidiance, 2020). Acknowledging smart cities as a viable solution to urbanization chal enges, Indonesia's central and local governments, along with key ministries, launched the 100 Smart Cities Movement. This nationwide initiative aims to implement smart cities across Indonesia by 2045, addressing unique challenges each city faces during the implementation phase. While the central government has provided comprehensive plans, regulations, and strategic national projects, the dynamic urban landscape and diverse characteristics present significant obstacles for individual cities. To overcome these challenges, cities adopt a collaborative and interdisciplinary approach, recognizing cross-disciplinary collaboration as a crucial instrument for realizing the goals of the smart city initiative (Bolívar, 2017; Herdiyanti et al., 2019) In contrast to other global cities, the progress of smart city concepts in many Indonesian cities is hindered by a lack of knowledge in identifying suitable action steps, resulting in a slower pace of development. Immediate support from the private sector is deemed crucial to expedite progress. The Indonesian government encourages collaboration with various entities, emphasizing an open approach to smart city development without limitations on collaboration types or involved companies. Significant technological gaps, such as IoT, Big Data, and AI/Machine Learning, present opportunities for both local and foreign companies with specialized expertise to contribute to smart city initiatives (YCP Solidiance, 2020). To practical y implement SPBE in the context of digital transformation, especial y in the Smart City framework, our research focuses on Jambi City in Jambi Province, Indonesia. As one of the early pioneers of smart cities in Indonesia since 2017, Jambi City, along with 25 other cities, aims to construct sustainable and competitive cities for the nation's future. The ultimate outcome of SPBE is envisioned as smart governance, guided by objectives such as digitizing public services, developing digital-capable public infrastructure, and facilitating increased business transactions, particularly through partnerships with technology companies and start-ups to enhance the growing digital ecosystem (YCP Solidiance, 2022). 44 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3.2 Smart Governance Transformation in Jambi City The Jambi City Government acknowledges the significance of incorporating digital technology and information systems into governance. This dedication and proactive stance are clearly articulated in Jambi City's Mission for the 2018-2023 period, emphasizing the foremost goal of fortifying the bureaucracy and improving community services through information technology. Commitment to digital transformation is underscored by the enactment of Regional Regulation Number 1 of 2019 on the Implementation of Smart City3. The development of Smart Governance in the city of Jambi is reflected in the Mayor of Jambi Regulation Number 89 of 2018 regarding the Jambi Smart City Masterplan. This master plan includes the Comprehensive Plan or Master Plan of Smart City Jambi for the years 2018 – 2028, outlining the vision, mission, and work program plans that serve as guidelines for the Jambi City Government, the community, and the business world in organizing and realizing Jambi as a Smart City over a 10-year period. Additionally, it serves as a direction and guide for the leaders of the Jambi City region in achieving the developmental performance of Jambi as a Smart City. Through the implementation of Smart Governance policies, it is anticipated that effective, efficient, communicative local government management wil be established, continually improving bureaucratic performance through innovation and the integrated adoption of technology (jambikota.go.id, 2022). The development plan for Smart Governance in Jambi is divided into three forms of planning: the Short-Term Plan for 2018, the Medium-Term Plan for 2018-2023, and the Long-Term Plan for 2018-2028. These plans aim to create a governance framework that is not only efficient and effective but also communicative, fostering continuous improvement in bureaucratic performance through the integration of innovative and technological advancements. 3 Interview results, Mayor of Jambi, 2023. R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 45 Table 4: Priority Program Plan of the City Government of Jambi in the Development of Smart Governance Short-Term Plan Medium-Term Plan Long-Term Plan • Strengthening the Local • Online social assistance • Public services using Government Work Plan fund management IoT (Internet of Things) Formulation system (bansos) • Community Application (RKPD) - • Electronic drug abuse participation in online, (E-musrenbang) counseling sustainable • Interoperability of e- • Government asset policymaking planning (SIMREDA) management • Optimizing the • Employee official travel information provider application portal for both citizens • Application for the and stakeholders - Open management of rooms data and assets (venue • Information management) Technology Competency Certification • Citizen problem reporting system through the application (siKesal) • Interoperability of e- government applications - Phase 1 • Management of information for the impoverished population • Management of information for Non- Governmental Organizations (ORMAS) and Political Parties Source: Own elaboration (2023) The incorporation of digital technology and information systems is a strategic initiative aimed at improving the efficiency, transparency, and accessibility of government services. This involves leveraging technology to streamline administrative processes, enhance citizen engagement, and support data-driven decision-making. At the forefront of the Jambi City Government's efforts to 46 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION embrace digital technology is the development of the SIKOJA (Sistem Informasi Kota Jambi) application, officially launched in July 2019. SIKOJA serves as the central hub for an integrated information system, consolidating various e- government service facilities in Kota Jambi. The primary objective of the SIKOJA application is to simplify public access to information about Kota Jambi, facilitating the seamless delivery of services to the community (jambiupdate.co, 2019). The SIKOJA application in Kota Jambi offers diverse services, including citizen complaints, food prices, live CCTV feeds, weather forecasts, licensing, health information, and exploring locations. As part of the Jambi City Government's digital transformation, the SIKOJA application integrates various services into a single platform. Each department (SKPD) is now required to coordinate with the Communication and Informatics Office (Diskominfo) for connectivity with SIKOJA, discontinuing independent information system development. Jambi City's notable digital progress earned recognition as one of the 25 pioneer Smart Cities in Indonesia for 2021. This achievement is attributed to the "Gerakan Menuju 100 Smart City" program, a collaborative effort involving key ministries and government offices. The program aims to guide regencies and cities in creating Smart City Masterplans, leveraging technology for improved public services and regional development (kominfo.go.id, 2017). Table 5. presents a comprehensive list of application-based government services in Jambi City, showcasing the city's commitment to leveraging technology for improved public administration. These services span various sectors, from licensing and mail management to healthcare, emergency assistance, and urban planning. The digital transformation in Jambi City signifies a profound shift in traditional government mechanisms, aligning with broader trends in Indonesia. In this era, technology serves as the cornerstone of public administration, blurring the boundaries between citizens and the government. The city's initiative reflects a commitment to widespread digitization across public service sectors, underscoring the government's dedication to technological y transforming service provision. Jambi City's digital ambitions extend to crucial areas like transportation, urban planning, security, and administrative services. The overarching goal is to enhance the efficiency and quality of public services, fostering a more modern and R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 47 competitive city4. This multifaceted approach mirrors Jambi City's resolute determination to stride confidently into a digital and inclusive future, where technology becomes a catalyst for positive change and improved governance. Table 5: List of application-based government services in Jambi City No Application Information Users’ classification 1 Silancar Jambi City One Stop Integrated Licensing Service Public and private Management of the mail system in the Jambi city 2 Sipadek Government government. Public services at district (kecamatan) and sub-district Government and 3 Sipaten (kelurahan) levels. public Sibapok, Information on prices of basic necessities in Jambi city 4 Public Siharko markets 5 e-SPPT PBB Payment of Land and Building Tax (PBB) Public and private Government and 6 Sipaduko Information on population administration services. public Services provision at the Abdul Manap Regional 7 Simenap Hospital, consist of information on bed availability, Public doctor schedules, online registration, and complaints. 8 Si Komedo Medical Consultation Services Public Ppdb Kota 9 School registration information based on zoning. Public Jambi 10 iJambikota Digital library. Public Satudata kota Management of sectoral statistical data for the Jambi Government, 11 jambi City Government. public, and privat Kliping digital Col ection of news on th Jambi City Government in 12 Public Kota Jambi print, online and television media 13 Sikesal Jambi city community complaints online. Public 14 112 Jambi City Emergency Assistance Services. Public 15 Simerahkoja Fire prevention and rescue in Jambi City. Public 16 Sp4anlapor Complaint application regarding government services. Public 17 Info Data ASN Government employee archives Government Government, 18 E Planning Digital-based regional development planning. public, and privat JCOC (Jambi 19 City Operation Integrated control and monitoring center. Government Center) Source: Own elaboration (2023) 4 Interview results, Mayor of Jambi, 2023. 48 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Various programs that have been established and implemented require strong synergy between the central and regional governments. In Jambi, in addition to support from the central government, it must be acknowledged that the commitment of the regional government plays a crucial role. In this context, the role of the mayor is crucial in allocating additional resources that are not always available in the State Budget (APBN). Sometimes, written statements are needed as a form of commitment from the regional government to provide everything needed. Regions also greatly rely on assistance from the central government to support the implementation of ongoing programs5. The digitization process in Jambi is systematically initiated through meticulous digitalization planning, which commences at the Regional Development Planning Agency (Bappeda). Serving as the epicenter for formulating strategies and programs related to digital transformation, Bappeda orchestrates specific missions for various stakeholders from related units, spanning sectors such as transportation, communication, and public services. Key players in this coordinated effort include the Communication and Informatics Department (Diskominfo) and the Transportation Office (Dishub), with the Joint Command Operation Center (JCOC) playing a pivotal role in overal coordination. The SIPADEK application stands out as a practical tool for online document management in Jambi City. This application facilitates the efficient handling of correspondence documents, whether incoming or outgoing, and their management at various levels, including the city mayor. SIPADEK empowers seamless document processing, providing flexibility for users to engage with correspondence from any location and at any time. The designation of Jambi City as one of the pioneer smart cities has spurred competition among Local Government Agencies (OPD) to actively engage in digital transformation by developing various applications. However, the success of these efforts’ hinges on the quantity and quality of available human resources within each institution. Jambi City Communications, and Information Department (Diskominfo), tasked with preparing IT professionals within the city government, employs strategic approaches such as recruiting contract personnel based on the city's needs and engaging them in training activities. The recruitment process is conducted professionally, involving higher education institutions and offering competitive compensation to attract high-quality talent. 5 Interview results, Head of Jambi City Communications, and Information Department, 2023. R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 49 Efforts to enhance human resource capabilities extend to participation in diverse training programs. Diskominfo collaborates with entities like the Regional Human Resources Development Agency (BPSDM) and organizes international training programs such as the Digital Technical Scholarship (DTS) and the Government Talent Academy. These initiatives underscore the city's commitment to nurturing a skilled workforce capable of driving digital transformation effectively6. To address citizen concerns, Jambi City offers accessible complaint services such as Sikesal, E-Lapor, and the 112-telephone service. Sikesal, initiated in 2017, serves as a dedicated platform for citizens to lodge complaints, facilitating the identification of prevalent issues in the city and the al ocation of budgets for necessary assistance. The 112 service provides emergency assistance, while E-Lapor caters to general complaints, which may not be exclusive to Jambi City. These services col ectively reflect the city's dedication to open communication and citizen engagement in the digital era. 4 Discussion Challenges On Smart Governance Implementation The research provides a comprehensive overview of the journey towards Smart Governance in Jambi City, highlighting both the city's commitment to innovation and the chal enges it faces. The discussion emphasizes the importance of leveraging technology for urban development and the mayor's leadership in driving this agenda forward. However, it also acknowledges the persistent chal enges, such as educational disparities and resistance to technology adoption, which hinder progress. In analyzing the situation, it becomes clear that achieving Smart Governance requires addressing various obstacles, as noted by Tangi et al. (2021), including skills deficiencies, organizational complexities, and coordination issues. These challenges align with the broader context of the concepts, where the delay in updating and disseminating government data underscores the need for improved skil s, streamlined processes, and better coordination among government agencies. It's essential to recognize that overcoming the chal enges identified requires a holistic approach. Jambi City's efforts towards Smart Governance should not only focus on 6 Interview results, Mayor of Jambi, 2023. 50 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION technological solutions but also address underlying issues related to skills development, organizational effectiveness, and inter-agency collaboration. By acknowledging and addressing these chal enges, Jambi City can better position itself to achieve its goals of sustainable progress and effective governance. Other chal enges arise regarding the delay in document submission from other government entities, attributed to bureaucratic procedures, administrative bottlenecks, and coordination chal enges, poses a significant obstacle to transparency and accountability initiatives, hindering the timely updating and accessibility of crucial information for the public. Addressing this challenge requires efforts to enhance data synergy and streamline information dissemination processes. Moreover, as emphasized by Wirtz & Daiser (2015), cognitive barriers rooted in individual perceptions hinder the implementation of open government data initiatives. These barriers, along with cultural norms like risk aversion and bureaucratic practices, intertwine with structural chal enges, necessitating comprehensive efforts across governance, organizational, managerial, and technical domains to overcome them. Connecting these points, it becomes evident that addressing the delay in document submission is not merely a technical issue but also involves addressing cognitive and cultural barriers, aligning with the broader goal of fostering Smart Governance practices in Jambi City. Persistent educational and skill disparities, along with resistance to technology adoption, underscore the need for a comprehensive examination of prevailing chal enges. This analysis should encompass their impact on efficiency metrics, employee compliance levels, and any unforeseen consequences in the adoption process, aiming for a nuanced understanding of barriers to technology integration. Moreover, the ongoing pursuit of an on-demand innovation program aligns with adaptive governance principles, highlighting the importance of further research into community awareness, responsiveness, perceptions, participation factors, and the effectiveness of communication strategies. Investigating these aspects wil yield valuable insights into societal dynamics and interactions shaping program success, facilitating a more informed and adaptive governance approach. Tangi et al. (2021) emphasizes deficiencies in skills, organizational intricacies, and coordination issues as primary obstacles, echoing findings on human resource and institutional aspects such as unclear institutional approaches, insufficient funding, leadership gaps, resistance to change, lack of policy guidelines, and shortages of skilled personnel R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 51 (Falco & Kleinhans, 2018; Gil-García & Pardo, 2005; Knox & Janenova, 2019; Ndou, 2004; Odat, 2012). From the research we found challenges arising from regulations and legislation, influencing development projects, and potentially leading to conflicts, highlight the complexities of governance processes. A thorough exploration of these chal enges is essential for suggesting targeted solutions, enhancing public policy effectiveness, and ensuring alignment with regulatory frameworks. This nuanced examination lays the groundwork for a more resilient and adaptable governance system, capable of navigating evolving regulatory landscapes and fostering sustainable development. Nonetheless, consensus on the implementation of smart governance remains elusive. While some research also emphasizes political involvement and administrative efficiency (Giffinger et al., 2007), others concentrate on gathering various types of data and information related to public management (Schuchmann & Seufert, 2015). Smart governance assumes a crucial role in facilitating accessibility to smart city initiatives for citizens while upholding transparency in decision-making and execution. Yet, the essence of e-governance within a smart city should center on being citizen-centric and driven by citizen participation. The expanding influence of technology in urban systems prompts governments to reevaluate their position in a knowledge-based society (Giffinger et al., 2007). Smart governance underscores the need to understand community participation dynamics, considering socio-economic factors shaping involvement levels. Proposed strategies for fostering more equitable engagement can gauge the impact of community participation on socio-economic factors, ensuring governance initiatives inclusive of diverse communities' specific needs in the future. While this research provides valuable insights into the role of digital government evolution in enhancing local government performance, it is not without limitations. Firstly, the study's focus on Indonesia's transformative journey on smart governance may limit the generalizability of findings to other contexts. Additionally, the qualitative approach employed in this research may offer rich insights but could also limit the breadth and depth of analysis compared to quantitative methods. Furthermore, the study acknowledges regulatory barriers and resource constraints as challenges but may not fully explore their nuanced impacts on smart cities initiatives. Finally, while the research identifies opportunities for success through stakeholder 52 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION collaboration and public-private partnerships, it may not fuly address the potential risks and limitations associated with these approaches. Overal , these limitations highlight areas for further research and consideration in future studies on the intersection of technology and government in developing nations. 5 Conclusion This research il uminates the ongoing evolution of smart governance in Jambi City, driven by the national Electronic-Based Government System (SPBE). This transformative initiative not only supports local governance but also aligns with the broader Smart Cities Movement, positioning Jambi City as a participant in this nationwide endeavor. SPBE, by harnessing Information and Communication Technologies (ICTs), elevates data processing efficiency and decision-making capabilities, embodying the principles of the Smart City concept, particularly in the provision of public services. This technology-driven approach, embodied by SPBE, contributes significantly to the development of intel igent and sustainable urban environments. The system integrates data and technology, providing valuable insights to address urban chal enges and enhance residents' quality of life. By facilitating effective data col ection across various city facets, SPBE enables targeted strategies for resource optimization and sustainability, aligning seamlessly with the Smart City framework and fostering comprehensive urban transformation. Jambi City exemplifies its commitment to this digital journey through initiatives like the SIKOJA Application and the City Operation Center (JCOC), showcasing technological advancements aimed at efficient governance and improved public services. While centralistic strategies continue to demonstrate effectiveness, persistent challenges, such as the digital divide and difficulties in data integration due to a lack of interoperability and single data integration, highlight the need for sustainable solutions. Jambi City's dedication to digital transformation is commendable; however, addressing unique socio-cultural landscapes is imperative. Chal enges related to infrastructure and human resources underscore the complexity of this journey. The city emerges as a representative case, emphasizing the necessity for inclusive approaches and comprehensive strategies in diverse societies pursuing digital transformation. As Jambi City navigates through these challenges, its experiences serve as valuable lessons for other regions undertaking similar transformative endeavors. R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 53 References Anderson, D., Wu, R., Cho, J. S., & Schroeder, K. (2015). E-Government Strategy, ICT and Innovation for Citizen Engagement. Springer New York. https://books.google.si/books?id=VIdNCwAAQBAJ Bertot, J., & Jaeger, P. (2008). The E-Government paradox: Better customer service doesn’t necessarily cost less. Government Information Quarterly - GOVT INFORM QUART, 25, 149– 154. https://doi.org/10.1016/j.giq.2007.10.002 Bolívar, M. P. R. (2017). Smart Technologies for Smart Governments: Transparency, Ef iciency and Organizational Issues. Springer International Publishing. https://books.google.si/books?id=vYotDwAAQBAJ Bolívar, M. P. R., & Meijer, A. J. (2016). Smart governance: Using a literature review and empirical analysis to build a research model. Social Science Computer Review, 34(6), 673–692. BP2D Jawa Barat. (2022). Kajian Evaluasi Optimalisasi Kebijakan Sistem Pemerintahan Berbasis Elektronik (SPBE) di Provinsi Jawa Barat. Badan Penelitian dan Pengembangan Daerah Provinsi Jawa Barat Badan Penelitian dan Pengembangan Daerah Provinsi Jawa Barat. Brayda, W. C., & Boyce, T. D. (2014). So you real y want to interview me?: Navigating “sensitive” qualitative research interviewing. International Journal of Qualitative Methods, 13(1), 318–334. Carayannis, E. G., & Hanna, N. K. (2016). Mastering Digital Transformation: Towards a Smarter Society, Economy, City and Nation. Emerald Group Publishing Limited. https://books.google.si/books?id=g2BoCwAAQBAJ Creswel , J. W. (2013). Qualitative Inquiry and Research Design: Choosing Among Five Approaches. SAGE Publications. https://books.google.si/books?id=Ykruxor10cYC Creswel , J. W., & Creswel , J. D. (2018). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. SAGE Publications. https://books.google.si/books?id=s4ViswEACAAJ Dias, R., & Gomes, M. (2021). From Electronic Government to Digital Governance: Transformation Governance Models and Strategies. Ciências e Políticas Públicas / Public Sciences & Policies, 7, 119–143. https://doi.org/10.33167/2184-0644.CPP2021.VVIIN1/pp.119-143 Eisenhardt, K. M. (1989). Building theories from case study research. Academy of Management Review, 14(4), 532–550. Falco, E., & Kleinhans, R. (2018). Beyond technology: Identifying local government chal enges for using digital platforms for citizen engagement. International Journal of Information Management, 40, 17–20. https://doi.org/10.1016/j.ijinfomgt.2018.01.007 Gano, G. (2013). The New Face of Government: Publics and Policymaking Online. Journal of Public Administration Research and Theory: J-PART, 23(3), 755–758. JSTOR. Giffinger, R., Fertner, C., Kramar, H., Kalasek, R., Milanović, N., & Meijers, E. (2007). Smart cities— Ranking of European medium-sized cities. Gil-García, J. R., & Pardo, T. A. (2005). E-government success factors: Mapping practical tools to theoretical foundations. Government Information Quarterly, 22(2), 187–216. Hancock, D. R., Algozzine, B., & Lim, J. H. (2021). Doing case study research: A practical guide for beginning researchers. Herdiyanti, A., Hapsari, P., & Susanto, T. (2019). Modelling the Smart Governance Performance to Support Smart City Program in Indonesia. Procedia Computer Science, 161, 367–377. https://doi.org/10.1016/j.procs.2019.11.135 Howes, S., & Bishop, T. K. (2018). The hidden obstacles to government digital transformation. https://www.instituteforgovernment.org.uk/publications/hidden-obstacles-government- digital-transformation indonesia.go.id. (2022, February 1). Pemerataan Infrastruktur TIK Dukung Akselerasi Transformasi Digital [Government]. Https://Www.Indonesia.Go.Id/. https://www.indonesia.go.id/kategori/editorial/3580/pemerataan-infrastruktur-tik-dukung- akselerasi-transformasi-digital%20 jambikota.go.id. (2022, Agustus). Visi & Misi Kota Jambi (2018—2023) [Government]. Jambikota.Go.Id. https://jambikota.go.id/content/visi_misi 54 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION jambiupdate.co. (2019, July 29). Pemkot Jambi Launching Aplikasi SIKOJA, Ini Tujuannya. https://www.jambiupdate.co/artikel-pemkot-jambi-launching-aplikasi-sikoja-ini- tujuannya.html Janowski, T. (2015). Digital government evolution: From transformation to contextualization. Government Information Quarterly, 32, 221–236. https://doi.org/10.1016/j.giq.2015.07.001 Knox, C., & Janenova, S. (2019). The e-government paradox in post-Soviet countries. International Journal of Public Sector Management, 32(6), 600–615. kominfo.go.id. (2017, November 28). Langkah Menuju “100 Smart City.” https://www.kominfo.go.id/content/detail/11656/langkah-menuju-100-smart- city/0/sorotan_media kominfo.go.id. (2020, October 10). Penerapan SPBE dan Rencana Pembangunan Pusat Data Nasional. https://aptika.kominfo.go.id/2020/10/penerapan-spbe-dan-rencana-pembangunan-pusat- data-nasional/ Meijer, A., Rodríguez, B., & Pedro, M. (2015). Governing the smart city: A review of the literature on smart urban governance. International Review of Administrative Sciences, 82. https://doi.org/10.1177/0020852314564308 Mergel, I., Edelmann, N., & Haug, N. (2019). Defining digital transformation: Results from expert interviews. Government Information Quarterly, 36(4), 101385. https://doi.org/10.1016/j.giq.2019.06.002 Nam, T., & Pardo, T. (2011). Conceptualizing smart city with dimensions of technology, people, and institutions. ACM International Conference Proceeding Series, 282–291. https://doi.org/10.1145/2037556.2037602 Ndou, V. (2004). E – GOVERNMENT FOR DEVELOPING COUNTRIES:OPPORTUNITIES AND CHALLENGES. Electronic Journal of Information Systems in Developing Countries, 18, 1–24. Odat, A. M. (2012). E-government in developing countries: Framework of chal enges and opportunities. 2012 International Conference for Internet Technology and Secured Transactions, 578– 582. Pittaway, J. J., & Montazemi, A. R. (2020). Know-how to lead digital transformation: The case of local governments. Government Information Quarterly, 101474. https://doi.org/10.1016/j.giq.2020.101474 Rakhmanov, E. (2009). The barriers affecting e-government development in Uzbekistan. 2009 Fourth International Conference on Computer Sciences and Convergence Information Technology, 1474–1480. Saldana, J. (2014). Thinking Qualitatively: Methods of Mind. SAGE Publications. Savoldelli, A., Codagnone, C., & Misuraca, G. (2014). Understanding the e-government paradox: Learning from literature and practice on barriers to adoption. Government Information Quarterly, 31, S63–S71. https://doi.org/10.1016/j.giq.2014.01.008 Schuchmann, D., & Seufert, S. (2015). Corporate Learning in Times of Digital Transformation: A Conceptual Framework and Service Portfolio for the Learning Function in Banking Organisations. International Journal of Advanced Corporate Learning (iJAC), 8. https://doi.org/10.3991/ijac.v8i1.4440 setneg.go.id. (2022). Peraturan Presiden (PERPRES) Nomor 132 Tahun 2022 Tentang Arsitektur Sistem Pemerintahan Berbasis Elektronik Nasional. LN.2022/No.233, jdih.setneg.go.id: 5 hlm. Shaheen, A., & Tassabehji, R. (2007). Measuring E-Government: Chal enges and Opportunities for the Developing World Measuring E-Government: Chal enges and Opportunities for the Developing World. Strauss, A., & Corbin, J. (1998). Basics of qualitative research techniques. Tangi, L., Janssen, M., Benedetti, M., & Noci, G. (2021). Digital government transformation: A structural equation modelling analysis of driving and impeding factors. International Journal of Information Management, 60, 102356. https://doi.org/10.1016/j.ijinfomgt.2021.102356 UNPA and ASPA. (2001). Benchmarking E-government: A Global Perspective 2001. American Society for Public Administration. https://desapublications.un.org/publications/benchmarking-e-government-global-perspective-2001 R. Y. Maulana et al.: Smart Governance Chal enges in Indonesian Local Government 55 Wirtz, B. W., & Daiser, P. (2015). e-Government: Strategy process instruments. Speyer : German University of Administrative Sciences Speyer, Chair for Information and Communication Management. http://nbn-resolving.de/urn:nbn:de:101:1-2015091658 YCP Solidiance. (2020). Can Indonesia Achieve ‘100 Smart Cities’ by 2045? https://ycpsolidiance.com/white-paper/can-indonesia-achieve-100-smart-cities-by-2045 YCP Solidiance. (2022, March). The Digital Transformation of Indonesia’s Government. Https://Ycpsolidiance.Com. https://ycpsolidiance.com/article/indonesia-government- digital-transformation Yin, R. K. (2017). Case Study Research and Applications (International Student Edition): Design and Methods. SAGE Publications. https://books.google.si/books?id=tHuiswEACAAJ 56 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION METAVERSE IN HIGHER EDUCATION – A SYSTEMATIC LITERATURE REVIEW ADRIAN WOICK,1, 2 HEIDI RINN,1 LINDA GROGORICK,3 TAMARA MÜHLEISEN,1 DANIEL MARKGRAF1 1 AKAD University, Institute for Digital Expertise and Assessment, Stuttgart, Germany adrian.woick@akad.de, heidi.rinn@akad.de, tamara.muehleisen@akad.de, daniel.markgraf@akad.de 2 Brandenburg University of Technology, Institute of Business and Economics, Cottbus, Germany woickadr@b-tu.de 3 Technische Universität Braunschweig, Institute for Business Information Systems, Braunschweig, Germany l.grogorick@tu-bs.de Due to the COVID-19 pandemic and the resulting restrictions, the need for a rapid conversion of teaching to digital formats has increased significantly. Not al teaching formats and content are suitable for traditional video conferencing, so the Metaverse, an interconnection of virtual worlds, has experienced a significant upswing in the education sector. Therefore, we conduct a systematic literature review to determine the current state of research on the Metaverse in higher education and to identify its definitions, benefits and chal enges, types, and technologies. The Keywords: initially found 5,539 papers were systematically filtered to 92 fully metaverse, coded articles. Our findings reveal a lack of standardized higher education, definitions, early-stage prototyping, a lack of prescriptive design immersive knowledge, and a lack of pedagogical and methodological learning, concepts and blueprints. These findings reveal significant systematic literature research gaps and lead to the derivation of future research review, streams. pandemic DOI https://doi.org/10.18690/um.fov.4.2024.4 ISBN 978-961-286-871-0 58 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Amid the COVID-19 pandemic and its resulting restrictions, the Metaverse, an interconnection of virtual worlds, saw a significant surge in the education sector, as teaching swiftly shifted to digital formats (Chamorro-Atalaya et al., 2023; John Lemay et al., 2023). The Metaverse offers numerous benefits in education, such as social and collaborative aspects (Lin et al., 2022; López et al., 2022), but also faces challenges like high requirements and costs (J.-E. Yu, 2022). Understanding the development and current state of the Metaverse research in education is crucial, especially given its impact not just as a technological innovation, but also on pedagogical concepts and learning methods (Lin et al., 2022; Prakash et al., 2023). There are already several reviews that address education in the Metaverse (e.g., Roy et al., 2023; Samala et al., 2023; Sunardi et al., 2022; Tlili et al., 2022). Recent literature reviews focus on bibliometric aspects (X. Chen et al., 2023; De Felice et al., 2023) or a limited number of articles (Asiksoy, 2023; Chamorro-Atalaya et al., 2023; López- Belmonte et al., 2023). They also cover specific solutions like Roblox in educational settings (J. Han et al., 2023), student engagement in the Metaverse (Asiksoy, 2023), or virtual/augmented reality (Chua & Yu, 2023). Roy et al. (2023) and Tlili et al. (2022) provide comprehensive insights into the Metaverse in education but identify the need for further research. De Felice et al. (2023) recommend continuously reviewing developments in the Metaverse. The Metaverse in higher education is only considered by Chamorro-Atalaya et al. (2023). However, they only included 16 articles in their systematic literature review (SLR), all published before 2020. Higher education is a constantly evolving sector, making it a favorable field for applying disruptive technologies (Zuñiga et al., 2021). Our work aims to present a current holistic overview of the Metaverse research in higher education by conducting a SLR. To this aim, we derive two main research questions: RQ 1: What is the status quo of research on Metaverse in higher education? RQ 2: Which future research directions exist in Metaverse in higher education? Our paper is structured as follows: detailed methodology (section 2), results and future research areas (sections 3 & 4), and key findings in the conclusion. A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 59 2 Methodology We conducted a SLR based on Page et al. (2021) and Schoormann et al. (2021). Our search spanned the databases AIS eLibrary, Taylor&Francis, ACM Digital Library, Scopus, and IEEE Xplore to include journal articles and conference proceedings from interdisciplinary fields such as computer science, education, and pedagogy. The SLR was conducted in December 2023 using the fol owing search phrase: ALL ("Metaverse" AND "Education"). Our search query yielded 5,539 hits. Figure 1, which is based on the PRISMA flowchart by Page et al. (2021), describes our selection process. Exclusions were based on criteria aligning with our research focus and article timeliness, namely: children, disability, the virtual world (VW), Second Life, systematic literature reviews, and languages other than English. VWs, like Second Life, experienced their hype around 2009 (Rinn, Khosrawi-Rad, et al., 2023). We consider these as precursors and part of the history of the Metaverse development. Due to the technological progress since then, we exclude both. Applying our predefined exclusion criteria, we narrowed down our dataset to 92 articles. Four coders, each with a background in business studies and information systems, systematical y analyzed these using MAXQDA software, following Bandara et al. (2015). Their expertise spanned several relevant fields including game-based learning, virtual reality learning, artificial intelligence in education, and design science. We drew our initial deductive coding scheme from Duncan et al. (2012) taxonomy and our research goals. We utilized a coding manual as Mayring (2015) recommended for consistency. A peer review process was established for objectivity. 3 Results The following morphological box following Ritchey (2011) previews the resulting coding categories. 60 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION AIS eLibrary Taylor & Francis ACM Digital Library Scopus IEEE Xplore (n = 140) (n = 423) (n = 447) (n = 4.271) (n = 258) 5,539 hits in total 14 duplicates removed 5,525 titels screened inclusion criteria 4,836 publications removed Multiuser Environment, higher/further Education, Research/Conference after title screening* Paper, Interoperability 689 abstracts screened 569 publications removed after abstract screening* ications 120 publications read 3 publications removed 0 publications added d publ due to lack of access* after backward/forward search 25 publications removed 117 publications read exclude due to non-matching content* *exclusion criteria 92 final y coded publications Children, disability, virtual worlds, Second Life, systematic literature review, withdrawn, no access Figure 1: PRISMA Statement Table 1: Morphological box on resulting categories Definitions Metaverse Edu-Metaverse Virtual World Type of Paper Concept Paper Prototype Paper Model Development Architecture Development In General Risk-free Fun & Students Learning Motivating Activation Advantages Game-based Adaptive & High Learning Learning Individualized Outcomes Stimulating Curriculum Transferability to Learning Flexibility Practice High Requirements Users Health Concerns Chal enges High Costs Usability Data Privacy Availability of Content Fear of Losing Focus Ethics & Principles Accessibility Inclusiveness Type of Metaverse Virtual World AR Mirror Worlds Life Logging Technologies VR AI AR Blockchain & NFT 3.1 Definition of Metaverse There is yet no uniform definition for the Metaverse. 77 of the 92 coded papers contain a definition of the term Metaverse, 10 do not define it (e.g., C. Ho, 2022; Iakovides et al., 2022; D. Yu, 2022), 1 defines VWs and 4 define the Edu-Metaverse A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 61 instead. Regarding the term origin, there are two approaches found: the science fiction novel that spawned the term (e.g., Mantoro et al., 2022; Troja et al., 2023; Y. Zhang et al., 2022) and the word creation (e.g., Iwanaga et al., 2023; J. Lee & Jang, 2023; J.-E. Yu, 2022). Since the Metaverse is yet an outlook in many aspects, the definitions are at least partial y prescriptive. There are two directions and a combination of both that was found. First, the technology-oriented definitions that describe how the Metaverse should be built. Second, the vision-oriented perspective explains in varying degrees of detail what action the Metaverse will allow us to do. These can be general like communication and social interaction (e.g., Jacobs et al., 2023; Mitra, 2023) or specific like synchronous learning (e.g., Almarzouqi et al., 2022) or virtual field trips and museum visits (Abraham et al., 2023). The 77 definitions contain one or more references. The most cited references are Mystakidis (2022) with 11 citations, Kye et al. (2021) with 7, Park & Kim (2022), and Hwang & Chien (2022) with 5 citations each. There are 31 definitions without a reference (e.g., W. Ho & Lee, 2023; Kim et al., 2023; Yue, 2022). The aspects taken from Mystakidis (2022) are the Metaverse being described as a multi-user VW combining physical and virtual reality. This computer-generated world is decentralized and persistent, enabling inhabitants to communicate and interact with each other (e.g., Al-Kfairy et al., 2022; Mitra, 2023). Economic and cultural usage examples are named (Wu et al., 2023). Technologies cited are virtual, augmented, and mixed reality, AI, and blockchain (e.g., Mitra, 2023; Onecha et al., 2023). Kye et al. (2021) are referenced when defining the Metaverse as an interactive, three-dimensional environment not limited to the VW, entered with a smartphone or computer via the internet (e.g., Iwanaga et al., 2023; J.-E. Yu, 2022). Park & Kim (2022) are cited with aspects combining reality and virtuality with technologies like augmented and virtual reality (e.g., Z. Chen, 2022; Joshi & Pramod, 2023). But they are also referenced for Second Life being classified as Metaverse (Al-Kfairy et al., 2022). Hwang & Chien (2022) are referenced for the Metaverse being a “new social connection method” (Pangsapa et al., 2023, p. 2). 3.2 Types of Paper 53 papers (58%) do not cover one specific field of education (e.g., Raj et al., 2023; Yuan et al., 2023) but are on education in general. Furthermore, there are four main types of research papers identified (Table 2). 62 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 2: Types of Paper Types of Paper n Reference Examples Concept Paper 48 (Yuan et al., 2023; X. Zhang et al., 2022) Prototype Paper 26 (Sin et al., 2023; Song et al., 2023) Model Development 16 (Jacobs et al., 2023; Jang & Kim, 2023) Architectural Development 2 (Abraham et al., 2023; Joshi & Pramod, 2023) Concept papers dominate the analyzed literature with 48 mentions. These papers classify the Metaverse and its educational applications, emphasizing their potential and chal enges (e.g., Al-Adwan & Al-Debei, 2023). Prototype papers present instantiations and evaluations. It is noticeable that virtual reality (VR) is primarily used (e.g., Araújo et al., 2023). It is often combined with augmented reality (AR) (e.g., López et al., 2022) or artificial intelligence (AI) (e.g., Z. Chen, 2022). These papers showcase diverse applications of the Metaverse, ranging from enhancing attention in virtual design classes (Araújo et al., 2023) to improving language skills (Cantone et al., 2023). The most common applications are virtual 3D classrooms. Ibili et al. (2023) describe a virtual classroom in “Spital” to teach computer hardware. They also explore the personalization of learning with AI-based systems (D. Yu, 2023) and the increase of student collaboration and playful engagement (Guillén- Yparrea & Hernández-Rodríguez, 2023). Gamification is used, e.g., in Damaševičius and Sidekersniene (2023) who added list rankings, badges, betting lists, and achievement levels for engagement and better learning outcomes. None of the prototype papers apply the design science research paradigm which ensures the prototype has a decent theoretical grounding (Hevner, 2007). Short papers dominate prototype papers (16 from 26) and concept papers (29 from 48). Model development papers focus on creating, formulating, or validating models. These models are mainly (10 out of 16 documents) based on wel -established acceptance models like the Technology Acceptance Model (TAM) and Unified Theory of Acceptance and Use of Technology (UTAUT). For instance, Alhalaybeh et al. (2023) propose a model to assess user satisfaction, while Jacobs et al. (2023) add additional influencing factors in the context. Laine and Lee (2023) investigate and evaluate the presence, opportunities, chal enges, and potential of collaborative applications of VR by simultaneous users. 13 of the 92 articles (14%; e.g., Al-Adwan & Al-Debei, 2023; Kalınkara & Özdemir, 2023) deal with aspects of acceptance research that A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 63 emerged in the 1980s (e.g., Davis, 1989). Architectural development papers focus on creating and analyzing digital structures and frameworks. For instance, Joshi & Pramod (2023) describe a decentralized architecture. 3.3 Advantages of Metaverse Advantages of the Metaverse were coded if they could be evaluated as a result of the analyzed article, but not if they came exclusively from references. We clustered the exploratory col ected benefits into two areas: “Metaverse in General” and “Metaverse in Education”. This division allows us to distinguish underlying general and context-specific characteristics reflected in Table 3 and Table 4. In total, 54 of the 92 articles (59%) analyzed benefits. Table 3: Advantages of the Metaverse in General Advantages of Metaverse in General n Reference Examples Immersive Experience 28 (N. Lee & Jo, 2023; Y. Zhang et al., 2022) Social & Collaborative 22 (G.-J. Hwang et al., 2023; Lin et al., 2022) Location Flexibility 16 (Al-Kfairy et al., 2022; Hussain, 2023) Time Flexibility 13 (Hines & Netland, 2022; Lin et al., 2022) Saving Costs 11 (Al-Kfairy et al., 2022; Braguez et al., 2023) Customization/Creation 9 (Kim et al., 2023; López et al., 2022) Equality 8 (Braguez et al., 2023; Hussain, 2023) Visualization 8 (Lin et al., 2022; Onecha et al., 2023) Saving Environment 7 (Al-Kfairy et al., 2022; Hussain, 2023) Interactivity 5 (Braguez et al., 2023; Zhao et al., 2022) Due to educational innovation, the most frequently cited benefit is that the Metaverse enhances students' fun and motivation. Furthermore, the possibility of adaptive and individualized learning (e.g., personalized content) is often highlighted, which is also associated with a high level of student activation. Practicing dangerous situations without risk is another advantage. Stimulating learning experience, partially playful, reports improved learning outcomes in early studies. The flexibility and transferability of learning from the virtual to the real world are other advantages that have not been discussed in detail in many articles. 64 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 4: Advantages of the Metaverse in Education Advantages of Metaverse in Education n Reference Examples Fun & Motivating 22 (Alvarez et al., 2023) Students’ Activation 15 (Hedrick et al., 2022; W. Ho & Lee, 2023) Game-based Learning 12 (Kim et al., 2023) Adaptive & Individualized 11 (Fu & Pan, 2022; Kurniawan et al., 2023) High Learning Outcomes 9 (Alvarez et al., 2023; Kshetri et al., 2022) Risk-free Learning 8 (Kshetri et al., 2022; Ruwodo et al., 2022) Stimulating Learning 8 (W. Ho & Lee, 2023; Y. Hwang, 2023) Curriculum Flexibility 6 (Z. Chen, 2022; Lin et al., 2022) Transferability to Practice 4 (Braguez et al., 2023) 3.4 Challenges of Metaverse We explored the chal enges and then clustered them. Table 5 shows the results. 36 (39 %) analyzed articles addressed the chal enges of the Metaverse. Table 5: Chal enges of Metaverse Chal enges of Metaverse n Reference Examples High Requirements 22 (Onecha et al., 2023; Troja et al., 2023) High Costs 17 (Braguez et al., 2023; Z. Chen, 2022) Data Privacy 16 (Al-Kfairy et al., 2022; López et al., 2022) Ethics & Principles 9 (Z. Chen, 2022; Iwanaga et al., 2023) Users Health Concerns 9 (Hines & Netland, 2022; Raj et al., 2023) Usability 7 (Hedrick et al., 2022; Kim et al., 2023) Availability of Content 6 (Onecha et al., 2023; Zhao et al., 2022) Accessibility 5 (Abraham et al., 2023; Hussain, 2023) Fear of Losing Focus 4 (Al-Kfairy et al., 2022; Troja et al., 2023) Inclusiveness 4 (Hussain, 2023; Lin et al., 2022) A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 65 High requirements and costs were the top chalenges mentioned. The requirements include high computational demands (e.g., Lin et al., 2022), but also equipment requirements (e.g., Abraham et al., 2023), necessary support services for different operating systems and devices (e.g., Z. Chen, 2022), time resources (e.g., Braguez et al., 2023), and digital literacy skills (e.g., Lin et al., 2022). Costs are mainly high development costs (e.g., Z. Chen, 2022), but also labor-intensive preparation (e.g., Hines & Netland, 2022). Students' concerns about their data and the security of the system were also frequently addressed. Ethical aspects and principles must be considered or created to regulate the Metaverse. In addition, the consideration of users' health concerns is important and integrates e.g., cybersickness (e.g., Braguez et al., 2023), disorientation, and risk of addiction (e.g., Z. Chen, 2022). Other challenges include current usability (often as beta software) and availability of educational content, in part due to high production costs. Accessibility, especially in remote areas, users' fear of losing concentration, and the risk that the Metaverse may offer less social interaction compared to current educational methods are some of the chal enges mentioned as wel as inclusiveness, but less often addressed in the analyzed articles. 3.5 Types of Metaverse Out of a total of 92 fully analyzed articles, 90 addressed the types of the Metaverse according to the 2006 Metaverse Roadmap (Smart et al., 2007). Multiple nominations were possible. VW is mentioned 42 times (e.g., Pangsapa et al., 2023; Raj et al., 2023), AR is mentioned 20 times (e.g., X. Han et al., 2022; López et al., 2022), mirror worlds (MW) is mentioned 11 times (e.g., C. Ho, 2022; Iakovides et al., 2022) and lifelogging (LL) is mentioned 9 times (e.g., Mantoro et al., 2022; Wu et al., 2023). VWs simulate a virtual environment, parallel to the physical world, in which users can interact via digital avatars. (Areepong et al., 2022). Studies highlight the role of VR, AR, and MR in fostering immersive learning (Al-Adwan & Al-Debei, 2023; Alhalaybeh, Alkhatib, et al., 2023). Platforms like “FrameVR” and “Virbela” offer virtual campus experiences including out-of-class activities (Frydenberg & Ohri, 2023; Liang et al., 2023). AR enriches the physical environment with interactive 3D elements in real time (BenedettDörr & BeatrysRuizAylon, 2023). It fosters immersive, col aborative experiences in a hybrid setting, meaning the synchronous encounter of physical and virtual participants (Alhalaybeh, Alkhatib, et al., 2023). AR 66 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION can be combined with VR for extended reality (XR) (Alkhwaldi, 2023). MWs are VWs that copy a real area or building 1:1 (BenedettDörr & BeatrysRuizAylon, 2023), e.g., the virtual copy of Limassol University Library (Iakovides et al., 2022). These MWs are supplemented by technologies like “Azure Digital Twins” and the Internet of Things (IoT). These technologies provide synchronization with physical locations in addition to physical representation (Kryvenko & Chalyy, 2023). MWs may even include real-time location data (López et al., 2022). LL uses wearable technology to col ect personal data (e.g., heart rate, sleep duration, steps, calorie expenditure), aiding in sectors like health, education, and well-being. It enables the recording of activity patterns, levels of engagement, and the impact on learning activities. (BenedettDörr & BeatrysRuizAylon, 2023; López et al., 2022). 3.6 Technologies Frequent topics are VR with 23 mentions (e.g., Purahong et al., 2022; D. Yu, 2022), AI with 14 mentions (e.g., Z. Chen, 2022; Lin et al., 2022), AR with 12 mentions (e.g., López et al., 2022; Onecha et al., 2023) and blockchain and non-fungible tokens (NFT) with 12 mentions (e.g., Fu & Pan, 2022; Mantoro et al., 2022). Other technologies mentioned include 5G/6G (9 mentions), digital twins (8 mentions), XR (7 mentions), IoT (6 mentions), and others. VR enhances immersive learning experiences with the use of head-mounted displays (Riva et al., 2007). Araujo et al. (2023) describe the use of “FrameVR“, a software that al ows 15 people to collaborate with or without a head-mounted display. Yu (2023) emphasizes that VR enables multisensory experiences through wearable devices and motion sensors e.g., for training chemical experiments and archeological excavations. Sin et al. (2023) show that VR engages students more, and their engagement can improve student performance, while Hines and Netland (2022) point to physical chal enges caused by prolonged VR use. Furthermore, papers illustrate the growing role of AI in education within the Metaverse. For instance, AI-controlled NPC tutors are intended to support students individually in learning (Agrati, 2023). The accessibility and distribution of technological resources pose chal enges especial y when transferring large amounts of data (J. Lee & Kim, 2023). AI-enhanced educational programs use adaptive mechanisms to tailor learning based on individual student needs, capabilities, weaknesses, and interests (Weng et al., 2023; D. Yu, 2023). Generative AI is also used to design and adapt the environment A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 67 and thus support teachers (W. Ho & Lee, 2023). AR is seen by Damaševičius & Sidekersniene (2023) as a key technology for interactive learning. They emphasize the need for simple, accessible AR content without programming skil s. These include, e.g., 3D models of the environment, characters, and interactive objects or entire simulation scenarios, which may be adapted without much effort. Onecha et al. (2023) and Raj et al. (2023) emphasize the benefits of AR in augmenting the physical learning environment and providing real-time feedback. Blockchain technology is highlighted in theory to secure and personalize learning content while ensuring transparent and secure transactions in the Metaverse (Al-Adwan & Al- Debei, 2023; Mourtzis et al., 2023). It enables authentication and protection of digital rights (Weng et al., 2023) and forms the basis for economic interactions using NFTs and digital currencies (Joshi & Pramod, 2023; López et al., 2022). 4 Discussion of Research Gaps We systematical y elicited research gaps from the findings within our code system and gave hints on potential future research streams. There is no clear definition of the Metaverse. Such inconsistency is not new and was already encountered by the predecessor VW in education (e.g., Girvan, 2018). However, it means that there are no clear distinction criteria for Metaverse towards VW. Hence, a comparison of study results e.g., for meta-studies is impossible. Instead, many papers paint a vision of the metaverse being the next generation of the internet being more social and added by technologies like VR, blockchain and other emerging technologies. Concept papers predominate over practical papers. These rather visionary concept papers reveal three times more benefits than chal enges. This unbalanced view might arise from the lack of practical implementations. Disadvantages or challenges are likely to be still unknown. We recommend further practical research in combination with al compatible technologies. Technologies such as blockchain and 5G/6G have not yet arrived in practical research. However, these technologies are necessary to implement the vision of the Metaverse, which includes lawful contracts or ubiquitous access. This lack of emerging technologies in practice also reflects the level of prototypes that ignore data security, data protection, and ethical considerations. Furthermore, the educational field is not specified in most papers, which leads to a generalization that is insufficient for a successful transfer to practice. Since short papers dominate in 68 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION concept and prototype papers, we conclude that the development is currently rapidly progressing, and the topic has high relevance within the research community. Since prototypes are stil infant, disillusionment is likely to occur. In the context of model development type of papers, the predominant source model originates from acceptance research which evaluates the probability of future and regular use of an IT artefact within the target group. Furthermore, the prototypes lack an appropriate theoretical foundation in terms of design research. As a result, the instantiations remain context-bound and there is a lack of generalized and prescriptive design knowledge on future prototypes that could be built upon. Furthermore, this lack also exists in the pedagogical design of content for the Metaverse. In the context of VW, Rinn et al. (2023) proposed a fair-like course design for academic writing as part of a design science research study. Such blueprints or even reference books for teaching methodologies in the Metaverse are missing. These are necessary to increase the adoption of lecturers and scalability for faculties. As a result, 3D classroom environments are used in many contributions. These are often adopted without reflection and unchanged from face-to-face teaching. Consequently, the potentials such as virtual labs or gamified content are not ful y exploited yet. These application examples require interactive 3D objects. Such labor-intensive adaptations and developments require low or no code editors for their broad application. Standardized formats for exchanging these between different Metaverses would further increase scalability and efficiency. 5 Conclusion We conducted an SLR and identified 92 relevant articles we included in our analysis. To answer RQ1 we found the following main categories: Definitions, advantages, and chal enges, different types of the Metaverse, and included emerging technologies. To answer RQ2 we identified research gaps, discussed them with research findings from the community, and derived research streams for future research. These are summarized in the fol owing figure. A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 69 Current Gap Research Streams Arising Potential 1 Clear differentiation Lack of clear definition criteria VW vs. Comparability of results Metaverse 2 Infancy of prototypes Holistic integration of Increasingly realistic emerging technologies assessment of potential 3 Lack of prescriptive Deriving design Supporting researchers design knowledge principles and practitioners in metaverse design Lack of pedagogical and 4 Lowering the barrier to methodological concept Expand current entry and increased and blueprints methodology references scalability Figure 2: Overview of future research streams This publication is subject to restrictions. Additional search databases, languages, and search term synonyms may lead to different results. Despite the peer review for paper selection and coding, the process still produces subjectivity. The study reveals implications for research and practice. We identified research gaps and future research directions. The integrations of technologies like 5G/6G and blockchain are rather theoretical at this point in time, practical evaluations and artifacts are still missing. Furthermore, theoretical y sound guidelines should be developed to address spatial design in the Metaverse. In practice, our study provides a starting point for the conceptualization and implementation of the Metaverse in educational contexts. For a practical application in a regular operation at educational institutions, data privacy and security should be considered. Acknowledgments This paper is part of the project AVILAB2, funded by the German Federal Ministry of Education and Research (BMBF); Grant # 16INB2005A. 70 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION References Abraham, A., Suseelan, B., Mathew, J., Sabarinath, P., & K, A. (2023). A Study on Metaverse in Education. 2023 7th International Conference on Computing Methodologies and Communication (ICCMC), 1570–1573. https://doi.org/10.1109/ICCMC56507.2023.10083910 Agrati, L. S. (2023). Tutoring in the metaverse. Study on student-teachers’ and tutors’ perceptions about NPC tutor. Frontiers in Education, 8. Scopus. https://doi.org/10.3389/feduc.2023.1202442 Al-Adwan, A. S., & Al-Debei, M. M. (2023). The determinants of Gen Z’s metaverse adoption decisions in higher education: Integrating UTAUT2 with personal innovativeness in IT. Education and Information Technologies. Scopus. https://doi.org/10.1007/s10639-023-12080-1 Alhalaybeh, A., Alkhatib, A. A. A., Albustanji, R. N., Elmanaseer, S., Kanan, T., & Alsakarnah, R. (2023). Facilitating Metaverse Integration in Education via Extended Reality and the Internet of Things: AN OVERVIEW. In Jaber K.M. (Ed.), Int. Conf. Inf. Technol.: Cybersecur. Chal enges Sustain. Cities, ICIT - Proceeding (pp. 38–42). Institute of Electrical and Electronics Engineers Inc.; Scopus. https://doi.org/10.1109/ICIT58056.2023.10226029 Alhalaybeh, A., Althunibat, A., Al-Mahadeen, B. M., Habashneh, K. K., & Daradkeh, A. M. (2023). Towards a Conceptual Model for Assessing User Satisfaction in Metaverse-Based Learning. In Jaber K.M. (Ed.), Int. Conf. Inf. Technol.: Cybersecur. Chal enges Sustain. Cities, ICIT - Proceeding (pp. 355–360). Institute of Electrical and Electronics Engineers Inc.; Scopus. https://doi.org/10.1109/ICIT58056.2023.10226119 Al-Kfairy, M., Al-Fandi, O., Alema, M., & Altaee, M. (2022). Motivation and Hurdles for the Student Adoption of Metaverse-based Classroom: A Qualitative Study. In Alja’Am J.M., AlMaadeed S., Elseoud S.A., & Karam O. (Eds.), Proc. Int. Conf. Comput. Appl., ICCA - Proc. Institute of Electrical and Electronics Engineers Inc.; Scopus. https://doi.org/10.1109/ICCA56443.2022.10039672 Alkhwaldi, A. F. (2023). Understanding learners’ intention toward Metaverse in higher education institutions from a developing country perspective: UTAUT and ISS integrated model. Kybernetes. Scopus. https://doi.org/10.1108/K-03-2023-0459 Almarzouqi, A., Aburayya, A., & Sal oum, S. A. (2022). Prediction of User’s Intention to Use Metaverse System in Medical Education: A Hybrid SEM-ML Learning Approach. IEEE Access, 10, 43421–43434. Scopus. https://doi.org/10.1109/ACCESS.2022.3169285 Alvarez, J., Angel, D. D., & Martínez, M. (2023). Tec Virtual Campus, a metaverse for engineering learning. 2023 IEEE World Engineering Education Conference (EDUNINE), 01–06. https://doi.org/10.1109/EDUNINE57531.2023.10102856 Araújo, L., Soares, M. M., Tori, R., & Zhang, Y. (2023). The Attention of Students in the Metaverse: Practical Application. In Marcus A., Rosenzweig E., & Soares M.M. (Eds.), Lect. Notes Comput. Sci.: Vol. 14033 LNCS (pp. 3–23). Springer Science and Business Media Deutschland GmbH; Scopus. https://doi.org/10.1007/978-3-031-35708-4_1 Areepong, T., Nilsook, P., & Wannapiroon, P. (2022). A Study of a Metaverse Interdisciplinary Learning Community. 2022 Research, Invention, and Innovation Congress: Innovative Electricals and Electronics (RI2C), 290–296. https://doi.org/10.1109/RI2C56397.2022.9910268 Asiksoy, G. (2023). Empirical Studies on the Metaverse-Based Education: A Systematic Review. International Journal of Engineering Pedagogy, 13(3), 120–133. Scopus. https://doi.org/10.3991/ijep.v13i3.36227 Bandara, W., Furtmuel er, E., Gorbacheva, E., Miskon, S., & Beekhuyzen, J. (2015). Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support. Communications of the Association for Information Systems, 37. https://doi.org/10.17705/1CAIS.03708 BenedettDörr, J., & BeatrysRuizAylon, L. (2023). A Survey on the Metaverse Aspects and Opportunities in Education. 2023 International Conference on Intel igent Metaverse Technologies & Applications (IMETA), 1–8. https://doi.org/10.1109/iMETA59369.2023.10294573 A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 71 Braguez, J., Braguez, M., Moreira, S., & Filipe, C. (2023). The possibilities of changes in learning experiences with Metaverse. In Martinho R., Rijo R., Cruz-Cunha M.M., Domingos D., & Peres E. (Eds.), Procedia Comput. Sci. (Vol. 219, pp. 504–511). Elsevier B.V.; Scopus. https://doi.org/10.1016/j.procs.2023.01.318 Cantone, A. A., Francese, R., Sais, R., Santosuosso, O. P., Sepe, A., Spera, S., Tortora, G., & Vitiel o, G. (2023). Contextualized Experiential Language Learning in the Metaverse. Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter. https://doi.org/10.1145/3605390.3605395 Chamorro-Atalaya, O., Durán-Herrera, V., Suarez-Bazalar, R., Gonzáles -Pacheco, A., Quipuscoa-Silvestre, M., Hernández-Hernández, F., Huaman-Flores, E., Chaccara-Contreras, V., Palacios-Huaraca, C., & Guía-Altamirano, T. (2023). The Metaverse in University Education during COVID-19: A Systematic Review of Success Factors. International Journal of Learning, Teaching and Educational Research, 22(5), 206–226. Scopus. https://doi.org/10.26803/ijlter.22.5.10 Chen, X., Zou, D., Xie, H., & Wang, F. L. (2023). Metaverse in Education: Contributors, Cooperations, and Research Themes. IEEE Transactions on Learning Technologies, 1–18. https://doi.org/10.1109/TLT.2023.3277952 Chen, Z. (2022). Exploring the application scenarios and issues facing Metaverse technology in education. Interactive Learning Environments, 0(0), 1–13. https://doi.org/10.1080/10494820.2022.2133148 Chua, H. W., & Yu, Z. (2023). A systematic literature review of the acceptability of the use of Metaverse in education over 16 years. Journal of Computers in Education. Scopus. https://doi.org/10.1007/s40692-023-00273-z Damaševičius, R., Maskeliūnas, R., & Misra, S. (2023). Supporting and Shaping Human Decisions Through Internet of Behaviors (IoB): Perspectives and Implications. In Stud. Comput. Intel . (Vol. 1105, pp. 115–144). Springer Science and Business Media Deutschland GmbH; Scopus. https://doi.org/10.1007/978-3-031-37454-8_6 Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008 De Felice, F., Petril o, A., Iovine, G., Salzano, C., & Baffo, I. (2023). How Does the Metaverse Shape Education? A Systematic Literature Review. Applied Sciences (Switzerland), 13(9). Scopus. https://doi.org/10.3390/app13095682 Frydenberg, M., & Ohri, S. (2023). Designing a Metaverse for an Immersive Learning Experience. Int. Conf. High. Educ. Adv. , 1139–1146. Scopus. https://doi.org/10.4995/HEAd23.2023.16080 Fu, W., & Pan, J. (2022). Exploration of classroom teaching reform in higher vocational col eges from the perspective of metaverse. 2022 International Conference on Cloud Computing, Big Data and Internet of Things (3CBIT), 178–181. https://doi.org/10.1109/3CBIT57391.2022.00044 Girvan, C. (2018). What is a virtual world? Definition and classification. Educational Technology Research and Development, 66(5), 1087–1100. Scopus. https://doi.org/10.1007/s11423-018-9577-y Guillén-Yparrea, N., & Hernández-Rodríguez, F. (2023). Reinforcement of Collaborative Work in the Metaverse Through Playful Activities. 2023 IEEE/ACIS 21st International Conference on Software Engineering Research, Management and Applications (SERA), 307–312. https://doi.org/10.1109/SERA57763.2023.10197690 Han, J., Liu, G., & Gao, Y. (2023). Learners in the Metaverse: A Systematic Review on the Use of Roblox in Learning. Education Sciences, 13(3). Scopus. https://doi.org/10.3390/educsci13030296 Han, X., Hu, Y., Li, Y., Tan, B., Tu, X., & Jiang, Y. (2022). Design and Research of Campus Culture Application Based on AR and Metaverse Technology. In Meen T.-H. (Ed.), Proc. Int. Conf. Comput., Big-Data Eng., ICCBE (pp. 97–101). Institute of Electrical and Electronics Engineers Inc.; Scopus. https://doi.org/10.1109/ICCBE56101.2022.9888114 Hedrick, E., Harper, M., Oliver, E., & Hatch, D. (2022). Teaching & Learning in Virtual Reality: Metaverse Classroom Exploration. 2022 Intermountain Engineering, Technology and Computing (IETC), 1–5. https://doi.org/10.1109/IETC54973.2022.9796765 72 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Hevner, A. (2007). A Three Cycle View of Design Science Research. Scandinavian Journal of Information Systems, 19(2), 87–92. Hines, P., & Netland, T. H. (2022). Teaching a Lean masterclass in the metaverse. International Journal of Lean Six Sigma. Scopus. https://doi.org/10.1108/IJLSS-02-2022-0035 Ho, C. (2022). Research on Teaching of Metaverse Technology Flipped the MICE Education. Proc. - Int. Conf. Educ., Knowl. Inf. Manag., ICEKIM, 592–596. Scopus. https://doi.org/10.1109/ICEKIM55072.2022.00136 Ho, W., & Lee, D. (2023). Enhancing Engineering Education in the Roblox Metaverse: Utilizing chatGPT for Game Development for Electrical Machine Course. International Journal on Advanced Science, Engineering and Information Technology, 13(3), 1052–1058. Scopus. https://doi.org/10.18517/ijaseit.13.3.18458 Hussain, S. (2023). Metaverse for education – Virtual or real? Frontiers in Education, 8. Scopus. https://doi.org/10.3389/feduc.2023.1177429 Hwang, G.-J., & Chien, S.-Y. (2022). Definition, roles, and potential research issues of the metaverse in education: An artificial intelligence perspective. Computers and Education: Artificial Intel igence, 3, 100082. https://doi.org/10.1016/j.caeai.2022.100082 Hwang, G.-J., Tu, Y.-F., & Chu, H.-C. (2023). Conceptions of the metaverse in higher education: A draw-a-picture analysis and surveys to investigate the perceptions of students with different motivation levels. Computers and Education, 203. Scopus. https://doi.org/10.1016/j.compedu.2023.104868 Hwang, Y. (2023). When makers meet the metaverse: Effects of creating NFT metaverse exhibition in maker education. Computers and Education, 194. Scopus. https://doi.org/10.1016/j.compedu.2022.104693 Iakovides, N., Lazarou, A., Kyriakou, P., & Aristidou, A. (2022). Virtual library in the concept of digital twin. Int. Conf. Interact. Media, Smart Syst. Emerg. Technol., IMET - Proc. 2022 International Conference on Interactive Media, Smart Systems and Emerging Technologies, IMET 2022 - Proceedings. Scopus. https://doi.org/10.1109/IMET54801.2022.9929598 İbili, E., Ölmez, M., Cihan, A., Bilal, F., İbili, A. B., Okumus, N., & Bil inghurst, M. (2023). Investigation of learners’ behavioral intentions to use metaverse learning environment in higher education: A virtual computer laboratory. Interactive Learning Environments. Scopus. https://doi.org/10.1080/10494820.2023.2240860 Iwanaga, J., Muo, E. C., Tabira, Y., Watanabe, K., Tubbs, S. J., D’Antoni, A. V., Rajaram-Gilkes, M., Loukas, M., Khalil, M. K., & Tubbs, R. S. (2023). Who real y needs a Metaverse in anatomy education? A review with preliminary survey results. Clinical Anatomy, 36(1), 77–82. Scopus. https://doi.org/10.1002/ca.23949 Jacobs, A., Pan, Y.-C., & Joel-Edgar, S. (2023). Are you in or not? A study into student intention to use immersive metaverse for learning. UK Academy for Information Systems Conference Proceedings 2023. https://aisel.aisnet.org/ukais2023/15 Jang, J., & Kim, J. (2023). Exploring the Impact of Avatar Customization in Metaverse: The Role of the Class Mode on Task Engagement and Expectancy-Value Beliefs for Fashion Education. Mobile Information Systems, 2023. Scopus. https://doi.org/10.1155/2023/2967579 John Lemay, D., Doleck, T., & Bazelais, P. (2023). Transition to online teaching during the COVID-19 pandemic. Interactive Learning Environments, 31(4), 2051–2062. https://doi.org/10.1080/10494820.2021.1871633 Joshi, S., & Pramod, P. J. (2023). A Col aborative Metaverse based A-La-Carte Framework for Tertiary Education (CO-MATE). Heliyon, 9(2). Scopus. https://doi.org/10.1016/j.heliyon.2023.e13424 Kalınkara, Y., & Özdemir, O. (2023). Anatomy in the metaverse: Exploring student technology acceptance through the UTAUT2 model. Anatomical Sciences Education. Scopus. https://doi.org/10.1002/ase.2353 Kim, H.-S., Kim, N.-Y., & Cha, Y. (2023). Exploring the Potential of Metaverse as a Future Learning Platform for Enhancing EFL Learners’ English Proficiency. Korean Journal of English Language and Linguistics, 23, 220–236. Scopus. https://doi.org/10.15738/kjell.23. 202303.220 A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 73 Kryvenko, I., & Chalyy, K. (2023). Phenomenological toolkit of the metaverse for medical informatics’ adaptive learning. Educacion Medica, 24(5). Scopus. https://doi.org/10.1016/j.edumed.2023.100854 Kshetri, N., Rojas-Torres, D., & Grambo, M. (2022). The Metaverse and Higher Education Institutions. IT Professional, 24(6), 69–73. https://doi.org/10.1109/MITP.2022.3222711 Kurniawan, Y., Jayamuni, J. H., Kwandou, N., & Anwar, N. (2023). Advancing Education Through Metaverse Virtual Reality Simulation. 2023 8th International Conference on Business and Industrial Research (ICBIR), 801–806. https://doi.org/10.1109/ICBIR57571.2023.10147723 Kye, B., Han, N., Kim, E., Park, Y., & Jo, S. (2021). Educational applications of metaverse: Possibilities and limitations. Journal of Educational Evaluation for Health Professions, 18. Scopus. https://doi.org/10.3352/jeehp.2021.18.32 Laine, T. H., & Lee, W. (2023). Collaborative Virtual Reality in Higher Education: Students’ Perceptions on Presence, Chal enges, Affordances, and Potential. IEEE Transactions on Learning Technologies, 1–14. Scopus. https://doi.org/10.1109/TLT.2023.3319628 Lee, J., & Jang, S. (2023). Metaverse-based education service adoption and preference study using conjoint analysis. Int. Conf. Comput., Netw. Commun., ICNC, 273–277. Scopus. https://doi.org/10.1109/ICNC57223.2023.10074583 Lee, J., & Kim, Y. (2023). Sustainable Educational Metaverse Content and System Based on Deep Learning for Enhancing Learner Immersion. Sustainability (Switzerland), 15(16). Scopus. https://doi.org/10.3390/su151612663 Lee, N., & Jo, M. (2023). Exploring problem-based learning curricula in the metaverse: The hospitality students’ perspective. Journal of Hospitality, Leisure, Sport and Tourism Education, 32. Scopus. https://doi.org/10.1016/j.jhlste.2023.100427 Liang, J., Li, G., Zhang, F., Fan, D., & Luo, H. (2023). Benefits and chal enges of the educational metaverse: Evidence from quantitative and qualitative data. Journal of Educational Technology Development and Exchange, 16(1), 71–91. Scopus. https://doi.org/10.18785/jetde.1601.04 Lin, H., Wan, S., Gan, W., Chen, J., & Chao, H.-C. (2022). Metaverse in Education: Vision, Opportunities, and Chal enges. 2022 IEEE International Conference on Big Data (Big Data), 2857–2866. https://doi.org/10.1109/BigData55660.2022.10021004 López, G. A. M., Chaux, H. R., & Alvarez, F. A. C. (2022). The University in the Metaverse. Proposal of application scenarios and roadmap model. Int. Conf. Technol., Learn. Teach. Electron., TAEE - Proc. 15th International Conference of Technology, Learning and Teaching of Electronics, TAEE 2022 - Proceedings. Scopus. https://doi.org/10.1109/TAEE54169.2022.9840630 López-Belmonte, J., Pozo-Sánchez, S., Moreno-Guerrero, A.-J., & Lampropoulos, G. (2023). Metaverse in Education: A systematic review. Revista de Educación a Distancia, 23(73). Scopus. https://doi.org/10.6018/RED.511421 Mantoro, T., Tarigan, W. H., & Ayu, M. A. (2022). Analysis of the Metaverse in Society 5.0 for Learning Systems using Meta. IEEE Int. Conf. Comput., Eng. Des., ICCED. 2022 IEEE 8th International Conference on Computing, Engineering and Design, ICCED 2022. Scopus. https://doi.org/10.1109/ICCED56140.2022.10010350 Mitra, S. (2023). Metaverse: A Potential Virtual-Physical Ecosystem for Innovative Blended Education and Training. Journal of Metaverse, 3(1), 66–72. Scopus. https://doi.org/10.57019/jmv.1168056 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2023). Metaverse and Blockchain in Education for collaborative Product-Service System (PSS) Design towards University 5.0. In Liu A. & Kara S. (Eds.), Procedia CIRP (Vol. 119, pp. 456–461). Elsevier B.V.; Scopus. https://doi.org/10.1016/j.procir.2023.01.008 Mystakidis, S. (2022). Metaverse. Encyclopedia, 2(1), Article 1. https://doi.org/10.3390/encyclopedia2010031 Onecha, B., Cornadó, C., Morros, J., & Pons, O. (2023). New Approach to Design and Assess Metaverse Environments for Improving Learning Processes in Higher Education: The Case of Architectural Construction and Rehabilitation. Buildings, 13(5). Scopus. https://doi.org/10.3390/buildings13051340 74 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanvil e, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLOS Medicine, 18(3), e1003583. https://doi.org/10.1371/journal.pmed.1003583 Pangsapa, P., Yun Wong, P. P., Wai Chung Wong, G., Techanamurthy, U., Wan Mohamad, W. S., & Shen Jiandong, D. (2023). Enhancing Humanities Learning with Metaverse Technology: A Study on Student Engagement and Performance. Int. Conf. Inf. Educ. Technol., ICIET, 251– 255. Scopus. https://doi.org/10.1109/ICIET56899.2023.10111125 Park, S.-M., & Kim, Y.-G. (2022). A Metaverse: Taxonomy, Components, Applications, and Open Chal enges. IEEE Access, 10, 4209–4251. Scopus. https://doi.org/10.1109/ACCESS.2021.3140175 Prakash, A., Haque, A., Islam, F., & Sonal, D. (2023). Exploring the Potential of Metaverse for Higher Education: Opportunities, Chal enges, and Implications. Metaverse Basic and Applied Research, 2, 40–40. https://doi.org/10.56294/mr202340 Purahong, B., Anuwongpinit, T., Kanjanasurat, I., Chansuthirangkool, M., Singto, K., Somdock, N., Damrongsak, P., Khunthawiwone, P., & Archevapanich, T. (2022). Engineering Education Roadmap of the Future Trend of Basic Metaverse based on VR with cooperation between the government and the private sector. 2022 7th International STEM Education Conference (ISTEM-Ed), 1–5. https://doi.org/10.1109/iSTEM-Ed55321.2022.9920775 Raj, A., Sharma, V., Rani, S., Singh, T., Shanu, A. K., & Alkhayyat, A. (2023). Demystifying and Analysing Metaverse Towards Education 4.0. Proc. Int. Conf. Innov. Pract. Technol. Manag., ICIPTM. Proceedings of 2023 3rd International Conference on Innovative Practices in Technology and Management, ICIPTM 2023. Scopus. https://doi.org/10.1109/ICIPTM57143.2023.10118054 Rinn, H., Khosrawi-Rad, B., Grogorick, L., Robra-Bissantz, S., & Markgraf, D. (2023). Virtual Worlds in Education—A systematic Literature Review. ECIS 2023 Research Papers. https://aisel.aisnet.org/ecis2023_rp/277 Rinn, H., Merkel-Kiss, M., Robra-Bissantz, S., & Markgraf, D. (2023, December 10). Digital Measures to Decrease Thesis Procrastination in Distance Education. International Conference on Information Systems Education and Research (ICISER 2023), Hyderabad, India. Ritchey, T. (2011). Modeling Alternative Futures with General Morphological Analysis. World Future Review, WFR, 83–94. https://doi.org/10.1177/194675671100300105 Riva, G., Mantovani, F., Capidevil e, C. S., Preziosa, A., Morganti, F., Vil ani, D., Gaggioli, A., Botel a, C., & Alcañiz, M. (2007). Affective Interactions Using Virtual Reality: The Link between Presence and Emotions. CyberPsychology & Behavior, 10(1), 45–56. https://doi.org/10.1089/cpb.2006.9993 Roy, R., Babakerkhell, M. D., Mukherjee, S., Pal, D., & Funilkul, S. (2023). Development of a Framework for Metaverse in Education: A Systematic Literature Review Approach. IEEE Access, 11, 57717–57734. https://doi.org/10.1109/ACCESS.2023.3283273 Ruwodo, V., Pinomaa, A., Vesisenaho, M., Ntinda, M., & Sutinen, E. (2022). Enhancing Software Engineering Education in Africa through a Metaversity. 2022 IEEE Frontiers in Education Conference (FIE), 1–8. https://doi.org/10.1109/FIE56618.2022.9962729 Samala, A. D., Usmeldi, Taali, Ambiyar, Bojic, L., Indarta, Y., Tsoy, D., Denden, M., Tas, N., & Dewi, I. P. (2023). Metaverse Technologies in Education: A Systematic Literature Review Using PRISMA. International Journal of Emerging Technologies in Learning (IJET), 18(05). https://doi.org/10.3991/ijet.v18i05.35501 Schoormann, T., Behrens, D., Fel mann, M., & Knackstedt, R. (2021). On Your Mark, Ready, Search: A Framework for Structuring Literature Search Strategies in Information Systems. In F. Ahlemann, R. Schütte, & S. Stieglitz (Eds.), Innovation Through Information Systems (Vol. 46, pp. 558–575). Springer International Publishing. https://doi.org/10.1007/978-3-030-86790-4_38 A. Woick et al.: Metaverse in Higher Education – A Systematic Literature Review 75 Sin, Z. P. T., Jia, Y., Wu, A. C. H., Zhao, I. D., Li, R. C., P. H. F. Ng, X. Huang, G. Baciu, J. Cao, & Q. Li. (2023). Towards an Edu-Metaverse of Knowledge: Immersive Exploration of University Courses. IEEE Transactions on Learning Technologies, 1–14. https://doi.org/10.1109/TLT.2023.3290814 Smart, J., Cascio, J., & Paffendorf, J. (2007). A Metaverse Roadmap: Pathways to the 3D Web. https://www.academia.edu/266307/A_Metaverse_Roadmap_Pathways_to_the_3D_Web_2 007 Song, Y., Cao, J., Wu, K., Yu, P. L. H., & Lee, J. C.-K. (2023). Developing ‘Learningverse’-a 3D Metaverse Platform to Support Teaching, Social and Cognitive Presences. IEEE Transactions on Learning Technologies, 1–14. https://doi.org/10.1109/TLT.2023.3276574 Sunardi, A. N. Hidayanto, Meyliana, & H. Prabowo. (2022). Discipline, Impact, And Challenges Of Virtual Reality In Higher Education: A Systematic Literature Review. 2022 International Conference on Information Management and Technology (ICIMTech), 476–481. https://doi.org/10.1109/ICIMTech55957.2022.9915242 Tlili, A., Huang, R., Shehata, B., Liu, D., Zhao, J., Metwal y, A. H. S., Wang, H., Denden, M., Bozkurt, A., Lee, L.-H., Beyoglu, D., Altinay, F., Sharma, R. C., Altinay, Z., Li, Z., Liu, J., Ahmad, F., Hu, Y., Salha, S., … Burgos, D. (2022). Is Metaverse in education a blessing or a curse: A combined content and bibliometric analysis. Smart Learning Environments, 9(1). Scopus. https://doi.org/10.1186/s40561-022-00205-x Troja, E., Debel o, J. E., & Truong, L. M. (2023). Teaching Effective and Gamified Cybersecurity using the Metaverse: Chal enges and Opportunities. In da Rocha Brito C. & Ciampi M.M. (Eds.), EDUNINE - IEEE World Eng. Educ. Conf.: Reimaging Eng. - Toward Next Gener. Eng. Edu., Merging Technol. A Connect. World, Proc. Institute of Electrical and Electronics Engineers Inc.; Scopus. https://doi.org/10.1109/EDUNINE57531.2023.10102900 Weng, L., Wei, Z., Li, F., & Lam, F. I. (2023). Research on the Innovative Integration Application of Metaverse and Education. Proceedings of the 9th International Conference on Education and Training Technologies. https://doi.org/10.1145/3599640.3599660 Wu, X., Chen, Y., & Wu, Y. (2023). Exploration of Mathematics Education by Metaverse Technology. IEEE Int. Conf. Educ. Inf. Technol., ICEIT, 173–178. Scopus. https://doi.org/10.1109/ICEIT57125.2023.10107894 Yu, D. (2022). Practical Exploration of Educational Metaverse—- A Teacher Training System Based on Virtual Reality. Int. Conf. Orange Technol., ICOT. 2022 10th International Conference on Orange Technology, ICOT 2022. Scopus. https://doi.org/10.1109/ICOT56925.2022.10008186 Yu, D. (2023). AI-Empowered Metaverse Learning Simulation Technology Application. 2023 International Conference on Intel igent Metaverse Technologies & Applications (IMETA), 1–6. https://doi.org/10.1109/iMETA59369.2023.10294830 Yu, J.-E. (2022). Exploration of Educational Possibilities by Four Metaverse Types in Physical Education. Technologies, 10(5). Scopus. https://doi.org/10.3390/technologies10050104 Yuan, J., Liu, Y., Han, X., Li, A., & Zhao, L. (2023). Educational metaverse: An exploration and practice of VR wisdom teaching model in Chinese Open University English course. Interactive Technology and Smart Education. Scopus. https://doi.org/10.1108/ITSE-10-2022-0140 Yue, K. (2022). Breaking down the Barrier between Teachers and Students by Using Metaverse Technology in Education: Based on A Survey and Analysis of Shenzhen City, China. ACM Int. Conf. Proc. Ser. , 40–44. Scopus. https://doi.org/10.1145/3514262.3514345 Zhang, X., Chen, Y., Hu, L., & Wang, Y. (2022). The metaverse in education: Definition, framework, features, potential applications, chal enges, and future research topics. Frontiers in Psychology, 13, 1016300. https://doi.org/10.3389/fpsyg.2022.1016300 Zhang, Y., Yang, S., Hu, X., & He, P. (2022). Metaverse Teaching Overview. Proc. - Int. Conf. Inf. Technol. Med. Educ., ITME, 104–109. Scopus. https://doi.org/10.1109/ITME56794.2022.00032 76 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Zhao, Z., Zhao, B., & Wan, X. (2022). Research on personalized learning space in educational metaverse. Fourth International Conference on Computer Science and Educational Informatization (CSEI 2022), 2022, 245–248. https://doi.org/10.1049/icp.2022.1479 Zuñiga, K. M., Rodríguez, A. R., & Velázquez, R. V. (2021). Tecnologías e innovación disruptiva en la educación superior. Serie Científica de la Universidad de las Ciencias Informáticas, 14(3), Article 3. HIGHER EDUCATION STUDENT'S SELF- EFFICACY BELIEFS DURING AND POST PANDEMIC: AN EXPLORATIVE LEARNING ANALYTICS STUDY SATU AKSOVAARA,1 TOMMI KÄRKKÄINEN,2 MINNA SILVENNOINEN1 1 Jamk University of Applied Sciences, Professional Teacher Education, Jyväskylä, Finland satu.aksovaara@jamk.fi, minna.silvennoinen@jamk.fi 2 Tommi Kärkkäinen, University of Jyväskylä, Faculty of Information Technology, Educational Technology and Cognitive Science, Jyväskylä, Finland tommi.karkkainen@jyu.fi The COVID-19 era massively accelerated digitalization of higher education and afterwards higher education institutions have partially reverted to their pre-pandemic modes of operation. In this study, we applied learning analytics to gain understanding of higher education students' experiences. We analyzed data on their self- efficacy beliefs and teamwork experiences. Data from 654 students were col ected from two temporal y distinct, identical courses, first at the beginning of the COVID-19 pandemic in fal 2020 and then after lockdowns ended in 2023. Our findings reveal a significant increase in self-efficacy post-pandemic, indicating that pandemic period may have influenced higher education students' self-efficacy beliefs. A moderately positive relationship between the students' self-efficacy and their self-assessed team contributions was found. These insights deepen understanding of higher education students' Keywords: study experiences and support the development of evidence-based COVID-19, educational practices applying learning analytics. The results higher highlight the need for higher education institutions to consider the education, self-efficacy, development of students' self-efficacy when designing col aborative self-assessment, learning processes, as supporting self-efficacy improves the study learning experience and presumably has an impact on teamwork. analytics DOI https://doi.org/10.18690/um.fov.4.2024.5 ISBN 978-961-286-871-0 78 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Higher education is undergoing digital transformation, while responding to both future workforce and technological demands (Gaebel et al., 2021). Digitalized education has increasingly required students to be self-directed (Song & Hill, 2007). In 2020 the COVID-19 pandemic catalyzed a massive shift towards online learning, leaving students to navigate their educational paths in isolation. Students were forced to develop new learning strategies to succeed as their contact with teachers and peers diminished, and opportunities for interaction and feedback became less frequent (Holzer et al., 2021; Koh & Daniel, 2022). There is a paradoxical dual impact enhanced by the pandemic on the student’s lives. As the world entered the post-COVID era, educational institutions began gradually reverting to their pre-pandemic modes of operation without, however, completely returning to the old way (e.g. Zancajo et al., 2022). Indeed, studies have shown that COVID-19 changed higher education students’ learning strategies to a more continuous habit, improving their efficiency (Gonzalez et al., 2020; Martin et al., 2023). The pandemic accelerated the digitalization of education significantly, which led to a decline in student wel -being (Holzer et al., 2021; Schmits et al., 2021). This decrease led higher education institutions to invest into student wel -being more heavily than before (Sarasjärvi et al., 2022; Van de Velde et al., 2021). Within an era of transformation, this shift in the learning paradigm necessitates research-based knowledge to understand students' experiences thoroughly, which is essential to implement meaningful, learner-centered education. There is a need for studies investigating students' resilience, changes in learning requirements, and abilities and regulation during COVID-19 and in post-pandemic era (Holzer et al., 2021; Müller et al., 2021). The digital transformation has opened new avenues to examine learning processes and student's experiences through learning analytics, which refers to the col ection, analysis, and reporting of educational data on learners and their environments to better understand and optimize learning (Long & Siemens, 2011). For example, data from student’s learning paths can be used to explore study experiences during learning processes (Ifenthaler et al., 2017; Heilala et al., 2020). In recent years, learning analytics has attracted significant interest in the field of higher education, as it is expected to contribute to the development of high-quality, learner-centered S. Aksovaara, T. Kärkkäinen, M. Silvennoinen: Higher Education Student's Self-Ef icacy Beliefs During and Post Pandemic: an Explorative Learning Analytics Study 79 education (Axelsen et al., 2020; Nunn et al., 2016; Oliva-Cordova, 2021) by informing decisions related to learning processes through insights into learners’ behaviors and preferences (Jayashanka et al., 2019). However, to assess study experiences and their changes in higher education, relevant data must be collected and analyzed, for example, by gathering information on students’ experiences through self-assessments (Aksovaara et al., 2024). Study experience is an important indicator of a successful learning process, with a positive correlation with both academic achievement and learning outcomes (El iott & Shin, 2002; Goh et al., 2017; Heilala et al., 2020). Study experience in higher education results from many factors, including individual traits (e.g., self-efficacy, competence beliefs, and motivation), relational aspects (e.g., interactions with peers and instructors), and participatory perspectives (e.g., opportunities to influence and personalize learning processes) (Goh et al., 2017; Jääskelä et al., 2021). Self-efficacy refers to individuals’ beliefs in their own abilities to succeed in specific tasks or activities (Bandura, 1993) and has emerged as an important construct in research over the last 30 years. Self-efficacy is known to play a predictive and mediating role in relation to students’ achievements, motivation, and learning (Parpala & Lindblom- Ylänne, 2012). Self-efficacy significantly impacts on students learning outcomes by increasing the ambitions in goal setting and positively affecting self-regulation of learning and study performance (e.g., Coutinho & Neuman, 2008; Kryshko et al., 202; Papinczak et al., 2008; Pintrich, 2003; Prat‐Sala & Redford, 2010, 2). In this study, we applied explorative learning analytics to gain understanding of higher education students' experiences. We analyzed data on students’ self-efficacy beliefs and teamwork experiences. Data was col ected from two identical implementations at the very beginning of the COVID-19 pandemic in fall 2020 and after educational practices had stabilized to their post-pandemic form in 2023. We consider the following research questions: RQ1 How has the COVID-19 era affected higher education students' self- efficacy? RQ2 How does self-efficacy relate to higher education students’ self-assessed contributions on teamwork? 80 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 2 Methods and analysis The research context was a two ECTS blended learning course, which is a mandatory course for first- and second-year undergraduates studying at a university of applied sciences (UAS). Our focus is on the collaborative phase (Phase 2; see Figure 1). This phase occurs between two asynchronous, online learning phases (Phases 1 and 3) of the course. The study was approved by the research review board of the UAS institution under study. Figure 1: Course structure Source: Own 2.1 Data col ection and participants Data collection was carried out across two identical implementations of the course. The structurally identical collaborative processes guiding self-directed teamwork and studying were built on the Moodle platform in both implementations and they included identical learning tasks and reflections. The first course implementation (A) took place in fal 2020 at the beginning of the COVID-19 pandemic, and the second (B) in 2023 after the pandemic had subsided. Participants (N = 654) represented a diverse range of educational fields, and the gender distribution among the 654 participants was nearly even, with males constituting 55% (194) and females 45% (159) implementation A. Implementation B maintained a similar balance, with males at 54% (162) and females at 46% (137). The col ection of research data was seamlessly integrated into the course workflow in Moodle. During the learning tasks, the students reflected on their own actions within a team, four times during the collaborative working week, at the end of each day (see Figure 1). This enabled understanding of the daily variations of students’ experiences. In addition, the S. Aksovaara, T. Kärkkäinen, M. Silvennoinen: Higher Education Student's Self-Ef icacy Beliefs During and Post Pandemic: an Explorative Learning Analytics Study 81 students’ evaluated their self-efficacy beliefs as part of the learning task within Phase 3 (see Figure 1). 2.2 Measures The self-efficacy scale by Parpala & Lindblom-Ylänne (2012) is an effective instrument for measuring self-efficacy in higher education, whose reliability and validity have been tested. The scale consists of five statements (see Table 1), where the agreement level is measured using a five-point Likert scale (1=Fully disagree, 2=Somewhat disagree, 3=Neutral, 4=Somewhat agree and 5=Fully agree). Based on students' responses to these five statements, the mean variable for self-efficacy beliefs (SES) was calculated (Cronbach’s alpha .893). The SES scores derived from this calculation represent the students' overall self-efficacy beliefs, with higher scores denoting stronger convictions in their academic abilities. Table 1: The statements of self-efficacy beliefs and Cronbach's alpha values Statements of self-efficacy beliefs Cronbach's Alpha I expect to be successful in my studies. .863 I am confident that I can understand even the most difficult things related to my studies. .875 I am sure that I can understand the basic concepts in my field. .880 I believe I wil succeed in my studies. .864 I am sure I can learn the skil s required for my field wel . .863 During daily learning tasks (R1-R4, see Figure 1), students reflected on their own activity and contributions within a team using two selected statements. An overal self-assessed mean variable (SAA) was calculated by combining the responses of the individual statements (SAA1 & SAA2) (see Table 2). 82 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 2: Mean variables for self-assessed own activity Variable Statements reflecting activity and contributions Items Mean Std. Dev. SAA1 “I was an active team member.” 4 4.01 0.585 SAA2 “My actions benefitted the team's work.” 4 4.30 0.544 reSAA Overall self-assessed contribution to teamwork from SAA1 & SAA2 8 4.13 0.510 3 Results As defined by Chatti et al. (2012), data mining, social network analysis, information visualization, and statistical techniques belong to the methodological landscape of learning analytics. Here, we confine ourselves to statistical methods, more precisely to comparisons and correlation analysis, where the tests were conducted using the IBM SPSS v. 28.0 in a pseudonymized form. Concerning RQ1, we start our investigation of the impact of the COVID-19 pandemic on students' self-efficacy beliefs by comparing the SES profiles between implementations A and B (see Table 3). The starting point is that in both instances, SES was always rated high, above four on a one-to-five scale. Table 3: SES at the beginning of (A) and after (B) COVID-19 SES N Mean Std. Dev. Mean Rank Sum of Ranks A 353 4.11 .618 278.84 98,432.00 B 301 4.43 .562 384.56 115,753.00 Initial analyses confirmed the non-normal distribution of SES data, as evidenced by the Shapiro-Wilk tests (statistic = .914, df = 654, p < .001). Consequently, Mann- Whitney U test was used to assess the differences between the SES datasets in A and B. The results of this test were profound, showing a significant disparity in SES between A and B (U = 35951.000, Z = -7.187, p < .001), with the mean ranks indicating a higher SES in implementation B, as detailed in Table 5. This difference is not only statistically significant but also of medium to large practical importance (Cohen's d = .593), as depicted in Figure 2. S. Aksovaara, T. Kärkkäinen, M. Silvennoinen: Higher Education Student's Self-Ef icacy Beliefs During and Post Pandemic: an Explorative Learning Analytics Study 83 Figure 2: Boxplots of SES for instances A and B Source: Own In addressing RQ2, a correlation analysis was conducted. Self-efficacy SES and self- assessed own contribution SAA both demonstrate relatively even distributions around their means of 4.2560 (SD = 0.61277) and 4.1270 (SD = 0.50993). Initial analyses confirmed the non-normal distribution of both SES and SAA. Consequently, the nonparametric Spearman's rho was used in the correlation analysis, revealing a moderate positive relationship between SES and SAA (ρ = .444, p < .001, two-tailed). It can be concluded that there is an association between SES and SAA, such that higher self-efficacy indicates higher self-assessed contribution. Nonetheless, the correlation is not so significant that the value of one variable could be predicted by the other. When comparing early and post-pandemic measures separately, the correlation between SES and SAA was consistently positive (A: ρ = .465, p < .001; B: ρ = .337, p < .001), though it decreased slightly post-pandemic. The same holds true for the individual aspects of the self-assessed contribution; namely, for the Spearman's rhos between SES and SAA1/SAA2 (SAA1 in A: ρ = .392, p < .001 and in B: ρ = .252, p < .001; SAA2 in A: ρ = .451, p < .001 and in B: ρ = .336, p < .001). Again, in both cases, the moderate correlation decreases slightly in the post-pandemic case. This means that factors other than self-efficacy, whose level was found to be very high in case B, explain more of the variability of the self-assessed contribution. To this end, interesting negative correlations between the second statement (SAA2 “My actions benefitted the team's work.”) and its standard deviation (SAA2_SD) were found: in A: ρ = -.398, p < .001 and in B: ρ = -.384, p < .001. This means that lower variability 84 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION of the self-assessed benefit was associated with the higher overal level. Again, this association was slightly stronger at the beginning of the COVID-19 period than in 2023. 4 Discussion and conclusions In this study, learning analytics provided a data-driven exploration to understand the study experiences (see e.g. Heilala et al., 2020; Jääskelä et al., 2021; Silvola et al., 2021). The data collection was integrated into students’ daily reflections of their own activity and contributions within a team enabling daily based tracking of their experiences. This data-driven approach provides opportunities for more learner- centered teaching and learning design (Cohen, 2018; Neelen & Kirschner, 2020), by enriching our understanding of students' varying experiences during collaborative, blended learning processes. Increasingly, learning analytics are being utilized to enable personalized learning and improve learning experiences with cost effective manners (Wong et al., 2023). Integrating data collection into the reflection process additional y supports the student's workflow and it has been shown how continuous reflection enables maintaining the student’s activity, which is on the other hand known to improve the learning curve (Millar et al., 2021). Enabling the seamless integration and tailoring of analytics to students' varying learning processes can be identified as an area for further development. The role of digitalization increased during COVID–19 and was a significant catalyst for changes that have profoundly impacted educational environments, even triggering a paradigm shift in how teaching and learning are organized (Gaebel et al., 2021; Holzer et al., 2021). The results of the present study offer insights into higher education students' self-efficacy at the beginning of and after the COVID-19 pandemic and the relationship between self-efficacy beliefs and student’s reflections of their own activity and contributions within a team. Even if the level of self-efficacy was high in general, it was found that the post- pandemic self-efficacy was significantly higher than it was at the beginning of the pandemic era. The results indicate that the COVID-19 era might have impacted on students' self-efficacy beliefs. Theories supporting this view may relate to the understanding that self-efficacy positively influences metacognitive learning strategies and academic performance (Hayat et al., 2020). It is known that the S. Aksovaara, T. Kärkkäinen, M. Silvennoinen: Higher Education Student's Self-Ef icacy Beliefs During and Post Pandemic: an Explorative Learning Analytics Study 85 pandemic brought significant changes to study routines, including remote learning which necessitated more independent studying. Our results may indicate that the increased role and amount of online learning during the COVID-19 period in general has built up their confidence in relation to self-regulated learning and engagement (Gonzalez et al., 2020; Martin et al., 2023; Mou, 2023). Students might have also developed new learning strategies to succeed (Holzer et al., 2021; Koh & Daniel, 2022). Our results also showed a moderate positive relationship between the students' self- efficacy and their self-assessed own activity and contributions in teamwork, suggesting that student’s self-efficacy beliefs might offer an indicator of their abilities to contribute to team outcomes. This finding is in line with the results from studies on professional skills development through collaborative learning (including teamwork), where self-efficacy was the only significant predictor of the learning results (Yadav et al., 2021). Self-efficacy is strengthened through positive feedback and experiences of success (see Bandura 1997), which improves study experiences and the development of skills. The results are related to the students’ self-efficacy having a predictive and mediating role in relation to their achievements, motivation, and learning (see, e.g., Dinther et al., 2011). Therefore, it is necessary for higher education institutions to pay more attention to the development of students’ self- efficacy when designing collaborative learning processes and offer support strategies for building student's self-confidence. Supporting self-efficacy could improve the quality of teamwork and vice versa. The COVID-19 pandemic and the era of digital transformation have prompted higher education institutions to develop peer learning solutions as student wel -being has declined. Collaborative learning and studying in small groups are increasingly emphasized in both higher education and in workplace learning (Guo et al., 2020). However, collaborative learning is a complex, multidimensional phenomenon influenced by several factors. Therefore, it is beneficial to gain a deeper understanding of the interplay between the various student related elements affecting team dynamics and contributions. Future research should also continue to explore the effects of other psychological factors, e.g., motivation, on collaborative learning dynamics using larger datasets (see also Charalambous et al., 2021; Hannam University & Shin, 2018). Higher education institutions should pay more attention to creating curriculums that bolster students' well-being and academic success (van 86 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Dinther et al., 2011). In this, the information produced by analytics about the student experience is a key constituent. It is crucial to connect learning analytics to the reflection process so that understanding the student experience is possible dynamically and during learning processes within its varying phases. Knowledge of this up-to-date experience would also enable the provision of up-to-date guidance and targeted support to the student. A more diverse and in-depth examination of the learning experience would benefit from large datasets, from which integrated learning analytics could be used to identify factors influencing students' experiences. References Aksovaara, S., Määttä, S., Kärkkäinen, T., & Silvennoinen, M. (2024). Improving Learning Design Using Learning Analytics in Relation to Study Experience. Submitted for publication, Seminar.net, Special Issue for MEC 2023, In review. Axelsen, M., Redmond, P., Heinrich, E., & Henderson, M. (2020). The evolving field of learning analytics research in higher education. Australasian Journal of Educational Technology, 36(2), 1–7. https://doi.org/10.14742/ajet.6266 Bandura, A. (1993). Perceived Self-Efficacy in Cognitive Development and Functioning. Educational Psychologist, 28(2), 117–148. Charalambous, M., Hodge, J. A., & Ippolito, K. (2021). Statistically significant learning experiences: Towards building self-efficacy of undergraduate statistics learners through team-based learning. Educational Action Research, 29(2), 226–244. https://doi.org/10.1080/09650792.2020.1782240 Chatti, M. A., Dyckhoff, A. L., Schroeder, U., & Thüs, H. (2012). A reference model for learning analytics. International Journal of Technology Enhanced Learning, 4(5–6), 318–331. https://doi.org/10.1504/IJTEL.2012.051815 Cohen, J. A. (2018). Evidence based learning design – the opportunities afforded by learning analytics. Development and Learning in Organizations: An International Journal, 32(4), 10. https://doi.org/10.1108/dlo-10-2017-0084 Coutinho, S. A., & Neuman, G. (2008). A model of metacognition, achievement goal orientation, learning style and self-efficacy. Learning Environments Research, 11(2), 131–151. https://doi.org/10.1007/s10984-008-9042-7 El iott, K. M., & Shin, D. (2002). Student Satisfaction: An alternative approach to assessing this important concept. Journal of Higher Education Policy and Management, 24(2), 197–209. https://doi.org/10.1080/1360080022000013518 Gaebel, M., Zhang, T., Stoeber, H., & Morrisroe, A. (2021). Digital y Enhanced Learning and Teaching in European Higher Education Institutions. Survey Report. European University Association. Goh, C., Leong, C., Kasmin, K., Hi , P., & Tan, O. (2017). Students’ Experiences, Learning Outcomes and Satisfaction in e-Learning. Journal of E-Learning and Knowledge Society, 13(2). https://www.learntechlib.org/p/188116/ Gonzalez, T., de la Rubia, M. A., Hincz, K. P., Comas-Lopez, M., Subirats, L., Fort, S., & Sacha, G. M. (2020). Influence of COVID-19 confinement on students’ performance in higher education. PLOS ONE, 15(10), e0239490. https://doi.org/10.1371/journal.pone.0239490 S. Aksovaara, T. Kärkkäinen, M. Silvennoinen: Higher Education Student's Self-Ef icacy Beliefs During and Post Pandemic: an Explorative Learning Analytics Study 87 Guo, P., Saab, N., Post, L. S., & Admiraal, W. (2020). A review of project-based learning in higher education: Student outcomes and measures. International Journal of Educational Research, 102, 101586. https://doi.org/10.1016/j.ijer.2020.101586 Hannam University, & Shin, M.-H. (2018). Effects of Project-based Learning on Students’ Motivation and Self-efficacy. English Teaching, 73(1), 95–114. https://doi.org/10.15858/engtea.73.1.201803.95 Heilala, V., Jääskelä, P., Kärkkäinen, T., & Saarela, M. (2020). Understanding the Study Experiences of Students in Low Agency Profile: Towards a Smart Education Approach. In A. El Moussati, K. Kpalma, M. Ghaouth Belkasmi, M. Saber, & S. Guégan (Eds.), Advances in Smart Technologies Applications and Case Studies (Vol. 684, pp. 498–508). Springer International Publishing. https://doi.org/10.1007/978-3-030-53187-4_54 Holzer, J., Lüftenegger, M., Korlat, S., Pelikan, E., Salmela-Aro, K., Spiel, C., & Schober, B. (2021). Higher Education in Times of COVID-19: University Students’ Basic Need Satisfaction, Self- Regulated Learning, and Wel -Being. AERA Open, 7, 233285842110031. https://doi.org/10.1177/23328584211003164 Ifenthaler, Dirk, Gibson, David, & Dobozy, Eva. (2017). The synergistic and dynamic relationship between learning design and learning analytics. ASCILITE 2017, 1:5, 112–116. Jääskelä, P., Heilala, V., Kärkkäinen, T., & Häkkinen, P. (2021). Student agency analytics: Learning analytics as a tool for analysing student agency in higher education. Behaviour & Information Technology, 40(8), 790–808. https://doi.org/10.1080/0144929X.2020.1725130 Jayashanka, R., Hewagamage, K. P., & Hettiarachchi, E. (2019). An Intelligent Interactive Visualizer to Improve Blended Learning in Higher Education. 2019 Twelfth International Conference on Ubi-Media Computing (Ubi-Media), 69–73. https://doi.org/10.1109/Ubi- Media.2019.00022 Koh, J. H. L., & Daniel, B. K. (2022). Shifting online during COVID-19: A systematic review of teaching and learning strategies and their outcomes. International Journal of Educational Technology in Higher Education, 19(1), 56. https://doi.org/10.1186/s41239-022-00361-7 Kryshko, O., Fleischer, J., Grunschel, C., & Leutner, D. (2022). Self-efficacy for motivational regulation and satisfaction with academic studies in STEM undergraduates: The mediating role of study motivation. Learning and Individual Differences, 93, 102096. https://doi.org/10.1016/j.lindif.2021.102096 Long, P., & Siemens, G. (2011). What is learning analytics. Proceedings of the 1st International Conference Learning Analytics and Knowledge, LAK, 11. Martin, A. J., Ginns, P., & Col ie, R. J. (2023). University students in COVID-19 lockdown: The role of adaptability and fluid reasoning in supporting their academic motivation and engagement. Learning and Instruction, 83, 101712. https://doi.org/10.1016/j.learninstruc.2022.101712 Millar, S.-K., Spencer, K., Stewart, T., & Dong, M. (2021). Learning Curves in COVID-19: Student Strategies in the ‘new normal’? Frontiers in Education, 6, 641262. https://doi.org/10.3389/feduc.2021.641262 Mou, T.-Y. (2023). Online learning in the time of the COVID-19 crisis: Implications for the self-regulated learning of university design students. Active Learning in Higher Education, 24(2), 185–205. https://doi.org/10.1177/14697874211051226 Mül er, F. H., Thomas, A. E., Carmignola, M., Dittrich, A.-K., Eckes, A., Großmann, N., Martinek, D., Wilde, M., & Bieg, S. (2021). University Students’ Basic Psychological Needs, Motivation, and Vitality Before and During COVID-19: A Self-Determination Theory Approach. Frontiers in Psychology, 12. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2021.775804 Neelen, M., & Kirschner, P. A. (2020). Evidence-informed Learning Design: Creating Training to Improve Performance. KoganPage. https://books.google.fi/books?id=fyutxAEACAAJ Nunn, S., Avel a, J. T., Kanai, T., & Kebritchi, M. (2016). Learning Analytics Methods, Benefits, and Challenges in Higher Education: A Systematic Literature Review. Online Learning, 20(2). https://doi.org/10.24059/olj.v20i2.790 88 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Oliva-Cordova, L. M. (2021). Learning Analytics to Support Teaching Skills: A Systematic Literature Review. IEEE Access, 9, 58351–58363. https://doi.org/10.1109/ACCESS.2021.3070294 Papinczak, T., Young, L., Groves, M., & Haynes, M. (2008). Effects of a Metacognitive Intervention on Students’ Approaches to Learning and Self-Efficacy in a First Year Medical Course. Advances in Health Sciences Education, 13(2), 213–232. https://doi.org/10.1007/s10459- 006-9036-0 Parpala, A., & Lindblom-Ylänne, S. (2012). Using a research instrument for developing quality at the university. Quality in Higher Education, 18(3), 313–328. https://doi.org/10.1080/13538322.2012.733493 Pintrich, P. R. (2003). A Motivational Science Perspective on the Role of Student Motivation in Learning and Teaching Contexts. Journal of Educational Psychology, 95(4), 667–686. https://doi.org/10.1037/0022-0663.95.4.667 Prat‐Sala, M., & Redford, Paul. (2010). The interplay between motivation, self‐efficacy, and approaches to studying. British Journal of Educational Psychology, 80(2), 283–305. https://doi.org/10.1348/000709909X480563 Sarasjärvi, K. K., Vuolanto, P. H., Solin, P. C. M., Appelqvist-Schmidlechner, K. L., Tamminen, N. M., Elovainio, M., & Therman, S. (2022). Subjective mental well-being among higher education students in Finland during the first wave of COVID-19. Scandinavian Journal of Public Health, 50(6), 765–771. https://doi.org/10.1177/14034948221075433 Schmits, E., Dekeyser, S., Klein, O., Luminet, O., Yzerbyt, V., & Glowacz, F. (2021). Psychological Distress among Students in Higher Education: One Year after the Beginning of the COVID-19 Pandemic. International Journal of Environmental Research and Public Health, 18(14), 7445. https://doi.org/10.3390/ijerph18147445 Silvola, A., Näykki, P., Kaveri, A., & Muukkonen, H. (2021). Expectations for Supporting Student Engagement with Learning Analytics: An Academic Path Perspective. Computers & Education, 168, 104192. https://doi.org/10.1016/j.compedu.2021.104192 Song, L., & Hil , J. R. (2007). A conceptual model for understanding self-directed learning in online environments. Journal of Interactive Online Learning, 6(1), 27–42. Van de Velde, S., Buffel, V., Bracke, P., Van Hal, G., Somogyi, N. M., Wil ems, B., Wouters, E., & for the C19 ISWS consortium#. (2021). The COVID-19 International Student Well-being Study. Scandinavian Journal of Public Health, 49(1), 114–122. https://doi.org/10.1177/1403494820981186 van Dinther, M., Dochy, F., & Segers, M. (2011). Factors affecting students’ self-efficacy in higher education. Educational Research Review, 6(2), 95–108. https://doi.org/10.1016/j.edurev.2010.10.003 Wong, B. T., Li, K. C., & Cheung, S. K. S. (2023). An analysis of learning analytics in personalised learning. Journal of Computing in Higher Education, 35(3), 371–390. https://doi.org/10.1007/s12528-022-09324-3 Yadav, A., Mayfield, C., Moudgalya, S. K., Kussmaul, C., & Hu, H. H. (2021). Col aborative learning, self-efficacy, and student performance in cs1 pogil. 775–781. Zancajo, A., Verger, A., & Bolea, P. (2022). Digitalization and beyond: The effects of Covid-19 on post-pandemic educational policy and delivery in Europe. Policy and Society, 41(1), 111–128. https://doi.org/10.1093/polsoc/puab016 INVESTIGATING THE BENEFITS OF 5G TO LEVERAGE THE DIGITAL TRANSFORMATION IN HEALTHCARE: A SYSTEMATIC REVIEW ON PERSONALIZED DIABETES SELF- MANAGEMENT AYESHA NILASHINI, NILMINI WICKRAMASINGHE La Trobe University, School of Computing, Engineering and Mathematical Sciences, Victoria, Australia a.wahalathanthrige@latrobe.edu.au, n.wickramasinghe@latrobe.edu.au This systematic literature review explores the innovative potential of 5G technology in global healthcare, focusing on personalized diabetes self-management. The COVID-19 pandemic increased the adoption of digital solutions related to health, creating new opportunities. This review examines the role of 5G technology in overcoming the specific chal enges associated with diabetes self-care, emphasizing its vital features, such as low latency and high reliability, enabling real-time data transmission and remote monitoring for improving patient care. Adhering to PRISMA guidelines, it synthesizes findings from reliable databases, exploring 5G's diverse influence. Key research questions include its contribution to healthcare digital transformation, SWOT analysis in diabetes management, Keywords: challenges in various diabetes types, and its role in designing 5G Technology, digital solutions. Findings reveal significant advancements, such digital as increased data transmission speeds, supporting real-time transformation, healthcare, remote patient monitoring and telemedicine. A proposed personalized framework guides digital solution development, highlighting diabetes future research areas and implementation chal enges, s elf-management, systematic emphasizing the need to ful y employ 5G's potential features in literature healthcare. review DOI https://doi.org/10.18690/um.fov.4.2024.6 ISBN 978-961-286-871-0 90 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The emergence of 5th Generation (5G) technology signifies a new phase in healthcare, providing unique opportunities for digital transformation and personalized medicine. This transformation has been improved by the COVID-19 pandemic, which has highlighted the opportunity for healthcare delivery that is not restricted by time or place (Ostovari et al., 2023; Williams et al., 2023). The need for innovation more pressing than in the management of chronic diseases like diabetes, which impact over 537 mil ion people worldwide and costing healthcare systems more than USD 966 billion annually (IDF Diabetes Atlas, 9th edition, 2021). With its features such as low latency, high speed, and extensive device connectivity, 5G technology stands as a powerful initiator for chronic disease management (Turab et al., 2023; Wersényi, 2022). As shown in Figure 1, 5G promises to improve chronic disease management, particularly in diabetes management, through its capabilities in facilitating real-time data transmission, personalized care, and facilotating more meaningful user interactions. This Systematic Literature Review (SLR), investigates the impact of 5G on healthcare, focusing specifical y on personalized diabetes self-management. Figure 1: 5G Technology in Healthcare Source: Own A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 91 The SLR aims to investigate four key research questions: • How does the integration of 5G technology contribute to the digital transformation in healthcare, with a focus on personalized diabetes self- management? • What are the Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis of 5G in the context of healthcare, particularly with respect to the chal enges and opportunities it presents for personalized diabetes self- management? • What are the chal enges and impacts on healthcare delivery posed by diabetes types 1, 2, and gestational, and how can 5G supported digital solutions address these chal enges? • How do digital solutions, especially those leveraging 5G capabilities, influence and support self-management in diabetes care, and what features are essential for the design of an effective dashboard for personalized diabetes self-management? The paper explores key components, beginning with background exploration (Section 2), followed by methodology (Section 3) and findings synthesis (Section 4). Discussion (Section 5), limitations and future research directions are outlined in Section 6. Ultimately, Section 7 concludes the paper by summarizing key insights and contributions. 2 Background and Related Work The shift towards digital healthcare represents a great shift in patient-centric care models. Outstanding innovations such as electronic health records (EHRs), AI- driven diagnostics, and telemedicine platforms have been helpful in enhancnig the accessibility and quality of healthcare services (Kruse et al., 2023). These innovations have considerably improved patient outcomes and operational efficiencies (Aliberti et al., 2022; Lauman & Dennis, 2021). Similarly, prsonalized medicine, incorporating genetic information and predictive analytics have also gained considerable importance, particularly in the management of chronic conditions like diabetes (Burford et al., 2019; El-Gayar et al., 2021; Rohilla et al., 2023). The arrival of 5G technology empowers real-time health data analytics, 92 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION supports the Internet of Medical Things (IoMT), and facilitates advanced telemedicine services. These capabilities offer numerous opportunities for the advancement of personalized medicine and patient-centered care (Chen et al., 2021; Moglia et al., 2022). The digital health innovations, while pioneering, often face chal enges related to data latency, connectivity issues, and limited real-time capabilities (Blonde et al., 2022; Lightfoot et al., 2022). 5G technology, with its low latency and high reliability, can mitigate these constraints, providing a strong foundation for personalized medicine applications. Previous studies have pointed out the use of 5G in enhancing remote patient monitoring, and implement comprehensive telemedicine services for diabetes care (Makroum et al., 2022). These findings highlight the potential of 5G and digital health technologies, which together can create more comprehensive, effective personalized solutions for diabetes management (Min et al., 2021; Taimoor & Rehman, 2022). 3 Methodology This study employed a SLR fol owing the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to investigate the integration of 5G technology in healthcare, specifical y on personalized diabetes self-management (See Appendix 3 for the PRISMA flow diagram). A search strategy was executed to capture the most relevant literature across four major academic databases namely, MEDLINE, Scopus, CINAHL, and IEEE Xplore. These databases were preferred for their comprehensive coverage of healthcare management, medical, and technological literature. The inclusion criteria were crafted to cover studies involving individuals with diabetes that investigated the role of 5G technology in enhancing digital healthcare solutions related to diabetes self-management. Also, the exclusion criteria were applied to filter the studies not directly related to diabetes or 5G technology, articles published before 2017 and non-english articles to ensure relevance and latest findings. A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 93 Search strings utilized a combination of related keywords and MeSH terms, structured around the PICO framework (Patient, Intervention, Comparison, Outcome), to optimize the database search. This approach was helpful in identifying a notable increase in research publications from 2017 onwards, signifying a significant contribution to the field of 5G technology and related concepts (See Appendix 2). The screening process involved a thorough review of titles and abstracts, followed by a full-text review of filtered studies. This approach ensured the inclusion of studies that directly addressed the integration of 5G in personalized diabetes self- management. Data extraction focused on key variables such as the technological aspects of 5G, its impact on diabetes management, and the features of 5G-enabled healthcare solutions. (See Appendix 1 for a more details related to the methodology, search string and inclusion and exclusion criteria). 4 Synthsesis of Results 4.1 Publication Trend Analysis Analysis reveals an interest in the application of 5G technology in healthcare, with publication peaks in 2019 and continued through to 2023. This trend highlights the progress in 5G technologies such as Internet of Things (IoT) in healthcare, stressed during the COVID-19 pandemic, reflecting an accelerated shift towards digital health solutions specially during the crisis. (See Appendix 4). 4.2 Qualitative Content Analysis Key components identified through qualitative analysis, such as remote patient monitoring (RPM), real-time data transmission, and personalized treatment plans demonstrate the significant impact of 5G in healthcare (Rghioui et al., 2020). Studies demonstrate 5G's capability to enhance diabetes self-management through continuous monitoring, immediate medical interventions, and data-driven care plans, marking a significant increase towards personalized healthcare (El-Rashidy et al., 2021; Magsi et al., 2018). 94 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION RPM: This element is recognized as a crucial application of 5G, improving healthcare system efficiency. By integrating machine learning (ML) algorithms for real-time data analysis, 5G supports timely interventions and contribute for future healthcare models (Subramanian & Thampy, 2021). Real-Time Data Transmission: 5G facilitates real-time data transmission, enabling remote surgeries and telemedicine (Coats-Thomas et al., 2022). AI and IoT device integration enhance patient monitoring and support tactile Internet systems, promising comprehensive mobile medicine solutions for various demographics, including the elderly people (Mohanta et al., 2019; Wu et al., 2021; Zhu et al., 2023). Personalized Treatment Plans: Leveraging 5G's high-speed data capabilities, innovations like the 5G-Smart Diabetes system have emerged, incorporating sensing, diagnosis, and data-sharing layers (Chen et al., 2018). These innovations contribute for sustainable and intel igent healthcare solutions (Latif et al., 2017; Taimoor & Rehman, 2022). Figure 2: Key Features of 5G Technology and Their Benefits for Healthcare Source: Own The capabilities of 5G technology, including high-speed data transmission, network reliability, and low latency significantly enhance health service delivery, facilitating self-care, monitoring, and remote surgery (Chih-Ping et al., 2017; Gupta et al., 2021) A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 95 as shown in Figure 2. (Detailed tables and figures with exmaple raw data can be found in Appendix 5). 4.3 SWOT Analysis of 5G in Healthcare The integration of 5G technology into diabetes management offers numerous strengths, such as high-speed data transmission and low latency. These capabilities facilitate real-time monitoring and efficient communication of vital health data (Moglia et al., 2022). The 5G networks offer enhanced connectivity and reliability, ensuring stable platforms for continuous remote monitoring and personalized interventions (Magsi et al., 2018). The capacity of 5G can handle massive data volumes also creates opportunities for comprehensive diabetes management and efficient telemedicine (Chen et al., 2021). However, there are weaknesses, such as infrastructure chal enges in terms of coverage and high deployment costs, security concerns regarding sensitive health data (Mohanta et al., 2019), and potential dependency on technological competence among healthcare professionals (Latif et al., 2017). Opportunities include advancements in telemedicine, innovation in wearable devices (Vesselkov et al., 2018), and efficient data analytics (Chen et al., 2018), which can enhance diabetes care. Nonetheless, there are threats to consider, including privacy concerns (Moglia et al., 2022), regulatory and compliance chal enges. Figure 3 dipicted the summary of SWOT analysis. Addressing these threats and weaknesses through strategic approaches is essential to utilize the full potential of 5G in diabetes management (Yangan Zhang et al., 2020). Figure 3: SWOT Analysis Summary: Integration of 5G in Healthcare Source: Own 96 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.4 Diabetes Care Challenges and Thematic Analysis Below Table 1 demontrates the summary of thematic analyses on how 5G technology can address the unique chal enges associated with different types of diabetes, offering tailored solutions that align with the specific healthcare needs of each group. Table 1: Chal enges and 5G Solutions for Different Types of Diabetes Type of Diabetes Type 1 Diabetes Continous insulin therapy, blood glucose monitoring, Healthcare Needs carbohydrate management (Morone, 2019) Constant need for blood glucose monitoring and insulin Chal enges administration. Education for self-management is crucial (Morone, 2019) Theme 1: Immediate Data Access and Response - 5G enables real-time monitoring and insulin dose adjustments, facilitating Thematic Analysis real-time blood glucose monitoring (Min Chen et al., 2018; Mohanta et al., 2019) Real-time monitoring for accurate insulin adjustments 5G Solutions (Rghioui et al., 2020) Type of Diabetes Type 2 Diabetes Lifestyle modifications, medications, monitoring for Healthcare Needs comorbidities (Bertsimas et al., 2017) Management complicated by comorbidities and lifestyle Chal enges intervention needs. Ensuring patient adherence to treatment plans (Twohig et al., 2019) Theme 2: Accessibility and Convenience - 5G supports remote management and counseling with seamless, real-time Thematic Analysis interaction between patients and healthcare providers (Chen et al., 2021; Moglia et al., 2022) Facilitates educational content delivery and improves medication adherence. Supports regular follow-ups and 5G Solutions personalized care plans (El-Rashidy et al., 2021; Taimoor & Rehman, 2022) Type of Diabetes Gestational Diabetes Blood glucose monitoring, dietary adjustments, postpartum Healthcare Needs monitoring (Alqudah et al., 2019; Fareed et al., 2023) A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 97 Limited intervention time and need for monitoring to prevent Chal enges complications during pregnancy (Fareed et al., 2023) Theme 3: Continuous Health Tracking and IoT Integration - 5G supports continuous monitoring for timely interventions, Thematic Analysis optimizing maternal and fetal health (Chen et al., 2018; Zhu et al., 2023) Supports remote monitoring, reducing hospital visits. 5G Solutions Provides infrastructure for remote monitoring solutions (Alqudah et al., 2019) 4.5 Emerging Trends and Gap Analysis The evolution of diabetes care has been greatly influenced by digital technologies, marking a shift from traditional manual tracking to advanced real-time monitoring and management systems enabled by advancements such as 5G technology (Chen et al., 2018; Zhu et al., 2023). Historical analysis reveals a transformative journey from self-management practices to the predictive analytics for preventive care systems significantly enhancing patient independence and care outcomes (Chen et al., 2018; Zhu et al., 2023). As shown in figure 4, early adoption and theoretical development were prominent before 2017, focusing on developing healthcare and personalized diabetes management (Bertsimas et al., 2017; Latif et al., 2017). The subsequent period (2018- 2021) shows an expansion of practical applications and pilot studies, notably in remote patient monitoring and mobile health, showcasing the growing role of 5G in healthcare (El-Rashidy et al., 2021; Rghioui et al., 2020). Integration of 5G with digital health tools gained attention, especial y with wearable devices and IoT for real-time monitoring (Chen et al., 2018; Zhu et al., 2023), and the COVID-19 pandemic accelerated the adoption of 5G-enabled telemedicine applications (Moglia et al., 2022). Chal enges such as infrastructure and security were addressed, with a growing focus on personalized healthcare solutions in recent years, particularly in type 2 diabetes management (Devi et al., 2023; Fareed et al., 2023; Mohanta et al., 2019). Ongoing research is exploring advanced applications and AI integration with 5G for predictive analytics in diabetes care, indicating promising future prospects (Taimoor & Rehman, 2022). 98 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The gap analysis highlights several areas in current research on 5G-enabled diabetes healthcare. First, there is a lack of unified patient-centric dashboard that integrates various complex data streams, presenting an opportunity for future research to develop interactive and personalized interfaces that encourage patient engagement (Dagliati et al., 2018). Second, there's a need for integration of machine learning, 5G, and the Internet of Medical Things (IoMT) to enhance predictive analytics and patient outcomes, which has not been extensively examined (Rghioui et al., 2020; Riaz et al., 2023). Third, research should focus on patient-centered models using blockchain, 5G, and IoMT for personalized care, addressing data privacy, security concerns, and cultural sensitivity in diabetes management (Morone, 2019; Subramanian & Sreekantan Thampy, 2021). Additionally, understanding the impact of 5G and IoMT technologies on healthcare utilization and cost across diverse patient populations in chronic disease management requires further investigation (Ames et al., 2021). Addressing these gaps wil contribute to more comprehensive, secure, and user-friendly healthcare solutions, ultimately improving patient care and outcomes. Figure 4: Evolution and Current Trends of 5G Technology in Diabetes Healthcare Source: Own 4.6 Feature Analysis for the Development of an Effective 5G-Enabled Dashboard for Personalized Diabetes Self-Management The advanced capabilities of 5G technology offer significant advantages in facilitating a comprehensive and real-time approach to personalized diabetes self- management. Utilizing feature analysis from the literature, some essential features identified and outlined in Table 2 for a 5G-enabled dashboard specifically tailored A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 99 for type 2 diabetes. These features empower individuals to actively engage in their care, leading to more informed decisions and improved health outcomes. Additionally, the dashboard should include user-friendly interfaces, multilingual support, secure data management protocols, integrate with medical devices, and community and social support features to enhance accessibility, security, and patient engagement (El-Rashidy et al., 2021; Mohanta et al., 2019; Twohig et al., 2019; Morone, 2019; Ames et al., 2021; Fareed et al., 2023). By leveraging the advanced capabilities of 5G technology, personalized diabetes self- management can be further enhanced, promoting better health outcomes for individuals with type 2 diabetes. Table 2: Essential Features for a 5G-Enabled Dashboard for Personalized Diabetes Self- Management (Type 2) Feature 5G Advantage Continuous Remote Real-time monitoring for timely adjustments Monitoring AI-Driven Personalized High-speed data processing for personalized Insights recommendations Telemedicine Effective virtual consultations for regular check-ins Integration Wearable Device Real-time data from sensors for holistic health Compatibility monitoring Medication Adherence Swift data transfer for effective medication management Tracking 5 Discussion The SLR clarifies the transformative potential of 5G technology in healthcare, with a particular emphasis on personalized diabetes self-management. Integrating the proposed framework, as shown in Figure 5 into the discussion, it was observed the alignment of 5G capabilities with healthcare digital transformation needs, highlighting 5G's high-speed data transmission, low latency, and robust connectivity, and the requirements for effective diabetes self-management systems. 100 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The SWOT analysis within the framework recognises the strengths, weaknesses, opportunities, and threats of 5G in healthcare, echoing the findings from Moglia et al. (2022) and Magsi et al. (2018). It highlights the 5G's role in enhancing personalized care models through real-time data analysis and remote monitoring, addressing the unique chal enges posed by different types of diabetes as discussed by Chen et al. (2018). Figure 5: Proposed Framework Source: Own Upon the framework's insights, expanding personalized diabetes self-management, it becomes evident that the integration of 5G technology holds substantial promise for enhancing patient care. Real-time data transmission and remote monitoring, as A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 101 essential features for a 5G-enabled diabetes management dashboard, are crucial in facilitating immediate medical interventions and data-driven care plans. Thus, advancing the overall quality of healthcare services. This aligns with the contributions and the predictive analytics in developing personalized care plans (Taimoor & Rehman, 2022). Moreover, the framework serves as a valuable guide for the development of digital solutions addressing the diverse healthcare delivery chal enges associated with various types of diabetes. The ability of 5G to support differentiation in healthcare needs enhances the patient-centric approach, which is vital for effective diabetes management and improved patient outcomes. 6 Limitations and Future Research While our study highlights the potential advantages of 5G in healthcare innovation, it is important to recognize its limitations and areas for further exploration. One key implication for practice is the necessity for focused research to fully realize the benefits of 5G technology. This should also include investigating cost-effective deployment strategies, particularly in remote and underserved regions, to ensure access to digital healthcare (Devi et al., 2023). Additionally, comprehensive studies are needed to develop stong cybersecurity frameworks that safeguard data privacy in 5G-enabled systems (Mohanta et al., 2019). Furthermore, the discovery and implementation of AI and ML algorithms will ensure enhancing predictive analytics and tailoring personalized diabetes management plans, leveraging the advanced capabilities of 5G (Taimoor & Rehman, 2022). The development and assessment of patient-centered care models that incorporate 5G for improved patient engagement and care outcomes are crucial (Giordanengo et al., 2019). Research into developing regulatory policies that ensure the safe integration of 5G in healthcare is equally important (Moglia et al., 2022). Therefore, interdisciplinary research and collaboration remain significant for attending existing chal enges and unlocking the full potential of 5G in revolutionizing digital healthcare and chronic disease management. 102 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 7 Conclusion In conclusion, this SLR has shown the potential of 5G technology in healthcare, specifical y in the domain of personalized diabetes self-management. In accordance with PRISMA guidelines and the synthesis of insights from esteemed databases highlight the pivotal role of 5G in enabling real-time data transmission and remote patient monitoring. Beyond healthcare, exploring the implications involves considerations of patient privacy and data security, which paral el the broader societal impacts similar to the exploration of AI's influence. This perspective highlights ongoing research to overcome technological and regulatory obstacles, paving the way for innovative and advanced solutions in personalized healthcare that can improve patient outcomes and operational efficacy while evolving the future of healthcare delivery. Acknowledgements I would like to express my sincere gratitude to Dr.Bruce Lu and Dr.John Zelcer for their invaluable comments and insights on this conference paper. References Aliberti, G. M., Bhatia, R., Desrochers, L. B., Giliam, E. A., & Schonberg, M. A. (2022). Perspectives of primary care clinicians in Massachusetts on use of telemedicine with adults aged 65 and older during the COVID-19 pandemic [Article]. Preventive Medicine Reports, 26, Article 101729. https://doi.org/10.1016/j.pmedr.2022.101729 Alqudah, A., McMul an, P., Todd, A., O'Doherty, C., McVey, A., McConnel , M., O'Donoghue, J., Gal agher, J., Watson, C. J., & McClements, L. (2019). Service evaluation of diabetes management during pregnancy in a regional maternity hospital: potential scope for increased self-management and remote patient monitoring through mHealth solutions. BMC health services research, 19(1), 662. https://doi.org/https://dx.doi.org/10.1186/s12913-019-4471- 9 Ames, J. L., Massolo, M. L., Davignon, M. N., Qian, Y., & Croen, L. A. (2021). Healthcare service utilization and cost among transition-age youth with autism spectrum disorder and other special healthcare needs. Autism: The International Journal of Research & Practice, 25(3), 705-718. https://doi.org/10.1177/1362361320931268 Bertsimas, D., Kal us, N., Weinstein, A. M., Ying Daisy, Z., & Zhuo, Y. D. (2017). Personalized Diabetes Management Using Electronic Medical Records. Diabetes care, 40(2), 210-217. https://doi.org/10.2337/dc16-0826 Blonde, L., Umpierrez, G. E., Reddy, S. S., McGil , J. B., Berga, S. L., Bush, M., Chandrasekaran, S., DeFronzo, R. A., Einhorn, D., Galindo, R. J., Gardner, T. W., Garg, R., Garvey, W. T., Hirsch, I. B., Hurley, D. L., Izuora, K., Kosiborod, M., Olson, D., Patel, S. B., Pop-Busui, R., Sadhu, A. R., Samson, S. L., Stec, C., Tamborlane, W. V., Tuttle, K. R., Twining, C., Vel a, A., Vellanki, P., & Weber, S. L. (2022). American Association of Clinical Endocrinology Clinical A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 103 Practice Guideline: Developing a Diabetes Mellitus Comprehensive Care Plan—2022 Update [Article]. Endocrine Practice, 28(10), 923-1049. https://doi.org/10.1016/j.eprac.2022.08.002 Burford, S. J., Park, S., & Dawda, P. (2019). Small data and its visualization for diabetes self-management: Qualitative study [Article]. JMIR diabetes, 4(3), Article e10324. https://doi.org/10.2196/10324 Chen, H., Pan, X., Yang, J., Fan, J., Qin, M., Sun, H., Liu, J., Li, N., Ting, D. S. W., & Chen, Y. (2021). Application of 5G Technology to Conduct Real-Time Teleretinal Laser Photocoagulation for the Treatment of Diabetic Retinopathy. JAMA ophthalmology, 139(9), 975-982. https://doi.org/https://dx.doi.org/10.1001/jamaophthalmol.2021.2312 (Comment in: JAMA Ophthalmol. 2022 Feb 1;140(2):205-206 PMID: 34913946 [https://www.ncbi.nlm.nih.gov/pubmed/34913946] Comment in: JAMA Ophthalmol. 2022 Feb 1;140(2):205 PMID: 34913954 [https://www.ncbi.nlm.nih.gov/pubmed/34913954]) Chen, M., Yang, J., Zhou, J., Hao, Y., Zhang, J., & Youn, C.-H. (2018). 5G-Smart Diabetes: Toward Personalized Diabetes Diagnosis with Healthcare Big Data Clouds. IEEE Communications Magazine, 56(4), 16-23. https://doi.org/10.1109/MCOM.2018.1700788 Chih-Ping, L., Jing, J., Chen, W., Tingfang, J., & Smee, J. (2017, 12-15 June 2017). 5G ultra-reliable and low-latency systems design. 2017 European Conference on Networks and Communications (EuCNC), Coats-Thomas, M. S., Bail argeon, E. M., Ludvig, D., Marra, G., Perreault, E. J., & Seitz, A. L. (2022). No Strength Differences Despite Greater Posterior Rotator Cuff Intramuscular Fat in Patients With Eccentric Glenohumeral Osteoarthritis. Clinical orthopaedics and related research, 480(11), 2217-2228. https://doi.org/https://dx.doi.org/10.1097/CORR.0000000000002253 (Comment in: Clin Orthop Relat Res. 2022 Nov 1;480(11):2229-2231 PMID: 35767817 [https://www.ncbi.nlm.nih.gov/pubmed/35767817]) Dagliati, A., Sacchi, L., Tibol o, V., Cogni, G., Teliti, M., Martinez-Mil ana, A., Traver, V., Segagni, D., Posada, J., Ottaviano, M., Fico, G., Arredondo, M. T., De Cata, P., Chiovato, L., & Bellazzi, R. (2018). A dashboard-based system for supporting diabetes care. Journal of the American Medical Informatics Association : JAMIA, 25(5), 538-547. https://doi.org/https://dx.doi.org/10.1093/jamia/ocx159 Devi, D. H., Duraisamy, K., Armghan, A., Alsharari, M., Aliqab, K., Sorathiya, V., Das, S., & Rashid, N. (2023). 5G Technology in Healthcare and Wearable Devices: A Review. Sensors, 23(5), 2519. https://doi.org/10.3390/s23052519 El-Gayar, O., Ofori, M., & Nawar, N. (2021). On the efficacy of behavior change techniques in mHealth for self-management of diabetes: A meta-analysis. Journal of biomedical informatics, 119, 103839. https://doi.org/https://dx.doi.org/10.1016/j.jbi.2021.103839 El-Rashidy, N., El-Sappagh, S., Riazul Islam, S. M., El-Bakry, H. M., & Abdelrazek, S. (2021). Mobile health in remote patient monitoring for chronic diseases: Principles, trends, and chal enges. Diagnostics (Basel), 11(4), 607. https://doi.org/10.3390/diagnostics11040607 Fareed, N., Swoboda, C., Singh, P., Boettcher, E., Wang, Y., Venkatesh, K., & Strouse, R. (2023). Developing and testing an integrated patient mHealth and provider dashboard application system for type 2 diabetes management among Medicaid-enrol ed pregnant individuals based on a user-centered approach: Mixed-methods study [Article]. Digital Health, 9. https://doi.org/10.1177/20552076221144181 Giordanengo, A., Årsand, E., Woldaregay, A. Z., Bradway, M., Grottland, A., Hartvigsen, G., Granja, C., Torsvik, T., & Hansen, A. H. (2019). Design and prestudy assessment of a dashboard for presenting self-collected health data of patients with diabetes to clinicians: Iterative approach and qualitative case study [Article]. JMIR diabetes, 4(3), Article e14002. https://doi.org/10.2196/14002 Gupta, N., Juneja, P. K., Sharma, S., & Garg, U. (2021, 6-8 May 2021). Future Aspect of 5G-IoT Architecture in Smart Healthcare System. 2021 5th International Conference on Intel igent Computing and Control Systems (ICICCS), 104 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Kruse, C. S., Molina-Nava, A., Kapoor, Y., Anerobi, C., & Maddukuri, H. (2023). Analyzing the Effect of Telemedicine on Domains of Quality through Facilitators and Barriers to Adoption: Systematic Review [Review]. Journal of medical Internet research, 25, Article e43601. https://doi.org/10.2196/43601 Latif, S., Qadir, J., Farooq, S., & Imran, M. A. (2017). How 5G wireless (and Concomitant Technologies) will revolutionize healthcare? Future Internet, 9(4), 93. https://doi.org/10.3390/fi9040093 Lauman, P., & Dennis, J. J. (2021). Advances in Phage Therapy: Targeting the Burkholderia cepacia Complex. Viruses, 13(7), 1331. https://doi.org/10.3390/v13071331 Lightfoot, C. J., Wilkinson, T. J., Hadjiconstantinou, M., Graham-Brown, M., Barratt, J., Brough, C., Burton, J. O., Hainsworth, J., Johnson, V., Martinez, M., Nixon, A. C., Pursey, V., Schreder, S., Vadaszy, N., Wilde, L., Wil ingham, F., Young, H. M. L., Yates, T., Davies, M. J., & Smith, A. C. (2022). The Codevelopment of "My Kidneys & Me": A Digital Self-management Program for People With Chronic Kidney Disease. Journal of medical Internet research, 24(11), N.PAG-N.PAG. https://doi.org/10.2196/39657 Magsi, H., Sodhro, A. H., Chachar, F. A., Abro, S. A. K., Sodhro, G. H., & Pirbhulal, S. (2018, 3-4 March 2018). Evolution of 5G in Internet of medical things. 2018 International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), Makroum, M. A., Adda, M., Bouzouane, A., & Ibrahim, H. (2022). Machine Learning and Smart Devices for Diabetes Management: Systematic Review [Review]. Sensors, 22(5), Article 1843. https://doi.org/10.3390/s22051843 Min, J., Chen, Y., Wang, L., He, T., & Tang, S. (2021). Diabetes self-management in online health communities: an information exchange perspective. BMC medical informatics and decision making, 21(1), 201. https://doi.org/https://dx.doi.org/10.1186/s12911-021-01561-3 Moglia, A., Georgiou, K., Marinov, B., Georgiou, E., Berchiol i, R. N., Satava, R. M., & Cuschieri, A. (2022). 5G in Healthcare: From COVID-19 to Future Chal enges [Article]. IEEE journal of biomedical and health informatics, 26(8), 4187-4196. https://doi.org/10.1109/JBHI.2022.3181205 Mohanta, B., Das, P., & Patnaik, S. (2019, 25-26 May 2019). Healthcare 5.0: A Paradigm Shift in Digital Healthcare System Using Artificial Intelligence, IOT and 5G Communication. 2019 International Conference on Applied Machine Learning (ICAML), Morone, J. (2019). Systematic review of sociodemographic representation and cultural responsiveness in psychosocial and behavioral interventions with adolescents with type 1 diabetes. Journal of diabetes, 11(7), 582-592. https://doi.org/https://dx.doi.org/10.1111/1753-0407.12889 Ostovari, M., Zhang, Z., Patel, V., & Jurkovitz, C. (2023). Telemedicine and health disparities: Association between the area deprivation index and primary care telemedicine utilization during the COVID-19 pandemic [Article]. Journal of Clinical and Translational Science, 7(1), Article e168. https://doi.org/10.1017/cts.2023.580 Rghioui, A., Lloret, J., Sendra, S., & Oumnad, A. (2020). A smart architecture for diabetic patient monitoring using machine learning algorithms [Article]. Healthcare (Switzerland), 8(3), Article 348. https://doi.org/10.3390/healthcare8030348 Riaz, A., Khan, S., & Arslan, T. (2023). Design and Model ing of Graphene-Based Flexible 5G Antenna for Next-Generation Wearable Head Imaging Systems [Article]. Micromachines, 14(3), Article 610. https://doi.org/10.3390/mi14030610 Rohil a, U., Ramarao, J. P., Lane, J., Khatri, N. N., Smith, J., Yin, K., & Lau, A. Y. S. (2023). How general practitioners and patients discuss type 2 diabetes mel itus and cardiovascular diseases concerns during consultations: Implications for digital health [Article]. Digital Health, 9. https://doi.org/10.1177/20552076231176162 Subramanian, G., & Sreekantan Thampy, A. (2021). Implementation of Blockchain Consortium to Prioritize Diabetes Patients' Healthcare in Pandemic Situations [Article]. IEEE Access, 9, 162459-162475. https://doi.org/10.1109/ACCESS.2021.3132302 A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 105 Subramanian, G., & Thampy, A. S. (2021). Implementation of Blockchain Consortium to Prioritize Diabetes Patients’ Healthcare in Pandemic Situations. IEEE Access, 9, 162459-162475. https://doi.org/10.1109/ACCESS.2021.3132302 Taimoor, N., & Rehman, S. (2022). Reliable and Resilient AI and IoT-Based Personalised Healthcare Services: A Survey [Article]. IEEE Access, 10, 535-563. https://doi.org/10.1109/ACCESS.2021.3137364 Turab, N. M., Al-Nabulsi, J. I., Abu-Alhaija, M., Owida, H. A., Alsharaiah, M., & Abuthawabeh, A. (2023). Towards fostering the role of 5G networks in the field of digital health [Article]. International Journal of Electrical and Computer Engineering, 13(6), 6595-6608. https://doi.org/10.11591/ijece.v13i6.pp6595-6608 Twohig, P. A., Rivington, J. R., Gunzler, D., Daprano, J., & Margolius, D. (2019). Clinician dashboard views and improvement in preventative health outcome measures: a retrospective analysis. BMC health services research, 19(1), 475. https://doi.org/https://dx.doi.org/10.1186/s12913-019-4327-3 Vesselkov, A., Hämmäinen, H., & Töyli, J. (2018). Technology and value network evolution in telehealth [Article]. Technological Forecasting and Social Change, 134, 207-222. https://doi.org/10.1016/j.techfore.2018.06.011 Wersényi, G. (2022). Evaluation of the HoloLens for Medical Applications Using 5G-connected Mobile Devices [Article]. Infocommunications Journal, 14(4), 11-17. https://doi.org/10.36244/ICJ.2022.4.2 Wil iams, M. S., Cigaran, E., Martinez, S., Marino, J., Barbero, P., Myers, A. K., DiClemente, R. J., Goris, N., Gomez, V. C., Granvil e, D., Guzman, J., Harris, Y. T., Kline, M., Lesser, M. L., Makaryus, A. N., Murray, L. M., McFarlane, S. I., Patel, V. H., Polo, J., Zeltser, R., & Pekmezaris, R. (2023). COVID-19 stressors for Hispanic/Latino patients living with type 2 diabetes: a qualitative study. Frontiers in clinical diabetes and healthcare, 4, 1070547. https://doi.org/https://dx.doi.org/10.3389/fcdhc.2023.1070547 Wu, J., Chang, L., & Yu, G. (2021). Effective Data Decision-Making and Transmission System Based on Mobile Health for Chronic Disease Management in the Elderly. IEEE systems journal, 15(4), 5537-5548. https://doi.org/10.1109/JSYST.2020.3024816 Zhang, Y., Chen, G., Du, H., Yuan, X., Cheriet, M., & Kadoch, M. (2020). Real-time remote health monitoring system driven by 5G MEC-IOT. Electronics (Basel), 9(11), 1-17. https://doi.org/10.3390/electronics9111753 Zhang, Y., Qu, M., Yi, X., Zhuo, P., Tang, J., Chen, X., Zhou, G., Hu, P., Qiu, T., Xing, W., Mao, Y., Chen, B. T., Wu, J., Zhang, Y., & Liao, W. (2020). Sensorimotor and pain-related alterations of the gray matter and white matter in Type 2 diabetic patients with peripheral neuropathy [Article]. Human Brain Mapping, 41(3), 710-725. https://doi.org/10.1002/hbm.24834 Zhu, T., Kuang, L., Daniels, J., Herrero, P., Li, K., & Georgiou, P. (2023). IoMT-Enabled Real-Time Blood Glucose Prediction With Deep Learning and Edge Computing [Article]. IEEE Internet of Things Journal, 10(5), 3706-3719. https://doi.org/10.1109/JIOT.2022.3143375 106 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix1: PRISMA Flow Diagram Appendix 2: Publications Per Year A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 107 Appendix 3: Detailed Methodology Components 1. Search String based on PICO Search Strategy Diabetes AND (5G Technology OR Fifth Generation Wireless Technology) AND (Digital healthcare OR Health Information Technology) AND (Non-digital healthcare OR Traditional healthcare) AND (Self-management OR Self-care OR Healthcare) 2. Inclusion and Exclusion Criteria To maintain the relevance and quality of the studies included in the review, specific inclusion and exclusion criteria were applied. Inclusion Criteria • Studies involving individuals diagnosed with diabetes • Research specifical y examining the integration of 5G technology or digital healthcare solutions • Studies published from 2017 onwards to ensure relevance to the research scope • Articles written in English for accessibility Exclusion Criteria • Studies not related to diabetes • Studies not directly related to 5G technology in healthcare • Publications before 2017 to prioritize recent developments • Non-English articles to maintain consistency and accessibility 108 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix 4: Publication trend analysis Appendix 5: Tables and Figures with Example Raw Data Qualitative Analysis- Summary of Articles Included in the Review Key Element Author and Year Study Summary/ Findings Remote Advocated for advanced healthcare systems with 5G Patient Rghioui et networks at their core to enhance RPM's efficiency. Monitoring al. (2020) Proposed an architecture for monitoring diabetic (RPM) patients using 5G, integrating ML algorithms for real- time data analysis and timely emergency notifications. A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 109 Key Element Author and Year Study Summary/ Findings Conducted a comprehensive survey on RPM systems, focusing on data acquisition, transmission protocols, Remote and cloud-based processing. Emphasized EHR Patient El- integration and early intervention as critical aspects for Monitoring Rashidy et enhancing RPM systems. Provided insights for (RPM) al. (2021) potential improvements in privacy and data transmission speed. Explored how 5G has revolutionized medical healthcare, especially RPM and its implications for Remote diabetes care. Introduced a 5G-based sensor node Patient Magsi et architecture for efficient health monitoring. Monitoring al. (2018) Highlighted the role of 5G in providing healthcare (RPM) services anywhere and improving the diagnosis of critical conditions. Explored AI, IoT, and 5G integration for real-time patient monitoring, emphasizing telemedicine and remote surgeries. Highlighted the role of AI in smart Real-Time Mohanta wearables and how IoT devices with AI overcome Data et al. limitations in fourth-generation healthcare systems. Transmission (2019) Focused on the urgent healthcare needs of remote surgeries and Tactile Internet, facilitated by 5G. Addressed the demand for real-time blood glucose prediction in managing type 1 diabetes, leveraging IoMT and deep learning. Introduced an innovative Real-Time deep learning model integrated into an IoMT-enabled Data Zhu et al. wearable device for accurate blood glucose prediction Transmission (2023) and hypoglycemia detection. Demonstrated IoMT's potential in revolutionizing type 1 diabetes management. Presented a comprehensive mobile medicine system for elderly health monitoring, focusing on real-time data transmission. Described a system architecture Real-Time involving data collection, analysis, decision-making, Data Wu et al. and transmission, emphasizing early warnings for Transmission (2021) doctors. Addressed the importance of remote medical services for elderly populations living alone. 110 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Key Element Author and Year Study Summary/ Findings Introduced the 5G-Smart Diabetes system, incorporating sensing, personalized diagnosis, and data-sharing layers. Enabled sustainable, cost- Personalized M. Chen effective, and intel igent diabetes diagnosis through Treatment et al. wearables, ML, and big data, with seamless data Plans (2018) transmission through the 5G network. Aimed to provide comprehensive sensing and analysis for diabetic patients. Emphasized 5G's potential in revolutionizing global Personalized healthcare systems, including personalized treatment Treatment Latif et al. plans, with IoT, AI, big data, and ML as catalysts. Plans (2017) Highlighted technical advancements and research opportunities in realizing this transformative vision. Explored the evolving landscape of personalized healthcare services driven by IoT and 5G technology. Advocated for interconnected healthcare systems Personalized Taimoor tailored to individual needs, with a focus on AI and Treatment and ML algorithms. Outlined key requirements for Plans Rehman Comprehensive Personalized Healthcare Services (2022) (CPHS) and a three-layer architecture for IoT-based healthcare systems. A. Nilashini, N. Wickramasinghe: Investigating the Benefits of 5g to Leverage the Digital Transformation in Healthcare: A Systematic Review on Personalized Diabetes Self-Management 111 Feature Author and Year Study Findings High-Speed 5G technology enables high-speed data transmission Data Riaz et al. for wearable head imaging devices, enhancing real- Transmission (2023) time monitoring. 5G's low latency and high-speed data transmission Devi et al. improve RPM by immediately transmitting patient (2023) data to medical staff. 5G facilitates remote consultations with fast Zhang et transmission of medical image data, improving al. (2020) diagnosis accuracy. 5G integrates with sensors for real-time data Rghioui et transmission and processing, revolutionizing diabetes al. (2020) management. 5G enables real-time transmission of medical data, Chen et al. improving remote medical procedures and patient (2021) outcomes. 5G's low latency and high device capacity optimize Low Latency Rghioui et healthcare communication, improving patient al. (2020) monitoring and diagnosis. 5G supports URLLC, reducing latency for real-time Devi et al. data transmission in healthcare applications. (2023) Chen et al. 5G's low latency is crucial for real-time retinopathy (2021) treatment via telemedicine. 5G ensures reliable connectivity for wearable health Network Magsi et devices, enhancing personal health monitoring and Reliability al. (2018) disease management. 5G's reliability supports continuous monitoring, Devi et al. clinical decision-making, and AI integration in (2023) wearable devices. Taimoor Strategies are discussed to improve network reliability and in IoT, including healthcare applications. Rehman (2022) 112 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION DATA INNOVATION EXPLORER: DESIGN OF A DATA-DRIVEN BUSINESS MODEL TOOL JOSCHKA ANDREAS HÜLLMANN,1 TORSTEN GOLLHARDT,2 ANDREAS HERMANN,2 PAUL KRUSE2 1 University of Twente, Faculty of Behavioral, Management and Social Sciences, Enschede, Netherland j.huel mann@utwente.nl 2 University of Muenster, Department of Information Systems, Münster, Germany {gollhardt, hermann, kruse}@ercis.de We are now living in the data economy with data as the central fuel for operating data-driven business models. Especially incumbent companies are constantly chal enged by rapid technological change and emerging business models that utilize data for value creation. Consequently, every company must rethink and, possibly, renew its business model over time to remain successful. Various tools have been proposed by practice Keywords: and academia in order to enable and facilitate business model digital innovation. Although IT tools for supporting business model transformation, business innovation proved to be meaningful, IT tools for data-driven model business model innovation are relatively scarce. Hence, we aim innovation, for the design of an IT tool to enable and facilitate data-driven data-driven business business model innovation. To reach the research objective, we model, employ a design science research approach accompanied by an business experimental evaluation design. In this research, we propose four model tool, design design features for IT tools supporting data-driven business science model innovation. research DOI https://doi.org/10.18690/um.fov.4.2024.7 ISBN 978-961-286-871-0 114 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction We are living in the data economy, where almost al aspects of everyday life are increasingly digitized, and a plethora of data is stored for analysis and subsequent value generation. The emergence of new technologies leads to an exponential increase in available data (Spiekermann 2019). At the same time, advances in data analysis, data storage, data sharing, and computing power accelerate the rise of the data economy (Zuboff 2019). The data economy depicts an economic perspective that understands data as an economic good with two primary purposes: the use and the trade of data (Bründl et al. 2015; Hüllmann et al. 2021). We focus on the use of data, where data is monetized by building a value chain around it (Spiekermann 2019). Consequently, data-driven business models are Business Models (BMs) that create value from data and data processing. Prominent sources of data include sensors, financial transactions, social media logs, or web-tracking. These data sources enable organizations to explore new data-based ways of creating value–ultimately al owing them to rethink and renew their traditional BMs toward data-driven BMs (Barann et al. 2019). However, innovating an organization’s BM toward a data- driven BM is a nontrivial undertaking (Barann et al. 2019; Foss and Saebi 2018). As a remedy, IT tools may facilitate the process of data-driven BM innovation (Bouwman et al. 2020). Fruhwirth et al. (2020) have already explored existing tools for ideating and evaluating data-driven BMs. Although various propositions (e.g., taxonomies, patterns, and visual tools) have been found in the literature, IT tools for supporting data-driven BM innovation remain scarce (Fruhwirth et al. 2020). IT tools can be useful for supporting BM innovation processes in many forms (e.g., Hermann et al. 2021; de Reuver et al. 2016; Spiekermann et al. 2018). However, creating value from data is fundamentally different from digital business models, which merely understand IT as an enabler. It remains unclear how a data-driven perspective can be integrated into such IT tools to provide meaningful support for creative tasks like data-driven BM innovation (cf. Fruhwirth et al. 2020; Szopinski et al. 2020). Hence, we address one research gap emphasized by Fruhwirth et al. (2020, p. 16) and aim to design “software tools as an IT support for developing, evaluating[,] and managing […] [data-driven BM innovation] based on information systems design methods.” J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 115 To address this research gap, we employ the design science research methodology by Peffers et al. (2007), which is appropriate for addressing real-world problems in the area of digital business models. We use and enhance the literature review by Fruhwirth et al. (2020) and analyze existing contributions concerning the design of an IT tool to support data-driven BM innovation. Based on these insights, design features are articulated and transferred into mockups. These features are integrated on a conceptual level into an existing software tool previously developed by the focal research team. The next step is evaluating the design features regarding user satisfaction and how they support innovation and creativity during the ideation process. We contribute to both academia and practice. First, the design features extend the design knowledge with respect to IT tools supporting data-driven BM innovation. Second, practitioners gain a better understanding of the interplay between their BMs as wel as the available and necessary data for data-driven BM innovation. The contributions are helpful for people collaborating on the design of data-driven business models. 2 Related Work 2.1 Data-driven Business Models Data-driven innovation represents a significant shift away from traditional approaches to BM development, where technology is viewed primarily as an enabler rather than a source of value creation. Creating value from data requires a fundamental y different approach, one that goes beyond simply integrating IT systems and instead focuses on leveraging data as a strategic asset. In this respect, three different types of data-driven innovation can be distinguished (Barann et al. 2019). First, data can be utilized to improve the organization’s business processes, which can enable subsequent innovation (Heberle et al. 2017; Schüritz and Satzger 2016; Sorescu 2017). Second, data-driven innovation may occur by enhancing individual BM components with data-driven aspects, e.g., product or service innovation (Heberle et al. 2017; Wiesböck and Hess 2020). Third, data-driven innovation can lead to entirely novel data-driven BMs (Barann et al. 2019; Schüritz and Satzger 2016; Sorescu 2017). Organizations no longer think of data as merely a by-product (Hess and Lamla 2019) but use it to invent new ways of value creation, which involve the generation, collection, storage, processing, search, analysis, and possibly the trade of data (Hartmann et al. 2016). The value generated by col ecting 116 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION and processing data can be captured through novel products and services. Compared to traditional value chains, data scales up and never depletes (Shapiro and Varian 1999; Spiekermann 2019). Just how lucrative data as an economic good is, is being showcased by the financial success of major players (e.g., Google and Facebook) and start-ups that purely operate on data and deliver data-related products and services (Klein and Hüllmann 2018). 2.2 Tools for Data-driven Business Model Innovation Recently, researchers have been focusing on data-driven BM innovation for incumbent organizations (Fruhwirth et al. 2020). In a comprehensive literature review, Fruhwirth et al. (2020) identified tools for ideating and evaluating data-driven BMs. Recurring types of contribution, among others, are taxonomies, frameworks, patterns, types, and visual tools (Fruhwirth et al. 2020). Taxonomies and frameworks support structuring and analyzing an organization’s BM regarding existing key concepts of data-driven BMs (Fruhwirth et al. 2020; Hartmann et al. 2016). Patterns and types can help position an organization’s BM and “serve as an inspiration and blueprint” (Hartmann et al. 2016, p. 1400) for data-driven BMs. “Visual tools mediate collaboration and support ideation for data-driven innovations” (Fruhwirth et al. 2020, p. 14). Other contributions, which are underrepresented in the literature, are IT tools for data-driven BMs innovation (cf. Fruhwirth et al. 2020). IT tools have proven useful to support creative tasks in the context of innovating an organization’s BM (Ebel et al. 2016; Osterwalder and Pigneur 2013; Veit et al. 2014). Furthermore, academia has proposed different design possibilities concerning IT tools for supporting BM innovation, i.e., BM development tools (e.g., Ebel et al. 2016; Schoormann et al. 2021; Szopinski et al. 2020). Szopinski et al. (2020) elaborated a taxonomy of 43 functions of BM development tools. These functions provide a useful template for designing IT-supported BM tools. However, digital BMs and data-driven BMs have a significant difference. A digital business model creates value through using (innovative) digital technologies. The means of production typically focus on software and information systems. Conversely, a data-driven BMs creates value through data and data processing (Spiekermann 2019). The means of production are not about software but statistics and data science. Key activities include col ection, preprocessing, analysis, presentation of data. Since the key resources and activities J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 117 differ between BMs and data-driven BMs, BM tools are not very helpful for innovating an organization’s business logic toward a data-driven BM (Fruhwirth et al. 2020). At the same, research on IT tools that support data-driven BM innovation is scarce. We address this lack and design an integrated IT tool for supporting data- driven BM ideation and evaluation. 3 Research Approach We employ the design science research methodology by Peffers et al. (2007), which is suitable for tackling real-world design problems. Our research plan is visualized in Figure 1, with preliminary results and future research indicated by checkmarks and empty circles, respectively. In this paper, we derive design features from the literature for an IT tool supporting data-driven BM innovation. The IT tool extends an existing tool that has been developed by the focal research team. The design features are conceptualized and visualized as mock-ups. In follow-up work, the features shall be implemented as a software prototype, iteratively improved by feedback retrieved through focus groups. Afterward, the prototype shal be evaluated in a laboratory experiment as an artificial evaluation episode (cf. Venable et al. 2016). Finally, a naturalistic evaluation period shall test the prototype in digital innovation projects with small and medium-sized enterprises (SMEs) (cf. Venable et al. 2016). To better inform our design science research plan, we opt for identifying the state of the art of data-driven BM tools. Thus, we perform a systematic literature review that extends the results by Fruhwirth et al. (2020) from May 2019. Our literature review methodology follows the recommendations by Webster and Watson (2002) and vom Brocke et al. (2009, 2015). We update the set of relevant articles within the time frame from early 2019 until October 2021 (see Appendix). The search string is adopted from Fruhwirth et al. (2020) and queried in the AIS Electronic Library, IEEE Xplore, Science Direct, Scopus, and Web of Science. For Google Scholar, a simplified search string (“data-driven business model”) is used. The articles are filtered according to literature valuation by Fruhwirth et al. (2020) and extended by a forward and backward search (Webster and Watson 2002). Based on the literature review, three researchers elaborated the design features for the intended IT- supported data-driven BM tool within four workshops. More specifically, the researchers jointly analyzed the literature to abstract recurring tool concepts into design knowledge for data-driven BM tools and, afterward, derive design features. 118 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Thereby, the resulting design features had to comply with two meta-requirements: First, potential features should support the ideation or evaluation of data-driven BM innovations. Second, potential features should integrate well into the existing concept of the already implemented IT tool. Figure 1: Research Approach adapted from Peffers et al. (2007) 4 Designing Tools for Data-driven Business Models 4.1 Initial Artifact Fol owing the objective to extend the existing IT-supported BM tool by Hermann et al. (2021) with a data-driven perspective, this section gives an overview of the initial tool’s logic and composition. PlanDigital has been developed in the course of multiple small-scale digital innovation projects in SMEs. PlanDigital integrates three selected BM tool concepts into one comprehensive toolset, i.e., a roadmapping tool, a BM documentation tool, and a tool for documenting company goals. The roadmapping tool can be considered the tool’s nexus, connecting the two other tools. On a roadmap, the user can orchestrate new BM innovation ideas along four time ranges: as-is, short-term, mid-term, and long-term. The continuous implementation of new ideas brings along changes to (components of) a company’s BM. To depict such changes, the user can model explicit links between innovation ideas on the roadmap and affected BM components. Moreover, PlanDigital lets users document various versions of a company’s BM. Besides the effects on a company’s BM, digitalization ideas are ultimately meant to contribute to the company goals. Company goals describe contextual boundaries that new ideas must adhere to. PlanDigital provides features to define company goals and assign new ideas from the roadmap to the fulfil ment of those. These core features of PlanDigital are implemented as two different modes, i.e., (1) an explorative, single J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 119 page dashboard-like view and (2) a stepper view that navigates the user in a wizard- like fashion. 4.2 Data-driven Design Features In our literature review, we found additional 40 articles. With the 33 articles already identified by Fruhwirth et al. (2020), in sum, 73 articles are considered relevant for our research objective (see appendix for further details). Despite numerous articles published only in the past two years, the results reflect previous findings by Fruhwirth et al. (2020). First, there is an imbalance between tools supporting ideation and those supporting idea evaluation tasks. Second, while most types of contributions are somewhat equally distributed in the literature, IT tools are again underrepresented. Next, we present new features that are planned to be integrated into the Data Innovation Explorer to enable and facilitate data-driven BM innovations (see Figure 2). Data-driven BM innovation needs to consider different perspectives, such as data (DF1) and business (DF2) while incorporating the temporal dynamics (DF3). DF4 brings everything together in an overview with best practices. 1. Describe and assess the potential of available data sources: According to the initial version of the software tool (cf. Section 4.1), every digital innovation is connected to an enabling technology. Those innovation- technology combinations continuously generate data available for new data- driven innovations. Since BM elements are explicitly linked to digital innovations (cf. Section 4.1), each element is, in turn, linked to those available data. Thus, existing data sources can be visualized for the various BM components. To include the underlying data in developing new data- driven innovation ideas, the Data Innovation Explorer integrates features that enable the description and assessment of existing sources. Such sources are depicted in the form of a dedicated data potentials profile. The profile reports on data origin (e.g., Hunke et al. 2019), data format (e.g., Kronsbein and Mueller 2019), source technology (e.g., Rizk et al. 2018), data entity (e.g., Weking et al. 2020), and data quality (e.g., Kühne and Böhmann 2019). 2. Document data requirements and relevant capabilities for new data- driven innovation ideas: It is usually more cost-effective for organizations to exploit the current resource base instead of starting on the green field. 120 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Therefore, the Data Innovation Explorer integrates features for evaluating the gap between available and required data and capabilities, following a bottom-up approach for generating new data-driven innovation ideas (Barann et al. 2019). The Data Innovation Explorer visualizes the gap between existing and required data regarding the dimensions data origin, data format, source technology, data entity, and data quality (e.g., Kayabay et al. 2022). A green color indicates a match, and a red color highlights a gap. For every new idea, the Data Innovation Explorer shows which data sources are considered for assessing the (mis-)match. Customer Segments 1 2 Key Partners Core Activities Value Proposition Customer Relationships Data Requirements Origin Internal As-is Situation Short Term Mid Term Long Term Data Potential Quality innovation Idea i> Quality Entity Customer Sales Source ERP System Entity Customer Sales Data Capabilities Format Structured Data modeling Data processing Costs Revenue Streams Based on… 3 4 Key Partners Core Activities Value Proposition Customer Customer Segments Relationships Technology- driven Innovation Key Resources Channels exploration Business model- driven Costs Revenue Streams Company Goals Business Model Roadmap Technologies Analytics-as-a-service Data generation & analysis Data col ector Figure 2: Mock-ups of the Design Features Source: Own 3. Enable different entry points for defining data-driven innovation ideas: To account for the dynamics of creative ideation processes (cf. Section 2.2), the Data Innovation Explorer offers three different paths for the generation of new data-driven ideas: idea generation may be technology- driven, BM-driven, or driven by innovation exploration (e.g., Rashed and J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 121 Drews 2021). This feature is integrated into the Data Innovation Explorer by pre-configuring the stepper-view (cf. Section 4.1). For instance, if idea generation follows a technology-driven approach, existing technologies are documented and analyzed early in the process, that is, immediately after defining company goals. Defining company goals is fixed as the first activity to set the contextual boundaries of the innovation process (e.g., Benta et al. 2017). 4. Provide templates of best practice data-driven BMs: Since the generation of data-driven innovation ideas is a creative and non-routine process, the Data Innovation Explorer integrates a col ection of best practice data-driven BM types (e.g., Hartmann et al. 2016). Especial y in early ideation activities, organizations benefit from overviews of best practices and pre-filled tools (e.g., Barann et al. 2019). The set of pre-filled canvases is thought of as a source of inspiration and orientation for developing data- driven BMs. 5 Evaluation We plan to evaluate our prototype in an iterative two-phase process for the artificial evaluation phase. The first phase consists of developmental feedback for improving the prototype and is based on conducting focus group discussions. After polishing the prototype, we plan to implement a controlled laboratory experiment to quantify the outcomes of user satisfaction as wel as creativity and innovation support. 5.1 Focus Group Our approach to focus groups follows established methodological recommendations (Crossley 2002; Krueger and Casey 2015). Inviting up to ten experts with experience in digital transformation, data-driven BMs, and data analytics, we conduct a focus group session lasting about 1-2 hours. During this focus group session, we divide the group into two subgroups and devise a hypothetical scenario in which we describe an existing business. The groups are assigned the task to develop and flesh out a data-driven BM based on the description of the business and using the tool’s various perspectives and features. The data- driven BM should incorporate the existing technologies and capabilities and propose 122 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION a mechanism for generating and capturing value. A guide is assigned to each group to help and advise concerning the tool’s features or the task’s peculiarities. Then, the groups are pooled, and in a focus discussion with al participants, further feedback is gathered and discussed. The discussion is recorded, transcribed, and coded (Kuckartz 2014; Saldaña 2015). The research team derives insights regarding the design features from the coded results and adapts the design and implementation of the tool accordingly. 5.2 Experiment After implementing the suggested changes derived from the focus group discussion, we set up an experiment. We adapt the same scenario that is used in the focus groups, making changes as necessary to reflect updates in the tool. Participants (n=50) work in groups of five for about 2-3 hours. They develop and flesh out a data-driven BM based on the description of the business according to the scenario depicted before. The experiment follows the randomized controlled trial protocol (Schulz et al. 2010) and randomly al ocates the groups to a treatment or control intervention, equally distributed. The treatment groups make use of the newly implemented version of the tool. In contrast, the control groups use the initial artifact of the tool that does not have any specific features supporting data-driven BM innovation. We record important control variables such as age, gender, colocation, and job role, to account for confounding effects (Maier, Laumer, Tarafdar, et al. 2021; Maier, Laumer, Thatcher, et al. 2021). The outcome variables are user satisfaction and innovation & creativity support, collected through a survey instrument. For user satisfaction, we adapt the SUS user satisfaction scale by Brooke (1996), which has been empirical y validated extensively (Bangor et al. 2008; Borsci et al. 2015). For innovation and creativity support, we aggregate and adapt the scales from Lukes and Stephan (2017), Janssen (2000), and Zhou and George (2001). We further model an interaction effect for IT affinity towards both user satisfaction and creativity and innovation support. The hypothesized model is illustrated in Figure 3. We estimate the model using SEM-CB, which is adequate for exploring causal inferences from survey data (Bollen and Pearl 2013). Since the participants are assigned to groups, we model unobserved group effects by adding a group fixed effect. J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 123 User Satisfaction Controls: − age Tool Use − gender − colocation Creativity & − job role Innovativeness IT affinity Figure 3: Hypothesized Evaluation Model Source: Own Follow-up interviews provide qualitative insights into how the participants experienced working with the tool. Drawing upon the quantitative and qualitative results, we conclude about the effectiveness of the designed and implemented prototype regarding user satisfaction and its effect on supporting creativity and innovation for developing data-driven BMs. Furthermore, we identify further potential for improvement left for the natural evaluation episode. 6 Concluding Remarks The ideation phase of the BM innovation process has received considerable attention in academic literature (Foss and Saebi 2018). Recently, scholars have focused on the emerging sub-domain of data-driven BM innovation (Fruhwirth et al. 2020). While there are many tools for supporting the ideation of digital BM innovations, tools for supporting data-driven BM innovation are scarce and only lately being designed and developed (Fruhwirth et al. 2020). In this paper, we use the literature review by Fruhwirth et al. (2020) to derive four core features of IT- supported data-driven BM innovation tools that can be integrated into PlanDigital: (1) describe and assess existing data potential, (2) document data requirements and relevant capabilities, (3) enable different entry points for data-driven innovation, and (4) provide templates of best practice data-driven BMs. The Data Innovation Explorer shal increase the effectiveness of the data-driven BM innovation process and lead to higher success rates in developing new data-driven BMs compared to using no such tool. Until today, we have developed a conceptual design for the Data Innovation Explorer, comprising a definition and graphical representation of the four features. Therefore, the effectiveness of the Data 124 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Innovation Explorer has not been proven yet. Following Venable et al. (2016), we intend to evaluate a prototype in an artificial evaluation episode by conducting focus groups and a laboratory experiment. With the experiment, we estimate how the prototype improves user satisfaction and supports creativity and innovation. Final y, rather than proposing a new tool to support data-driven BM innovation, we integrate propositions from the literature into an already existing IT solution to ultimately help practitioners in ideating and evaluating new data-driven BMs. The limitations of our study include that the prototype is in development but not yet finalized. Consequently, the design is conceptual and has not been empirically evaluated yet. Since there are no comparable data-driven BM tools available, there will be no baseline to compare the evaluation results. Although the scientific risk includes the possibility of negative evaluation, the Data Innovation Explorer builds upon a previous iteration of PlanDigital which was positively evaluated. Furthermore, the Data Innovation Explorer builds upon standard frameworks and packages. By building the tool, design knowledge about data-driven BM tools is generated. The evaluation of the Data Innovation Explorer generates theoretical insights about the success factors of data-driven BMs and their components. References Agrawal, A., Gans, J., and Goldfarb, A. 2018. “A Simple Tool to Start Making Decisions with the Help of AI,” Harvard Business Review. Bangor, A., Kortum, P. T., and Mil er, J. T. 2008. An Empirical Evaluation of the System Usability Scale, (24:6), pp. 574–594. Barann, B., Hermann, A., Cordes, A.-K., Chasin, F., and Becker, J. 2019. “Supporting Digital Transformation in Small and Medium-Sized Enterprises: A Procedure Model Involving Publicly Funded Support Units,” in Proc. 52nd Hawaii Int. Conf. Syst. Sci., Wailea, HI, pp. 4977–4986. Benta, C., Wilberg, J., Hol auer, C., and Omer, M. 2017. “Process Model for Data-Driven Business Model Generation,” in Proc. 21st Int. Conf. Eng. Des., Vancouver, Canada. Bock, M., and Wiener, M. 2017. “Towards a Taxonomy of Digital Business Models – Conceptual Dimensions and Empirical Il ustrations,” in 38th Int. Conf. Inf. Syst., Seoul, South Korea. Bollen, K. A., and Pearl, J. 2013. Eight Myths about Causality and Structural Equation Models, Handbooks of Sociology and Social Research, S. L. Morgan (ed.), Dordrecht: Springer Netherlands, pp. 301–328. (https://doi.org/10.1007/978-94-007-6094-3_15). Borsci, S., Federici, S., Bacci, S., Gnaldi, M., and Bartolucci, F. 2015. Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience, (31:8), pp. 484–495. Bouwman, H., de Reuver, M., Heikkilä, M., and Fielt, E. 2020. “Business Model Tooling: Where Research and Practice Meet,” Electronic Markets (30), pp. 413–419. J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 125 Brillinger, A.-S. 2018. “Mapping Business Model Risk Factors,” International Journal of Innovation Management (22:5). vom Brocke, J., Simons, A., Niehaves, Bjoern, Niehaves, Bjorn, Reimer, K., Plattfaut, R., and Cleven, A. 2009. “Reconstructing the Giant: On the Importance of Rigour in Documenting the Literature Search Process,” in Proc. 17th Eur. Conf. Inf. Syst. (ECIS 2009), Verona, Italy, pp. 2206–2217. vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut, R., and Cleven, A. 2015. Standing on the Shoulders of Giants: Chal enges and Recommendations of Literature Search in Information Systems Research, (37). Brooke, J. 1996. “SUS: A ‘Quick and Dirty’ Usability Scale,” in Usability Eval. Ind., P. W. Jordan, B. Thomas, B. A. Weerdmeester, and I. L. McClel and (eds.), London, United Kingdom: Taylor & Francis, pp. 189–194. Brownlow, J., Zaki, M., and Neely, A. 2015. Data and Analytics - Data-Driven Business Models - A Blueprint for Innovation, Working Paper., Cambridge, UK: Cambridge Service Al iance, pp. 1–15. Bründl, S., Hess, T., and Matt, C. 2015. “White Paper: Wertschöpfung in Datenmärkten - Eine Explorative Untersuchung Am Beispiel Des Deutschen Marktes Für Persönliche Daten,” No. January 2016. Crossley, M. L. 2002. “‘Could You Please Pass One of Those Health Leaflets along?’: Exploring Health, Morality and Resistance through Focus Groups,” Social Science & Medicine (55:8), pp. 1471–1483. Ebel, P., Bretschneider, U., and Leimeister, J. M. 2016. Leveraging Virtual Business Model Innovation: A Framework for Designing Business Model Development Tools, (26:5), pp. 519–550. Enders, T., Schüritz, R., and Frey, W. 2019. “Capturing Value from Data - Exploring Factors Influencing Revenue Model Design for Data-Driven Services,” in 14th Int. Conf. Wirtschaftsinformatik, Siegen, Germany. Engelbrecht, A., Gerlach, J., and Widjaja, T. 2016. “Understanding The Anatomy Of Data Driven Business Models - Towards An Empirical Taxonomy,” in 24th Eur. Conf. Inf. Syst., Istanbul, Turkey. Exner, K., Stark, R., Kim, J. Y., and Stark, R. 2017. “Data-Driven Business Model - A Methodology To Develop Smart Services,” in 2017 Int. Conf. Eng. Technol. Innov., Madeira, Portugal, pp. 146–154. Förster, M., Bansemir, B., and Roth, A. 2019. “Understanding the Role of Data for Innovating Business Models - A System Dynamics Perspective,” in 14th Int. Conf. Wirtschaftsinformatik, Siegen, Germany. Foss, N. J., and Saebi, T. 2018. “Business Models and Business Model Innovation: Between Wicked and Paradigmatic Problems,” Long Range Planning (51:1), pp. 9–21. Fruhwirth, M., Ropposch, C., and Pammer-Schindler, V. 2020. Supporting Data-Driven Business Model Innovations - a Structured Literature Review on Tools and Methods, (8:1). Hartmann, P. M., Zaki, M., Feldmann, N., and Neely, A. 2016. Capturing Value From Big Data – A Taxonomy Of Data- Driven Business Models Used By Start-up Firms, (36:10), pp. 1382– 1406. Heberle, A., Lowe, W., Gustafsson, A., and Vorrei, O. 2017. “Digitalization Canvas-towards Identifying Digitalization Use Cases and Projects.,” Journal of Universal Computer Science (23:11), pp. 1070–1097. Hermann, A., Gol hardt, T., Cordes, A.-K., and Kruse, P. 2021. “PlanDigital: A Software Tool Supporting the Digital Transformation,” in Proc. 16th Int. Conf. Des. Sci. Res. Inf. Syst. Technol. (DESRIST 2021), L. Chandra Kruse, S. Seidel, and G. I. Hausvik (eds.), Kristiansand, Norway [hybrid conference], pp. 356–361. Hess, T., and Lamla, J. 2019. “Einführung: Die Zukunft Der Datenökonomie. Zwischen Geschäftsmodel , Kollektivgut Und Verbraucherschutz,” in Die Zukunft Der 126 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Datenökonomie Zwischen Geschäftsmodell, Kol ekt. Und Verbraucherschutz, C. Ochs, M. Friedewald, T. Hess, and J. Lamla (eds.), Springer VS, pp. 1–10. Hüllmann, J.A., Sivakumar, A., and Krebber, S. 2021. “Data Management Platforms: An Empirical Taxonomy,” in Proceedings of the 34th Bled eConference, Bled, Slovenia. Hunke, F., Engel, C., Schüritz, R., and Ebel, P. 2019. “Understanding the Anatomy of Analytics-Based Services - A Taxonomy to Conceptualize the Use of Data and Analytics in Service,” in Proc. 27th Eur. Conf. Inf. Syst., Stockholm & Uppsala, Sweden. Hunke, F., and Schüritz, R. M. 2021. “Smartere Produkte Durch Analysebasierte Dienstleistungen – Ein Methodisches Werkzeug Zur Strukturierten Entwicklung,” in IoT - Best Pract., Wiesbaden, Germany: Springer Viewing, pp. 277–292. (https://doi.org/10.1007/978-3-658- 32439-1_16). Hunke, F., Seebacher, S., Schüritz, R., and Il i, A. 2017. “Towards a Process Model for Data-Driven Business Model Innovation,” in IEEE 2017 - 19th Conf. Bus. Informatics, CBI, Thessaloniki, Greece. Hunke, F., and Wambsganß, T. 2017. “Turning Data into Value: Towards an Ideation Tool for Key Activities of Data-Driven Business Models,” in 3rd Karlsruhe Serv. Summit Res. Work., Karlsruhe, Germany. Janssen, O. 2000. Job Demands, Perceptions of Effort-Reward Fairness and Innovative Work Behaviour, (73:3), pp. 287–302. Kayabay, K., Gökalp, M. O., Gökalp, E., Erhan Eren, P., and Koçyiğit, A. 2022. “Data Science Roadmapping: An Architectural Framework For Facilitating Transformation Towards A Data-Driven Organization,” Technological Forecasting and Social Change (174), p. 121264. Klein, S., and Hül mann, J. A. 2018. “Datenkapitalismus akademischer Wissenschaftsverlage,” Wirtschaftsdienst, 98(7), pp. 477–480. Kronsbein, T., and Muel er, R. 2019. “Data Thinking: A Canvas for Data-Driven Ideation Workshops,” in 52nd Hawai Int. Conf. Syst. Sci., Grand Wailea, Hawai , pp. 561–570. Krueger, R. A., and Casey, M. A. 2015. Focus Groups: A Practical Guide for Applied Research, (5th ed.), Thousand Oaks, CA, US: SAGE Publications Ltd, pp. 1–560. Kuckartz, U. 2014. Qualitative Text Analysis: A Guide to Methods, Practice & Using Software, London, United Kingdom: SAGE Publications Ltd. (https://doi.org/10.4135/9781446288719). Kühne, B., and Böhmann, T. 2019. Data-Driven Business Models - Building the Bridge Between Data and Value. (https://www.researchgate.net/publication/332687256). Lukes, M., and Stephan, U. 2017. Measuring Employee Innovation, (23:1), pp. 136–158. Maier, C., Laumer, S., Tarafdar, M., Mattke, J., Reis, L., and Weitzel, T. 2021. “Chal enge and Hindrance IS Use Stressors and Appraisals: Explaining Contrarian Associations in Post-Acceptance IS Use Behavior,” Journal of The Association for Information Systems (22:6), pp. 1590–1624. Maier, C., Laumer, S., Thatcher, J. B., Wirth, J., and Weitzel, T. 2021. Trial-Period Technostress: A Conceptual Definition and Mixed-Methods Investigation, (forthcoming), pp. 1–51. Osterwalder, A., and Pigneur, Y. 2013. “Designing Business Models and Similar Strategic Objects: The Contribution of IS,” Journal of The Association for Information Systems (14:5), pp. 237–244. (https://doi.org/10.17705/1jais.00333). Peffers, K., Tuunanen, T., Rothenberger, M. A., and Chatterjee, S. 2007. “A Design Science Research Methodology for Information Systems Research,” Journal of Management Information Systems (24:3), pp. 45–77. Rashed, F., and Drews, P. 2021. “Pathways Of Data-Driven Business Model Design And Realization - A Qualitative Research Study,” in Proceedings of the 54th Hawai International Conference on System Sciences. (http://hdl.handle.net/10125/71309). de Reuver, M., Athanasopoulou, A., Haaker, T., Roelfsema, M., Riedl, A., and Breitfuss, G. 2016. “Designing an ICT Tooling Platform to Support SME Business Model Innovation: Results of a First Design Cycle,” in BLED 2016 - 29th Bled eConference, Bled, Slovenia, pp. 556– 570. J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 127 Rizk, A., Bergvall-Kåreborn, B., and Elragal, A. 2018. “Towards a Taxonomy of Data-Driven Digital Services,” in Proc. 51st Hawaii Int. Conf. Syst. Sci. (HICSS 2018), Waikoloa Village, USA. Saldaña, J. 2015. The Coding Manual for Qualitative Reseachers, (3rd ed.), London, UK: SAGE Publications Ltd, p. 368. Schoormann, T., Stadtländer, M., and Knackstedt, R. 2021. “Designing Business Model Development Tools for Sustainability—a Design Science Study,” Electronic Markets (forthcoming). Schulz, K. F., Altman, D. G., and Moher, D. 2010. “CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials,” BMJ (Clinical Research Ed.) (340:mar23 1), C332—-C332. Schüritz, R., and Satzger, G. 2016. “Patterns of Data-Infused Business Model Innovation,” in Proc. 18th Conf. Bus. Informatics, Paris, France, pp. 133–142. Shapiro, C., and Varian, H. R. 1999. Information Rules: A Strategic Guide to the Network Economy, Boston, MA, USA: Harvard Business School Press. (https://doi.org/10.1145/776985.776997). Sorescu, A. 2017. Data-Driven Business Model Innovation, (34:5), pp. 691–696. Spiekermann, M. 2019. Data Marketplaces: Trends and Monetisation of Data Goods, Leibniz Information Centre for Economics, pp. 208–216. (https://doi.org/10.1007/s10272-019- 0826-z). Spiekermann, M., Tebernum, D., Wenzel, S., and Otto, B. 2018. “A Metadata Model for Data Goods,” in Multikonferenz Wirtschaftsinformatik 2018, Lüneburg, Germany. Szopinski, D., Schoormann, T., John, T., Knackstedt, R., and Kundisch, D. 2020. “Software Tools for Business Model Innovation: Current State and Future Chal enges,” Electronic Markets (30:3), pp. 469–494. Veit, D., Clemons, E., Benlian, A., Buxmann, P., Hess, T., Kundisch, D., Leimeister, J. M., Loos, P., and Spann, M. 2014. “Business Models: An Information Systems Research Agenda,” Business & Information Systems Engineering (6:1), pp. 45–53. Venable, J., Pries-Heje, J., and Baskerville, R. 2016. FEDS: A Framework for Evaluation in Design Science Research, (25:1), pp. 77–89. Webster, J., and Watson, R. T. 2002. “Analyzing the Past to Prepare for the Future: Writing a Literature Review,” MIS Q. (26:June). Weking, J., Stöcker, M., Kowalkiewicz, M., Böhm, M., and Krcmar, H. 2020. “Leveraging Industry 4.0 - A Business Model Pattern Framework,” International Journal of Production Economics (225), p. 107588. Wiesböck, F., and Hess, T. 2020. “Digital Innovations: Embedding in Organizations,” Electronic Markets (30:1), pp. 75–86. Zhou, J., and George, J. M. 2001. When Job Dissatisfaction Leads to Creativity: Encouraging the Expression of Voice, (44:4), pp. 682–696. Zuboff, S. 2019. “Surveillance Capitalism - Überwachungskapitalismus,” Aus Polit. Und Zeitgeschichte Daten-Ökonomie (69), Bonn, Germany, pp. 24–26. 128 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix Table 1: Identified Artefacts in the Literature Review Fruhwir Design Title Author Outlet th et al. Feature (2020) (DF) A Simple Tool to Start Making Agrawal et al. Harvard Business Decisions with the Help of AI 2018 Review X 4 Process Model for Data- 21st International Driven Business Model Benta et al. Conference on X 1 & 2 Generation 2017 Engineering Design Towards a Taxonomy of Digital Business Models - Bock and 38th International Conceptual Dimensions and Wiener 2017 Conference on X 1 Empirical Illustrations Information Systems Mapping Business Model Risk International Journal Factors Brillinger 2018 of Innovation X 2 Management Data and Analytics - Data- Driven Business Models - A Brownlow et Blueprint for Innovation al. 2015 Working Paper X 2 & 3 Capturing Value from Data - Exploring Factors Influencing Enders er al. 14th International Revenue Model Design for 2019 Conference on X 2 Data-Driven Services Wirtschaftsinformatik Understanding The Anatomy Of Data Driven Business Engelbrecht et 24th European Models - Towards An al. 2016 Conference on X 1 Empirical Taxonomy Information Systems 23rd International Data-driven Business Model - Conference on A Methodology To Develop Exner et al. Engineering, X 2 & 4 Smart Services 2017 Technology and Innovation Understanding the Role of Data for Innovating Business Förster et al. 14th International Models - A System Dynamics 2019 Conference on X 3 Perspective Wirtschaftsinformatik Capturing Value From Big International Journal Data – A Taxonomy Of Data- Hartmann et al. of Operations & Driven Business Models Used 2016 Production X 1 & 4 By Start-up Firms Management Smartere Produkte durch analysebasierte IoT – Best Practices. Dienstleistungen – Ein Hunke and Edition HMD. X 3 methodisches Werkzeug zur Schüritz 2021 Springer Vieweg, strukturierten Entwicklung Wiesbaden Turning Data into Value: Hunke and Karlsruhe Service Towards an Ideation Tool for Wambsganß 2017 Summit X 3 J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 129 Fruhwir Design Title Author Outlet th et al. Feature (2020) (DF) Key Activities of Data-Driven Business Models Towards a Process Model for Data-Driven Business Model Hunke et al. 19th Conference on Innovation 2017 Business Informatics X 3 Understanding the Anatomy of Analytics-Based Services - 27th European A Taxonomy to Conceptualize Hunke et al. Conference on X 1 the Use of Data and Analytics 2019 Information Systems in Service Leveraging The Value Of Data-driven Services Systems 27th European In Manufacturing - A Graph- Kammler, 2019 Conference on X 1 based Approach Information Systems Data Science as an Innovation Technology Chal enge. From Big Data to Kayser, 2018 Innovation X 2 & 3 Value Proposition Management Review Data Thinking: A Canvas for 52nd Hawaii Data-Driven Ideation Kronsbein & International X 4 Workshops Müller, 2019 Conference on System Sciences Requirements for Representing Data-Driven Kühne & 24th Americas Business Models - Towards Böhmann, Conference on X 2 Extending the Business Model 2018 Information Systems Canvas Data-Driven Business Models Kühne & 27th European - Building the Bridge Between Böhmann, Conference on X 1 & 2 Data and Value 2019 Information Systems Data-Need Fit – Towards Service Design and Data-Driven Business Model Mathis & Innovation X 1 Innovation Köbler, 2016 Conference The Data Value Map: A Framework for Developing Nagle & 25th European Shared Understanding on Sammon, 2017 Conference on X 1 & 2 Data Initiatives Information Systems 51st Hawaii Towards a Taxonomy of Rizk et al., International Data-driven Digital Services 2018 Conference on System X 4 Sciences Charting the Emerging Financial Services Ecosystem 51st Hawaii of Fintechs and Banks - Six Schmidt et al., International Types of Data-Driven 2018 Conference on System X 4 Business Models in the Sciences Fintech Sector Patterns of Data-Infused Schüritz & 18th Conference on Business Model Innovation Satzger, 2016 Business Informatics X 4 130 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Fruhwir Design Title Author Outlet th et al. Feature (2020) (DF) How To Cultivate Analytics Capabilities Within An 25th European Organisation - Design And Schüritz et al., Conference on X 2 Types Of Analytics 2017 Information Systems Competency Centers Capturing Value from Data - 50th Hawaii Revenue Models for Data- Schüritz et al., International X 4 Driven Services 2017 Conference on System Sciences A Metadata Model for Data Spiekermann et Multikonferenz Goods al., 2018 Wirtschaftsinformatik X 1 On the Utility of E-Health 24th European Business Model Design Sprenger & Conference on X 4 Patterns Mettler, 2016 Information Systems Design, Implement, Repeat: Essays on Business Model Terrenghi, Management in Offline-Born 2019 PhD Thesis X 1 Organizations Data Value Assessment: Recognizing Data as an Wixom & MIT CISR Research Enterprise Asset Markus, 2015 Briefing X 2 Making Money from Data Wrapping: Insights from Wixom & MIT CISR Research Product Managers Schüritz, 2018 Briefing X 2 Business Model 24th European Transformation Patterns of Zolnowski et Conference on X 4 Data-Driven Innovations al., 2016 Information Systems Towards a Cost-Benefit- 13. Internationale Analysis of Data-Driven Zolnowski et Tagung X 1 Business Models al., 2017 Wirtschaftsinformatik 40th International Archetypes For Data-driven Conference on Business Models For Müller & Information Systems, Manufacturing Companies In Buliga, 2019 Special Interest Group 4 Industry 4.0 on Big Data Proceedings DDI: A Novel Technology Design, Automation & And Innovation Model For Armengaud et Test in Europe Dependable, Col aborative al., 2021 Conference & 3 And Autonomous Systems Exhibition Decision Framework for Journal of Cases on Engaging Cloud-Based Big Ayaburi et al., Information 1 Data Analytics Vendors 2020 Technology A Taxonomy for Data-Driven 24th Pacific Asia Services in Manufacturing Azkan et al., Conference on 1 & 2 Industries 2020 Information Systems Accountable Algorithms? The Ethical Implications Of Data- Breidbach & Journal of Service driven Business Models Maglio, 2020 Management 2 J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 131 Fruhwir Design Title Author Outlet th et al. Feature (2020) (DF) The Data-Driven Business 32nd Bled Value Matrix - A Classification Breitfuss et al., eConference Scheme for Data-Driven 2020 Humanizing 3 Business Models Technology for a Sustainable Society AI-Enabled Business-model Innovation And Transformation In Industrial Burström et al., Journal of Business Ecosystems: A Framework, 2021 Research 4 Model And Outline For Further Research Creating Value From Energy Data: A Practitioner’s Perspective on Data-Driven Chasin et al., Schmalenbach Smart Energy Business 2020 Business Review 4 Models What Makes A Data-driven 29th European Business Model? A Dehnert et al., Conference on 1 & 2 Consolidated Taxonomy 2021 Information Systems 32 Ways to Innovate Business Models Through Data: Emerging Data-Driven 54th Hawaii Solution Business Model Eber et al., International 4 Patterns From a Study of 471 2021 Conference on System Late Stage Data-Driven Sciences Startups A Systematic Mapping Study Faber et al., 25th Americas on Business Ecosystem Types 2019 Conference on 2 Information Systems The Data Product Canvas - A Visual Collaborative Tool for Fruhwirth et Designing Data-Driven al., 2020 BLED 1 & 2 Business Models Pre-ICIS SIGDSA To Sell or Not to Sell_ (Symposium on Knowledge Risks in Data- Fruhwirth et Inspiring mindset for 2 Driven Business Models al., 2019 Innovation with Business Analytics and Data Science) A Network-based Tool for 54th Hawaii Identifying Knowledge Risks Fruhwirth et International in Data-driven Business al., 2021 Conference on System 1 Models Sciences Applying Frameworks for Journal of Systems Cognitive Services in IIoT Gain, 2021 Science and Systems 3 Engineering Developing Data Driven 30th ISPIM Business Models for Haaker et al., Innovation 2 Interactive Media Companies 2019 Conference: 132 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Fruhwir Design Title Author Outlet th et al. Feature (2020) (DF) Celebrating Innovation - 500 Years Since Da Vinci Designing for Digital Please Tel Me What to Do – Transformation. Co- Towards a Guided Creating Services with Orchestration of Key Hunke et al., Citizens and Industry. 1 Activities in Data-Rich Service 2020 DESRIST 2020. Systems Lecture Notes in Computer Science Pathways from Data to Value: Identifying Strategic Hunke et al., Wirtschaftsinformatik Archetypes of Analytics-Based 2020 (Zentrale Tracks) 4 Services The Role Of Analytics In Data-driven Business Models Technology Of Multi-sided Platforms: An Isabel e et al., Innovation 1 Exploration In The Food 2020 Management Review Industry Data Science Roadmapping: An Architectural Framework Technological For Facilitating Kayabay et al., Forecasting and Social 2 Transformation Towards A 2021 Change Data-driven Organization Realizing Value with Data and Analytics - A Structured 54th Hawaii Literature Review on Kayser et al., International 2 Classification Approaches of 2021 Conference on System Data-Driven Innovations Sciences Data Col ection Map - A Canvas for Shared Data Kayser et al., Pre-ICIS SIGDSA Awareness in Data-Driven 2019 Symposium 1 Innovation Projects Formative Evaluation of Data- Kühne & 53rd Hawaii Driven Business Models–The Böhmann, International 1 & 2 Data Insight Generator 2020 Conference on System Sciences Making Data Tangible for 25th Americas Data-driven Innovations in a Kühne et al., Conference on 1 Business Model Context 2019 Information Systems From Ideation to Realization - Essential Steps and Activities Lange & 22nd Conference on for Realizing Data-Driven Drews, 2020 Business Informatics 1 Business Models Business Model of Energy Big 4th Conference on Data Service Based on Li et al., 2020 Energy Internet and 4 Business Canvas Theory Energy System Integration J. A. Hül mann et al.: Data Innovation Explorer: Design of a Data-Driven Business Model Tool 133 Fruhwir Design Title Author Outlet th et al. Feature (2020) (DF) A Data-driven Reversible Framework For Achieving Sustainable Smart Product Li et al., 2021 Journal of Cleaner Production 3 Service Systems Operationalizing Business Model Innovation through Big Minatogawa et Data Analytics for Sustainable al., 2019 Sustainability 2 Organizations Data-driven Business Models 53rd Hawaii in Logistics: A Taxonomy of Möller et al., International Optimization and Visibility 2020 Conference on System 1 & 2 Services Sciences Trading Social Visibility for Economic Amenability: Data- Ochs et al., Science, Technology, based Value Translation on a 2021 & Human Values 2 “Health and Fitness Platform” Pathways Of Data-driven 54th Hawaii Business Model Design And Rashed & International Realization - A Qualitative Drews, 2021 Conference on System 3 Research Study Sciences How Does Enterprise Architecture Support the International Design and Realization of Rashed & Conference on 3 Data-Driven Business Drews, 2021 Wirtschaftsinformatik Models? An Empirical Study Implications of Service-related 26th CIRP Conference Business Models on Product Riesener et al., on Life Cycle 2 Development Processes 2019 Engineering Big Data Business Models: Schroeder, Chal enges And Opportunities 2016 Cogent Social Sciences 1 How AI Capabilities Enable Business Model Innovation - Scaling AI Through Co- Sjödin et al., Journal of Business evolutionary Processes And 2021 Research 3 Feedback Loops Redesigning Business Models For Data-driven Innovation: Troisi et al., Research and A Three Layered Framework 2020 Innovation Forum 3 Industrial Data-Driven Business Models- Towards a Voigt et al., Goods-Service-Data 2021 Book Section 1 & 2 Continuum Leveraging Industry 4.0 - A International Journal Business Model Pattern Weking et al., of Production 4 Framework 2020 Economics Opportunity Complementarity in Data-Driven Business Xu et al., 2020 Journal of Business Models Models 4 134 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Fruhwir Design Title Author Outlet th et al. Feature (2020) (DF) Innovation Capabilities As A Mediator Between Big Data Ylijoki et al., Journal of Enterprise And Business Model 2018 Transformation 1 UNMASKING BIASES AND MAPPING THE LANDSCAPE OF AI ADOPTION IN DIVERSE BUSINESS FUNCTIONS THROUGH A DELPHI STUDY AMOGHA AMMAVA SUDARSHAN, ISABELLA SEEBER 1 Grenoble Ecole de Management, Information Systems for Society, Grenoble, France amogha.sudarshan@grenoble-em.com, isabel a.seeber@grenoble-em.com Artificial Intel igence (AI) tools are exceedingly being introduced in various business sectors as a way to improve efficiency and drive overal organisational performance. Prior research has uncovered many success and failure factors influencing the adoption of these tools. However, in the absence of a common understanding between practitioners and researchers, factors deemed theoretical y significant do not always align with reality, resulting in a researcher bias in AI adoption literature. Additionally, these factors and their priorities depend on specific business functions, deeming existing one-size-fits-all AI adoption theories incapable of explaining these nuances. To address these shortcomings, this study investigates the existence of a potential researcher bias and establishes factors influencing Keywords: AI adoption in different business functions through a 2-fold, 3- artificial round, 3-panel Delphi study. The findings establish a potential intelligence (AI), adoption of AI, researcher bias and confirm that factors influencing adoption, delphi and their priorities, differ by business functions. This study study, contributes to literature by first establishing the potential AI-driven organisations, researcher bias and then furthering the understanding of factors AI-driven influencing adoption for different business contexts. In a pivotal innovation, contribution to practice, this study enables organisations to AI-driven marketing, foster better adoption practices based on different business HR and functions. finance DOI https://doi.org/10.18690/um.fov.4.2024.8 ISBN 978-961-286-871-0 136 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Artificial Intelligence (AI) is revolutionising the way organisations function in different business sectors. In the form of business tools, this technology has permeated several domains like manufacturing (Li et al., 2017), hospitality (Nam et al., 2021; Price, 2019), finance (Ahmed et al., 2022; Bahrammirzaee, 2010), marketing (Chintalapati & Pandey, 2022; Davenport et al., 2020; Wierenga, 2010), and administration (Brougham & Haar, 2018; Kolbjørnsrud et al., 2016). For example, analysts at the McKinsey Global Institute estimate that 56 percent of typical “hire-to-retire” tasks could be automated using machine learning and cognitive agents (Bustamante & Gandhi, 2018). Related research suggests various factors that lead to successful AI adoption in workplaces, such as trust (Bedué & Fritzsche, 2021), technology readiness (Brock & Khan, 2017; Janssen et al., 2020), top management support (Duan et al., 2017; Kurup & Gupta, 2022; Saberi et al., 2019; Yang et al., 2015) and explainability (Lee & Shin, 2020; Solaimani et al., 2023). Though the various advantages of using AI technology at work are established, there exist several chal enges to its adoption in the workplace like limited understanding of the technology (Nam et al., 2021; Volkmar, 2020; Zerfass et al., 2020), lack of skilled personnel (Hair Jr. & Sarstedt, 2021; Zerfass et al., 2020), implementation issues (Kolbjørnsrud et al., 2016; Nam et al., 2021) and technological inadequacies (Jarek & Mazurek, 2019; Wierenga, 2010). Such insights into the success and failure factors are usually gained by researchers who study how professionals use AI tools in the workplace. Hence, the findings are the result of an interpretation phase by the researchers who analyse data based on their own experiences, values, opinions, and knowledge (Chenail, 2016; Miyazaki & Taylor, 2008; Romano et al., 2021). This can bias the inquiry, misinterpret the results and reduce the trustworthiness of research (Chenail, 2016). Moreover, researchers may be unfamiliar with AI tools and may potential y infuse biases either in the data col ection or the analysis stage (Romano et al., 2020). Potential biases among researchers could also be exacerbated by the popular AI narrative perpetuated in the general media (Ouchchy et al., 2020), even though training in the scientific method should lower this influence (Chenail, 2016). A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 137 When a researcher bias exists, subsequent findings may not be in line with the views practitioners hold, posing a problem for the development of theory that sufficiently reflects reality. It could mean that a developed theory cannot adequately explain the real world. While there exists research on the success and failure factors on AI adoption, we are not aware of studies that investigate if researchers and practitioners share a similar view on success and failure factors. If both populations share a similar view, current research wil not suffer from the consequences of a researcher bias. However, if researchers and practitioners have different views, then a potential research bias may be a threat to theory development on AI adoption. Therefore, this study investigates the following research question, “How do practitioners and researchers dif er in their views on the success and failure factors of adoption of AI-based tools?” Additional y, factors influencing adoption can vary across different business functions. However, current research on AI adoption often builds on general theories such as Diffusion of Innovation (DOI), Unified Theory of Acceptance and Use of Technology (UTAUT) and Technology, Organisation and Environment (TOE) (Radhakrishnan & Chattopadhyay, 2020), that are domain-independent. We posit that distinct factors influence the adoption of AI tools in different business functions. For example, efficiency and scalability may be the priorities for AI tools aiding in organisational management, while creativity and experimentation may be important for AI used to aid innovation, whereas personalisation and trainability may be more important for the core business domains of marketing, finance and HR. This phenomenon remains largely unaddressed in current literature, leading to our second research question, “What are the distinct sets of factors influencing AI adoption across diverse business functions, and how do they vary in their prioritisation? ” To answer these research questions, we conducted a 2-fold, 3-round, 3-panel Delphi study with 16 experts, consisting of practitioners and researchers. The findings of this research show that there indeed exists researcher bias in identifying the drivers and barriers of AI adoption in workplaces. Further, it shows that the AI adoption factors and their priorities vary according to the respective business functions. 138 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 2 Background 2.1 Researcher bias When a researcher influences the outcome of a study based on their expectations, either consciously or subconsciously, it results in a researcher bias (Romano et al., 2021). They are of two types, either stemming from a biased research design (Fagenson, 1990; McDonald, 2000) or by interaction with participants (Houston & Gremler, 1993; Kahn & Cannell, 1957). These biases could be the product of questionable research practices, or the researchers’ expectation of a positive outcome to the concepts being studied, or the immaturity of the field under investigation (Romano et al., 2020). Further, simply the way a researcher interacts with a respondent can bias the answers of the respondent and potential y compromise the results (Miyazaki & Taylor, 2008). Yet, achieving complete control over researcher biases is considered infeasible (Miyazaki & Taylor, 2008). This underscores the need to examine disparities in perspectives between researchers and practitioners, in the context of AI adoption, because it could reveal potential areas of misalignment in theory. Further, the lasting impact such researcher bias has on research discourse and its cascading implications on policy-related decisions is documented in many cases. For example, clinical intervention in empirical studies was affected by such a bias (Crossley et al., 2008), leading to findings that drove decisions favouring pro-industry outcomes (Berkman et al., 2014; Kunz et al., 2007). Researcher bias also leads to exaggerated effects, with around 80% of the effects reported in economics research being inflated (Ioannidis et al., 2017). Shepperd et al., (2014) used 600 empirical studies and established that researcher bias influences the outcome of the model being built. Similar critiques of research establishing the concept of biological determinism, which posits that human intel igence is dependent on race and economic status (Morton & Combe, 1839), show how empirical research by scientists exhibiting researcher bias could influence decades of public policy (Gould, 2003). One such example in the USA was the enacting of the Immigration Act of 1924 including the Asian Exclusion Act, that favoured immigrants from Northern and Western Europe while reducing the number of immigrants from other parts of the world, who were deemed to have lower intel igences based on these experiments that suffered from researcher bias (Gould, 2003). A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 139 Establishing such a bias, if it exists in IS research, wil urge researchers to employ techniques to reduce it, thereby increasing the practical impact of research into the AI adoption process. This constitutes the first objective of this study. 2.2 Adoption factors hinging on business functions Adoption factors of AI-based technology are usually studied for a business as a whole (Dasgupta & Wendler, 2019; Kar & Kushwaha, 2023; Kurup & Gupta, 2022; Solaimani et al., 2023). However, several sub categories of business functions like organisational management, innovation, and marketing, finance and HR possess unique operational contexts and objectives. In the context of organisational management, extant research shows that factors like effective implementation (Richards et al., 2019), technical compatibility, organisational readiness and users’ expertise (Nguyen et al., 2022) enable successful adoption of AI tools. In the innovation context, other success factors play a role, such as availability of data (Liu et al., 2020; Mikalef & Gupta, 2021; Trocin et al., 2021), reduced cost (Liu et al., 2020) and organisational readiness (Mikalef & Gupta, 2021). The adoption of AI tools in the core business functions of marketing, finance and HR are yet again driven by other success factors like trust, usefulness, top management support (Pan et al., 2021; Wang et al., 2023), regulatory support, and financial readiness (Gupta et al., 2022). However, studies continue to build theories for AI as a whole. Establishing the unique factors and the priorities of each of them wil benefit both researchers and practitioners (Radhakrishnan & Chattopadhyay, 2020). This is the second objective of this study. In summary, the review of extant literature shows that the potential presence of a researcher bias is a chal enge for theorising AI adoption factors. Moreover, domain- specific research on AI-adoption shows that adoption factors vary across business functions, which chal enges the applicability of existing technology adoption models as theoretical lenses. 140 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3 Methodology This study adopted the Delphi method of inquiry, as prescribed by (Okoli & Pawlowski, 2004). It is a systematic and iterative approach, which elicits expert consensus on complex and uncertain topics (Okoli & Pawlowski, 2004). It is a suitable method to answer the outlined research questions, because it allows an investigation into the potential researcher bias affecting research on success and failure factors (pertaining to RQ1) and the differences in these factors between business functions and their prioritisation (pertaining to RQ2). The study was conducted over 10 months, spanning over 3 rounds of contemplation, in 3 different panels, with 2 types of experts and 16 participants. 3.1 Participants 16 experts participated in al of the rounds of the Delphi study, of which eight were practitioners and eight were researchers. As suggested by Okoli & Pawlowski, (2004), the first step in the study was to develop a Knowledge Resource Nomination Worksheet (KRNW). This list included 84 experts, of both researchers and practitioners. They were purposeful y selected based on their expertise in organisational management, innovation, or core business operations like marketing, finance and HR. To participate, the researchers had to have published at least two papers in AI-based applications and the practitioners had to have at least two years of experience in working with AI-based tools. 58 of these experts were invited to participate in the study. We incentivised participation with gift vouchers equivalent to 80 Euros. 14 of the final participants accepted the gift voucher, two of them refused compensation. Al the participants gave informed consent. Two participants withdrew from the study after round 1 resulting in 16 experts that finished round 3. In an effort to reduce social y desirable answers, in each of the 3 surveys the participants were reminded that their responses would remain anonymous, and they were not primed by quoting any success or failure factor beforehand (Joinson, 1999). 3.2 Procedure The first round of the Delphi study commenced when the first Qualtrics questionnaire was sent to these experts. The objective of this stage of the study was to identify both success and failure factors of adoption of AI tools in workplaces, A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 141 thus necessitating a two-fold study. All the experts were sent the same questionnaire that briefly explained the purpose of this research and asked them to list and explain factors that successfully and unsuccessfully influenced the adoption of AI tools at their workplaces (for the practitioners) and as dictated by their research (for the researchers). This resulted in 18 of the 58 experts returning the first round of the study. In total, the experts returned 278 responses, 107 for success factors and 171 for failure factors of adoption of AI. The responses were qualitatively analysed, using the template-based thematic coding technique (Cassell & Gillian, 2004). This method of analysis was found to be suitable because the responses identified themes of success and failure factors, and were viewed with a conventional positivistic position of quantitative social science. This method entailed reading each response and creating themes of factors discussed therein or adding them to existing themes and updating these for each of the consecutive responses. Final y, the language was unified, resulting in distilling these responses into a total of 22 success factors and 16 failure factors. In the second and third rounds, the 18 participants were split into 3 panels, based on their expertise and research interests, as seen in Appendix A: the AI-driven organisational management panel (ORM), the AI-driven innovation panel (INN) and a panel of core business applications cal ed the AI-driven marketing, finance and HR panel (MFHR), with 6 experts in each panel. The objective of the second round was to solicit the priorities of each of the factors, as identified by the experts and their fel ow panel ists. For example, an expert in the INN panel was given one list of success factors and one of failure factors, that included their own responses from the first round and the ones from their fel ow panel ists, but not those of the ORM panel. The expert was asked to rank these factors by perceived priorities. In this round, al experts were informed of the panel they were added to, but during no part of the study were they informed about the names or any other identifiers of the other experts. This resulted in two lists (one each for success and failure factors) as ranked by each expert, exclusive to each panel. The objective of the third round was to build a consensus of factors and their priorities among the panel ists. The experts were now informed of the mean ranks of each of the factors (as calculated from the results of the previous round) and were given the opportunity to reconsider the ranks of each factor, based on the opinions 142 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION of their felow panelists. Each expert was a sent separate questionnaire (see appendix B) that included the list of factors, the mean panel ranks of each of the factors, a short explanation of the factor (as derived from al the responses of the panel) along with the rank they themselves had assigned to the factors in the previous round. The questionnaire mentioned that the experts could choose to retain the rank they had assigned to each factor or rerank the factors. This resulted in two lists of exhaustive factors that were first ranked and then reconsidered and reranked, thereby building a concordance of priorities of these drivers and barriers of AI adoption. The Delphi process employed in this study is visualised in Appendix C. 4 Results The first objective of this study was to understand if there exists a potential researcher bias in the investigation of success and failure factors of AI adoption in the workplace (RQ1). The second was to investigate if and to what extent these factors differ between business functions (RQ2). 4.1 Researcher bias in AI adoption We investigated a potential researcher bias by assessing the average ranks the researchers and practitioners assigned to the success and failure factors of AI adoption. A researcher bias is indicated if the average importance of factors differs drastically between researchers and practitioners. We assessed the Kendal 's W factor as suggested by (Okoli & Pawlowski, 2004), which is a non-parametric measure of concordance, to elicit how much practitioners and researchers agree or disagree with each other. The factor ranges from 0 to 1; 0 indicating complete disagreement and 1 indicating absolute agreement. Tables 1 and 2 visualise the average ranks per factor for practitioners (P) and researchers (R) for each subpanel and for rounds 2 (before the consensus-making phase) and 3 (after the consensus-making phase). Kendall's W factors are indicated in the two bottom rows of Tables 1 and 2, to help interpret the consensus per group. In the final round, the panel ists of the ORM and INN panels had a substantial agreement (Schmidt, 1997) on the success and failure factors (with Kendall’s W > 0.6). The panellists of MFHR had a fair agreement with Kendall’s W of 0.346 and 0.386 for the success and failure factors, respectively. To assess the possibility of a A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 143 researcher bias, each panel is broken down into subpanels of researchers and practitioners. In support of a researcher bias, both the researchers and practitioners, among themselves, had a better agreement about the factors affecting adoption and their priorities. This agreement level drops when they are considered as a whole panel (i.e., combining researchers and practitioners). This is evidenced by 9 out of 12 (75%) of these subpanels having a higher agreement level, when compared to the whole panel, for the success factors and 11 out of 12 (92%) for the failure factors. In the context of success factors, some of them like availability of data and the fear of missing out in ORM, bet er forecasting in INN and ease of use and ef iciency of the tool in MFHR have perfect agreement among the subpanels. However, factors like ease of implementation in ORM and ease of use in INN have vastly different priorities. These differences drive the researcher bias, because practitioners and researchers do not consider the same factors to be as important. Though the failure factors like lack of trust and high implementation cost in ORM, algorithmic bias in both INN and MFHR and false outcomes in MFHR show agreement, some others like lack of user control in ORM, lack of trust in both INN and MFHR and its unnecessity in MFHR show substantial divergence in prioritisation. In summary, the agreement level within the subpanels (researchers and practitioners separately) is higher when compared to the panels. To assess the level of difference, the percentage change between the agreement levels of the subpanels to the panels were calculated for each round and each panel. This difference ranges from the lowest of 3.83% (the success factors as ranked by the researchers in the INN) to the highest change of 105.96% (the failure factors as ranked by the researchers in the MFHR panel). Hence, the data supports the existence of a researcher bias. 144 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 1: The difference in success factor rankings among researchers and practitioners AI-driven organisational AI-driven innovation AI-driven marketing, management (INN) finance and HR (MFHR) Success factors (ORM) Round 2 Round 3 Round 2 Round 3 Round 2 Round 3 rank rank rank rank rank rank P R P R P R P R P R P R Availability of data 4 3 1 1 Better forecasting 6 2 11 11 10 12 12 12 6 5 5 6 Competitors usage 16 15 9 10 8 16 14 16 Cost reduction 3 7 2 5 11 2 3 2 Ease of implementation 10 6 12 15 Ease of use 5 8 4 3 1 6 1 6 2 1 1 1 Easier task automation 7 8 9 8 Easy implementation 16 9 16 9 Efficiency of the tool 1 1 3 6 2 1 2 1 1 2 2 2 Fear of missing out 13 13 16 16 13 15 15 15 Frequency of usage 9 10 8 4 Human control 14 14 7 8 Impact on deliverables 14 3 6 3 5 6 6 5 Knowledge of 15 4 13 4 3 4 4 4 A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 145 AI-driven organisational AI-driven innovation AI-driven marketing, management (INN) finance and HR (MFHR) Success factors (ORM) Round 2 Round 3 Round 2 Round 3 Round 2 Round 3 rank rank rank rank rank rank P R P R P R P R P R P R the technology/ AI literacy Meticulous implementation 11 12 13 14 Potential for innovation 7 9 10 9 Quality of information 12 5 7 5 Task automation 2 4 5 7 Top management 8 5 6 2 9 14 11 14 support/ Team support Trainability of the tool 3 13 8 13 Transparency 15 11 14 13 6 11 10 11 4 3 3 3 Trust 4 7 4 7 Kendall's W 0.44 0.64 0.25 (for each 2 7 0.919 0.81 6 1 0.38 9 1 0.786 0.327 0.36 5 0 0.629 subpanel) Kendall's W (for each panel) 0.195 0.820 0.389 0.757 0.217 0.346 Note: P = practitioners, R = researchers 146 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 2: The difference in failure factor rankings among researchers and practitioners AI-driven AI-driven marketing, organisational AI-driven finance and HR innovation (INN) Failure management (ORM) (MFHR) factors Round 2 Round 3 Round 2 Round 3 Round 2 Round 3 rank rank rank rank rank rank P R P R P R P R P R P R Algorithmic bias 9 12 12 12 10 6 9 9 Blackbox/ lack of 5 11 7 10 10 10 10 9 6 8 6 7 transparency Data privacy concerns/ 1 2 1 2 1 3 1 2 5 7 7 3 regulations False outcomes 10 6 6 9 4 5 5 5 Fear of missing out 13 13 13 12 High implementatio 4 8 5 5 5 4 4 1 1 1 1 2 n cost Implementatio n time 2 2 2 3 constraints Incompatibility with existing IT 3 1 3 5 infrastructure Inefficient tool 8 5 9 7 8 7 7 6 2 3 2 4 Lack of clean data 6 3 2 3 12 11 11 8 8 11 11 8 Lack of data 3 1 3 1 7 10 10 6 A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 147 AI-driven AI-driven marketing, organisational AI-driven finance and HR innovation (INN) Failure management (ORM) (MFHR) factors Round 2 Round 3 Round 2 Round 3 Round 2 Round 3 rank rank rank rank rank rank P R P R P R P R P R P R Lack of expertise 2 9 4 4 Lack of trust 7 7 8 8 4 9 8 11 9 2 8 10 Lack of user control 11 4 10 6 6 5 5 4 Not necessary 9 12 12 13 11 9 4 11 Operational chal enges 12 10 11 11 7 6 6 7 Kendall's W 0.79 (for each 0.438 0.395 0.553 0.74 8 1 0.38 2 1 0.63 0 0.168 0.335 0.48 1 5 subpanel) Kendall's W (for each 0.249 0.603 0.259 0.608 0.074 0.386 panel) Note: P = practitioners, R = researchers 4.2 AI adoption in different business functions To answer RQ2, from the second round onwards, we broke up the experts into 3 panels. The success and failure factors of adoption exhibited notable discrepancies across the three panels, in both their identification and prioritisation. This supports RQ2, by highlighting the inherent differences in AI adoption across different business functions. The specific factors are visualised in Figure 1 and are discussed next. The five success factors common among al panels are bet er forecasting, ease of use, ef iciency of the tool, transparency of the tool and the AI literacy of the users. The MFHR panel has no unique driver or barrier of AI adoption, suggesting that the 148 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION operationalisation of AI tools in such work functions highly overlap with other AI systems, like the ones used to drive innovation or organisational management. Further, the INN and ORM panels have the highest level of bi-panel overlap of five factors ( competitors’ usage, cost reduction, fear of missing out, human control and top management or team support). This indicates that the drivers of business functions that require efficiency and scalability (for organisational management) and creativity and generative models (for innovation) share operational commonalities. In support of RQ2, 50% of success factors were unique and panel-specific, while only 23% of them were common across al the panels. Figure 1: Success and failure factors of AI adoption identified by the 3 panels Source: Own The seven failure factors common to al three subpanels were: AI being a black box, data privacy regulations, high implementation costs, inef icient tools, lack of clean data, lack of trust and lack of user expertise. The ORM and MFHR panels share three common failure factors, including AI tools providing false outcomes, lack of data and it not being necessary. In summary, 43% of the failure factors were common among al the panels and 19% were unique panel-specific factors. This indicates that in contrast to success factors, most failure factors are applicable across business functions. We complemented this descriptive analysis with a correlation analysis on the normalised ranks of the 5 success and 7 failure factors common between the 3 panels. This revealed a high correlation (r = 0.879) between the priorities of the common success factors as determined by the ORM and INN panels. The same was A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 149 high between the INN and MFHR panels (r = 0.859) and the ORM and MFHR panels (r = 0.844). Hence, this indicates that experts prioritised success factors which are common across al application domains similarly important. The agreement on the priority levels of the 7 common failure factors is more contrastive. The correlation of the ranks between the panels was significantly lesser, with r = 0.307 between the ORM and INN panels, r = 0.365 between the INN and MFHR panels and r = 0.002 between the ORM and MFHR panels. This suggests that experts ranked the importance of the common failure factors dissimilarly. The contrast in these agreement levels across panels underscores the diverse nature of AI's application in different business contexts and highlights the need for tailored evaluation frameworks within distinct application areas. In summary, providing further support to RQ2, the analysis shows that the unsuccessful adoption of AI tools is influenced by domain-specific objectives and priorities, and the success factors seem to stem from systemic issues that cut across business functions. 5 Discussion This study identified a potential researcher bias in the examination of the adoption factors of AI tools in workplaces. Beginning with the success factors, several disparities were identified, first on the matter of implementation of AI tools, where a researcher (RE026) who ranked it high, claimed, “. .should be an incremental process ensuring inclusion”, while a practitioner (PE011) who ranked it low wrote, “Integrating into existing software…as for any software.” Similarly, a researcher (RE021) discussing the ease of use ranked it low, and wrote, “Usability… as for any software”, while a practitioner (PE010) who ranked it high wrote, “. .a key, especially for a non-technical person.” Further, a similar trend of disagreement was seen in the discussion of failure factors like lack of user control, when a practitioner (PE040) who ranked it high claimed, “. .some functions should be overridable by human intelligence”, but a researcher (RE010) who ranked it low opined, “lack of control in the decision-making process…is hard to analyse why a particular output is given.” Trust in the AI tool was another contentious factor, with a practitioner (PE040) claiming, “. .trust is a huge factor in research but not in industry…”. This claim seems to be true as a researcher (RE022) wrote, “I thought this (trust) would be top 150 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1…”. The other disputed failure factor was the unnecessity of AI tools. A practitioner (PE010) who ranked it high claimed, “. .would not want to switch…if necessity is not seen.”, however a researcher (RE018) who ranked it low, wrote, to justify the low rank, “AI adoption comes from a strategic level”, insinuating that the actual users may not make decisions about its necessity. Given these inconsistencies in the identification of adoption factors between researchers and practitioners, and the fact that researchers’ opinions could shape the outcome of a study (Romano et al., 2021), it is beneficial to reconsider the research direction in order to ensure that future research on the matter sufficiently represents the reality it examines. Secondly, this study also shows that success and failure factors are unique to different business functions and that they are varied in their prioritisation. The reason for the differences in AI adoption could be explained by the unique capabilities that AI offers for different contexts of usage. For example, in organisational management it is used to streamline processes and enhance employee efficiency, but in the innovation domain, it is used to foster creativity and research, but in core business domains of marketing, finance, and HR, it is used majorly as predictive analytics for advertising strategies and talent acquisition through natural language processing algorithms. To further this point, current research in organisational management focuses on effective implementation (Richards et al., 2019), technical compatibility, organisational readiness and users’ expertise (Nguyen et al., 2022). But in the same context, this research brings to light other success factors like the AI tool having a potential to drive innovation and its frequency of usage. Similar observations can be made for failure factors and the 2 other business functions being studied here, but are not elucidated due to space constraints. These findings underscore the need for future research to explore these newfound factors in specific contexts of usage, to build theory that more closely reflects reality. A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 151 6 Contribution 6.1 Theoretical contributions Our findings uncovered the presence of a potential researcher bias in studies of AI adoption factors in the workplace. The ramifications of such a researcher bias could range from potential distortion of research findings (Miyazaki & Taylor, 2008), to wrongly impacting policy creation, to perpetuating certain perspectives that may not reflect the reality they represent (Chenail, 2016). This study serves as a stepping stone to acknowledging this bias so that researchers can employ methods to counteract this bias, as suggested in (Chenail, 2016; Romano et al., 2020). Our findings also show that AI adoption factors differ across business functions and are assigned different priorities. Most success factors are not common across business functions and mostly have dissimilar priorities. In contrast, different business functions often share the same failure factors but are assigned different priorities. With this, we enrich the current discourse on AI adoption in the literature, which study AI adoption for business as a whole (Bérubé et al., 2021; Dasgupta & Wendler, 2019; Kar & Kushwaha, 2023; Kurup & Gupta, 2022; Solaimani et al., 2023), because existing cross-domain adoption theories can only cover a part of the bigger picture. However, this study encourages researchers to step away from examining the drivers and barriers of AI adoption in workplaces holistical y, and take into consideration the unique operational contexts and objectives. This implies that domain-specific theories may be needed to best explain and predict AI adoption. 6.2 Practical contribution For practitioners, this research offers curated lists of success and failure factors that can serve as a navigational framework for AI adoption initiatives. For example, if a customer segmentation tool is slated to be adopted for marketing purposes in an organisation, they can prioritise the ease of use and the efficiency of the tool (ranked 1 and 2 respectively, by the MFHR panel), as opposed to a supervised learning model being used to track employee performance in an organisation, where the availability of data should be taken into consideration first (ranked 1 by the ORM panel). This extends to the factors that hinder the adoption of AI tools as wel . This study is rich with ranked lists in 3 different business functions, discussed over 3 rounds, 152 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION providing valuable insights to all stakeholders involved to foster the adoption of such tools in the workplace. 6.3 Methodological contribution Final y, in a contribution to research methodology, to the best of the knowledge of the authors, this is the first paper to employ a dual-fold Delphi study. Delphi studies are usual y conducted over one phenomenon and build consensus on the matter at hand. In this study, two distinct but related themes of the success and failure factors of AI adoption were discussed and a consensus was drawn. As shown in the results, the factors that foster better adoption do not necessarily hinder the adoption in their absence, in the same level of priority. For example, the efficiency of the tool was discussed to have the highest priority as a success factor by the INN panel, however, inefficiency of the tool was not the highest ranked failure factor, but was ranked as the 6th most important failure factor by the same panel. Expanding the Delphi study to encompass a dual focus (both success and failure factors), researchers can uncover nuanced insights of the matter, by unearthing asymmetric causal relationships (Wagemann et al., 2016). 7 Limitations and future research This study suffers from a few limitations. 16 experts were involved in all three rounds. Even though this is a fairly adequate sample size of experts in a Delphi study (Okoli & Pawlowski, 2004), we would have liked a larger sample to better represent the underlying populations. For example, the practitioners in the INN panel were al involved in software development, which could explain why their assessment of AI- literacy was rather low. Adding practitioners with a business background could overcome this limitation. To add to the robustness of the findings, future research could verify the findings through a survey of practitioners. Another limitation of this study is that we could not definitively assess researcher bias for the MFHR subpanels due to the limited number of factors generated and that concordance levels within subpanels were general y low. One of the reasons could be that business disciplines such as marketing, HR and finance may involve more diverse interpretations and understanding of the AI technology and could result in lower concordance levels due to the inherent complexities and variations in A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 153 assessing performance or impact. Thus, we recommend splitting marketing, finance and HR into separate panels. Future researchers can extend the expert pool to include decision makers in organisations. This wil produce a more rounded understanding of the third stakeholder involved, the decision-makers in institutions, as opposed to the 2 primary stakeholders in the research of AI adoption factors, the practitioners and the researchers. 8 Conclusion This paper employs the Delphi method to demonstrate the researcher bias present in studies that examine the adoption of AI tools in the workplace. Further, the findings show that diverse business functions of organisational management, innovation, and marketing, finance and HR possess both similar and dissimilar drivers and barriers to adoption of AI tools, and that these vary in their priorities across the panels. Further, through a two-fold study, this paper elucidates that a factor which could drive the adoption of an AI tool, need not necessarily hinder the adoption in its absence, in the same level of priority. Finally, this paper provides lists of drivers and barriers of AI adoption and their priorities, as decided by both researchers and practitioners, for 3 distinct business functions, which could help practitioners and policy makers foster better adoption practices at their workplaces. Acknowledgement The authors would like to acknowledge the Digital Organisation and Society chair, operating under the aegis of the Grenoble Ecole de Management, for funding the data collection of this study. References Ahmed, S., Alshater, M. M., Ammari, A. E., & Hammami, H. (2022). Artificial inteligence and machine learning in finance: A bibliometric review. Research in International Business and Finance, 61, 101646. https://doi.org/10.1016/j.ribaf.2022.101646 Bahrammirzaee, A. (2010). A comparative survey of artificial intelligence applications in finance: Artificial neural networks, expert system and hybrid intelligent systems. Neural Computing and Applications, 19(8), 1165–1195. https://doi.org/10.1007/s00521-010-0362-z Bedué, P., & Fritzsche, A. (2021). Can we trust AI? An empirical investigation of trust requirements and guide to successful AI adoption. Journal of Enterprise Information Management, 35(2), 530–549. https://doi.org/10.1108/JEIM-06-2020-0233 Berkman, N. D., Santaguida, P. L., Viswanathan, M., & Morton, S. C. (2014). The empirical evidence of bias in trials measuring treatment differences. 154 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION https://europepmc.org/article/nbk/nbk253181 Bérubé, M., Giannelia, T., & Vial, G. (2021). Barriers to the Implementation of AI in Organizations: Findings from a Delphi Study. Hawai International Conference on System Sciences. https://doi.org/10.24251/HICSS.2021.805 Brock, V., & Khan, H. U. (2017). Big data analytics: Does organizational factor matters impact technology acceptance? Journal of Big Data, 4(1), 21. https://doi.org/10.1186/s40537-017- 0081-8 Brougham, D., & Haar, J. (2018). Smart Technology, Artificial Intelligence, Robotics, and Algorithms (STARA): Employees’ perceptions of our future workplace. Journal of Management and Organization, 24(2), 239–257. https://doi.org/10.1017/jmo.2016.55 Bustamante, M & Gandhi, N. (2018, January 23). Human resources in the age of automation. HR Analytics and Technology. https://www.mckinsey.com/capabilities/people-and- organizational-performance/our-insights/the-organization-blog/human-resources-in-the- age-of-automation Cassell, C., & Gillian, S. (2004). Essential Guide to Qualitative Methods in Organizational Research. Sage. Chenail, R. (2016). Interviewing the Investigator: Strategies for Addressing Instrumentation and Researcher Bias Concerns in Qualitative Research. The Qualitative Report. https://doi.org/10.46743/2160-3715/2009.2821 Chintalapati, S., & Pandey, S. K. (2022). Artificial intelligence in marketing: A systematic literature review. International Journal of Market Research, 64(1), 38–68. https://doi.org/10.1177/14707853211018428 Crossley, N. A., Sena, E., Goehler, J., Horn, J., van der Worp, B., Bath, P. M. W., Macleod, M., & Dirnagl, U. (2008). Empirical Evidence of Bias in the Design of Experimental Stroke Studies. Stroke, 39(3), 929–934. https://doi.org/10.1161/STROKEAHA.107.498725 Dasgupta, A., & Wendler, S. (2019). AI adoption strategies. Retrieved September, 18, 2022. Davenport, T., Guha, A., Grewal, D., & Bressgott, T. (2020). How artificial intelligence will change the future of marketing. Journal of the Academy of Marketing Science, 48(1), 24–42. https://doi.org/10.1007/s11747-019-00696-0 Duan, Y., Miao, M., Wang, R., Fu, Z., & Xu, M. (2017). A framework for the successful implementation of food traceability systems in China. The Information Society, 33(4), 226– 242. https://doi.org/10.1080/01972243.2017.1318325 Fagenson, E. A. (1990). At the heart of women in management research: Theoretical and methodological approaches and their biases. Journal of Business Ethics, 9(4–5), 267–274. https://doi.org/10.1007/BF00380326 Gould, S. J. (2003). The Mismeasure of Man. In Arguing About Science. Routledge. Gupta, S., Ghardallou, W., Pandey, D. K., & Sahu, G. P. (2022). Artificial intelligence adoption in the insurance industry: Evidence using the technology–organization–environment framework. Research in International Business and Finance, 63, 101757. https://doi.org/10.1016/j.ribaf.2022.101757 Hair Jr., J. F., & Sarstedt, M. (2021). Data, measurement, and causal inferences in machine learning: Opportunities and chal enges for marketing. Journal of Marketing Theory and Practice, 29(1), 65–77. https://doi.org/10.1080/10696679.2020.1860683 Houston M B & Gremler D D. (1993). Biases in the Researcher/Informant Interaction in the Col ection of Marketing Research Data: A Cognitive Framework. Enhancing Knowledge Development in Marketing - AMA Educators’ Proceedings, 4(American Marketing Association, Chicago), 311–319. Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2017). The Power of Bias in Economics Research. The Economic Journal, 127(605), F236–F265. https://doi.org/10.1111/ecoj.12461 Janssen, M., Weerakkody, V., Ismagilova, E., Sivarajah, U., & Irani, Z. (2020). A framework for analysing blockchain technology adoption: Integrating institutional, market and technical factors. International Journal of Information Management, 50, 302–309. A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 155 Jarek, K., & Mazurek, G. (2019). Marketing and Artificial Intelligence. Central European Business Review, 8(2), 46–55. https://doi.org/10.18267/j.cebr.213 Joinson, A. (1999). Social desirability, anonymity, and internet-based questionnaires. Behavior Research Methods, Instruments, & Computers, 31(3), 433–438. https://doi.org/10.3758/BF03200723 Kahn, R. L., & Cannel , C. F. (1957). The dynamics of interviewing; theory, technique, and cases. https://psycnet.apa.org/record/1957-07878-000 Kar, A. K., & Kushwaha, A. K. (2023). Facilitators and Barriers of Artificial Intel igence Adoption in Business – Insights from Opinions Using Big Data Analytics. Information Systems Frontiers, 25(4), 1351–1374. https://doi.org/10.1007/s10796-021-10219-4 Kolbjørnsrud, V., Amico, R., & Thomas, R. (2016). The promise of artificial intelligence: Redefining management in the workforce of the future. Kunz, R., Vist, G. E., & Oxman, A. D. (2007). Randomisation to protect against selection bias in healthcare trials. In The Cochrane Col aboration (Ed.), Cochrane Database of Systematic Reviews (p. MR000012.pub2). John Wiley & Sons, Ltd. https://doi.org/10.1002/14651858.MR000012.pub2 Kurup, S., & Gupta, V. (2022). Factors Influencing the AI Adoption in Organizations. Metamorphosis, 21(2), 129–139. https://doi.org/10.1177/09726225221124035 Lee, I., & Shin, Y. J. (2020). Machine learning for enterprises: Applications, algorithm selection, and challenges. Business Horizons, 63(2), 157–170. Li, B., Hou, B., Yu, W., Lu, X., & Yang, C. (2017). Applications of artificial intelligence in intelligent manufacturing: A review. Frontiers of Information Technology & Electronic Engineering, 18(1), 86–96. https://doi.org/10.1631/FITEE.1601885 Liu, J., Chang, H., Forrest, J. Y.-L., & Yang, B. (2020). Influence of artificial intelligence on technological innovation: Evidence from the panel data of china’s manufacturing sectors. Technological Forecasting and Social Change, 158, 120142. McDonald, G. (2000). Cross-Cultural Methodological Issues in Ethical Research. In J. Sójka & J. Wempe (Eds.), Business Chal enging Business Ethics: New Instruments for Coping with Diversity in International Business (pp. 89–104). Springer Netherlands. https://doi.org/10.1007/978-94-011-4311-0_10 Mikalef, P., & Gupta, M. (2021). Artificial intelligence capability: Conceptualization, measurement calibration, and empirical study on its impact on organizational creativity and firm performance. Information & Management, 58(3), 103434. Miyazaki, A. D., & Taylor, K. A. (2008). Researcher Interaction Biases and Business Ethics Research: Respondent Reactions to Researcher Characteristics. Journal of Business Ethics, 81(4), 779– 795. https://doi.org/10.1007/s10551-007-9547-5 Monica Wu. (2023, September 1). Artificial Intelligence in R&D. https://www.sanofi.com/en/our- science/rd-focus-areas/artificial-intelligence-rd Morton, S. G., & Combe, G. (1839). Crania Americana; or, a comparative view of the skulls of various aboriginal nations of North and South America: To which is prefixed an essay on the varieties of the human species. Philadelphia: J. Dobson; London: Simpkin, Marshal . https://books.google.com/books?hl=en&lr=&id=VX5PAQAAMAAJ&oi=fnd&pg=PA1& dq=Crania+Americana&ots=G-XYB0GUJO&sig=3o1yfQHNbghHjI6uR-QNQcohMng Nam, K., Dutt, C. S., Chathoth, P., Daghfous, A., & Khan, M. S. (2021). The adoption of artificial intelligence and robotics in the hotel industry: Prospects and chal enges. Electronic Markets, 31(3), 553–574. https://doi.org/10.1007/s12525-020-00442-3 Nguyen, T. L., Nguyen, V. P., & Dang, T. V. D. (2022). Critical Factors Affecting the Adoption of Artificial Intelligence: An Empirical Study in Vietnam. The Journal of Asian Finance, Economics and Business, 9(5), 225–237. https://doi.org/10.13106/JAFEB.2022.VOL9.NO5.0225 Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1), 15–29. https://doi.org/10.1016/j.im.2003.11.002 156 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Ouchchy, L., Coin, A., & Dubljević, V. (2020). AI in the headlines: the portrayal of the ethical issues of artificial intelligence in the media. AI & SOCIETY, 35, 927-936. Pan, Y., Froese, F., Liu, N., Yunyang, H., & Ye, M. (2021). The adoption of artificial intelligence in employee recruitment: The influence of contextual factors. The International Journal of Human Resource Management, 33, 1–23. https://doi.org/10.1080/09585192.2021.1879206 Price, K. (2019, August 2). Artificial intelligence: The future of hospitality? The Caterer, 207, 10–11. Radhakrishnan, J., & Chattopadhyay, M. (2020). Determinants and Barriers of Artificial Intelligence Adoption – A Literature Review. In S. K. Sharma, Y. K. Dwivedi, B. Metri, & N. P. Rana (Eds.), Re-imagining Diffusion and Adoption of Information Technology and Systems: A Continuing Conversation (pp. 89–99). Springer International Publishing. https://doi.org/10.1007/978-3-030-64849-7_9 Richards, G., Yeoh, W., Chong, A., & Popovič, A. (2019). Business Intel igence Effectiveness and Corporate Performance Management: An Empirical Analysis. Journal of Computer Information Systems, 59, 188–196. https://doi.org/10.1080/08874417.2017.1334244 Romano, S., Fucci, D., Scanniel o, G., Baldassarre, M. T., Turhan, B., & Juristo, N. (2020). Researcher Bias in Software Engineering Experiments: A Qualitative Investigation (arXiv:2008.12528). arXiv. http://arxiv.org/abs/2008.12528 Romano, S., Fucci, D., Scanniel o, G., Baldassarre, M. T., Turhan, B., & Juristo, N. (2021). On researcher bias in Software Engineering experiments. Journal of Systems and Software, 182, 111068. https://doi.org/10.1016/j.jss.2021.111068 Saberi, S., Kouhizadeh, M., Sarkis, J., & Shen, L. (2019). Blockchain technology and its relationships to sustainable supply chain management. International Journal of Production Research, 57(7), 2117–2135. https://doi.org/10.1080/00207543.2018.1533261 Schmidt, R. C. (1997). Managing Delphi surveys using nonparametric statistical techniques. decision Sciences, 28(3), 763-774. Solaimani, S., Dabestani, R., Prentice, T. H., El is, E., Kerr, M., Choudhury, A., & Bakhshi, N. (2023). Exploration and Prioritization of Critical Success Factors in Adoption of Artificial Intel igence: A mixed-methods study. Int. J. Bus. Inf. Syst.(Forthcom.). https://www.researchgate.net/profile/Sam- Solaimani/publication/367079699_Exploration_and_Prioritization_of_Critical_Success_Fact ors_in_Adoption_of_Artificial_Intelligence_a_mixed- methods_study/links/650c3052c05e6d1b1c1f424f/Exploration-and-Prioritization-of- Critical-Success-Factors-in-Adoption-of-Artificial-Intelligence-a-mixed-methods-study.pdf Shepperd, M., Bowes, D., & Hal , T. (2014). Researcher bias: The use of machine learning in software defect prediction. IEEE Transactions on Software Engineering, 40(6), 603-616. Trocin, C., Hovland, I. V., Mikalef, P., & Dremel, C. (2021). How Artificial Intelligence affords digital innovation: A cross-case analysis of Scandinavian companies. Technological Forecasting and Social Change, 173, 121081. https://doi.org/10.1016/j.techfore.2021.121081 Volkmar, G. V. (2020). How Can Marketing Managers Thrive in the Age of Artificial Intelligence? In Á. Rocha, J. L. Reis, M. K. Peter, & Z. Bogdanović (Eds.), Marketing and Smart Technologies (pp. 444–448). Springer. https://doi.org/10.1007/978-981-15-1564-4_41 Wagemann, C., Buche, J., & Siewert, M. B. (2016). QCA and business research: Work in progress or a consolidated agenda? Journal of Business Research, 69(7), 2531–2540. https://doi.org/10.1016/j.jbusres.2015.10.010 Wang, C., Ahmad, S. F., Bani Ahmad Ayassrah, A. Y. A., Awwad, E. M., Irshad, M., Ali, Y. A., Al- Razgan, M., Khan, Y., & Han, H. (2023). An empirical evaluation of technology acceptance model for Artificial Intelligence in E-commerce. Heliyon, 9(8), e18349. https://doi.org/10.1016/j.heliyon.2023.e18349 Wierenga, B. (2010). Marketing and Artificial Intelligence: Great Opportunities, Reluctant Partners. In J. Casil as & F. J. Martínez-López (Eds.), Marketing Intel igent Systems Using Soft Computing: Managerial and Research Applications (pp. 1–8). Springer. https://doi.org/10.1007/978-3-642-15606-9_1 A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 157 Yang, Z., Sun, J., Zhang, Y., & Wang, Y. (2015). Understanding SaaS adoption from the perspective of organizational users: A tripod readiness model. Computers in Human Behavior, 45, 254– 264. Zerfass, A., Hagelstein, J., & Tench, R. (2020). Artificial intelligence in communication management: A cross-national study on adoption and knowledge, impact, chal enges and risks. Journal of Communication Management, 24(4), 377–389. https://doi.org/10.1108/JCOM-10-2019- 0137 158 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix A Table A: The expertise of each of the panel ists Researcher Panel assigned Practitioner Research area Job title Gender Field of work Job title Gender AI in data driven decision support, auditing Project Project Professor Male AI and data- management manager Male driven business AI-driven models organisational management AI in Chief Senior (ORM) organisations/ Information Male researcher Male IT Department management Officer AI in Associate Product organisations professor Male Project management director Female Researcher Software Principal AI in policy in interactive Male development architect Male intelligence AI-driven AI in Senior Research innovation innovation, software Male associate Female Software (INN) development product design engineer AI in software Assistant Director of development professor Male Software development delivery Male Finance/ risk Financial AI in finance management consultant Female and risk Professor Male management Marketing and Sales AI-driven sales engineer Male marketing, finance and Human human resources Human resources Male (MFHR) AI in marketing Assistant Resources analyst and sales professor Female Data usage in Data project SME market lead Female A. A. Sudarshan, I. Seeber: Unmasking Biases and Mapping the Landscape of AI Adoption in Diverse Business Functions Through a Delphi Study 159 Appendix B Figure B: One of the unique questionnaires sent to an expert in the ORM panel, for the third round of the Delphi study Source: Own Appendix C Figure C: Visualisation of the Delphi process of this study Source: Own 160 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION ASSESSMENT INSTRUMENT OF TECHNOLOGY ACCEPTANCE AMONGST PEOPLE WITH MINOR INTELLECTUAL DISABILITIES MATHIJS PETRUS ANTONIUS VAN DIJK, ARJEN MARIS HU University of Applied Sciences, Utrecht, Netherland bas@basvandijk.com, arjen.maris@hu.nl For people with moderate intellectual disabilities (PID) and their carers, eHealth is becoming increasingly important. However, there are no technology acceptance instruments known to determine what PID need to properly deploy eHealth. Therefore, we developed a technology acceptance assessment for PID. A design research approach is applied to develop a conceptual model based on the UTAUT2-model. Based on the outcomes of seventeen interviews with PID experts, two determinants (Public Keywords: Financing & Voluntariness of Use) and two moderators (Health technology Literacy & Emotional State) are added to the conceptual model. acceptance The conceptual model is translated into a first assessment model, UTAUT2, prototype using the Universal Design technique and Goegan et assessment al's (2018) accommodating principles. The first tests that took instrument, place within this research confirm the applicability of the healthcare, mild instrument and provides the first clues for the explanatory value intellectual of the conceptual model for the adoption of eHealth by PID. disability DOI https://doi.org/10.18690/um.fov.4.2024.9 ISBN 978-961-286-871-0 162 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Approximately 1.1 million PID live in the Netherlands (Social and Cultural Planning Office 2023). Within the Netherlands the general consensus is that a person can be placed within de PID category if the person has an IQ between 50-70 or an IQ between 70-85 with additional problems within the area of self-management (e.g. managing personal hygiene) (National Knowledge Center for PID 2018). A significant part of this group needs support with their day-to-day activities (National Knowledge Center for PID 2018), or with personal- and mental healthcare. A large part of these support services (e.g. help with finding a suitable job or assisting with personal hygiene) are provided by organizations within the intellectual disability sector. Organizations within this sector are currently being chal enged by a growing shortage of staff (UWV 2020) and increasing budget cuts. Additionally, the demand for the services of these organizations is growing as PID are more and more struggling to find their way in our contemporary society, a society that is increasing in pace, getting more digital and emphasizes on personal responsibility and self- management (FWG Foundation 2022). The use of eHealth has long been touted as a part of the solution to meet the current chal enges in the healthcare sector (Dutch Government, 2022; Ossebaard & Gemert-Pijnen, 2016; Xiang & Venrick, 2017). Interest for eHealth within the intellectual disability sector has also been steadily growing (Oudshoorn et al. 2020). However, research regarding the specific determinants (factors that influence the behavioral intention to use) and moderators (factors that can influence the effect of one or more determinants) for a successful adoption of eHealth by PID is scarce and lagging (Frielink, Oudshoorn, and Embregts 2020; Oudshoorn et al. 2020) and if you would speak with PID and their carers they wil tel you they are struggling to discover these determinants together in a practical setting. Giving attention to these adoption determinants can potential y help PID and their carers to realize a higher adoption rate and a more targeted and cost-effective appliance of eHealth which better suits the need of the individual PID (Frielink et al. 2020). These findings underpin the need for an instrument that enables and facilitates the conversation between PID and their carers about the adoption determinants for a specific eHealth solution and to evaluate them over time. Considering this, the following central research was formulated: How may care providers determine the conditions for PID to successful y adopt eHealth? M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 163 In the remainder of this paper, the core concepts of this research, 'technology acceptance', 'eHealth adoption' and 'design principlesáre discussed, fol owed by a description of the research method. The results of this study are presented in the results section followed by a discussion. The last sections contain the discussion and limitations of this study and wraps up with the conclusion. 2 Literature 2.1 Technology acceptance The field of technology acceptance has a relatively long tradition. The first research publications in this field date from the late 1960’s (Abdul Aziz et al. 2020; Koul and Eydgahi 2017; Rondan-Cataluña, Arenas-Gaitán, and Ramírez-Correa 2015). Pioneer in this field was (Fishbein 1967) with his Theory of Reasoned Action. His theory resulted in the TAM (Technology Acceptance Model) (Davis, 1986). Since the introduction of the TAM-model it has been used in numerous studies on the adoption of technology and in very diverse contexts (Rondan-Cataluña et al. 2015). (Venkatesh, Thong, and Xu 2012) incorporated the TAM-model along with other relevant models in their UTAUT-model (Universal Theory of Acceptance and Use of Technology). Research by (Bergmo 2015; Li et al. 2013; Rondan-Cataluña et al. 2015) shows that TAM and UTAUT(2) are the dominant models to explain the adoption of technology in existing research. Although there has been critique from other researchers regarding the relative simplicity of the models and their explanatory value (Li et al. 2013; Shachak, Kuziemsky, and Petersen 2019) the general consensus is that both models do explain the adoption of technology in a general setting (Rahman et al. 2017; Rondan-Cataluña et al. 2015). The lower explanatory value in specific settings can be mitigated by extending/changing both models for use in the context in which they are to be used (Rahman et al. 2017). Numerous researchers have adapted both models to better suit the context of their studies and elevate the explanatory value (Magsamen-Conrad et al. 2019; Shachak et al. 2019). The concepts that are embedded within the TAM- and UTAUT-models are usually operationalized in the form of a questionnaire (Wil iams, Rana, and Dwivedi 2015; Yousafzai, Foxall, and Pallister 2007). The original authors of the TAM- en UTAUT-model also used questionnaires to gather research data for the validation of their models. In these questionnaires the various concepts of the model are operationalized using one or more questions which measure the performance of 164 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION the concept. Usually, the answering of the question(s) is done quantitatively using a 7- or 5-point likert scale (Davis 1985; Venkatesh et al. 2012). Researchers that use a modified/extended version of one of the two models usual y operationalize their modified/added concepts using existing research or using their own questions. (Ivanova 2022) for example, used existing research to operationalize additional determinants in the context of the adoption of mobile banking. In the context of eHealth adoption, the determinants ‘health literacy’ and ‘eHealth Literacy’ have shown the potential to elevate the explanatory value of the vanil a versions of both models (Chang et al. 2021; Magsamen-Conrad et al. 2019). In research on the adoption of eHealth both models are operationalized in the same way as they are operationalized in the general field of technology adoption (Li et al. 2013). Based on this we assume that (1) TAM, UTAUT or UTAUT2 are suitable candidates for the foundation of the conceptual model. 2.2 eHealth adoption Literature studies (Pagliari et al. 2005; Uribe-Toril 2021) show that the term ‘eHealth’ was introduced into research in 1999. Within this study eHealth is defined as ‘the appliance of digital information and communication with the intention to support and/or improve the healthcare sector general and the health of a specific individual’ (Lettow, Wouters, and Sinnige 2019). Some of the potential benefits that arise from existing research are providing more cost-effective healthcare, (Bergmo 2015; Swanepoel 2020) elevating healthcare quality (Ossebaard and Gemert-Pijnen 2016; Xiang and Venrick 2017) and elevating the self-management of the client/patient (Kelley et al. 2011; van Zelst et al. 2021) Other research however shows that implementing eHealth within existing healthcare processes and reaping the potential benefits isn’t always easy and the percentage of eHealth implementations that deliver on their promise is sometimes lower than initial y expected (Enam, Torres-Bonilla, and Eriksson 2018; Kraaijkamp et al. 2020). One key driver of a successful eHealth implementations that is mentioned across research is that of engaging with the client/patient en putting his/her needs central (Dutch Government 2022; Xiang and Venrick 2017; Zaagsma et al. 2021). TAM- and UTAUT(2) are the dominant models for explaining and researching the adoption of eHealth (Alqudah, Al-Emran, and Shaalan 2021; Heinsch et al. 2020). The TAM-model by (Davis 1985) defines ‘Perceived Usefulness’ and ‘Perceived Ease of Use’ as the most important M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 165 determinants (factors) that influence the behavioral intention for an individual to use a new technology. The UTAUT-model by (Venkatesh et al. 2003) is a modified and extended version of the original TAM-model which adds a set of additional determinants and a set of moderators (factors that influence the strength of one or more determinants). The key difference between the UTAUT and UTAUT2 (Venkatesh et al. 2012) models is that the original UTAUT model assumes non- voluntary use and UTAUT2 assumes voluntary use. Research in the field of eHealth adoption by PID is relatively new. Only a few studies have been conducted on this subject (Frielink et al. 2020; Oudshoorn et al. 2020). The studies that have been conducted mostly focus on the adoption of a specific eHealth-solution or category. An example within this context is research on the adoption of digital mental health interventions (Vereenooghe, Trussat, and Baucke 2021). The studies that focused on the adoption of eHealth by PID are qualitative, and only the TAM-model was employed within this context (Vereenooghe et al. 2021). (Frielink et al. 2020) used a focus group approach in their research on facilitating and impeding factors for eHealth adoption by PID. Most of their results can be plotted on the concepts of the UTAUT-model. The bottom line is that further research on this topic is needed, as underpinned by (Frielink et al. 2020) and (Oudshoorn et al. 2020). The study of (Frielink et al. 2020) demonstrates that focus groups consisting of PID and their carers are a suitable setting for conducting research on the topic of eHealth for PID. In addition to the determinants for technology adoption as stated by UTAUT2, existing research shows that there are additional factors to consider regarding the adoption of eHealth in the general population and more specific PID: (1) (Schuurman, Speet, and Kersten 2004) state that most PID have lower than average financial means. This suggests that PID are less tempted to buy eHealth themselves as they are struggling to get through their day to day lives financially. (2) (Alqudah et al. 2021) state that voluntariness of use of an eHealth solution can significantly influence adoption in the general population. (3) eHealth adoption in the general population can be influenced by ‘(e)Health literacy’ (Chang et al. 2021; Magsamen-Conrad et al. 2019). (4) Research in the general field of psychology (McCurdy, Scott, and Weems 2022) suggests that the emotional state of an individual can influence the intention and the skil s needed to try something new and bring this into practice within the own personal context. 166 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Based on this we assume that (2) Extension of TAM, UTAUT or UTAUT2 with the concepts of Health literacy, eHealth literacy and cognitive age wil elevate the explanatory value of the vanil a versions of these models. 2.3 Design principles for PID instrument There is a lot of diversity within the PID group and the individual needs regarding accessibility and literacy (National Knowledge Center for PID 2018, 2023). The goal of this research was to develop an instrument that would be inclusive for the broad PID group and could be used by every PID regardless of their literacy skil s and understanding of the subject. The Universal Design method (Preiser et al., 2011) underpins the need for inclusive products and environments and provides seven principles to accomplish this. Universal Design has its roots in physical architecture but has been used to accomplish inclusivity in a variety of contexts and products (Aslaksen et al. n.d.; Crow 1997; Oliveira, MUNSTER, and GONÇALVES 2019). (Goegan et al. 2018) have used the principles of universal design to accomplish inclusive educational tests for PID. They’ve extended the seven general principles of Universal Design with four PID specific principles which they cal accommodations. Publications by (Moonen 2018; National Knowledge Center for PID 2023; Schuurman et al. 2004) state that PID can benefit from visual and auditive support in the form of icons, answer options using smileys and spoken text. Another area that needs attention is the level of literacy skil s of PID. In the Netherlands the general level of literacy is classified as ‘B1’ on the CEFR-scale (Common Framework of Reference for Languages) (Council of Europe 2023), the general level literacy level of PID however is somewhere between A1-A2 on the CEFR-scale (Lee-A-Fong 2018). This for example translates into the need for more concrete text usage and shorter sentences (Moonen 2018; National Knowledge Center for PID 2023). Based on this we assume that (3) The principles of Universal Design are a solid foundation for developing an inclusive research instrument for PID and (4) Extending the standard set of principles of Universal Design with the principles by (Goegan et al. 2018) will add extra value for the inclusivity of the instrument. M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 167 3 Method A design science approach was used to conduct the research and develop the instrument. This approach enables working in iterations and smal increments (Hevner 2007; Peffers et al. 2006). Working in smal increments was an important precondition for this research because PID (can) have a lower-than-average attention span (National Knowledge Center for PID 2023) and can experience stress during long sessions (Schuurman et al. 2004). The research methodology approach of (McLaren and Buijs 2011) was chosen to overarch the research because of its balanced emphasis on both the theoretical foundation and the design principles for the research instrument to be developed. This research methodology uses the design science approach as described by (Hevner 2007) as it’s foundation to provide a framework for the development of a solid research instrument that is both theoretical solid and usable/applicable for the end-user population that is being targeted. Engaging with PID in a research context has its chal enges, there must be enough time and individual attention to keep PID engaged and accommodate their personal needs during a research project (Schuurman et al. 2004). Because of this a representative sample that can be generalized for the PID population wasn’t possible within the proposed timespan of this research. This resulted in a slightly modified version of the research methodology to better suit the qualitative nature of this research. The quantitative data gathering methods were swapped for qualitative equivalents and the order of the checkpoints was altered to better suit the needs of PID during the research, the design principles were checked first and the more theoretical checks in a later stage. For a more detailed overview of the research methodology that was used during this research please refer to appendix A. 3.1 Data gathering methods Literature review To find candidate determinants and moderators for the conceptual model and candidate design principles for the suitable form(s) for the instrument a literature review was conducted. This review resulted in a set of (four) assumptions (section 2) which were further validated during the later stages of the research. 168 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Interviews Seventeen interviews were conducted as an initial attempt to validate our four assumptions. The participant group consisted of six PID care providers, two behavioral scientists, one PID mental health nurse, one PID communication expert and seven PID. The interviews took place within three organizations in the intellectual disability sector and one organization in the mental-health sector. To ensure that al the interviews would cover the same topics a semi-structured topic list (appendix B) and interview guide were developed. The semi-structured character of the interview was careful y chosen to al ow PID to deviate from the central topics and felt the freedom to talk about anything they wanted. The interviews with the care providers took place first to allow the researchers to get more familiar with the PID group and how to appropriately interview them. The care providers confirmed the finding of the literature study regarding the need for visual support for PID. As a result of this a PowerPoint was developed to visually support the interviews with PID and provide them with a clear structure as advised by (Schuurman et al. 2004). The participants that asked for approval were provided with a copy of the transcript. Focus Group The first concept versions of the instrument were validated by a focus group which consisted of four PID and one care provider. The goal of the focus group was to validate the design principles and to choose the forms of the instrument that best suited PID. The focus group also extensively reviewed the operationalization of the concepts within the various concept versions of the instrument. In the context of the research methodology the results of the focus group were used to execute the check on ‘prescriptive utility’. The focus group took place in one session spanning a couple of hours. During this session the participants discussed the concept versions of the instrument with a researcher and provided feedback. The session was (again) supported using a PowerPoint. Test Phase A group of seven PID was used to assess the 'reliability', 'validity', and 'predictive utility' as described by (McLaren and Buijs 2011). Also, a second check on the ‘prescriptive utility’ was conducted. Data was gathered in the form of used versions M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 169 of the instrument. The test session took place at two separate moments across two separate organizations. The first test session took place in a group setting. At the request of the participants the second test session was done individual y. During both sessions a PowerPoint was used to guide the session and the PID participants could freely choose a couple of eHealth examples that they already were using at the time or potential y want to use in the future. A few examples of the examples that were chosen by the participants: a smartwatch, a personal digital healthcare environment and a care robot. Using the chosen examples, al the participants used the instrument and provided the results to the researchers. The test phase resulted in a total of 15 used versions of the instrument which were used as input for further analysis regarding the explanatory value of the conceptual model (appendix C) and the practical use/applicability of the instrument for PID. 3.2 Selection of participants Due to ethical considerations potential PID participants were not contacted directly. Contact with potential PID participants was established through the care managers and care coordinators of the participating organizations. Due to ethical considerations, no information about the background of the PID was shared with the researchers. The care managers and coordinators checked if the potential candidates fitted within the profile of PID. If the PID wanted to participate, the PID and a carer would fill in the informed consent form. The form and letter were tailored to PID, both were screened and approved by the Ethical Committee of HU University of Applied Sciences. There has not been any direct contact between the researchers and PID preceding the interview nor afterwards; if needed a care provider acted as proxy. For care-provider participants, the respective contacts within the participating organizations were contacted. 3.3 Data handling and analysis All participants that participated within the interviews and focus group allowed for recordings to be made. These recordings were securely erased after the transcription was finished. The transcripts were done in verbatim form. This al owed for the researchers to more precisely determine if any socially desirable response was given by the participant. The transcripts were done manual y to guarantee that none of the recordings would be available to third parties. The transcripts were anonymized by 170 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION assigning a unique code to the participant and removing any attributes that could potential y identify the participant. The transcripts were analyzed using the process of coding as described by (Baarda 2018). To streamline the coding process the application Atlas.ti was used. Atlas.ti enables researchers to visually guide and support the coding process. 4 Results The assumption that the general technology adoption models TAM, UTAUT and UTAUT2 could be suitable for explaining eHealth adoption by PID was confirmed during the interviews. UTAUT2 seemed to be the most suitable to explain the adoption of eHealth by PID. Figure 1 shows a couple of quotes by PID and care providers from the interview transcripts that confirm this. In addition to the main determinants, the UTAUT2-model also contains a set of moderators. These moderators (can) influence the main determinants and usage behavior. The influence of a couple of these moderators could be matched with the interview results, the influence of the other moderators could not be matched with the interview results but could also not be dismissed. For further details regarding the matching results of the existing determinants and moderators please refer to appendix D. In addition to the validation of the standard UTAUT2 determinants and moderators the assumed determinants ‘eHealth Literacy, ‘Health Literacy’ and ‘Cognitive Age’ were validated against the interview results. Both ‘eHealth Literacy’ and ‘Health Literacy’ were recognized in the interview results. A care provider on this subject: ‘usually PID have a lower-than-average level of health literacy skills.’ The recognition of ‘Cognitive Age’ was inconclusive and a couple of participants advised against the use of this determinant due to ethical reasons. A care provider on this subject: ‘if you ask someone, how old do you think you are in your head? I don’t know if that’s ehm, PID like to be seen as normal.’ M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 171 Additional determinants and moderators As a result of the (inductive) coding process two additional determinants and one additional moderator were discovered: voluntariness of use, public financing, and emotional state. A couple of example quotes are shown in figure 2. Figure 1: Standard UTAUT2-determinants with example quotes Determinant Example quote PID on a smartwatch: ‘Initial y I thought it was interesting to fol ow my Performance sleep rhythm and to monitor my heartbeat. It was fun in the beginning, Expectancy but it didn’t add any value for me in the long term.’ PID on the app ‘DigiD’ which must be used to access some healthcare Effort services: ‘I think it must be improved for people like us, regarding the Expectancy information they provide, easier language, so that PID people can easily understand it and use it.’ Social PID about a smartwatch: ‘I bought one because everyone that I know influence has one, so I thought that I also should get one.’ PID on the topic of a care robot: ‘I think that the irritating voice of the Facilitating robot keeps people from using it, we’ve said this months ago to the conditions people of the pilot, but nothing has been done about it and we didn’t hear anything about it since.’ PID on the pleasure experienced using a physical exercise app: ‘Usual y Hedonic I’m not good at doing things for a long time but with this app, it doesn’t Motivation feel like something that I must do, it feels like a game sometimes because you can win badges, I think it’s fun!’ PID on the pleasure that is experienced using a physical exercise app: Hedonic ‘Usual y I’m not good at doing things for a long time but with this app, Motivation it doesn’t feel likè PID on the topic of making a ful /partial contribution to the costs of Price Value eHealth: ‘I would take that into consideration, but it must have benefits for me to do that.’ 172 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 2: Additional determinants and moderator with example quotes Determinant Example quote PID: ‘No one can force me to do anything, in my life so far, I’ve had Voluntariness to do a lot of things that I didn’t want to do, I have my own opinion, of use so if something is forced on me, I wil not do it.’ Public PID: ‘ehm, I’m against al those things in healthcare that I must pay a Financing partly contribution for. Moderator Example quote Emotional Care provider: ‘if you’re mentally not well it can be difficult to start State using something new.’ 4.1 Suitable design principles for a research instrument for PID The assumption that the principles of Universal Design extended with the ‘accommodating’ principles by (Goegan et al. 2018) would provide a suitable framework for a research instrument for use by PID was validated positively by placing the interview results in the context of these principles. For an overview of the final design principles refer to appendix E. The first set of concept versions of the instrument encompassed these forms: a questionnaire, a checklist, a ‘praatplaat’ (Dutch word for a specific form of visual aid) and a presentation form. Al forms were developed as a physical and a digital variant. Different variants were developed for determining the determinants for adoption and to evaluate them over time. The digital version of the questionnaire also contained speech support. The review of all the concept forms was general y positive but modifications were made because of feedback that was given during the focus group and test phase. In the context of the research methodology, the result of the prescriptive utility check was positive. For examples of final versions of the instrument please refer to appendix F. 4.2 Performance of the instrument The check on reliability was found to be positive. A total of three answers were found to be potential y not reliable answered by the participants. One participant, for example, provided an answer using a not happy smiley but in the previous questions gave the impression that he/she was positive. This is in line with the question of another participant who asked for a textual explanation regarding what the smileys meant. In the definitive forms more explanation regarding the answer M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 173 options was added. The check on validity confirmed that the instrument captured results that were in line with what was expected regarding the concepts within the conceptual model. The dataset that was gathered was too smal to give a definitive answer on the predictive utility of the instrument. The results, however, provide the first clues for the predictive utility of the instrument. 5 Discussion and limitations This research proposes a research instrument that PID and their care providers can use to determine and evaluate the determinants for a successful adoption of eHealth by PID. This research also demonstrates that using the UTAUT2-model extended with the determinants ‘Voluntariness of Use’ and ‘Public Financing’ and the moderators ‘(e)Health Literacy’ and ‘Emotional State’ provide a solid foundation for a research instrument for the adoption of eHealth by PID. PID are a diverse group with special needs regarding the form of a research instrument (Schuurman et al. 2004). This research demonstrates that using a framework consisting of the principles of Universal Design extended with the principles of (Goegan et al. 2018) contributes to the development of inclusive forms for a research instrument for PID. These new insights contribute to future research in the field of eHealth and technology adoption by PID and the development of research instruments for PID in general. This study has several limitations. First, the most significant limitation is the smal sample size that was gathered and used during the performance checks of the instrument. The results as presented should be viewed as a first indication of the performance and explanatory value of the instrument and not as definitive results. We urge for further use and research on the performance of the instrument. Repeating the ‘Predictive Value’ check with a representative sample would be a good starting point for future research. Second, the supporting material used during the interviews contained a set of examples. This was necessary to give the PID participants an impression of what was to be discussed. These examples may however have led to unintentional steering of the answers that were given. To mitigate this the first two stages of the coding phase were done using an inductive approach. However, unintended steering on a couple of interview subjects cannot be excluded. The applicability of the design principles and translation that was made to the context of a research instrument for PID would also be a great topic for further research on how to create inclusive forms of research instruments for PID. 174 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 6 Conclusion PID are a vulnerable group that is often overlooked in our contemporary society (National Knowledge Center for PID 2023). (Chadwick et al. 2022) and (Lussier- Desrochers et al. 2017) describe a growing digital divide between people who can use and reap the benefits of digital technology and those who can’t. The growing staff shortage and increasing budget cuts in the intel ectual disability care sector can have a direct impact on the time that is available for the care and attention for PID (Dutch Healthcare and Youth Care Inspection 2023). It’s tempting for care providers in this sector to replace certain aspects of their services with eHealth solutions. It is however crucial that PID are engaged in this process to have a good change of adoption success (Dutch Government 2022). Implementing eHealth to counter personal shortage and budget cuts without PID engagement can potential y contribute to the growing digital divide in our society and that would not be in line with the ‘client central’ approach that many care providers in this sector have embedded within their organizational vision. So how can care providers bridge this digital divide and engage with PID to determine the conditions which are necessary for an individual PID to successfully adopt eHealth? This brings us to the answering of the central research question. Care providers may use the instrument which is proposed in this paper to engage with PID and together explore eHealth solutions and determine which adoption factors are necessary for PID to adopt these eHealth solutions. The proposed instrument can also be used to evaluate and monitor progression on the adoption determinants over time. The test phase of the instrument has proven that the developed forms are suitable for PID and provide enough flexibility to be tailored to specific situations. References Abdul Aziz, Mohd Nasiruddin, Siti Harun, Mohd Baharom, and Norfadilah Kamaruddin. 2020. “THE EVOLUTION OF THE TECHNOLOGY ACCEPTANCE MODEL (TAM).” 242. Alqudah, Adi, Mostafa Al-Emran, and Khaled Shaalan. 2021. “Technology Acceptance in Healthcare: A Systematic Review.” Applied Sciences 11:10537. doi: 10.3390/app112210537. Aslaksen, Finn, Steinar Bergh, Olav Bringa, and Edel Heggem. n.d. “Universal Design: Planning and Design for All.” GLADNET Collection. Baarda. 2018. Basisboek Kwalitatief onderzoek. Utrecht: Noordhoff Uitgevers. Bergmo, Trine. 2015. “How to Measure Costs and Benefits of eHealth Interventions: An Overview of Methods and Frameworks.” Journal of Medical Internet Research 17:e254. doi: 10.2196/jmir.4521. Chadwick, Darren, Kristin Alfredsson Ågren, Sue Caton, Esther Chiner, Joanne Danker, Marcos Gómez‐Puerta, Vanessa Heitplatz, Stefan Johansson, Claude Normand, Esther Murphy, M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 175 Piotr Plichta, Iva Strnadová, and Eva Flygare Wallén. 2022. “Digital Inclusion and Participation of People with Intellectual Disabilities during COVID ‐19: A Rapid Review and International Bricolage.” Journal of Policy and Practice in Intel ectual Disabilities 19. doi: 10.1111/jppi.12410. Chang, Yung-Tzung, Cheng-Min Chao, Chen-Wei Yu, and Fang-Chen Lin. 2021. “Extending the Utility of UTAUT2 for Hospital Patients’ Adoption of Medical Apps: Moderating Effects of e-Health Literacy.” Mobile Information Systems 2021:1–10. doi: 10.1155/2021/8882317. Council of Europe. 2023. “COMMON EUROPEAN FRAMEWORK OF REFERENCE FOR LANGUAGES: LEARNING, TEACHING, ASSESSMENT.” Crow, Kevin L. 1997. “Universal Design Principles: An Overview.” Davis, Fred. 1985. “A Technology Acceptance Model for Empirical y Testing New End-User Information Systems.” Dutch Government. 2022. eHealthmonitor 2021: Stand van zaken digitale zorg. Rijksoverheid. Dutch Healthcare and Youth Care Inspection. 2023. “Personeelstekorten in de zorg.” Retrieved (https://www.igj.nl/onderwerpen/personeelstekort#:~:text=De%20hele%20gezondheidszo rg%20kampt%20met,en%20jeugdhulp%20te%20laten%20doorgaan.). Enam, Amia, Johanna Torres-Bonilla, and Henrik Eriksson. 2018. Evidence-Based Evaluation of eHealth Interventions: Systematic Literature Review (Preprint). Fishbein, M. A. 1967. “Behavior Theory Approach to the Relations between Beliefs about an Object and the Attitude toward the Object, and Attitude and the Prediction of Behavior.” Readings in Attitude Theory and Measurement (Pp 389–99. doi: https://doi.org/10.1007/978-3-642-51565-1_25. Frielink, Noud, Cathelijn Oudshoorn, and P. Embregts. 2020. “eHealth in Support for Daily Functioning of People with Intellectual Disability: Views of Service Users, Relatives, and Professionals on Both Its Advantages and Disadvantages and Its Facilitating and Impeding Factors.” Journal of Intel ectual & Developmental Disability 46. doi: 10.3109/13668250.2020.1744878. FWG Foundation. 2022. De maatschappij, dat zijn zij. TRENDRAPPORT GEHANDICAPTENZORG. Goegan, Radil, and Daniels. 2018. “Design Accessibility in Questionnaire Research: Integrating Universal Design to Increase the Participation of Individuals With Learning Disabilities.” Learning Disabilities: A Contemporary Journal 16(2). Heinsch, Milena, Jessica Wyllie, Jamie Carlson, Hannah Wells, Campbell Tickner, and Frances Kay-Lambkin. 2020. “Theories Informing eHealth Implementation: Systematic Review and Typology Classification (Preprint).” Journal of Medical Internet Research 23. doi: 10.2196/18500. Hevner, Alan. 2007. “A Three Cycle View of Design Science Research.” Scandinavian Journal of Information Systems 19. Ivanova. 2022. “Acceptance and User of Mobile Banking in Central Asia: Evidence from Modified UTAUT Model.” Journal of Asian FInance. Kelley, Helen, Mike Chiasson, Angela Downey, and Danièle Pacaud. 2011. “The Clinical Impact of eHealth on the Self-Management of Diabetes: A Double Adoption Perspective.” J. AIS 12. doi: 10.17705/1jais.00263. Koul, Sahil, and Ali Eydgahi. 2017. “A Systematic Review of Technology Adoption Frameworks and Their Applications.” Journal of Technology Management and Innovation 12:106–13. doi: 10.4067/S0718-27242017000400011. Kraaijkamp, Jules, Eléonore van Dam van Isselt, Anke Persoon, Anke Versluis, Niels Chavannes, and Wilco Achterberg. 2020. “eHealth in Geriatric Rehabilitation: A Systematic Review of Effectiveness, Feasibility and Usability (Preprint).” Journal of Medical Internet Research 23. doi: 10.2196/24015. Lee-A-Fong. 2018. “Een snel e screening om het leesniveau van jongeren met LVB te bepalen.” Universiteit van Amsterdam. Lettow, Wouters, and Sinnige. 2019. “E-Health, Wat Is Dat?” 176 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Li, Junhua, Amir Talaei-Khoei, Holly Seale, Pradeep Ray, and Chandini Macintyre. 2013. “Health Care Provider Adoption of eHealth: Systematic Literature Review.” Interactive Journal of Medical Research 2:e7. doi: 10.2196/ijmr.2468. Lussier-Desrochers, Dany, Claude Normand, Alejandro Romero-Torres, Yves Lachapel e, Valérie Godin-Tremblay, Marie-Eve Dupont, Jeannie Roux, Laurence Pépin-Beauchesne, and Pascale Bilodeau. 2017. “Bridging the Digital Divide for People with Intellectual Disability.” Cyberpsychology: Journal of Psychosocial Research on Cyberspace 11. doi: 10.5817/CP2017-1-1. Magsamen-Conrad, Kate, Fang Wang, Dinah Tetteh, and Yen-I. Lee. 2019. “Using Technology Adoption Theory and a Lifespan Approach to Develop a Theoretical Framework for eHealth Literacy: Extending UTAUT.” Health Communication 35:1–12. doi: 10.1080/10410236.2019.1641395. McCurdy, Bethany, Brandon Scott, and Carl Weems. 2022. “The Associations Among Parent Anxiety, Emotion Regulation, and Parenting Behaviors.” Journal of Child and Family Studies 31. doi: 10.1007/s10826-022-02389-9. McLaren, Tim, and Paul Buijs. 2011. “A Design Science Approach for Developing Information Systems Research Instruments.” Moonen. 2018. “De regels voor Taal voor al emaal + Voor mensen met lage taalvaardigheden.” National Knowledge Center for PID. 2018. “Kenmerken en de gevolgen van diagnostisch onderzoek en (gedrags)interventies.” National Knowledge Center for PID. 2023. “Aansluiten bij een LVB, (hoe) doe jij dat?” Oliveira, Amália Rebouças de Paiva e, Mey de Abreu van MUNSTER, and Adriana Garcia GONÇALVES. 2019. “Desenho Universal Para Aprendizagem e Educação Inclusiva: Uma Revisão Sistemática Da Literatura Internacional1.” Revista Brasileira de Educação Especial 25(4):675–90. doi: 10.1590/s1413-65382519000400009. Ossebaard, Hans, and Julia Gemert-Pijnen. 2016. “EHealth and Quality in Health Care: Implementation Time.” International Journal for Quality in Health Care 28:mzw032. doi: 10.1093/intqhc/mzw032. Oudshoorn, Cathelijn, Noud Frielink, Sara Nijs, and P. Embregts. 2020. “eHealth in the Support of People with Mild Intellectual Disability in Daily Life: A Systematic Review.” Journal of Applied Research in Intel ectual Disabilities 33. doi: 10.1111/jar.12758. Pagliari, Claudia, David Sloan, Peter Gregor, Frank Sullivan, Don Detmer, James Kahan, Wija Oortwijn, and Steve Macgillivray. 2005. “What Is eHealth (4): A Scoping Exercise to Map the Field.” Journal of Medical Internet Research 7:e9. doi: 10.2196/jmir.7.1.e9. Peffers, Ken, Tuure Tuunanen, Charles Gengler, Matti Rossi, Wendy Hui, Vil e Virtanen, and Johanna Bragge. 2006. “The Design Science Research Process: A Model for Producing and Presenting Information Systems Research.” Proceedings of First International Conference on Design Science Research in Information Systems and Technology DESRIST. Rahman, Md, Mary Lesch, Wil iam Horrey, and Lesley Strawderman. 2017. “Assessing the Utility of TAM, TPB, and UTAUT for Advanced Driver Assistance Systems.” Accident Analysis & Prevention 108:361–73. doi: 10.1016/j.aap.2017.09.011. Rondan-Cataluña, Francisco, Jorge Arenas-Gaitán, and Patricio Ramírez-Correa. 2015. “A Comparison of the Different Versions of Popular Technology Acceptance Models: A Non- Linear Perspective.” Kybernetes 44. doi: 10.1108/K-09-2014-0184. Schuurman, Speet, and Kersten. 2004. Onderzoek met mensen met een verstandelijke beperking: Handreiking voor de praktijk. Shachak, Aviv, Craig Kuziemsky, and Carolyn Petersen. 2019. “Beyond TAM and UTAUT: Future Directions for HIT Implementation Research.” Journal of Biomedical Informatics 100:103315. doi: 10.1016/j.jbi.2019.103315. Social and Cultural Planning Office. 2023. “Het aantal mensen met een licht verstandelijke beperking: een schatting.” Swanepoel, DeWet. 2020. “EHealth Technologies Enable More Accessible Hearing Care.” Seminars in Hearing 41:133–40. doi: 10.1055/s-0040-1708510. M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 177 Uribe-Toril. 2021. “A Study of eHealth from the Perspective of Social Sciences.” doi: https://doi.org/10.3390/healthcare9020108. UWV. 2020. “Factsheet Zorg.” Retrieved (https://www.uwv.nl/overuwv/kennis-cijfers-en- onderzoek/arbeidsmarktinformatie/factsheet-zorg-maart-2020.aspx). Venkatesh, Viswanath, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003. “User Acceptance of Information Technology: Toward a Unified View.” MIS Quarterly 27(3):425– 78. doi: https://doi.org/10.2307/30036540. Venkatesh, Viswanath, James Thong, and Xin Xu. 2012. “Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology.” MIS Quarterly 36:157–78. doi: 10.2307/41410412. Vereenooghe, Leen, Felix Trussat, and Katja Baucke. 2021. “Applying the Technology Acceptance Model to Digital Mental Health Interventions: A Qualitative Exploration with Adults with Intellectual Disabilities.” Journal of Mental Health Research in Intel ectual Disabilities 14:1–26. doi: 10.1080/19315864.2021.1929597. Wil iams, Michael, Nripendra Rana, and Yogesh Dwivedi. 2015. “The Unified Theory of Acceptance and Use of Technology (UTAUT): A Literature Review.” Journal of Enterprise Information Management 28:443–88. doi: 10.1108/JEIM-09-2014-0088. Xiang, Jun, and Samantha Venrick. 2017. “From Online to Offline: Exploring the Role of e-Health Consumption, Patient Involvement, and Patient-Centered Communication on Perceptions of Health Care Quality.” Computers in Human Behavior 70. doi: 10.1016/j.chb.2016.12.072. Yousafzai, Shumaila, Gordon Foxal , and John Pal ister. 2007. “Technology Acceptance: A Meta-Analysis of the TAM: Part 1.” Journal of Model ing in Management 2. doi: 10.1108/17465660710834453. Zaagsma, Miriam, Dominique Van de Velde, Mark Koning, Karin Volkers, Alice Schippers, and Geert Van Hove. 2021. “‘ When I Need Them, I Cal Them and They Wil Be There for Me ’. Experiences of Independently Living People with Intellectual Disabilities with 24/7 Available Online Support.” Disability & Society 38:1–24. doi: 10.1080/09687599.2021.1932756. van Zelst, Cathelijne, Marise Kasteleyn, Esther van noort, Maureen Molken, Gert-Jan Braunstahl, Niels Chavannes, and Johannes In ’t Veen. 2021. “The Impact of the Involvement of a Healthcare Professional on the Usage of an eHealth Platform: A Retrospective Observational COPD Study.” Respiratory Research 22. doi: 10.1186/s12931-021-01685-0. 178 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix A Figure 3: The modified version of the methodology of Source: (McLaren & Buijs, 2011) Appendix B During the (semi structured) interviews a topic list was used to ensure that every interview fol owed the same structure, and no topics were left unaddressed. Separate topic lists were made for PID (figure 4) and their carers (figure 5). Both topic lists were translated into an interview guide and supporting PowerPoint. For this article the original topic lists were translated from their original language (Dutch) into English to enhance readability for international readers. M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 179 Topic # Topic (possible) sub-topics and questions - Introduction of the interviewer - Some personal information about the interviewer (e.g job and hobbies) - Introduction of the - Introduction interviewee - Some personal information about the interviewee (e.g. job and hobbies) - Age and gender of the interviewee (not mandatory - Introduction to the term 'eHealth - About the interview - Introduction to the purpose and structure of the interview - What devices does the interviewee use (daily)? Why does the interviewee use these devices and what for (to accomplish which tasks)? - What does the interviewee like about these devices? - What does the interviewee not like about these devices? Usage of - What kind of obstacles does the interviewee experience while using these devices? 1 technology by the - The usage of online (governmental) services by the interviewee interviewee - What kind of obstacles does the interviewee experience while using these services? - The usage of social media by the interviewee What does the interviewee like about social media? - What does the interviewee not like about social media? - What kind of obstacles does the interviewee experience while using social media? - Usage of eHealth by the interviewee (the examples in the PowerPoint can be used to start the conversation) - Why the interviewee uses these eHealth solutions Usage of eHealth 2 - Why the interviewee doesn't use some of the examples in the PowerPoint by the interviewee - What the interviewee likes about these eHealth solutions - What the interviewee does not like about these eHealth solutions - What can be improved on these eHealth solutions according to the interviewee - What would be the ideal eHealth solution for the interviewee according to the interviewee? The ideal eHealth - What would it look like? 3 solution for the - Which functionality must it provide? interviewee - For what purpose or development goals would the interviewee use it? Determinants for - Determinants of eHealth usage by PID (which factors are important for eHealth adoption and 4 the adoption of usage in the long term according to the interviewee?) eHHalith - Prioritizing these determinants (which are most important according to the interviewee? - Does the interviewee (actively) search for eHealth solutions that can help with their personal Selection of (healthcare) goals? 5 eHealth by and for - Do the carees of the interviewee help with searching for suitable eHealth solutions and/or the interviewee propose eHealth solutions to the interviewee? - Would the interviewee appreciate more initiative form their carers on the topic of eHealth? - If cares proposed an eHealth solution to the interviewee how would they like to explore this On the process of solution? Together with the care? Alone? First alone and then together with a cares? 6 selecting eHealth - What kind of aid would help the interviewee to explore an eHealth solution and talk about it with carers and with carer? How would this look like? talking about it - With what frequency would the interviewee like to evaluate the usage of Health with the carer? - Does the interviewee need help with the usage of technology and eHealth? - What kind of help would be appreciated? On the support - Does the interviewee think enough help is available/provided? 7 with the usage of - Is there already someone in their network (e.g. carer, parent, family member) that can provide eHealth help with technology and eHealth? Who is this? - Who would the interviewee first contact when help is needed with technology or eHealth? - What kind of role does the interviewee expect from their carers regarding eHealth? - Am questions? - Wrap up - Any suggestions or feedback regarding the interview contents of process? Figure 4: the topic list that was used for the interviews with the carers 180 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Topic Topic (possible) sub-topics and questions # - Introduction of the interviewer - Some personal information about the interviewer (e.g job and hobbies) - Introduction of - Introduction the interviewee - Some personal information about the interviewee (e.g. job and hobbies) - Age and gender of the interviewee (not mandatory - Introduction to the term 'eHealth - About the interview - Introduction to the purpose and structure of the interview - What devices does the interviewee use (daily)? Why does the interviewee use these devices and what for (to accomplish which tasks)? - What does the interviewee like about these devices? - What does the interviewee not like about these devices? - What kind of obstacles does the interviewee experience while using these devices? 1 Usage of technology by - The usage of online (governmental) services by the interviewee the interviewee - What kind of obstacles does the interviewee experience while using these services? - The usage of social media by the interviewee What does the interviewee like about social media? - What does the interviewee not like about social media? - What kind of obstacles does the interviewee experience while using social media? - Usage of eHealth by the interviewee (the examples in the PowerPoint can be used to start the conversation) - Why the interviewee uses these eHealth solutions 2 Usage of eHealth by the - Why the interviewee doesn't use some of the examples in the PowerPoint interviewee - What the interviewee likes about these eHealth solutions - What the interviewee does not like about these eHealth solutions - What can be improved on these eHealth solutions according to the interviewee - What would be the ideal eHealth solution for the interviewee according to the interviewee? The ideal eHealth - What would it look like? 3 solution for the - Which functionality must it provide? interviewee - For what purpose or development goals would the interviewee use it? - Determinants of eHealth usage by PID (which factors are important for eHealth adoption Determinants for the 4 and usage in the long term according to the interviewee?) adoption of eHHalith - Prioritizing these determinants (which are most important according to the interviewee? - Does the interviewee (actively) search for eHealth solutions that can help with their personal (healthcare) goals? Selection of eHealth by 5 - Do the carees of the interviewee help with searching for suitable eHealth solutions and/or and for the interviewee propose eHealth solutions to the interviewee? - Would the interviewee appreciate more initiative form their carers on the topic of eHealth? - If cares proposed an eHealth solution to the interviewee how would they like to explore this On the process of solution? Together with the care? Alone? First alone and then together with a cares? - What kind of aid would help the interviewee to explore an eHealth solution and talk about 6 selecting eHealth with carers and talking about it with carer? How would this look like? it - With what frequency would the interviewee like to evaluate the usage of Health with the carer? - Does the interviewee need help with the usage of technology and eHealth? - What kind of help would be appreciated? - Does the interviewee think enough help is available/provided? 7 On the support with the - Is there already someone in their network (e.g. carer, parent, family member) that can usage of eHealth provide help with technology and eHealth? Who is this? - Who would the interviewee first contact when help is needed with technology or eHealth? - What kind of role does the interviewee expect from their carers regarding eHealth? - Am questions? - Wrap up - Any suggestions or feedback regarding the interview contents of process? M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 181 Figure 5: the topic list that was used for the interviews with PID Appendix C Figure 6: conceptual model on which the instrument is based 182 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix D Figures 7 and 8 show the results of the matching process between the interview transcripts and the existing determinants and moderators that were hypothesized from the literature review. Figure 7: the recognition of existing determinants by the interviewees M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 183 UTAUT2 Moderator Influences Mapping to interview results The younger PID interviewees actively search for apps and devices that can Age Habit help them with their health, (intellectual) development and self-management. The older PID interviewees don’t do this. Price value Not confirmed within the interview results but can also not be excluded based on the interview results. The younger PID interviewees are open and enthusiastic about trying out a care Hedonic robot. They think they wil enjoy and get pleasure out of the usage of the robot. motivation The older PID interviewees don’t want to try out the care robot and some find the idea of a care robot scary. The younger PID interviewees are enthusiastic about trying out virtual reality glasses and think it would be fun. The older PID interviewees don’t want to try out the virtual reality glasses and don’t think the usage wil be fun. Facilitating conditions Al the PID interviewees value good (technical) support regarding eHealth. The younger PID interviewees mostly use Google or another internet search to troubleshoot potential problems themselves before using an (external) helpline like a carer, family member or support desk. The older PID interviewees mostly contact a family member or carer to help with troubleshooting. Gender Habit Not confirmed within the interview results but can also not be excluded based on the interview results. Price value Not confirmed within the interview results but can also not be excluded based on the interview results. Hedonic Not confirmed within the interview results but can also not be excluded based motivation on the interview results. Facilitating Not confirmed within the interview results but can also not be excluded based conditions on the interview results. A couple of interviewees think that the usage of an eHealth solution for a Experience Habit longer time wil make the usage easier. They think the usage becomes more of a habit when they use the eHealth solutions for a longer time. Hedonic Not confirmed within the interview results but can also not be excluded based motivation on the interview results. A couple of interviewees think that people with prior experience with an eHealth solution wil need less support with the adoption process of a (new) Facilitating conditions eHealth solution. When they get more familiar with the usage it is needed. They also think they need less talks with others (e.g. carers) to evaluate the usage of the eHealth solution. A couple of interviewees think that prior experience with an eHealth solution can influence the adoption process of a (new) eHealth solution. This influence Use behavior can be positive or negative, it depends on prior experience. A couple of interviewees think that a negative prior experience with eHealth can raise bias extensively for PID. Figure 8: the matching of existing moderators with interview results 184 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION APPENDIX E Figure 9: the final design principles on which were used to develop the forms of the instrument Appendix F Figures 10 – 17 shows two examples of forms of the instrument; one of the questionnaire forms and one of the ‘praatplaat’ (a Dutch word for a visual aid to guide a conversation). Both can be used for determining the determinants prior to the first usage of an eHealth application together with PID. Please contact the authors for the other forms of the instrument. Al definitive forms of the instrument were made in the Dutch language. Because English forms of the instrument were not tested during the test phase the validity of an English translation cannot be guaranteed. Therefore, the Dutch versions are shown here. The icons used in this version of the instrument are property of ‘SpelPartners VOF’ and kindly provided for free for this research. M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 185 Figure 10: questionnaire example – page 1 186 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 11: questionnaire example – page 2 M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 187 Figure 12: questionnaire example – page 2 188 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 13: questionnaire example – page 3 M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 189 Figure 14: questionnaire example – page 4 190 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 15: praatplaat example – page 1 M. P. Antonius van Dijk, A. Maris: Assessment Instrument of Technology Ac eptance Amongst People with Minor Intel ectual Disabilities 191 Figure 16: praatplaat example – page 2 192 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 17: praatplaat example – page 3 CULTIVATING A DATA-DRIVEN CULTURE: EXPLORING THE ROLE OF DATA ANALYTIC CAPABILITY, EMPLOYEE EMPOWERMENT, AND TRANSFORMATIONAL LEADERSHIP ALINDA KOKKINOU,1,2 ONDREJ MITAS,3 ALBERT MANDEMAKERS1 1 Academy for Built Environment and Logistics, Breda University of Applied Sciences, Breda,Netherland kokkinou.a@buas.nl, mandemakers.a@buas.nl 2 Center of Expertise Wellbeing Economy and New Entrepreneurship, Avans University of Applied Sciences, Breda, Netherland kokkinou.a@buas.nl 3 Academy for Tourism, Breda University of Applied Sciences, Breda, Netherland mitas.o@buas.nl To remain competitive in an increasingly complex business environment, companies are turning to digitalization technologies. To benefit from these technologies, companies need to develop their Data Analytic Capability and Data-Driven Culture. This is a complex socio-technical process that in addition to technical aspects, also involves organizational aspects. The present study examines how two organizational aspects, Transformational Leadership and Employee K eywords: digital Empowerment contribute to companies harnessing their Data transformation, Analytic Capability to develop a Data Driven Culture. The data-driven findings of a cross-sectional survey design show that culture, transformational Transformational Leadership compounds the positive effect of leadership, Data Analytic Capability on Data Driven Culture. However, and employee contrary to what the theory predicts, Employee Empowerment empowerment, data-driven in combination with Transformational Leadership can have a decision negative impact on DDC. Possible explanations are proposed. making DOI https://doi.org/10.18690/um.fov.4.2024.10 ISBN 978-961-286-871-0 194 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction To compete in an increasingly digital world, companies are embracing digitalization as a way to reduce waste, add value for customers, and improve company performance (Rossi et al., 2022). Digitalization is a broad concept, referring to our society transforming from analog to digital, and the corresponding changes in customer behavior, and within and across communication and collaboration (Vial, 2019). Three distinct processes related to digitalization can be distinguished: (1) digitization as the process of converting analog into digital data, (2) digitalization as the improvement of business processes through the application of digitalization tools, and (3) digital transformation as the changes in the business model to take advantage of new opportunities stemming from digital technologies (Machado et al., 2019). By enabling companies to combine technological tools and data analytics, digitalization al ows them to improve their capacities and effectiveness (Garmaki et al., 2016; Gupta & George, 2016; Yu et al., 2021). This is called Data Analytic Capability (DAC) (Mikalef et al., 2018). For companies, DAC in combination with a Data Driven Culture (DDC), can be an avenue to competitive advantage (Kokkinou et al., 2023d). DAC enables companies to turn data into insights, improving data-driven decision making. DAC can thereby lead to improved supply chain agility (Dubey et al., 2019), supply chain robustness (Kokkinou et al., 2023d), more sustainable supply chain performance (Al-Khatib, 2022) and improved firm performance (Akter et al., 2016). Nevertheless, DAC alone is not sufficient as the generated insights wil only be of value if they are used as part of decision-making (Chatterjee et al., 2021). The concept of DDC was therefore introduced. DDC refers to whether the insights developed are actual y used by employees and managers in day-to-day decision-making (Yu et al., 2021). Even though the need to develop DDC is widely recognized (Chatterjee et al., 2021; Kokkinou et al., 2023; Yu et al., 2021), less is known about the factors affecting a company's development of DDC. Previous research has highlighted the role of leadership and employee empowerment in making changes and shifting organizational culture (Cortellazzo et al., 2019; Motamarri et al., 2017). Therefore, the purpose of this study is to examine how leadership and employee empowerment contribute to the development of DAC and DDC, a novel contribution to the digitalization literature. A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 195 2 Review of the Literature 2.1 Resource Based View and Dynamic Capability Theory The Resources Based View (RBV) of the firm and Dynamic Capability Theory (DCT) provide a comprehensive theoretical framework for how companies can orchestrate people, process, and technology resources in such a way as to create competitive advantage (Brandon-Jones et al., 2014; Kokkinou, 2023a; Kokkinou, 2024; Mikalef & Krogstie, 2020; Wu et al., 2006). RBV stipulates that companies possessing resources that are valuable, rare, and cannot be imitated or substituted can generate competitive advantage (Barney, 1991). While RBV is widely embraced, it is critized for considering assets and capabilities as being static, and ignoring the need for companies to evolve their resources in response to changes in their environment (Eisenhardt & Martin, 2000; Wang & Ahmed, 2007). DCT evolved to address this limitation by introducing the concept of dynamic capabilities, defined as their “ability to integrate, build, and reconfigure internal and external competencies to address rapidly changing environments” (Teece et al., 1997, p. 516). DAC has been described as both a capability, and an enabler of other capabilities (Helfat & Peteraf, 2009; Kokkinou, 2023b). DAC is designated as a capability as it al ows companies to sense their environment (e.g. identify new patterns) (Mikalef et al., 2019), seize opportunities (e.g. making data-driven decisions to assess investment opportunities), and to reconfigure and transform activities in response (Teece, 2012). DAC is also considered an enabler of other dynamic capabilities (Helfat & Peteraf, 2009). These include process innovation capability (Mikalef & Krogstie, 2020), sustainable innovation capability (Al-Khatib, 2022) and supply chain agility (Dubey et al., 2019; Ghasemaghaei et al., 2017). 2.2 Data Analytic Capability and Data Driven Culture DAC is a company’s ability to use data, technology, and people to quickly generate the insights needed to support complex decision-making (Yu et al., 2021). While this definition is often applied to big data in particular (Akter et al., 2016; Dubey et al., 2019; Singh & Singh, 2019; Yu et al., 2021), the rapid pace of technological developments shifts the definition of big data, broadening the significance and applicability of DAC (Kokkinou et al., 2023b). Grounding themselves in the RBV 196 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION and socio-materialism theories, Akter et al. (2016) identified three dimensions of DAC. These were, in order of relative importance: talent capability, management capability, and technology capability. Talent capability refers to the capabilities of analytics professionals, and encompasses technical, business, relational and technology management knowledge (Akter et al., 2016). Management capability refers to decisions being made by applying a proper management framework. Technology capability refers to the flexibility of IT platforms as enabled through connectivity and compatibility of the various applications. DDC refers to an organizational culture where managers prioritize in their decision- making the insights generated by data over their intuition and gut feeling (Gupta & George, 2016; Kokkinou et al., 2023d). Unlike Akter et al. (2016), who view management capability as a dimension of DAC, we adopt the view of Gupta and Geroge (2016) and Yu et al. (2021) that DAC is an antecedent of DDC. We apply Gupta and George’s (2016, p. 1053) definition of DDC as “the extent to which organizational members (including top-level executives, middle-managers, and lower-level employees) make decisions based on the insights extracted from data.” From this perspective, DDC is an intangible resource that enables companies to harness the benefits of DAC by facilitating the dissemination of data-driven insights across a company (Chatterjee et al., 2021; Gupta & George, 2016; Karaboga et al., 2023). This is supported by previous research that found the effects of DAC and DDC to synergize (Karaboga et al., 2023; Kokkinou et al., 2023d). Consistent with these findings, we hypothesize that: H1: DAC has a positive ef ect on DDC 2.3 Employee Empowerment The definition of DDC explicitly refers to employees at al levels of the company (Gupta & George, 2016), implying that employee empowerment may be a necessary condition for DDC. Employee empowerment is defined as the “downward movement of authority and relaxing the boundaries of vertical control” (Motamarri et al., 2017). Employee empowerment has been viewed alternatively as the act of granting power to the person being empowered, the process leading to the person experiencing the power, and a psychological state (Menon, 2001). However, Menon (2001) argued A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 197 that these views were not mutualy exclusive but instead together provided a comprehensive picture of employee empowerment. By extension, employee empowerment can be seen as the delegation of decision- making authority, in such a way that it remains in line with company values and objectives (Motamarri et al., 2017). Employee empowerment is a widely embraced strategy, especial y in companies embracing Total Quality Management (Ugboro & Obeng, 2000) and has been linked to numerous business outcomes. Employee empowerment positively links to employee and customer satisfaction (Ugboro & Obeng, 2000) and individual and team performance (Chen et al., 2007; Özaralli, 2003). Employee empowerment as an organizational strategy has also been hypothesized to contribute to greater supply chain integration (Shub & Stonebraker, 2009). It has also been found to contribute to supply chain innovativeness by reducing restrictions and enabling employees to conduct operational activities pro-actively (Jaouadi, 2022). As technology has become increasingly part of employees’ day-to-day work, the concept of employee empowerment has evolved to encompass new dimensions, including (a) decision- making, (b) discretionary skills, (c) information access, (d) knowledge and skills, (e) tools and technology, and (f) training and development. Motamarri et al. (2020) examined front-line employee empowerment in the context of analytics-driven services and emphasized the importance of training and equipping front-line employees with the knowledge and technology needed to perform their work. The development of a DDC requires employees at al levels of the company, from top management to front line employees (Gupta & George, 2016; Motamarri et al., 2017) to adopt data-driven decision-making, leading to the second hypothesis: H2: Employee empowerment has a positive ef ect on DDC The company environment, and leadership in particular, play an important role in whether employees feel empowered (Menon, 2001). 198 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 2.4 Transformational Leadership Transformational leadership has been proposed as an enabler of companies’ digitalization efforts (McCarthy et al., 2022). Where transactional leadership refers to an exchange relationship between a leader and fol ower where each seeks to meet their own self-interest; transformational leadership refers to a relationship where the leader moves the fol ower beyond immediate self-interest (Bass, 1999). This is done through a combination of inspiration (the leader sets the example to be followed), charisma (the leader provides vision, arouses and inspires), intellectual stimulation (the leader provides challenge and stimulates to think in a different way), and individual consideration (the leader provides coaching and feedback) (Bass, 1999; Özaralli, 2003). Transformational leadership is thus characterized by “raising an awareness of the importance and value of designated outcomes and by developing intel ectual y stimulating and inspiring fol owers to transcend their own self-interests for a higher col ective purpose, mission, or vision” (Özaralli, 2003, p. 335). Leaders and managers play an important role in company’s digitalization and development of DAC (Kokkinou et al., 2023c; Kokkinou et al., 2024; McAfee et al., 2012) by orchestrating aspects relevant to developing DAC (Tabesh et al., 2019). To successful y guide the company, leaders and managers need to take a people, process, technology and data perspective (McCarthy et al., 2022). Leaders need to take the role of digital enabler, recognizing the opportunities engendered by technologies, communicating their value, and enabling their implementation (Cortellazzo et al., 2019). Leaders also need to promote the development of knowledge and expertise across the company through hiring new employees, training existing employees, or creating access to outside expertise (Behl et al., 2019; Kokkinou et al., 2021). Leaders also need to demonstrate commitment, give support, and al ocate the right resources to the right people (Kokkinou et al., 2023b). Transformational leadership, with its emphasis on providing a vision, coaching, and intellectually stimulating employees is therefore expected to strengthen the impact of implementing DAC on the development of DDC, leading to the following hypothesis: H3a: Transformational leadership strengthens the positive ef ect of DAC on DDC A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 199 Transformational leaders are also responsible for creating an organizational culture that encourages employee empowerment (Donate & Guadamil as, 2011). In the context of digitalization, data and the insights it enables become easier to share, enabling more autonomy for employees through involvement in real time decision-making (Cortellazzo et al., 2019). A transformational leadership is therefore expected to strengthen the effect of employee empowerment on DDC, leading to the following hypothesis: H3b: Transformational leadership strengthens the positive ef ect of employee empowerment on DDC The ful theoretical framework and study hypotheses are shown in Figure 1. Figure 1: Conceptual Framework 3 Methods A cross-sectional survey design was used to test the study hypotheses. We describe below the design of the data collection instrument, the sampling method, and the data collection and analysis procedures. 200 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3.1 Sampling and Data Collection The unit of analysis consisted of companies operating in the Netherlands. To represent these companies, we recruited participants from a database consisting of 322 professionals acting as internship supervisors for students attending a logistics programme at a Dutch university of applied sciences. After one reminder, 120 responses were obtained. 22 responses were eliminated due to excessive missing data. The remaining 98 responses were retained for further analysis (a response rate of 30.4%)1. Consistent with the nature of the sample, the transportation and logistics sector was over-represented. Third party logistics providers, transportation/ warehousing, and wholesale/distribution companies accounted respectively for 24%, 7%, and 10% of the responses obtained. Nevertheless, a wide range of other industries were also represented, including retail (14%), pharmaceuticals (8%), automotive (7%), and professional services (9%). The companies included in the final response set were also diverse in terms of size (28% had fewer than 250 employees while 35% had more than 10,000 employees worldwide) and revenues (31% earned less than 100 million EUR, while 41% earned more than 500 million EUR per year worldwide). The responses were equal y divided between Dutch companies operating mostly in the Netherlands (27%), Dutch companies operating international y (31%), and Dutch subsidiaries of an international company (31%), with the remainder qualifying themselves as “other” (10%). Respondents reported an average tenure at the company of 6.453 years (SD = 4.757). 3.2 Survey Design To collect the data for this study, a survey consisting of four sections was designed in the software Qualtrics. The first section explained the survey objective, confidentiality procedures, and informed consent. The second section was used to col ect demographic data about the company. The third section was used to col ect data on the constructs of interest to the study. The fourth section consisted of demographic questions about the respondent. To measure the constructs of interest to this study, previously validated scales were obtained from the literature. The scales for DAC (Srinivasan & Swink, 2018) and 1 The 98 responses retained for further analysis had less than 5% missing data. However, this missing data explains discrepancies between sample sizes used across the remainder of the paper and percentages not adding to 100%. A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 201 DDC (Yu et al., 2021) each consisted of four items measured on a scale from 1 (completely disagree) to 5 (completely agree). The abbreviated scale for transformational leadership, the Global Transformational Leadership scale consisting of seven items developed by Carless et al. (2000) was used and adapted to the perspective of the respondent. Respondents were asked to indicate how frequently they engaged in the behavior described on a scale from 1 (never) to 5 (always). The complete scale to measure employee empowerment consisting of 15 items was used, anchored with 1 (completely disagree) to 5 (completely agree) (Motamarri et al., 2020). A team of native Dutch speakers consisting of the third author and four master students translated the survey to Dutch. Both Dutch and English surveys were pre-tested by individuals of equivalent experience as the intended sample and their feedback was used to revise the wording in the surveys. Two native English speakers including the second author backtranslated the Dutch survey. Any inconsistencies between the two surveys and feedback received from the pre-test were discussed and resolved by the research team (consisting of the three authors) in consultation with the four master students who translated and three industry professionals who pre-tested the survey. 4 Results For each construct, the internal consistency was first established using Cronbach Alpha (Fornell & Larcker, 1981). For all constructs, Cronbach Alpha ranged from .715 to .878 (see Table 1). Table 1: Descriptive Statistics Cronbach Mean Standard Alpha Deviation Data Analytic Capability (DAC) .816 3.665 0.896 Data Driven Culture (DDC) .743 3.840 0.695 Transformational Leadership (TL) .715 4.284 0.411 Employee Empowerment (EMP) .878 4.018 0.512 202 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Linear models were subsequently used to examine the relationships between DAC, employee empowerment and DDC, and the moderating role of transformational leadership. The linear regression analysis seen in table 2 showed a statististically significant model (F(5,83)=15.72, p<.001, R2 = .486, adj R2 = .455). Since the effects of both interaction terms (transformational leadership and DAC, transformational leadership and employee empowerment) on DDC were significant, the main effects wil not be interpreted. No conclusions wil therefore be drawn regarding H1 and H2. Table 2: Linear Regression Results Variable Estimate St. Error t-value p-value Constant -3.008 4.636 -0.649 0.518 DAC -1.639 0.783 -2.092 0.039* EMP 3.017 1.429 3.017 0.038* TL 1.139 1.083 1.052 0.296 DACxTL 0.489 0.181 2.686 0.009* EMPxTL -0.683 0.334 -2.048 0.044* * denotes p < 0.05 Transformational leadership was found to significantly impact the relationship between DAC and DDC (b= 0.489, SE = 0.181, p<.05). As the interaction plot in Figure 2 shows, when transformational leadership is high (one standard deviation above the mean), DAC has a stronger positive effect on DDC than when transformational leadership is low (one standard deviation below the mean), supporting H3a. Against expectations, transformational leadership was found to significantly impact the relationship between employee empowerment and DDC (b= -0.683, SE = 0.334, p<.05) in a negative way. As the interaction plot in Figure 3 shows, when transformational leadership is high (one standard deviation above the mean), employee empowerment has a negative effect on DDC. Conversely, when transformational leadership is low (one standard deviation below the mean), employee empowerment has a positive effect on DDC. This led to the rejection of H3b. A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 203 Figure 2: Interaction Effect of Data Analytic Capability and Transformational Leadership on Data Driven Culture Source: Own Figure 3: Interaction Effect of Employee Empowerment and Transformational Leadership on Data Driven Culture Source: Own 204 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 5 Discussion There is increasing attention to the role that employees play in shaping the company’s digitalization efforts (Blanka et al., 2022). The purpose of the present study was to examine how companies can develop their organizational culture towards being a DDC, by considering both technical aspects such as the development of DAC, and organizational aspects such as leadership and employee empowerment (Gupta & George, 2016). Our findings show that the development of DAC, in combination with transformational leadership has the greatest positive impact on the development of a DDC. Developing DAC contributes to turning data into actional insights (Chatterjee et al., 2021). Leaders leaning towards a more transformational leadership style in turn can inspire employees by sharing their vision of how DAC contributes to the company’s strategy and modeling data-driven decision-making (Kokkinou et al., 2023a; McCarthy et al., 2022; Özaralli, 2003). Unexpectedly, employee empowerment in combination with transformational leadership had a negative impact on DDC. While Cortellazzo et al. (2019), Jaouadi (2022) and Motamarri et al. (2020) advocate for employee empowerment and transformational leadership to promote a DDC (Cortellazzo et al., 2019; Jaouadi, 2022; Motamarri et al., 2017), our findings demonstrate the opposite. In the presence of transformational leadership, employee empowerment had a negative impact on developing a DDC. Several explanations can be proposed based on the literature. First, employee empowerment is a multi-dimensional concept (Motamarri et al., 2017) incorporating people aspects (e.g. discretionary skills), process aspects (e.g. decision-making), and technology aspects (e.g. tools and technology) (Motamarri et al., 2020). As such, transformational leadership only impacts certain aspects of employee empowerment, and could impact those which are unrelated to DDC, wasting precious resources. Second, as Motamarri et al. (2020) demonstrated in the context of frontline employees, employees need to be equipped with the training, knowledge, and technology to be able to perform. It is thus possible that when empowered and inspired by transformational leaders but without being given the necessary data analytic knowledge on a more practical level, employees resort to using their intuition instead of DAC. A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 205 5.1 Limitations and Further Research Further research should delve deeper into processes linking employee empowerment, transformational leadership, and DDC. A qualitative and more in- depth approach, through multiple case studies, should examine the mechanisms underlying these processes, and identify potential mediating variables. These variables could include other people related aspects such as intrapreneurial competency (Blanka et al., 2022; Vargas-Halabí et al., 2017) and technical training, knowledge, and skills (Kokkinou, 2023b). The present study used a cross-sectional survey design to examine how DAC, employee empowerment and transformational leadership contribute to companies’ development of a DDC. The sample was limited to companies operating the Netherlands and was relatively smal . The Netherlands is characterized by a low- power-distance culture (Hofstede, 1983), and an affinity for data-driven decision making. To improve the generalizability of the findings, the study should be replicated with a broader and more diverse sample. 5.2 Implications for Practitioners In their pursuit of competitive advantage, companies need to embrace digitalization and develop their DAC. To realize the advantage of DAC, companies need to invest in developing a DDC, so that these insights are used for decision-making and contribute to improving company performance, supply chain resilience and competitive advantage. To achieve this, companies need to not only focus on technical issues, but also the interlinked aspects of leadership and employee empowerment. Clearly, just promoting transformational leadership and employee empowerment practices is not sufficient. Instead, attention needs to be given to the fit between people, process, and technology, and the adequate provision of training, knowledge, and skills. Acknowledgements We would like to acknowledge funding received from the Dutch Taskforce for Applied Research (SIA). This publication is part of the project A roadmap for developing Data Analytic Capability for Digital Transformation (project number KIEM.K23.01.081) funded by the Taskforce for Applied Research 206 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION (SIA) and the Dutch Research Council (NWO). We would like to also thank the students and professionals who participated in the BUas Master in Supply Chain Management Learning Communities during the academic year 2023-2024. References Akter, S., Wamba, S. F., Gunasekaran, A., Dubey, R., & Childe, S. J. (2016). How to improve firm performance using big data analytics capability and business strategy alignment? International Journal of Production Economics, 182, 113–131. https://doi.org/10.1016/j.ijpe.2016.08.018 Al-Khatib, A. W. (2022). Big data analytics capabilities and green supply chain performance: Investigating the moderated mediation model for green innovation and technological intensity. Business Process Management Journal, 28(5/6), 1446–1471. https://doi.org/10.1108/BPMJ-07-2022-0332 Barney, J. (1991). Firm Resources and Sustained Competitive Advantage. Journal of Management, 17(1), 99–120. https://doi.org/10.1177/014920639101700108 Bass, B. M. (1999). Two Decades of Research and Development in Transformational Leadership. European Journal of Work and Organizational Psychology, 8(1), 9–32. https://doi.org/10.1080/135943299398410 Behl, A., Dutta, P., Lessmann, S., Dwivedi, Y. K., & Kar, S. (2019). A conceptual framework for the adoption of big data analytics by e-commerce startups: A case-based approach. Information Systems and E-Business Management, 17(2–4), 285–318. https://doi.org/10.1007/s10257-019-00452-5 Blanka, C., Krumay, B., & Rueckel, D. (2022). The interplay of digital transformation and employee competency: A design science approach. Technological Forecasting and Social Change, 178, 121575. https://doi.org/10.1016/j.techfore.2022.121575 Brandon-Jones, E., Squire, B., Autry, C. W., & Petersen, K. J. (2014). A Contingent Resource-Based Perspective of Supply Chain Resilience and Robustness. Journal of Supply Chain Management, 50(3), 55–73. https://doi.org/10.1111/jscm.12050 Carless, S. A., Wearing, A. J., & Mann, L. (2000). A short measure of transformational leadership. Journal of Business and Psychology, 14(3), 389–405. https://doi.org/10.1023/A:1022991115523 Chatterjee, S., Chaudhuri, R., & Vrontis, D. (2021). Does data-driven culture impact innovation and performance of a firm? An empirical examination. Annals of Operations Research. https://doi.org/10.1007/s10479-020-03887-z Chen, G., Kirkman, B. L., Kanfer, R., Al en, D., & Rosen, B. (2007). A multilevel study of leadership, empowerment, and performance in teams. Journal of Applied Psychology, 92(2), 331–346. https://doi.org/10.1037/0021-9010.92.2.331 Cortel azzo, L., Bruni, E., & Zampieri, R. (2019). The Role of Leadership in a Digitalized World: A Review. Frontiers in Psychology, 10, 1938. https://doi.org/10.3389/fpsyg.2019.01938 Donate, M. J., & Guadamil as, F. (2011). Organizational factors to support knowledge management and innovation. Journal of Knowledge Management, 15(6), 890–914. https://doi.org/10.1108/13673271111179271 Dubey, R., Gunasekaran, A., & Childe, S. J. (2019). Big data analytics capability in supply chain agility: The moderating effect of organizational flexibility. Management Decision, 57(8), 2092–2112. https://doi.org/10.1108/MD-01-2018-0119 Eisenhardt, K. M., & Martin, J. A. (2000). Dynamic capabilities: What are they? Strategic Management Journal, 21(10‐11), 1105–1121. Fornell, C., & Larcker, D. F. (1981). Structural Equation Models with Unobservable Variables and Measurement Error: Algebra and Statistics. Journal of Marketing Research, 18(3), 382–388. https://doi.org/10.1177/002224378101800313 Garmaki, M., Boughzala, I., & Wamba, S. F. (2016). The ef ect of Big Data Analytics Capability on Firm Performance. 301. A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 207 Ghasemaghaei, M., Hassanein, K., & Turel, O. (2017). Increasing firm agility through the use of data analytics: The role of fit. Decision Support Systems, 101, 95–105. https://doi.org/10.1016/j.dss.2017.06.004 Gupta, M., & George, J. F. (2016). Toward the development of a big data analytics capability. Information & Management, 53(8), 1049–1064. https://doi.org/10.1016/j.im.2016.07.004 Helfat, C. E., & Peteraf, M. A. (2009). Understanding dynamic capabilities: Progress along a developmental path. Strategic Organization, 7(1), 91–102. https://doi.org/10.1177/1476127008100133 Hofstede, G. (1983). The cultural relativity of organizational practices and theories. Journal of International Business Studies, 14(2), 75–89. Jaouadi, M. H. O. (2022). Investigating the influence of big data analytics capabilities and human resource factors in achieving supply chain innovativeness. Computers & Industrial Engineering, 168, 108055. https://doi.org/10.1016/j.cie.2022.108055 Karaboga, T., Zehir, C., Tatoglu, E., Karaboga, H. A., & Bouguerra, A. (2023). Big data analytics management capability and firm performance: The mediating role of data-driven culture. Review of Managerial Science, 17(8), 2655–2684. https://doi.org/10.1007/s11846-022-00596-8 Kokkinou, A. (2023a). How organizations can harness continuous improvement practices to develop their data analytic capability: A conceptual paper. Presented at the 2023 European Lean Educators Conference: Chal enging the Future with Lean. Kokkinou, A. (2023b). Planting the seed: Introducing knowledge of digitalization using continuous improvement capabilities. In J. Antony, J. Alsadi, A. Elmougi, M. Ramadan, & M. Bader (Eds.), Operational Excel ence in the era of Industry 4.0 and Industry 5.0 Proceedings from the Ninth International Conference on Lean Six Sigma (pp. 37–54). Northumbria University. Kokkinou, A. (2024). How organizations can harness continuous improvement practices to develop their data analytic capability: a conceptual paper, in van Kol enburg, T., Kokkinou, A. and McDermott, O. (Eds.), Chal enging the Future with Lean: 9th IFIP WG 5.7 European Lean Educator Conference, ELEC 2023, ’s-Hertogenbosch, The Netherlands, October 24-26, 2023, Proceedings, Springer. Kokkinou, A., Kollenburg, T. van, & Mandemakers, A. (2023a). Development of data analytic capability in organizations. Presented at the 2023 EurOMA Conference. European Operations Management Association (EurOMA), Leuven, Belgium. Kokkinou, A., Kol enburg, T. van, Mandemakers, A., Hopstaken, E., & Elderen, J. van. (2023b). The Data Analytic Capability Wheel: An implementation framework for digitalization. In A. Pucihar, M. Kljajić Borštnar, R. Bons, G. Ongena, M. Heikkilä, & D. Vidmar (Eds.), Proceeedings of the 36th Bled eConference Digital Economy and Society: Balancing act for digital innovation in times of instability: June 25—28, . Univerza v Mariboru, Univerzitetna založba. https://press.um.si/index.php/ump/catalog/book/786 Kokkinou, A., Kol enburg, T. van, van Doren, S., Matthijssen, G., Visser, E., Hoogendoorn, L., & Herpt, P. van. (2023c). Enablers and inhibitors of digitalization as part of continuous improvement. Presented at the 2023 EurOMA Conference. European Operations Management Association (EurOMA), Leuven, Belgium. Kokkinou, A., Mandemakers, A., & Mitas, O. (2023d). Developing Resilient and Robust Supply Chains through Data Analytic Capability. Continuity & Resilience Review, 5(3), 320–342. https://doi.org/10.1108/CRR-07-2023-0013 Kokkinou, A., van Kol enburg, T., & Touw, P. (2021). The Role of Training in the Implementation of Lean Six Sigma. Presented at the 2021 EurOMA Conference. European Operations Management Association (EurOMA), Berlin, Germany. Kokkinou, A., Quak, H., & Mandemakers, A. (2024). Driving digital transformation: enablers and inhibitors of data analytic capability in transportation and logistics companies. In Bijdragen Vervoerslogistieke Werkdagen 2024 (VLW). Vervoerslogistieke werkdagen (VLW). Machado, C. G., Winroth, M., Carlsson, D., Almström, P., Centerholt, V., & Hal in, M. (2019). Industry 4.0 readiness in manufacturing companies: Chal enges and enablers towards 208 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION increased digitalization. Procedia CIRP, 81, 1113–1118. https://doi.org/10.1016/j.procir.2019.03.262 McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D., & Barton, D. (2012). Big data: The management revolution. Harvard Business Review, 90(10), 60–68. McCarthy, P., Sammon, D., & Alhassan, I. (2022). Digital Transformation Leadership Characteristics: A Literature Analysis. Journal of Decision Systems, 32(1), 79–109. https://doi.org/10.1080/12460125.2021.1908934 Menon, S. (2001). Employee Empowerment: An Integrative Psychological Approach. Applied Psychology, 50(1), 153–180. https://doi.org/10.1111/1464-0597.00052 Mikalef, P., & Krogstie, J. (2020). Examining the interplay between big data analytics and contextual factors in driving process innovation capabilities. European Journal of Information Systems, 29(3), 260–287. https://doi.org/10.1080/0960085X.2020.1740618 Mikalef, P., Van de Wetering, R., & Krogstie, J. (2019). From Big Data Analytics to Dynamic Capabilities: The Ef ect of Organizational Inertia. 1–14. Motamarri, S., Akter, S., & Yanamandram, V. (2020). Frontline employee empowerment: Scale development and validation using Confirmatory Composite Analysis. International Journal of Information Management, 54, 102177. https://doi.org/10.1016/j.ijinfomgt.2020.102177 Motamarri, S., Akter, S., Yanamandram, V., & Wamba, S. F. (2017). Why is Empowerment Important in Big Data Analytics? Procedia Computer Science, 121, 1062–1071. Özaral i, N. (2003). Effects of transformational leadership on empowerment and team effectiveness. Leadership & Organization Development Journal, 24(6), 335–344. Rossi, A. H. G., Marcondes, G. B., Pontes, J., Leitão, P., Treinta, F. T., De Resende, L. M. M., Mosconi, E., & Yoshino, R. T. (2022). Lean Tools in the Context of Industry 4.0: Literature Review, Implementation and Trends. Sustainability, 14(19), 12295. Shub, A. N., & Stonebraker, P. W. (2009). The human impact on supply chains: Evaluating the importance of “soft” areas on integration and performance. Supply Chain Management: An International Journal, 14(1), 31–40. https://doi.org/10.1108/13598540910927287 Singh, N. P., & Singh, S. (2019). Building supply chain risk resilience: Role of big data analytics in supply chain disruption mitigation. Benchmarking: An International Journal, 26(7), 2318–2342. https://doi.org/10.1108/BIJ-10-2018-0346 Srinivasan, R., & Swink, M. (2018). An Investigation of Visibility and Flexibility as Complements to Supply Chain Analytics: An Organizational Information Processing Theory Perspective. Production and Operations Management, 27(10), 1849–1867. https://doi.org/10.1111/poms.12746 Tabesh, P., Mousavidin, E., & Hasani, S. (2019). Implementing big data strategies: A managerial perspective. Business Horizons, 62(3), 347–358. https://doi.org/10.1016/j.bushor.2019.02.001 Teece, D. J. (2012). Dynamic Capabilities: Routines versus Entrepreneurial Action: Routines versus Entrepreneurial Action. Journal of Management Studies, 49(8), 1395–1401. https://doi.org/10.1111/j.1467-6486.2012.01080.x Teece, D. J., Pisano, G., & Shuen, A. (1997). Dynamic capabilities and strategic management. Strategic Management Journal, 18(7), 509–533. https://doi.org/10.1002/(SICI)1097-0266(199708)18:7<509::AID-SMJ882>3.0.CO;2-Z Ugboro, I. O., & Obeng, K. (2000). Top management leadership, employee empowerment, job satisfaction, and customer satisfaction in TQM organizations: An empirical study. Journal of Quality Management, 5(2), 247–272. https://doi.org/10.1016/S1084-8568(01)00023-2 Vargas-Halabí, T., Mora-Esquivel, R., & Siles, B. (2017). Intrapreneurial competencies: Development and validation of a measurement scale. European Journal of Management and Business Economics, 26(1), 86–111. https://doi.org/10.1108/EJMBE-07-2017-006 Vial, G. (2019). Understanding digital transformation: A review and a research agenda. The Journal of Strategic Information Systems, 28(2), 118–144. https://doi.org/10.1016/j.jsis.2019.01.003 Wang, C. L., & Ahmed, P. K. (2007). Dynamic capabilities: A review and research agenda. International Journal of Management Reviews, 9(1), 31–51. https://doi.org/10.1111/j.1468-2370.2007.00201.x A. Kokkinou, O. Mitas, A. Mandemakers: Cultivating a Data-Driven Culture: Exploring The Role of Data Analytic Capability, Employee Empowerment, and Transformational Leadership 209 Wu, F., Yeniyurt, S., Kim, D., & Cavusgil, S. T. (2006). The impact of information technology on supply chain capabilities and firm performance: A resource-based view. Industrial Marketing Management, 35(4), 493–504. https://doi.org/10.1016/j.indmarman.2005.05.003 Yu, W., Wong, C. Y., Chavez, R., & Jacobs, M. A. (2021). Integrating big data analytics into supply chain finance: The roles of information processing and data-driven culture. International Journal of Production Economics, 236, 108135. https://doi.org/10.1016/j.ijpe.2021.108135 210 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION A MATURITY MODEL FOR EVALUATING DATA- DRIVEN SUSTAINABILITY MANAGEMENT ANNA SELL,1 JERKER BJÖRKQVIST,2 JANNE HAKALA,2 FREDRIK HELLMAN,2 XIANG HU,3 OSKAR KARLSTRÖM,3 ANDERS LINDGREN,3 ANDREAS LUNDELL,2 AMAIA MARTIN-GRANIZO2 1 Åbo Akademi University, Faculty of Social Sciences, Business and Economics, and Law, Turku, Finland anna.sel @abo.fi 2 Åbo Akademi University, Faculty of Science and Engineering, Turku, Finland jerker.bjorkqvist@abo.fi, janne.hakala@abo.fi, fredrik.hel man@abo.fi, andreas.lundel @abo.fi, amaia.martin-granizo@abo.fi 3 University of Turku, Faculty of Technology, Turku, Finland xiang.hu@utu.fi, oskar.karlstrom@utu.fi, anders.lindgren@utu.fi. In order to stay competitive, manufacturing companies seek to enhance the accuracy, timeliness, and transparency of their sustainability efforts. This can be achieved through implementing data-driven and dynamic sustainability measurement throughout product life cycles. We introduce a maturity model for assessing and improving data-driven sustainability management, encompassing eight technical and organizational dimensions derived from both theory and practitioner insights through a design science research approach. We detail the maturity levels within each dimension, providing insights into companies' progress. For instance, in data handling and data sensors, companies move from basic implementation to real-time integration and cloud connectivity. The model also highlights challenges, such as collecting sustainability Keywords: sustainability, background data, formulating sustainability KPIs, and how to tailor maturity sustainability communication. We emphasize the importance of model, aligning sustainability efforts with strategic business outcomes and data-driven communication, the role of a pervasive data culture within companies. The article data-driven concludes with considerations for future research and model sustainability, refinement. KPIs DOI https://doi.org/10.18690/um.fov.4.2024.11 ISBN 978-961-286-871-0 212 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Sustainability is increasingly important for manufacturing companies due to regulations, customer demands, and competition. To improve the accuracy, timeliness and transparency of their sustainability efforts, companies are turning to data-driven and dynamic sustainability measurement throughout their product life cycles. Data-driven sustainability management (DDSM) poses new chal enges on several dimensions compared to more static sustainability measures, ranging from more complex data capture and data processing to new questions regarding data analysis and communication. Companies that successful y implement data-driven sustainability management can comply with regulatory requirements but also achieve a competitive edge. They do this by leveraging sustainability data and insights for business development, credible sustainability communications, and the creation of new products and services. We identify and describe relevant technical and organizational dimensions and introduce a maturity model for assessing and improving the level of data-driven sustainability management in organizations, with a special focus on using data continuously and dynamical y. Maturity models are tools for continuous improvement and benchmarking, outlining the capabilities and conditions necessary to achieve a desired level of performance (Lasrado et al., 2015, Mettler, 2009). While some models use a life cycle perspective with the goal of al organizations reaching the highest level of maturity, modern models more frequently use a potential performance perspective where the organization itself decides which level of maturity is optimal for a given situation (Wendler, 2012). Maturity models general y comprise five components: (a) maturity levels or stages, (b) dimensions, (c) sub-categories, (d) paths to maturity, and (e) assessment questions (Lasrado et al., 2015). Taken together, they describe the current state of the organization, enable comparison against high-performing organizations and prescribe actions to be taken to advance maturity (Poeppelbuss et al., 2011). While previous models exist for evaluating the maturity of sustainability (e.g., Sari et al., 2021; Vásquez et al., 2021) and data-driven operations (e.g., Grossman, 2018; Gökalp et al., 2021), there is no previous model combining these, i.e., data-driven sustainability management. A. Sel et al.: A Maturity Model for Evaluating Data-driven Sustainability Management 213 2 Data-driven Sustainability Evaluation In manufacturing, sustainability is measured with key performance indicators (KPIs) (Neri et al., 2021) and evaluated with Life Cycle Assessment (LCA) (Curran, 2013). A limitation of both LCA and many KPIs is that results may suffer from too static input assumptions. Continuous measurement of data points, and continuous evaluation, can be done with Dynamic Life Cycle Assessment (DLCA) (Sohn et al., 2020). DLCA facilitates the continuous tracking of environmental parameters throughout the production process, exemplified by real-time carbon emission monitoring, which correlates closely with energy usage during production. DLCA and similar methodologies can be seamlessly integrated with Enterprise Resource Planning (ERP) in Industry 4.0 environments (Ferrari et al., 2021), as well as manufacturing system simulations, thereby facilitating comprehensive assessments of energy production and carbon emissions (Rödger et al., 2021). Additional y, DLCA facilitates the tracking of Global Warming Potential over time, offering valuable insights into evolving environmental dynamics (Levasseur et al., 2010). In the context of our Dynamic Data-driven Sustainability Management model, DLCA serves as a cornerstone for evaluating environmental impacts across diverse industries. Following data collection, the DLCA methodology is employed to concurrently assess carbon emissions and other environmental factors. Importantly, within the DDSM framework, environmental impact results are integrated seamlessly alongside data gathering, transformation, and communication processes, culminating in a comprehensive solution for industry sustainability evaluations. 3 Methodology We fol ow a design science research (DSR) methodology to construct the maturity model, as outlined by e.g., Mettler (2009). A central aspect of DSR is that the research wil produce an artifact “created to solve an important organizational problem” and address an identified business need (Hevner et al., 2004, p 82). Our research is conducted within an industry-academia project. In a DSR process (Peffers et al., 2007), this is especially important regarding problem identification and motivation (activity 1) , defining objectives for the solution (act. 2), demonstration (act. 4) and evaluation (act. 5). The maturity model, as presented in section 4, is thus based on both theory and practitioner insights. Demonstration of a proposed solution and subsequent evaluation of a more mature artifact are central components of DSR, 214 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION aiming to ensure rigor, utility, and quality (Mettler, 2009; Hevner et al., 2004). We have carried out a demonstration in a workshop with industry partners, collecting feedback on the model and its dimensions (discussed in section 5). Five privately owned companies are contributing with case studies, data and scenarios (Table 1). They are all manufacturing companies but with very different products, giving us the possibility to explore how wel the proposed methodology is applicable to different application domains. Most of the companies meet the European Commission criteria for medium-sized enterprises (turnover ≤50 mil ion, employees <250) (European Commission, 2020). Table 2: Companies participating in the research project Company Products A Abrasives and equipment for surface finishing B Large sailing yachts C Bicycle components D Mechanical biomass treatment equipment E Chemical solutions 4 The Data-Driven Sustainability Management Maturity Model Table 2 provides an overview of the Data-Driven Sustainability Management Maturity Model. The eight dimensions represent central maturity-influencing factors, based on literature and discussion with project companies. The level descriptions are abbreviated for presentation purposes to three main levels, from a more complete, five-level version of the model. 4.1 Data Handling and Data Sensors Data collection forms the basis of all subsequent analyses, reporting and decision making (Linke et al., 2019), and is thus a central aspect of data-driven sustainability management. At the Beginning stage, companies store sensor data on fundamental operational metrics from the manufacturing execution system (MES), such as energy and water usage, focusing more on operational efficiency than sustainability. In the Intermediate stage, there is a clear shift towards integrating data from various sources, automating data pipelines, and incorporating tools to calculate sustainability metrics, A. Sel et al.: A Maturity Model for Evaluating Data-driven Sustainability Management 215 such as LCA. This could include energy usage across different processes, monitoring the use of raw materials and waste processes more closely, and starting to assess the lifecycle impacts of products. Sensor technology advances to capture more nuanced sustainability-related data, including water and air quality metrics. At the Advanced stage, companies have implemented ful real-time data integrations and pipelines. Further, version handling and data verification is implemented to ensure accuracy and reliability. Sensors are implemented to cover al relevant energy and material flows, with cloud connectivity for real-time monitoring, and edge computing for high-speed data processing. Table 2: The Data-Driven Sustainability Management Maturity Model Beginning Intermediate Mature / Advanced Automated data Data is stored with pipeline available with Real-time data Data handling manual or offline support for integration, support for processing LCA/sustainability full sustainability calculations calculations in real-time Additional sensors, Data sensors Basic use of sensors Sensors for all material cloud connected and primarily from MES and energy flows can handle high-speed real-time data Collect some data, Inventory prepared for Use the collected data background data Try to collect data sustainability evaluation for sustainability using those data evaluation Process Basic process Process description for Mature process descriptions description sustainability evaluation description for sustainability evaluation Descriptive; basic Predictive; comparing analysis tools, no data points and Prescriptive; data- Analytics standardized historical data to show driven, dynamic processes future trends dashboards and self- service reporting Ad-hoc based, one- Automation and Internal directional, periodic Information shared on integration of communication reports internal digital communication platforms platforms, targeted, two-way External Unstructured data, Company website, Multi-channel, communication periodic reports regular updates dynamic, automated Group level sustainability KPIs, Financial, operational Management High-level financial Financial/operational and sustainability KPIs KPIs KPIs KPIs for product, on product, service, service, segment or site segment or site level level 216 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.2 Inventory Background Data Background data is crucial for sustainability evaluations, covering various datasets such as energy consumption, material flows, and other production-related metrics. This foundational data is integral to methodologies like Life Cycle Assessment (LCA) and underpins other sustainability evaluation practices. Sufficient and accurate background data is essential for ensuring the precision of sustainability assessments, as inadequate data may result in flawed assumptions. Presently, most LCAs operate within a static framework. However, with the advent of Industry 4.0, there is a burgeoning interest in real-time monitoring of environmental impacts throughout the production process (Ferrari et al., 2021). A clear and well-organized repository of background data is instrumental in facilitating this transition, ultimately aiding in the reduction of carbon emissions, and fostering more informed decision- making processes. 4.3 Process Descriptions In the realm of Data-driven Sustainability Management (DDSM), the term "process" refers to production or manufacturing activities directly pertinent to sustainability evaluations conducted within an organization. Process descriptions are vital for obtaining data for sustainability assessments. A wel -articulated process description facilitates the establishment of clear objectives and delineation of scope, thereby aiding in the systematic data inventory process for LCA. Conversely, inadequate process descriptions can lead to disarray in sustainability evaluations, thereby compromising the comprehensiveness of sustainability communication efforts. An ideal process description encompasses wel -structured processes, enables dynamic data linkage, is compatible with LCA methodologies, and provides a comprehensive overview of the entire production cycle. 4.4 Analytics Data analytics involves cleaning and interpreting data for actionable insights. In the Beginning level, companies typically face unstructured sustainability data and a lack of analysis processes. Some basic analysis tools are in use (i.e., spreadsheets) but sustainability data is analysed on an ad-hoc basis. The next step ( Intermediate) is to compare trends and determine possible cause-and-effect relationships with more A. Sel et al.: A Maturity Model for Evaluating Data-driven Sustainability Management 217 advanced tools and methods, like linear regression models and visualizations. Moreover, historical data is used to make predictions on future trends. The use of advanced sustainability dashboards and self-service Business Intel igence or automation, characterizes the Mature stage, in which sustainability data is being used to drive insight across the company and suggest actionable solutions. Data sources are integrated and ful y accessible, encouraging employee interaction through personalized, dynamic dashboards, where real-time sustainability data is managed and updated seamlessly (van Groenendale, 2022; Gudfinnsson et al., 2015). 4.5 Internal Communication Internal communication engages employees and fosters awareness of organizational sustainability goals (Sedej & Mumel, 2015). Communication has an integral function in converting sustainability data into actionable insights and clear messages to employees in real time. In the Beginning, sustainability information exchange is predominantly one-way, and data is underutilized. Sustainability information is shared at specific times through the company intranet or periodic reports. In the Intermediate stage, there is an enhanced understanding of internal communication as a lever of company sustainability strategy. Further communication capabilities may be developed, including the use of digital platforms with readily available sustainability data, and contribution to internal sustainability programs is encouraged for the individuals. Advanced maturity is reached when internal communication is embedded in the sustainability strategy and real-time tracking of life-cycle data across the value chain is actively shared through company platforms. Finally, automation of internal sustainability communication is achieved, and integrated data systems support the timely monitoring of metrics, upon which to act. Communication is more targeted and has evolved towards two-way exchange, actively including employees, who companies see as having an integral role in reinforcing sustainability targets (Donnellan et al., 2011; Robertson, 2024). 4.6 External Communication External communication focuses on brand awareness and conveying identity commitment to stakeholders (Piehler et al., 2018). Stakeholders expect organizations to contribute positively to society, minimize environmental impact, and cultivate authentic brands (Markovic et al., 2023). External sustainability communication is 218 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION shifting from periodic reports and environmental certifications, towards more integrated, dynamic forms of communicating, with the goal to reduce the risk of greenwashing and increase transparency. In the Beginning, sustainability data is communicated periodical y, within an annual report or sometimes in a specified sustainability report. Intermediate level is reached when sustainability communication is supported by processed sensor data and LCA. Communication may happen through the corporate website, where the most important information is displayed. The addition of multiple channels and web-based communication, such as social media, are some characteristics of an Advanced maturity level. At this stage, communication happens in a timely way and al ows for interaction between the company and the external stakeholders. It also marks the transition towards automated, data-driven sustainability communication. 4.7 Management KPIs Key Performance Indicators (KPIs) are a way to measure activities in organizations and help management make sound business decisions. KPIs can be based on statutory reporting statements with follow-up occurring either annual y or more frequently. An intermediate level might be achieved by detailed financial and operational KPIs split by either product, service, business segment or site. Frequency of KPI follow-up versus targets would occur either at month-end close or weekly. The sustainability reporting on the intermediate level would be performed on a group level following international standards. The advanced level involves more granular information on sustainability KPIs. This might include electricity use, water use, waste production and greenhouse gas emissions by product, service, business segment or site. Most of the environmental measurements could be achieved by sensors, and therefore be more data-driven, whereas for the sustainability aspects of social and governance would likely need more manual processing from HR systems (personnel turnover, workplace accidents, on-the-job training) and legal (claims, payment terms). A. Sel et al.: A Maturity Model for Evaluating Data-driven Sustainability Management 219 5 Demonstrating the DDSM Maturity Model 5.1 Data Handling and Data Sensors Al companies have some sensors implemented throughout the production process, and some are looking to expand their sensor network to cover al material and energy flows, indicating a clear desire for more comprehensive ways of capturing data to support data-driven sustainability management. Company D and B are at the initial phases of integrating basic sensors and enhancing data storage solutions, while Company E is advancing towards a more comprehensive sensor network. However, the common aspiration for al companies was for sensors and data storage solutions that could facilitate real-time data collection, directly feeding into cloud-based systems for real-time analysis. Cloud storage emerged as a major theme in the discussions, reflecting a trend towards more accessible and scalable data storage solutions. Company A is at the forefront of embracing cloud storage, demonstrating a commitment to modernizing data management. Meanwhile, Company C and D are transitioning from manual and offline data processing to automated data pipelines, capable of supporting real-time sustainability calculations, overcoming the existing limitations of their data systems. The pursuit of real-time data integration and the chal enges of achieving it were recurrent themes throughout the interviews. 5.2 Inventory Background Data The interviews indicate that most companies have progressed to the intermediate and mature stages of development, with many having already amassed energy and material flow data to facilitate carbon emission calculations and broader sustainability assessments. Notably, most of the companies demonstrate capability in managing Scope 1 and Scope 2 carbon emissions through the integration of gathered data. Companies B, C, and E have enlisted consulting firms to leverage background data for carbon emission calculations. In contrast, Company A has attained a more advanced level of proficiency, leveraging comprehensive data sets for life cycle assessments and carbon emission calculations. Utilization of these findings informs product design enhancements and modifications geared towards fostering more sustainable applications. 220 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION During the interviews, al companies voiced common chalenges with the acquisition of essential background data. The primary hurdle lies in the identification, collection, and categorization of pertinent data, particularly when faced with resource constraints. Data col ection emerges as the most time-intensive and arduous aspect of sustainability evaluations. Another significant challenge pertains to the management and utilization of collected data; there is a need for specific guidelines or handbooks delineating optimal data al ocation strategies for subsequent calculations or evaluations. While overarching standards such as ISO 14040 and 14044 exist to aid companies in conducting accurate life cycle assessments (LCA), their generic nature renders them insufficient for tailoring approaches to individual products and production lines. The calculation of Scope 3 emissions using background data presents a chal enge for all companies. Procuring data from suppliers is particularly arduous, especially for enterprises with global supply chains outside Europe. A potential solution entails companies strategically selecting suppliers capable of furnishing comprehensive sustainability data when procuring raw materials. Moreover, all companies express apprehension regarding environmental impact factors beyond carbon emissions, e.g. waste treatment, water consumption, and land use. These additional evaluation categories necessitate distinct considerations, underlining the multifaceted nature of sustainability assessments. Company A has pioneered the integration of Internet of Things (IoT) technology into its background data collection practices, enabling them to capture real-time energy consumption data. Other companies are deliberating the introduction of similar practices. 5.3 Process Descriptions The participating companies universaly endorse our conceptualization of Process Descriptions. Most companies have progressed beyond the initial stages and have reached an intermediate level of development: they have initiated efforts to delineate or depict their production or manufacturing processes in the context of sustainability evaluations. Notably, Company A stands out as being at an advanced stage in this regard. Multiple companies articulated their intentions to enhance their process descriptions to align with the requirements for calculating carbon emissions. The adoption of standardized descriptions across various processes holds promise for A. Sel et al.: A Maturity Model for Evaluating Data-driven Sustainability Management 221 facilitating data collection and advancing sustainability evaluations within companies. 5.4 Analytics Most companies employ daily analytics for key business operations and sales, relying on historical data for prognoses. However, sustainability reporting practices vary, with most relying on manual data gathering from manufacturing execution systems (MES) and spreadsheet calculations. Analytic tools availability depends on data sources; ERP systems offer standard reports or dashboards, while some use additional business intelligence tools like Power BI, Tableau, or Qlik Sense. Limited users in organizations can modify these tools. Company C, however, aims to make sustainability data accessible company wide. Company A aims to gather data in a Databricks cloud environment for more flexible analytics. Company B prioritizes utilizing ERP data for sustainability reporting. Company C utilizes Tableau for dashboards, and the data is synced there once per day; sustainability data is currently not synced, however. Company D excels in customer-facing analytics but aims to enhance internal sustainability reporting with a new ERP system. Company E employs Qlik Sense for some real-time analytics, with varying capabilities across different organizational sections. 5.5 External and Internal Communication Overall, the discussions reflected our initial level descriptions in the DDSM maturity model. The three main factors emerging when considering communication maturity, are: enhanced dynamic capabilities and data-driven approach, customization of messages and channels, and two- or multi-directional communication. Moreover, the division into two separate dimensions, internal and external communication, proved to be appropriate as each communication activity serves its own purpose and goals within the organization. In some case, a more Advanced maturity had been achieved in external communication but not yet internally, and vice versa. Several additional points of interest were raised during the discussions. Company D described their desire to share sustainability data internal y to promote employee agency and decision-making in line with sustainability goals, but tailoring the message in suitable ways to blue- and white-col ar workers proved a chal enge. Company A 222 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION referred to a product stewardship approach, aiming to extend the responsibility for reducing the environmental impact of products to al actors in the supply chain. Company B mentioned the importance of resources, and the lack thereof, as a crucial factor in reaching higher levels of communication maturity. On the same note, companies discussed the importance of knowing their target audiences, to understand which metrics or messages are the most relevant to each customer segment. Finally, the discussions touched upon the loss of control over the narrative, when communication is data-based, real-time and tailored to the recipient. As message creation is more dependent on real-time data, organizations must build an open and transparent data culture in which multiple internal actors and departments work in alignment to get the right information across. 5.5 Management KPIs The introduction of a new EU directive on corporate sustainability reporting wil make sustainability reporting mandatory for many companies and wil indirectly affect smal er companies. This new directive wil force companies towards measuring and reporting sustainability linked KPIs. Companies A, B and E are large enough to be directly affected by the new directive, whereas companies C and D wil be indirectly affected through their position in the value chain. Al companies are of a size that warrants monthly reporting of VAT and payroll related costs; hence al companies have month-end closing routines. These two factors, being monthly accounting cycles and upcoming mandatory sustainability reporting requirements, wil set a baseline for the companies to effectively ensure the capability of an intermediate level of management KPIs. The reporting frequency and follow-up of KPIs is general y at a good level of sophistication for the participating companies, with differences being somewhat attributable to the ownership base. The intermediate level should suffice to meet also regulatory sustainability reporting requirements whereas the advanced stage might generate more informed decisions and potentially allow for more value creation. A. Sel et al.: A Maturity Model for Evaluating Data-driven Sustainability Management 223 6 Discussion and conclusions Taken together, the interviews provided an initial validation of the proposed dimensions and maturity levels. The interview findings form the basis for developing the maturity model further, focusing on dimension sub-categories, paths to maturity and assessment questions. The model was valuable in fostering structured dialogues around companies’ current standings and future aspirations in data-driven sustainability management. Some interesting conclusions emerge. While companies excel in handling financial data, analytics, tools, and reporting, they struggle to apply similar practices and tools when it comes to sustainability. The companies described an ambition to report and communicate sustainability as real-time as possible, on a product-level and to have a data pipeline for sustainability, i.e., an automated flow from data sensors to visualization. Challenges included collecting background data, formulating sustainability KPIs, measuring sustainability on a granular level, the current need for manual data collection and analysis, and knowing how to communicate sustainability on a product and customer level. The dimensions in the maturity model proved helpful in identifying the chal enges, and the steps needed to achieve higher levels of maturity. In line with a potential performance perspective, it was clear that companies need not aim for the highest maturity level in the model if it does not provide tangible business value: technological advancements should not be just for show, but drive real, measurable value. For example, there is a keen interest in utilizing data to secure and maintain green labels, underscoring the significance of sustainability credentials in business operations and in communicating their green initiatives. The case companies describe that currently only a few employees know how to extract reports and do data analyses, while a higher level of maturity on many dimensions requires data literacy and analytics capabilities throughout the organization. In line with this, our evaluation has revealed that higher levels of maturity require not only technical investments and expertise, but also organizational changes, strategic commitment, social and human engagement and ethical positioning. While the initial model does not outline strategic, ethical and social issues explicitly, these are impossible to exclude from consideration as they permeate each dimension, and our development of the model wil take this into consideration. 224 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION While the maturity model describes eight separate dimensions, there are significant interactions between the dimensions. For example, decisions regarding which sensors to instal , which data to measure and which KPIs to calculate affect al dimensions. In developing the guidance for practical application of the maturity model, this should be taken into consideration. Guidance should be developed also on how the model can be adapted to different situations, organizations and domains. Endnote Authors have contributed equally to conceptualization, methodology, validation and analysis. Authors are listed in alphabetical order, except for the corresponding author (Sel ). Acknowledgements The authors grateful y acknowledge the support from Business Finland and the companies participating in the Data-Driven Sustainability Management project. References Curran, M. A. (2013). Life cycle assessment: a review of the methodology and its application to sustainability. Current Opinion in Chemical Engineering, 2(3), 273-277. Donnellan, B., Sheridan, C., & Curry, E. (2011). A capability maturity framework for sustainable information and communication technology. IT professional, 13(1), 33-40. European Commission (2020). User Guide to the SME Definition. Publications Office of the European Union, Luxembourg. Ferrari, A. M., Volpi, L., Settembre-Blundo, D., & García-Muiña, F. E. (2021). Dynamic life cycle assessment (LCA) integrating life cycle inventory (LCI) and Enterprise resource planning (ERP) in an industry 4.0 environment. Journal of Cleaner Production, 286, 125314. Gökalp, M. O., Gökalp, E., Kayabay, K., Koçyiğit, A., & Eren, P. E. (2021). Data-driven manufacturing: An assessment model for data science maturity. Journal of Manufacturing Systems, 60, 527-546. Grossman, R. L. (2018). A framework for evaluating the analytic maturity of an organization. International Journal of Information Management, 38(1), 45-51. Gudfinnsson, K., Strand, M., & Berndtsson, M. (2015). Analyzing business intel igence maturity. Journal of Decision Systems, 24(1), 37-54. Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. Management Information Systems Quarterly, 28(1), 6. Lasrado, L.A., Vatrapu, R., Andersen, K.N. (2015). Maturity Models Development in IS Research: A Literature Review. Selected Papers of the IRIS, Issue Nr 6. Levasseur, A., Lesage, P., Margni, M., Deschenes, L., & Samson, R. (2010). Considering time in LCA: dynamic LCA and its application to global warming impact assessments. Environmental science & technology, 44(8), 3169-3174. Linke, B. S., Garcia, D. R., Kamath, A., & Garretson, I. C. (2019). Data-driven sustainability in manufacturing: selected examples. Procedia Manufacturing, 33, 602-609. A. Sel et al.: A Maturity Model for Evaluating Data-driven Sustainability Management 225 Markovic, S., Koporcic, N., Samara, G., & Barkemeyer, R. (2023). Sustainability and interactive network branding in fast-changing business environments. Business Ethics, the Environment and Responsibility, 32(3), 877-881. Mettler, T. (2009). A design science research perspective on maturity models in information systems. St. Gal en: Institute of Information Management, University of St. Gal en. Neri, A., Cagno, E., Lepri, M., & Trianni, A. (2021). A triple bottom line balanced set of key performance indicators to measure the sustainability performance of industrial supply chains. Sustainable Production and Consumption, 26, 648-691. Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of management information systems, 24(3), 45-77. Piehler, R., Schade, M. & Burmann, C. (2018) Employees as a second audience: the effect of external communication on internal brand management outcomes. J Brand Manag 26, 445–460 (2019). Poeppelbuss, J., Niehaves, B., Simons, A., & Becker, J. (2011). Maturity models in information systems research: literature search and analysis. Communications of the Association for Information Systems, 29(1), 27. Robertson, J. (2024, January 16). What’s the digital maturity of your internal communication? [Blogpost]. Retrieved from https://www.steptwo.com.au/papers/whats-the-digital-maturity- of-your-internal-communications/ Rödger, J. M., Beier, J., Schönemann, M., Schulze, C., Thiede, S., Bey, N., . . & Hauschild, M. Z. (2021). Combining life cycle assessment and manufacturing system simulation: evaluating dynamic impacts from renewable energy supply on product-specific environmental footprints. International Journal of Precision Engineering and Manufacturing-Green Technology, 8, 1007-1026. Sari, Y., Hidayatno, A., Suzianti, A., Hartono, M., & Susanto, H. (2021). A corporate sustainability maturity model for readiness assessment: a three-step development strategy. International Journal of Productivity and Performance Management, 70(5), 1162-1186. Sedej, T. & Mumel, D. (2015). Evaluation of information awareness and understanding through different internal communication tools. Handbook of Research on Organizational Transformations through Big Data Analytics (pp. 146-165). IGI Global. Vásquez, J., Aguirre, S., Puertas, E., Bruno, G., Priarone, P. C., & Settineri, L. (2021). A sustainability maturity model for micro, smal and medium-sized enterprises (MSMEs) based on a data analytics evaluation approach. Journal of Cleaner Production, 311, 127692. Van Groenendale, P. (2022). Analytics Maturity Model: How to get the most out of your data strategy. [Blogpost]. Retrieved from https://supermetrics.com/blog/analytics-maturity- model Wendler, R. (2012). The maturity of maturity model research: A systematic mapping study. Information and software technology, 54(12), 1317-1339. 226 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION IMPROVING USER'S CONFIDENCE TO ACT WHEN USING ADVICE ALGORITHMS THROUGH INTERACTIVE USE OF COUNTERFACTUALS JEROEN VAN GRONDELLE,1 MARLIES VAN STEENBERGEN,1 ALETTA SMITS,2 MARCEL STALENHOEF,2 KOEN VAN TURNHOUT,2 TIJS TIMMERMAN,3 HARALD POL4 1 HU University of Applied Sciences Utrecht, Digital Ethics, Utrecht, Netherland jeroen.vangrondel e@hu.nl, marlies.vansteenbergen@hu.nl 2 HU University of Applied Sciences Utrecht, Human Experience & Media Design, Utrecht, Netherland aletta.smits@hu.nl, marcel.stalenhoef@hu.nl, koen.vanturnhout@hu.nl 3 HU University of Applied Sciences Utrecht, Marketing & Customer Experience, Utrecht, Netherland tijs.timmerman@hu.nl. 4 Hanze University of Applied Sciences Groningen, Marketing and Entrepreneurship, Groningen, Netherland h.g.pol@pl.hanze.nl In this paper, we explore the design of web-based advice robots to enhance users' confidence in acting upon the provided advice. Drawing from research on algorithm acceptance and explainable AI, we hypothesise four design principles that may encourage interactivity and exploration, thus fostering users' confidence to act. Through a value-oriented prototype experiment and value- oriented semi-structured interviews, we tested these principles, confirming three of them and identifying an additional principle. Keywords: advice The four resulting principles: (1) put context questions and robot, resulting advice on one page and allow live, iterative exploration, confidence to (2) use action or change oriented questions to adjust the input act, interactive, parameters, (3) actively offer alternative scenarios based on design counterfactuals, and (4) show al options instead of only the science recommended one(s), appear to contribute to the values of research, value agency and trust. Our study integrates the Design Science sensitive Research approach with a Value Sensitive Design approach. design DOI https://doi.org/10.18690/um.fov.4.2024.12 ISBN 978-961-286-871-0 228 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction As part of their digital transformation, public organisations increasingly offer personalised information and advice on complex services digital y and through algorithms (Chandra et al., 2022). However, organisations note that, when offered such digital advice, many citizens hesitate to act upon the advice. Before choosing or applying for the service, citizens often look for confirmation through the human channels first. In an earlier study, we argue that this phenomenon of lacking ‘confidence to act’ should, specifically in a public service context, be seen as more than a user experience (UX) or online conversion problem (Van Grondel e et al., 2023). We describe confidence to act as the extent to which a recipient feels that the service or advice offered is a solid basis to act upon, trusting both their own part and that of the service provider in the interaction. Confidence to act, or lack thereof, ties in with larger values, such as trust, agency and accessibility. The literature on acceptance of algorithmic outcomes suggests that being in control and able to override the algorithm (Dietvorst et al., 2018) and being part of a contrastive, interactive dialog rather than one-way communication (Miller, 2019; Smits & Van Turnhout, 2023) help to accept the outcomes. The research question we address in this paper is: How can a web-based advice robot be designed to instill confidence to act by supporting interactive exploration? To answer this question, we use concepts and insights from the field of explainable AI (XAI) and human-AI-interaction, and specifically existing knowledge on the role of counterfactuals in understanding cause-effect underneath algorithmic decisions. Based on existing theory, a Design Science Research (DSR) approach is combined with Value Sensitive Design (VSD) techniques to create a design artefact, evaluate it in a concrete organisational context, yet also offer a generalisation path towards similar problem contexts. Combining the VSD techniques ‘value-oriented prototype’ and ‘value-oriented semi-structured interview’ (Friedman & Hendry, 2019), we aim to involve people that may be less comfortable or even unable to contribute in a more conceptual, cognitive interview approach, while addressing the breadth and conceptual nature of the confidence to act construct. In the next section we discuss the theoretical background to our study, which is primarily rooted in explainable AI. Section 3 discusses how we employed a combination of Design Science Research and Value Sensitive Design as our research method. The findings of our study are presented in section 4, fol owed by discussion and conclusion in section 5. J. van Grondel e et al.: Improving User's Confidence to Act when Using Advice Algorithms Through Interactive Use of Counterfactuals 229 2 Theoretical background The degree to which people use available digital tools has been widely studied by the research field of technology adoption (Ajzen, 1991; Venkatesh et al., 2003; Rogers, 2003). In this study, however, we do not focus on the initial intention to use advice tools, but the intention, after using the tool, to follow up on the advice provided. This ‘confidence to act’ emerged in a previous study as a characteristic of what citizens regard personal public service (Van Grondelle et al., 2023), but is as yet underconceptualized in literature. It seems to combine elements of trust in the advice provided, and self-confidence of having provided the right information to the organisation and of having understood the advice correctly. Users being hesitant to adopt the outcomes of algorithms is a documented issue. A number of factors contribute to this. Logoni et al. (2019) introduced the construct of uniqueness neglect to capture the phenomenon that users expect algorithms to be less able to address their unique, individual situations. Castelo et al. (2019) found that the perception of high subjectivity in the task diminishes the level of trust in algorithms. Also, users may expect algorithms to “just” maximize a one-dimensional utility function and therefore trust algorithms less in ethically complex domains (Dietvorst & Bartels, 2022). Similarly, users are less likely to trust algorithms in uncertain domains where the task includes an element of prediction (Dietvorst & Bharti, 2020). There appears to be a level of debate to the question whether algorithm aversion is non-compensatory or can be compensated by other factors, such as objectively delivering better results and accuracy (Pezzo & Beckstead, 2020). Also, allowing users to exert some control over the algorithm, al owing them to (slightly) override or course-correct its decisions helps overcome algorithm aversion to an extent (Dietvorst et al., 2018; Hekman et al, 2022; Kleemann & Ziegler, 2023; Tintarev & Masthoff, 2015). In the context of XAI, much recent work focuses on explaining complex decisions taken by algorithms. Miller (2019) draws from social sciences when he indicates 4 criteria for successful explanation as it occurs in a human-to-human context: it is contrastive, in the sense that it takes into account and addresses the expectations of the explainee; it is social, in that it is part of an interactive dialog aiming to transfer knowledge between two parties; it is often selective, in the sense that it does not provide a complete list of factors, but instead presents a subset of relevant and 230 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION sufficient causes for the decision that is explained; and it should not try to convey underlying probabilistic relations, as they are often too subtle to be comprehended by many people. In addressing these objectives, different techniques are used to identify the attribution of different factors to a decision, identify the most decisive factors and use those as the basis for explanation. Determining the relative importance of factors in physical and economical models has often been done using variance-based sensitivity analysis. Techniques like Sobol (2001) and its successors introduce random disturbances in the inputs and quantify their impact on the result. Other techniques are based on the work of Shapley (1953) in economics on quantifying the contribution of a team member to the performance of the team. It has been applied for instance in explaining medical diagnostics (Ibrahim et al., 2020) and in financial forecasting (Jabeur et al., 2021). In machine learning context, often the gradients of underlying models are used to quantify the rate of change in the outcome in relation to the different inputs. In addition to identifying decisive factors, many of these techniques are also used to convey a measure of confidence: to which extent for instance a weather prediction can be expected to hold under slightly different conditions (Parker, 2010). A group of explanation techniques that heavily focus on conveying cause-effect relations underneath an algorithm-derived prediction or decision, is counterfactuals. Counterfactuals are scenarios where one or more of the inputs are different, and that (given the underlying model that is to be explained) have a different outcome. Although they have their background in philosophical logic concerning causality, more recently they have been studied and applied in the context of XAI, conveying underlying causality as part of explanations. Various criteria have been found for the successful application of counterfactuals in explaining decisions. Successful counterfactuals have proximity to the actual situation of the explainee, presenting smaller rather than larger deviations while still reaching a different outcome (Verma et al., 2019). Related, successful counterfactuals are sparse in the number of factors they change to reach a different outcome ( ibid. ). And successful counterfactuals are found to be actionable, in the sense that they offer the explainee a realistic course of action if an undesirable decision is explained (Poyiadzi et al., 2020). J. van Grondel e et al.: Improving User's Confidence to Act when Using Advice Algorithms Through Interactive Use of Counterfactuals 231 3 Research Method We applied a DSR approach (Hevner et al., 2004; Peffers et al., 2007) combined with VSD techniques (Friedman et al., 2006) to develop an artefact in the form of UX design principles that address the lack of confidence to act. We conducted the following steps: (1) Formulate design principles based on literature, (2) Develop a prototype based on the design principles, distinguishing four scenarios, (3) Conduct think-aloud sessions followed by interviews, looking for underlying values, (4) Analyse the sessions using a combination of open coding and template coding, and (5) Refine the design principles. From the literature as described in the theoretical background, we derived four basic principles. Next, we operationalised these principles in a web-based advice robot prototype that gives advice on public transport subscriptions for travelling in a large city in the Netherlands. To better distinguish the individual effects of the principles, we developed various versions of the prototype. Over two days we conducted 11 think-aloud sessions with current clients of the public transport organisation broadly ranging in situation and education. We opted for think-aloud sessions followed by interviews because that al owed us to explore the underlying values and beliefs motivating the participants’ actions. The participants were recruited from the transport organisation’s client panel. In this way we achieved a near-real life setting. Twelve persons were invited, but one person did not turn up. This number was primarily motivated by practical reasons. Each session lasted 60 minutes, starting with a brief introduction in which the setting was explained, including the independent role of the researcher, after which the participant was asked to find the best public transport subscription for three different cases. The cases were brief and written on cards. An example is: “You live on Mozartlaan in Rhoon and travel 4 days a week to the Melanchtonweg, for work. You are considering working one day less soon, but that is not yet certain. Find the most suitable product for you”. Each case referred to a different version of the prototype. Each participant was presented with three out of the four versions. The versions were presented in varying combinations and orders. Each case was fol owed by a brief semi-structured interview, following an interview guide. In the think-aloud exercises and interviews we applied the VSD techniques value-oriented prototype and value-oriented semi- structured interview (Friedman & Hendry, 2019), focusing not on the UX factors themselves, but on the effect they have on the participants in terms of human values. 232 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION We did so mainly by asking open ‘why’-questions. The sessions were conducted by two researchers, taking turns. The sessions were recorded on video. The recordings were transcribed and coded using Atlas.ti. First open coding was applied. The codes were then grouped into groups representing the design principles. In addition, a group was made for codes pertaining to confidence to act. Based on this an analysis was performed on the relation between each of the design principles and the participants’ confidence to act on the advice provided. 4 Results 4.1. Design principles From the literature we derived four design principles for interactive advice algorithms (table 1). Our first design principle addresses the lack of influence on the algorithm (Dietvorst et al, 2018) and lack of interactivity (Miller, 2019). Many current online advice modules apply a two-page approach: on the first page, al relevant context factors are entered through a form. On submittance, the algorithm computes an advice based on those factors and the second page presents the outcomes. Often, a back button is offered to return to the first page, to modify the factors if mistakes were made. Our first design principle proposes to remove the separation between input and output by combining them on one page. This al ows changes to the inputs to be made more easily, encouraging users to explore the algorithm by trying out alternative scenarios. Our second design principle is to formulate change-oriented questions to modify what essential y are “static” properties, instead of asking for the current situation. Often, questions in advice algorithms focus on the current situation, and users may be hesitant to change those values to fictitious ones, especial y in the context of government websites. To address this hesitation, questions can be worded in terms of potential future changes. For instance, the context factor “number of days per week you work” can be modified using a change-oriented question “do you anticipate your number of workdays to change?”. Design principle 3 uses sensitivity analysis to analyse the boundaries of the current advice. It distinguishes factors that require little change to impact the advice from factors that require a large change or that do not impact the advice at all. Visualising the boundaries of the input parameters in relation to the current advice guides users in effective scenario J. van Grondel e et al.: Improving User's Confidence to Act when Using Advice Algorithms Through Interactive Use of Counterfactuals 233 exploration. Design principle 4 uses counterfactual techniques to actively present alternative scenarios to the user, combining a potential change in input values and the new advice this change would generate. Table 1: Design Principles for Confidence to Act Design principle Description Example The questions about the 1. Put context A change in the input current context are on the parameters on the left-hand questions and resulting same page as the advice, side of the page immediately advice on one page and al owing for interactive leads to an update of the list of allow live, iterative recommended subscriptions exploration of the effects of exploration. on the right-hand side of the different inputs. page. Context can be asked “Do you consider working initially, after which the more or fewer days?” 2. Use action or change questions are formulated 3 days less >> 3 days more oriented questions to “incrementally” or as a delta (instead of having to correct change the context to the base context. the number of days you actual y work) 3. Guide exploration of Options that lead to a If the current advice is valid context factors based changed advice are regardless of the number of on advice boundaries highlighted. days you work, that change OR options that lead to the control is grayed out. Based on sensitivity same advice are disabled. analysis 4. Guide exploration of A counterfactual algorithm context factors based “If you go to your workplace runs in the background and on alternative scenarios one extra day each week, a 3- offers advice: If your star subscription would be situation was X, our advice Based on cheaper.” would be Y. counterfactuals Our second design principle is to formulate change-oriented questions to modify what essential y are “static” properties, instead of asking for the current situation. Often, questions in advice algorithms focus on the current situation, and users may 234 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION be hesitant to change those values to fictitious ones, especialy in the context of government websites. To address this hesitation, questions can be worded in terms of potential future changes. For instance, the context factor “number of days per week you work” can be modified using a change-oriented question “do you anticipate your number of workdays to change?”. Design principle 3 uses sensitivity analysis to analyse the boundaries of the current advice. It distinguishes factors that require little change to impact the advice from factors that require a large change or that do not impact the advice at all. Visualising the boundaries of the input parameters in relation to the current advice guides users in effective scenario exploration. Design principle 4 uses counterfactual techniques to actively present alternative scenarios to the user, combining a potential change in input values and the new advice this change would generate. 4.2 Operationalisation To test the design principles, we operationalised them in a prototype of a web-based advice robot for public transport subscriptions in a large city in the Netherlands. Although selecting a public transport subscription might seem a relatively simple task, in this particular case the complexity was considerable. The range of subscription models includes a geographical zoning system, pay-as-you-go components and discounts above a certain amount of travel. When combined, this leads to some unexpected, non-linear effects in total transportation cost, making the choice non-trivial for customers. Due to the zoning system, smal changes of the origin or destination of the commute could have a high impact on the resulting total cost. The existing web application fol owed the typical scenario of a page with questions and a consecutive page with advice. The questions explored the need for public transport, on workdays and for leisure/other trips. The advice recommended a subscription, based on optimal total cost per month. Although great care was given to offer optimal advice, many online users still contacted the service centers on the stations for confirmation. To implement principle 2 in the prototype, users are not asked to change source or destination address, but how many minutes they are wil ing to walk to and from the station. Similarly, they are asked whether they might decide to work more or less days in future. In the algorithm, this is used to shorten or lengthen the trip and/or change the frequency, and thus (depending on zone limits) increase or lower the J. van Grondel e et al.: Improving User's Confidence to Act when Using Advice Algorithms Through Interactive Use of Counterfactuals 235 subscription cost. Changing the answers to these questions leads to immediate updating of the advice portion of the screen (principle 1). To establish the advice boundaries, a naive, per-factor sampling of adjacent inputs is generated, and the outcome of the algorithm is compared to the outcome of the actual inputs. Based on this the input form shows, with a blue bar, which changes will not impact the outcome and are therefore less useful to explore (principle 3). Based on the same simulation a chatbot suggests input changes that are close to the current context yet lead to a different advice (principle 4). To be able to study the contribution of the four principles separately, four versions of the prototype were developed. In all versions input was provided by using sliders. The baseline version 0 reflected the current state, with two separate screens. All other versions offered questions and advice on a single page (principles 1 and 2). Version 2 additionally visualised with blue bars the advice boundaries in the input sliders (implementing principle 3). In version 3 a chatbot offered advice on how changes in input would lead to an alternative advice (principle 4). Figure 1 illustrates how all four principles were implemented in the versions of the prototype. Figure 1: The prototype, with al 4 principles implemented Source: Own printscreen 236 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.3 Evaluation Below, we discuss how the participants (referred to as P#) experienced the advice robot in relation to the design principles and their confidence to act. Al but one of the participants used version 0 and 1, while six participants used version 2 and five participants used version 3. One participant only used version 1 and 2. Input and output on one page Most participants appreciated the fact that their input and the robot’s advice appeared on the same page. In version 0 participants do use the back button, but it is an obstacle. Participant P1 understands the purpose of the back button and remarks that normal y they would make notes and then go back to change the parameters. When confronted with version 1 later, P1 appreciates the ease of use of the fact that changing the sliders back and forth immediately results in changes in advice on the same screen. Various participants in version 1 immediately focus on the initial list of subscriptions and reason about the possible effects of changing parameters instead of changing them in the prototype to see the effects. The back button is mainly used for confirmation of their own reasoning. P5, for instance, only starts to actively use the parameters in the one-page versions, subsequently expressing great enthusiasm for ‘playing with the sliders’. Playing with the sliders and seeing the immediate effects makes some participants reflect that there are more options than they had realized, which stimulates them to have a closer look at what they want and what subscription fits that need. Apart from just entering the parameters as they currently are, they can make conscious choices, such as choosing to walk farther, that may affect the price of their transportation costs. Some express great curiosity as to the effects of changing particular parameters. P2, for instance, freely plays with the sliders and uses the sliders to detect what parameters lead to a particular subscription, i.e. they reason from subscription to parameters instead of the other way round. The fact that al options are shown instead of only the most suitable is appreciated by al participants. The fact that they can see the differences between options for themselves instil s confidence. It disperses many doubts about whether the proposed subscription is real y the best for them. J. van Grondel e et al.: Improving User's Confidence to Act when Using Advice Algorithms Through Interactive Use of Counterfactuals 237 Change oriented questions Though only one participant explicitly remarked on the fact that they liked the fact that they could indicate the number of minutes they were prepared to walk instead of having to select a station, the wording of the sliders seemed to encourage the participants to better think about their options. Especial y the walking distance engendered deliberation. Not only about the time participants are prepared to walk, but it is also transferred to the option of taking the bicycle to the station. The precise wording of the sliders is extremely important though, as some participants had trouble interpreting some of them. Especial y the formulation ‘any other trips?’ generated some doubts about its meaning. Participants often used a mixed approach of reasoning about the list of subscriptions using their own previous experiences with public transport as wel as experimenting with the input parameters. They use the list to compare the subscriptions mentioned, often using their own experiences and knowledge to reflect on the differences and suitability in different circumstances in addition to playing with the sliders. Some use the parameters to generate new outcomes, others use them primarily for confirmation of their own reasonings. The balance between reasoning and exploring differs between participants, ranging from accepting the top advice without further ado to extensively exploring al parameters and their effects on the list of options. Passive presentation of advice boundaries Even though a legend was given at the bottom of the page, the showing of advice boundaries in the form of a blue bar was noticed by none of the participants except P2, who did not understand what it was for and found it irritating. All participants had to be alerted to the blue bar and even after explanation it was not perceived as being helpful. On the contrary, if anything, it was experienced as confusing. P1 remarks that it is yet another aspect to consider, without adding much value. It makes a clear interface more complicated. Only P3 thought that marking the boundaries of the advice might be useful. 238 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Active ofering of alternative scenarios The bot messages indicating that a particular change in parameters would lead to a cheaper subscription was better appreciated than the boundary markers in the sliders. P4 and P5 both spontaneously remarked on missing the messages when they moved from version 3 to version 1. P5 appreciates the bot messages especial y because of their open phrasing. This is also connected to the format of the bot: it is not a pop-up that must be clicked away but a smal panel integrated in the page and the wording is neutral, not trying to direct the user in a particular direction. Some participants, though, find the messages superfluous, confusing, or not applicable to their situation, and thus not actionable. Confidence to act Asked whether they would be confident to take a subscription based on the advice robot, most participants answered confirmatively. Various participants expressed that they feel confident to act upon the advice because they had been able to explore and check the various parameters by themselves. P2 appreciates having experienced for themselves that changing certain parameters doesn’t make a difference as to the best subscription for them. Also, the fact that al options are shown seems to instil confidence. As P5 remarks, the fact that the options are shown in a simple list, instead of one option being shown ‘ful screen with a few other options tucked away in a corner’ makes it more trustworthy for them. The fact that the option of not taking a subscription but buying individual tickets is included instills confidence, too. As do the tips provided by the bot in version 3, because the bot indicates what is needed to arrive at a cheaper subscription, instead of trying to promote a particular choice. The most frequent reason for participants hesitating to take a subscription is the wish for further information on the exact subscription conditions, which were not included in the prototype. Various ways of acquiring this additional information are suggested, ranging from an information button to speaking to a person via chat or telephone. J. van Grondel e et al.: Improving User's Confidence to Act when Using Advice Algorithms Through Interactive Use of Counterfactuals 239 5 Discussion and Conclusion The experiments provide some support for the usefulness of three of the four design principles to instil confidence to act. Factors that seem to increase confidence to act are: being able to explore the effects of changing parameters (enabled by design principles 1 and 2), being presented with al options, not only the recommended one, in a neutral manner (enabled by principle 1), and being actively informed about how changes in input might affect the advice (principle 4). Despite being grounded in the literature on sensitivity analysis in XAI, no support is found for design principle 3. This may be due to the way it is implemented in the prototype. This requires further study. Showing al options in a neutral manner is enabled by principle 1, but its importance may warrant a separate design principle. The most important factor that inhibits confidence to act seems to be the lack of detailed information on what the recommended subscription entails. Most participants seem not to doubt that the recommended subscription is indeed the best option for their situation, but need more information about what exactly it al ows or not. This gap seems easily mended by providing an information button for each of the subscriptions. Combining the VSD techniques ‘value-oriented prototype’ and ‘value-oriented semi- structured interview’ allowed us to investigate the underlying values that influence confidence to act with a wide range of citizens. Using the prototype al owed for a diverse range of participants, selected from the actual clientele of the public transport system. By asking open questions about experiences, intentions, and motivations we were able to elicit impact on underlying values, thus moving beyond a merely functional UX study. The values that emerged most prominently were agency and trust. Agency was positively affected by design principle 1 and 2: being stimulated to play with input parameters while immediately seeing changes in recommendations gave a feeling of being in control of the selection process. Trust was positively affected by design principle 1 and 4: being presented with all options and being actively informed of how to arrive at cheaper options helped prevent mistrust about being manipulated to a particular unfavorable outcome. As may be expected, the wording of the parameters and information messages proves to be extremely important. 240 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION In this study we aimed to answer the research question ‘ How can a web-based advice robot be designed to instil confidence to act by supporting interactive exploration?’. By deriving four design principles from literature followed by empirical validation, we arrived at three confirmed design principles: (1) put context questions and resulting advice on one page and allow live, iterative exploration, (2) use action or change oriented questions to adjust the input parameters, (3) actively offer alternative scenarios based on counterfactuals. In addition, from the experiments we derived a fourth principle: (4) show al options instead of only the recommended one(s). This study has its limitations. The design principles were operationalized in a single use case, with a limited number of participants. Applying the design patterns to a different advice algorithm could solidify the current qualitative outcomes. Finally, this could give rise to a quantitative study where the impact on confidence to act is measured in terms of the number of people that no longer seek human confirmation after receiving digital advice. We hope that our study contributes to the growing body of knowledge on the design and implementation of digital services that are of true service to citizens. Acknowledgements We would like to thank all citizens participating in our experiments. This research is co-financed by the Taskforce for Applied Research SIA, part of the Netherlands Organization for Scientific Research (NWO). References Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-t Castelo, N., Bos, M. W., & Lehmann, D. R. (2019). Task-dependent algorithm aversion. Journal of Marketing Research, 56(5), 809-825. Chandra, S., Verma, S., Lim, W. M., Kumar, S., Donthu, N. (2022). Personalization in personalized marketing: Trends and ways forward. Psychology & Marketing, 39(8), 1529-1562. Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management science, 64(3), 1155-1170. Dietvorst, B. J., & Bharti, S. (2020). People reject algorithms in uncertain decision domains because they have diminishing sensitivity to forecasting error. Psychological science, 31(10), 1302-1314. Dietvorst, B. J., & Bartels, D. M. (2022). Consumers object to algorithms making moral y relevant tradeoffs because of algorithms’ consequentialist decision strategies. Journal of Consumer Psychology, 32(3), 406-424. J. van Grondel e et al.: Improving User's Confidence to Act when Using Advice Algorithms Through Interactive Use of Counterfactuals 241 Friedman, B., Kahn, P. H., & Borning, A. (2006). Value sensitive design and information systems. In P. Zhang & D. Gal etta (Eds.), Human-computer interaction in management information systems: Foundations (pp. 348–372). M.E. Sharpe. Friedman, B., Hendry, D. G. (2019). Value Sensitive Design: Shaping Technology with Moral Imagination. The MIT Press. Hekman, E., Nguyen, D., Stalenhoef, M. & Van Turnhout, K. (2022). Towards a Pattern Library for Algorithmic Affordances. Joint Proceedings of the IUI 2022 Workshops (pp. 24-33). Hevner, A., March, S., Park, J., Ram, S. (2004). Design Science in Information Systems Research. MIS Quarterly 28, pp. 75—105 Ibrahim, L., Mesinovic, M., Yang, K. W., & Eid, M. A. (2020). Explainable prediction of acute myocardial infarction using machine learning and shapley values. Ieee Access, 8, 210410-210417. Jabeur, S. B., Mefteh-Wali, S., & Viviani, J. L. (2021). Forecasting gold price with the XGBoost algorithm and SHAP interaction values. Annals of Operations Research, (pp. 1-21). Kleemann, T. & Ziegler, J. (2023). Blending Conversational Product Advisors and Faceted Filtering in a Graph-Based Approach. In: Abdelnour Nocera, J., Kristín Lárusdóttir, M., Petrie, H., Piccinno, A., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2023. INTERACT 2023. Lecture Notes in Computer Science, vol 14144. Springer, Cham. https://doi.org/10.1007/978-3-031-42286-7_8 Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020, November). Problems with Shapley-value-based explanations as feature importance measures. In International Conference on Machine Learning (pp. 5491-5500). PMLR. Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019). Resistance to medical artificial intelligence. Journal of Consumer Research, 46(4), 629-650. Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267, 1-38. Parker, W. S. (2010). Predicting weather and climate: Uncertainty, ensembles and probability. Studies in history and philosophy of science part B: Studies in History and Philosophy of Modern Physics, 41(3), 263-272. Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of management information systems, 24(3), 45-77. Pezzo, M. V., & Beckstead, J. W. (2020). Algorithm aversion is too often presented as though it were non-compensatory: A reply to Longoni et al. (2020). Judgment and Decision Making, 15(3), 449-451. Poyiadzi, R., Sokol, K., Santos-Rodriguez, R., De Bie, T., & Flach, P. (2020, February). FACE: feasible and actionable counterfactual explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 344-350). Rogers, E.M. (2003). Diffusion of Innovations (5th ed.). New York: Free Press. Smits, A. & Van Turnhout, K. (2023). T owards a Practice-Led Research Agenda for User Interface Design of Recommender Systems. In: Abdelnour Nocera, J., Kristín Lárusdóttir, M., Petrie, H., Piccinno, A., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2023. INTERACT 2023. Lecture Notes in Computer Science, vol 14144. Springer, Cham. https://doi.org/10.1007/978-3-031-42286-7_10. Tintarev, N. & Masthoff, J. (2015). Explaining Recommendations: Design and Evaluation. In: Ricci, F., Rokach, L., Shapira, B. (eds) Recommender Systems Handbook. Springer, Boston, MA. https://doi.org/10.1007/978-1-4899-7637-6_10. Shapley, L. S. (1953). A value for n-person games. Sobol, I. M. (2001). Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and computers in simulation, 55(1-3), 271-280. 242 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION van Grondele, J., van Steenbergen, M., Smits, A., Timmerman, T., van Turnhout, K. & Pol, H. (2023). The Anatomy of a Personal Service: Eight dimensions of ‘Personal’. Proceedings of the 36th Bled eConference, 483–495. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. doi 10.2307/30036540 Verma, S., Boonsanong, V., Hoang, M., Hines, K. E., Dickerson, J. P., & Shah, C. (2020). Counterfactual explanations and algorithmic recourses for machine learning: A review. arXiv preprint arXiv:2010.10596. EMPOWERING DATA SOVEREIGNTY: STRATEGIES OF DATA INTERMEDIARIES IN DATA ECOSYSTEMS RUBEN D’HAUWERS, AUGUST BOURGEUS Imec-SMIT, VUB, Brussel, Belgium ruben.dhauwers@vub.be, august.bourgeus@vub.be This study delves into the intricacies of data control mechanisms within (personal) data ecosystems, with the goal of attaining a harmonious balance in data sovereignty between individuals and data providers, facilitated by data intermediaries. Employing a multiple-case study analysis involving eleven data intermediaries, the research utilizes axial coding and triangulation with existing literature to identify dimensions of the novel Taxonomy of Data Control Mechanisms for data intermediaries. These dimensions encompass three meta-dimensions: data access control, power Keywords: dynamics, and revenue sharing models. These meta-dimensions data consist of eight dimensions that can be harmonized to achieve sovereignty, data equilibrium in data sovereignty between data subjects and intermediary providers. This research contributes to both theoretical data comprehension and practical implementation in navigating the control data complexities of data sovereignty within dynamic data ecosystem ecosystems. taxonomy DOI https://doi.org/10.18690/um.fov.4.2024.13 ISBN 978-961-286-871-0 244 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction In today's data-driven world, the dominance of big tech companies in the data economy has raised concerns about the control and influence they exert over smal er actors (Zuboff, 2015). The concept of data ecosystems has emerged as a promising avenue for fostering collaboration within networks centered around the use of data (S. Oliveira et al., 2019). Particularly within personal data ecosystems, the focus shifts to individuals and their personal data, forming the cornerstone of these ecosystems (Moiso & Minerva, 2012). Moreover, data intermediaries have gained traction, overseeing data governance between data providers and data users (Janssen & Singh, 2022). To ensure the sustainability of personal data ecosystems, a balance in data sovereignty for both data subjects (Hummel et al., 2021) and data providers (Zrenner et al., 2019) is required. This can be facilitated by data intermediaries through access control and data governance mechanisms (Curry, 2020; Gelhaar et al., 2021), empowering data providers and subjects to dictate terms for data usage by data consumers. This paper addresses the research question: "Through which dimensions of data control mechanisms can data intermediaries influence the equilibrium of data sovereignty between individuals and data providers within personal data ecosystems?" A taxonomy that identifies data control mechanisms within data intermediaries to harmonize data sovereignty for both data providers and subjects is developed, encompassing various data intermediary types. To address this question, a multiple-case study analysis approach (Yin, 2013) is employed, involving eleven data intermediaries. The taxonomy (Nickerson et al., 2013) is developed through axial coding of case studies (Corbin & Strauss, 2008), and triangulated with existing literature (Patton, 1999). The paper begins with a review of literature on data ecosystems, data sovereignty, data intermediaries, and data governance taxonomies. It then outlines the methodology, presents results, discusses implications for academia and real-world applications, and concludes with key findings and future research directions. R. D’Hauwers, A. Bourgeus: Empowering Data Sovereignty: Strategies of Data Intermediaries in Data Ecosystems 245 2 Literature 2.1 (Personal) Data Ecosystems Various trends in big data, artificial intelligence, and the Internet of Things have increasingly drawn attention to data ecosystems focusing on data usage (Curry & Sheth, 2018). Within these trends, personal data is often used as the asset traded in data ecosystems (Spiekermann et al., 2015). In the context of business, ecosystems are networks of interacting organizations devoid of hierarchical management, instead bound together by their shared investments, facilitating coordination that eliminate the necessity for individual contractual agreements with each partner (Jacobides et al., 2018). Data ecosystems are business ecosystems (Adner, 2017) that aim to create a focal value proposition with the exchange of data at the center. Data ecosystems are characterized by their connected network structure, the presence of platforms facilitating value creation, and collaborative co-evolution among actors (Schreider 2023, Oliveira et al, 2019). In personal data ecosystems, individuals are integral in exchanging personal data (Ojasalo & Miskeljin, 2020; Spiekermann et al., 2015). A "user-centric" model is advocated, empowering individuals to control the gathering, management, use, and sharing of their data (Moiso & Minerva, 2012). The integration of humans in data ecosystems entails providing them with adequate information and power over their data while maintaining transparency, honesty, and security (Koskinen et al., 2023).In this context, stakeholders in the data ecosystem include data providers, who make data they control available, and data consumers, who receive this data (Otto & Teuscher, 2019). A data subject refers to an individual identifiable through personal data (Scheider et al., 2023). In personal data ecosystems, a data provider could be a private company controlling data or a data subject providing data. This research distinguishes between private data providers and data subjects seeking control over personal data. 2.2 Balancing Data Sovereignty in Data Ecosystems Big tech companies currently dominate and control the data economy, sidelining smaller actors affecting both individuals and organizations (Knaapi-Junnila et al., 2022; Koskinen et al., 2023). This power dynamic, termed data colonialism (Couldry & Mejias, 2019) or surveillance capitalism (Zuboff, 2015), involves exploiting personal data for profit and impacts less powerful actors by diminishing their 246 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION autonomy and control over their data. In this context, the concept of data sovereignty becomes crucial, defining entities' self-determination over their data, encompassing both individuals and businesses (Otto & Teuscher, 2019; Scheider et al., 2023).Regarding data sovereignty of the data subject, different authors advocate the need for individuals to have control over their personal data usage, including determining access and processing purposes, with clarity on data privacy and protection (Hummel et al., 2021). Also self-determined sharing and (monetary) incentivization for the sharing of personal data is required (Lauf et al., 2022). For businesses, enabling data sovereignty of the data subject may contradict with the data sovereignty of the company, as it involves granting data control to data subjects. As companies often perceive (personal) data as an asset crucial for enhancing competitiveness (Gupta & George, 2016), it leads them to prioritize its protection. As granting data control to data subjects potential y compromises their competitive advantage, they may be reluctant to share business-critical data (Tomi Dahlberg & Nokkala, 2017). To foster sustainable personal data ecosystems, a need arises to balance data sovereignty for both data subjects and data providers. This can be enabled through mechanisms like access control and data control (Zrenner et al., 2019), allowing data providers and data subjects to set terms for data usage by data consumers (Loebbecke et al., 2016; Scheider et al., 2023). This research focuses on identifying data - and access control mechanisms, enabling the balance of data sovereignty for the data subjects and data providers. 2.3 Data Intermediary Models Data intermediaries, a new model introduced in the Data Governance Act in Europe Act (Regulation (EU) 2022/868, 2022), act as mediators between data providers and users, governing the data and providing confidence in its usage (Janssen & Singh, 2022). Some of these intermediaries facilitate data sovereignty for both data subjects and providers. Firstly, in enabling data sovereignty for data providers, we consider data sharing pools and data marketplaces. Data-sharing pools (DSPs) (Micheli et al., 2023) are al iances among data providers that share data intending to improve their assets (data products, processes and services) by exploiting the complementarities of the pooled R. D’Hauwers, A. Bourgeus: Empowering Data Sovereignty: Strategies of Data Intermediaries in Data Ecosystems 247 data. The aliances have a shared purpose, context or application, and are intended to benefit al their participants. Data marketplaces (DM) (Janssen & Singh, 2022; Micheli et al., 2023) serve as platforms facilitating data exchange between buyers and sellers, allowing data sellers to monetize their data while retaining control over its usage and access. Secondly, various models facilitate data sovereignty for data subjects. In data cooperatives (DC) (Hartman et al., 2020; Micheli et al., 2023), the collective ownership of data is emphasized. They recognize data subjects as vital stakeholders and aim to rectify power imbalances and enable equitable benefit-sharing. Personal Information Management Systems (PIMS) (Micheli et al., 2023; Van Kleek & OHara, 2014) stress individuals' control over their data, countering private companies' influence and empowering users to determine their personal information's usage, fostering a balanced relationship with digital platforms. Additionally, data unions (Micheli et al., 2023) advocate for individuals or groups in the data economy by pooling their data and negotiating fair terms with data buyers, enhancing individuals' bargaining power (Micheli et al., 2023) (Micheli, 2023). Data trusts are based on trust law, which al ows data rights holders to delegate control of their data to a trustee (Micheli et al., 2023) based on a legal mechanism that permits the rights of data subjects/holders to be pooled to negotiate terms of use in data subjects’ favor (Sadowski et al., 2021). 2.4 Research Gap The lack of research on data governance in inter-organizational data exchange is evident (Abraham et al., 2019), which similarly applies to data governance within ecosystems. Typologies and taxonomies (Gelhaar et al., 2021; Lis & Otto, 2020) have been developed to address this gap, focusing on data governance within ecosystems, yet research should expand beyond organizational boundaries to encompass personal ecosystem models (Koskinen et al., 2023). While various studies have addressed data sovereignty for data subjects (Scheider et al., 2023) and data providers (Zrenner et al., 2019), there needs to be more research on achieving balance between them within the context of data intermediaries.Thus, the research question for this study is: "Through which dimensions of data control mechanisms can data intermediaries influence the equilibrium of data sovereignty between individuals and data providers within personal data ecosystems?". The focus is on a taxonomy 248 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION identifying data control mechanisms data intermediaries can use to balance data sovereignty. 3 Methdology Following a taxonomy development methodology (Nickerson et al., 2013), the researchers progressed through two iterations to discern the taxonomy's dimensions. An empirical-to-conceptual (E2C) was performed in the first iteration, deducing dimensions and characteristics from empirical studies (Nickerson et al., 2013). This involved analyzing real-life use cases of existing data intermediaries using a case study analysis method, which al ows for examination within authentic contexts (Yin, 2013). The case study methodology is suitable for explanatory and descriptive purposes (Runeson & Höst, 2009), particularly in industry-based scenarios (Verner et al., 2009). Eleven data intermediaries were selected based on the definition: "data intermediaries act as mediators between data providers and users, governing the data and providing confidence in its usage" (Janssen & Singh, 2022). Data intermediaries for this study were chosen using a snowbal ing approach (Berg, 2006), initially identified through web searches, and reviewing reports in academic and industry literature. Cases encompassing PIMS, data unions, DC, DMs, trusts, and DSPs were analyzed, focusing on their objectives regarding data sovereignty for either data subjects or data providers. This iterative process ensured diverse representation across models, functions, and industries for comprehensive analysis. Refer to Annex 1 for an overview of selected use cases. Qualitative data analysis was conducted using MAXQDA, examining textual descriptions on the intermediaries' websites and whitepapers. Axial coding, combining inductive and deductive thinking (Corbin & Strauss, 2008) was applied iteratively. As different use cases were analyzed, meta- dimensions and dimensions were refined accordingly. In the second iteration, oriented at the conceptual-to-empirical approach (C2E), methodological triangulation (Denzin, 1978; Patton, 1999) was performed to scrutinize the results to increase the credibility of the findings (Patton, 1999). Following the second iteration, all objective and subjective ending conditions were met (Nickerson et al., 2013). R. D’Hauwers, A. Bourgeus: Empowering Data Sovereignty: Strategies of Data Intermediaries in Data Ecosystems 249 4 Taxonomy of Data Control Mechanisms for Data Intermediaries Table 1 presents a taxonomy of Data Control Mechanisms for Data intermediaries, identifying modifiable dimensions that shape control and data sovereignty trade-offs among ecosystem actors. Table 1: Taxonomy of Data Control Mechanisms for Data Intermediaries 4.1 Data Access Control Data access and usage controls ensure data sovereignty, al owing data subjects and providers to effectively regulate access and usage rights (Bussard et al., 2010; Kelbert & Pretschner, 2012; Zrenner et al., 2019). Actor Access Control determines which actor has the authority to grant access to data, which can be wielded by data subjects or data providers. In PIMS, data subjects can control their data, especial y in ecosystems with sensitive personal data, like health information (WeAre), or primarily user-generated data (Karamel). Actors' access control can also be managed by the intermediary, (e.g. Swash and LunaDNA which handle group-based aggregated data). In certain Data Sharing Pools and Marketplaces, especial y in cases with competitive business data (NxtPort, Catena-X), data providers often have greater control over access. Dual actor control aims for a balanced sharing of control (DjustConnect and DataVil age) in 250 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION cases where sensitive personal data is also competitive business data. Data subjects grant access control to data consumers, while data providers can either give consent or open APIs. Data Type concerns the type of data consumers receive access to, which helps mitigate the sensitivity of the data (Abrams, 2014; Kugler & Plank, 2022). (Raw) data are inherently sensitive for both data subjects and providers, representing the lowest level of control. Providing processed data (anonymized, pseudonymized or aggregated data) reduces sensitivity for both parties (LunaDNA's aggregated genomic data). Sharing inferred or derived data alone, without divulging algorithms or raw data, is a method of selective sharing based on a need-to-know basis (Datavillage and Swash sIntelligence). Verification by the data provider (Karamel verified diplomas) transfers control to both the verifying agent and the data consumer, ensuring data validity. Data Access Criteria defines how access can be granted to various actors within an ecosystem and are different types of attribute –based (Gupta et al., 2018), relationship-based (Gates, 2007) or role-based (Ferraiolo & Kuhn, 1992) access control mechanisms. The actor that sets the access control criteria holds the highest data sovereignty. User preference-based access is set by the data subject, including preferences regarding the data consumer’s identity, the data, and the context at the time of the access request. It can happen through individual consent (DjustConnect, Karamel) or automatically (Consent-o-matic). In Collective policies, the rights of data subjects/holders can be pooled, and the terms of use for the data determined in the suppliers’ interests (datatrusts.uk), often based on democratic decision-making in data cooperatives (Midata.coop). Intermediary-based access control hinges on participation within a defined ecosystem, and it is determined by policies set by the intermediary (data sharing regulations set by NxtPort). Data provider relationship-based access control fosters controlled data sharing among trusted partners (DataVil age). In intricate ecosystems like Catena-X, data provider usage- based access control is established through legal y binding policies set by the data provider, delineating the conditions under which data consumers can utilize data assets. R. D’Hauwers, A. Bourgeus: Empowering Data Sovereignty: Strategies of Data Intermediaries in Data Ecosystems 251 4.2 Power Dynamics Data sharing within data ecosystems is heavily influenced by power imbalances among companies (Li & Lin, 2008). Data intermediaries can utilize control mechanisms like storage and processing location, representation, and legal rights to manage these power dynamics. Data Storage Location plays a crucial role in determining control and power dynamics within digital ecosystems (Gelhaar et al., 2021; Scheider et al., 2023). In the data provider central scenario (tech giants like Google), control lies firmly in the hands of the data holder. Conversely, in data subject decentral models (Karamel and WeAre), personal data is stored within individual-centric pod systems hosted by neutral data storage hosts. In intermediary-central storage models, trusted intermediaries store personal data on behalf of users, al owing them to exert power by aggregating data rights across multiple individual data subjects (Midata.coop, Swash and LunaDNA). In decentralized data provider storage (Catena-X), data is distributed across multiple data provider locations. Through connectors and interoperability standards, interoperable sharing is facilitated. Data Processing Location, also referred to as the distribution of intelligence (Ballon, 2007), is an essential consideration, particularly given the rising utilization of AI and data mining. Within data ecosystems, this concept pertains to the specific al ocation of processing power, control, and functionality across the system. Data may undergo processing locally at the data subject's end. While not exemplified in our sample, this scenario is common in applications such as privacy assistants (Morel & Fischer-Hübner, 2023). In an intermediated context, secure col aboration spaces are created where the algorithm is securely located and data remains encrypted (DataVil age), ensuring confidentiality on both the data and the algorithm within the collaboration. Alternatively, data can be processed at the data provider's end, where aggregated data is processed, or algorithms are run to generate insights (DjustConnect or NxtPort). Conversely, raw data can be provided to the data consumer, who then processes the data or runs their algorithms, typically observed in large, dominant companies like Facebook and Google. 252 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Representation within a collective can enable data subjects to exert greater control over their data (Delacroix & Lawrence, 2019). Thus, data intermediaries may facilitate a "communal approach to data sharing," involving the entire community in decision-making regarding data rights (Ho & Chuang, 2019). This representation by the data intermediary can take a collective or in a group form, where data from various subjects are pooled to enhance power, (Midata.coop, LunaDNA and Swash). Alternatively, representation can be individual, where data subjects independently determine data usage without leveraging community power (WeAre, Karamel, and Consent-o-matic). Legal Rights can be entrusted by a data subject to a data intermediary (World Economic Forum, 2022), serving as a legal mechanism that consolidates the rights of data subjects/holders and determines data usage terms in their favor (Sadowski et al., 2021). This arrangement can enhance protection against privacy infringements and unethical handling of personal data (Micheli et al., 2023). Among data intermediaries, we note a distinction: some actively advocate for the enforcement of data subjects' legal rights (Datatrusts.uk). Conversely, others primarily focus on ensuring legal compliance without actively enforcing legal rights, by aligning data handling with legal standards (DjustConnect, NxtPort). 4.3 Revenue Sharing Model The revenue sharing model is the extent to which revenues are shared within the ecosystem can facilitate fair profit-sharing among its members (Lauf et al., 2022), potentially leading to a more equitable distribution of benefits for the data subjects. In scenarios where data subjects do not receive a share of revenue, profits generated from data sales remain unal ocated. Conversely, when revenue sharing occurs, profits from data sales are redistributed, promoting equitable sharing. The individual-interest model (Fox, 2020) al ows each data subject to receive a portion of revenues based on the amount of shared data. This may involve data subjects receiving stocks, leading to potential monetary returns (LunaDNA), or redistributing revenues generated from data sales (Swash). In contrast, the col ective-interest model (Fox, 2020) involves community trusts, directing value redistribution col ectively towards specific groups. R. D’Hauwers, A. Bourgeus: Empowering Data Sovereignty: Strategies of Data Intermediaries in Data Ecosystems 253 5 Discussion This paper makes a significant contribution to both the data sovereignty literature (Hummel et al., 2021; Zrenner et al., 2019) and the data governance literature (Gelhaar et al., 2021; Lis & Otto, 2020) by introducing a taxonomy that serves two primary purposes: Firstly, it facilitates the mapping of various data governance mechanisms aimed at achieving a balance of data sovereignty between data subjects and data providers. Secondly, it provides the building blocks for developing data governance models for data intermediaries. This framework involves the creation of mechanisms for data control, which are closely linked to the governance models implemented by these intermediaries. Additionally, this paper contributes to the literature on data intermediary models (Janssen & Singh, 2022; Micheli et al., 2023) by uncovering the different mechanisms for data intermediaries to achieve a balance of data sovereignty tailored to the specific purposes of the data intermediary. The purpose of the data intermediary can be control over sensitive or competitive data (Hummel et al., 2021; Zrenner et al., 2019) equitable benefit sharing (Fox, 2020; Lauf et al., 2022) or decentralizing power in the data economy towards the data subject (Zuboff, 2015).First, if the purpose is to effectively manage personal sensitive or competitive data, the control mechanisms depend on two key factors: the company's data competitiveness level (Enders et al., 2020; Kugler & Plank, 2022) and the individual's data sensitivity (Belen Saglam et al., 2022). In this case, data access management is the major control mechanism which aims to balance data sovereignty and mitigate data sensitivity during sharing. Depending on data sensitivity and competitiveness, control varies. Data providers hold the most control in the case of low-sensitivity, high-competitiveness data (e.g., Catena-X), data subjects in the case of highly sensitive, low-competitive data (e.g., Karamel), and dual mechanisms are needed for highly sensitive, highly competitive data (e.g., DjustConnect). Second, value redistribution can be the primary purpose of the data intermediary, as seen in data unions where intermediaries empower data subjects by aggregating their data to create value for consumers. The revenue sharing control mechanism is crucial to enable this purpose. Third, if the purpose is to decentralize power in the data economy, the focus shifts to empowering data subjects and rebalancing dynamics. The power dynamics mechanics involves forming user groups in data unions and DCs to collectively negotiate data access and 254 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION centralize storage and access control with PIMS, giving individuals greater control. Moreover, in data trusts, legal rights are enforced by the data intermediary. 6 Conclusion Various entities like PIMS, data unions, data trusts, data cooperatives, data pools, and DMs aim to achieve a balanced data sovereignty between individuals and data providers, facilitated by mechanisms which enable data sovereignty. These mechanisms were examined in this paper, leading to a taxonomy with three meta- dimensions: data access control, power dynamics, and revenue sharing models, identifying eight dimensions essential for reaching an equilibrium in data sovereignty, depending on the purpose of the data intermediary. These dimensions enable intermediaries to tailor control mechanisms for sensitive data protection, changing power dynamics towards the user, and equitable benefit sharing. Various entities like PIMS, data unions-, trusts and - cooperatives, DSPs and DMs have different data sovereignty balance points, with control mechanisms facilitating this purpose. This research contributes to theoretical understanding of data governance, data ecosystems and data intermediary literature. The practical applications enable data intermediaries to navigate data sovereignty complexities within evolving data ecosystems by providing data control mechanism building blocks. Limitations to the research include the case study depth and breadth; deeper analysis can reveal underlying control mechanism characteristics, while broader case studies can support evaluating the taxonomy's validity. Future research could augment the taxonomy with value creation and governance dimensions and model different intermediary types using taxonomy dimensions. R. D’Hauwers, A. Bourgeus: Empowering Data Sovereignty: Strategies of Data Intermediaries in Data Ecosystems 255 7 Annex: overview of data intermediaires and mapping of data control mechanisms * In these instances, the characteristics were determined from the context or were not conclusive based on the available information accessible to the researchers. 256 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION References Abraham, R., Johannes Schneider, & vom Brocke, J. (2019). Data Governance: A conceptual framework, structured review, and research agenda. International Journal of Information Management, 49, 424–438. https://doi.org/10.1016/j.ijinfomgt.2019.07.008 Abrams, M. (2014). The Origins of Personal Data and its Implications for Governance. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2510927 Adner, R. (2017). Ecosystem as a structure: An actionable construct for strategy. Journal of Management, 43(1), 39–58. Ballon, P. (2007). Business Modelling Revisited: The Configuration of Control and Value. The Journal of Policy, Regulation and Strategy for Telecommunications, Information and Media, 9(5), 6–19. Belen Saglam, R., Nurse, J. R. C., & Hodges, D. (2022). Personal information: Perceptions, types and evolution. Journal of Information Security and Applications, 66, 103163. https://doi.org/10.1016/j.jisa.2022.103163 Berg, S. (2006). Snowball Sampling—I. In Encyclopedia of Statistical Sciences. John Wiley & Sons, Ltd. https://doi.org/10.1002/0471667196.ess2478.pub2 Bussard, L., Neven, G., & Preiss, F.-S. (2010). Downstream Usage Control. 2010 IEEE International Symposium on Policies for Distributed Systems and Networks, 22–29. https://doi.org/10.1109/POLICY.2010.17 Corbin, J. M., & Strauss, A. (2008). Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory. Couldry, N., & Mejias, U. A. (2019). Data Colonialism: Rethinking Big Data’s Relation to the Contemporary Subject. Television & New Media, 20(4), 336–349. https://doi.org/10.1177/1527476418796632 Curry, E. (2020). Fundamentals of Real-time Linked Dataspaces. In E. Curry (Ed.), Real-time Linked Dataspaces: Enabling Data Ecosystems for Intel igent Systems (pp. 63–80). Springer International Publishing. https://doi.org/10.1007/978-3-030-29665-0_4 Curry, E., & Sheth, A. (2018). Next-Generation Smart Environments: From System of Systems to Data Ecosystems. IEEE Intel igent Systems, 33(3), 69–76. https://doi.org/10.1109/MIS.2018.033001418 Delacroix, S., & Lawrence, N. D. (2019). Bottom-up data Trusts: Disturbing the ‘one size fits al ’ approach to data governance. International Data Privacy Law, 9(4), 236–252. https://doi.org/10.1093/idpl/ipz014 Denzin, N. (1978). Sociological methods: A sourcebook. McGraw-Hill. Enders, T., Wolff, C., & Satzger, G. (2020). Knowing What to Share: Selective Revealing in Open Data. European Conference on Information Systems. https://www.semanticscholar.org/paper/Knowing-What-to-Share:-Selective-Revealing-in- Open-Enders-Wolff/04ba82f4971c731287e2e576c2be23bf6f35c577 Ensign, P. C., & Hébert, L. (2009). Competing explanations for knowledge exchange: Technology sharing within the global y dispersed R&D of the multinational enterprise. The Journal of High Technology Management Research, 20(1), 75. Ferraiolo, D., & Kuhn, R. (1992). Role-Based Access Controls. In Proceedings of the 15th National Computer Security Conference (pp. 554–563). National Institute of Standards and Technoloty. https://doi.org/10/13/rolebased-access-controls/final Fox, K. (2020). The Illusion of Inclusion—The “Al of Us” Research Program and Indigenous Peoples’ DNA. New England Journal of Medicine, 383(5), 411–413. https://doi.org/10.1056/NEJMp1915987 Gates, C. (2007). Access Control Requirements for Web 2.0 Security and Privacy. Gelhaar, J., Groß, T., & Otto, B. (2021). A Taxonomy for Data Ecosystems. Hawai International Conference on System Sciences. Gupta, M., & George, J. F. (2016). Toward the development of a big data analytics capability. Information & Management, 53(8), 1049–1064. https://doi.org/10.1016/j.im.2016.07.004 R. D’Hauwers, A. Bourgeus: Empowering Data Sovereignty: Strategies of Data Intermediaries in Data Ecosystems 257 Gupta, M., Patwa, F., & Sandhu, R. (2018). An Attribute-Based Access Control Model for Secure Big Data Processing in Hadoop Ecosystem. Proceedings of the Third ACM Workshop on Attribute-Based Access Control, 13–24. https://doi.org/10.1145/3180457.3180463 Hartman, T., Kennedy, H., Steedman, R., & Jones, R. (2020). Public perceptions of good data management: Findings from a UK-based survey. Big Data & Society, 7(1), 2053951720935616. https://doi.org/10.1177/2053951720935616 Ho, C., & Chuang, T.-R. (2019). Governance of Communal Data Sharing. Institute of Network Cultures. http://ir.sinica.edu.tw/handle/201000000A/55961 Hummel, P., Braun, M., & Dabrock, P. (2021). Own Data? Ethical Reflections on Data Ownership. Philosophy & Technology, 34(3), 545–572. https://doi.org/10.1007/s13347-020-00404-9 Jacobides, M. G., Cennamo, C., & Gawer, A. (2018). Towards a theory of ecosystems. Strategic Management Journal, 39(8), 2255–2276. https://doi.org/10.1002/smj.2904 Janssen, H., & Singh, J. (2022). The Data Intermediary (SSRN Scholarly Paper 4070971). https://papers.ssrn.com/abstract=4070971 Kelbert, F., & Pretschner, A. (2012). Towards a policy enforcement infrastructure for distributed usage control. Proceedings of the 17th ACM Symposium on Access Control Models and Technologies, 119–122. https://doi.org/10.1145/2295136.2295159 Knaapi-Junnila, S., Rantanen, M. M., & Koskinen, J. (2022). Are you talking to me? – Calling laypersons in the sphere of data economy ecosystems. Information Technology & People, 35(8), 292–310. https://doi.org/10.1108/ITP-01-2021-0092 Koskinen, J., Knaapi-Junnila, S., Helin, A., Rantanen, M. M., & Hyrynsalmi, S. (2023). Ethical governance model for the data economy ecosystems. Digital Policy, Regulation and Governance, 25(3), 221–235. https://doi.org/10.1108/DPRG-01-2022-0005 Kugler, P., & Plank, T. J. (2022). Coping with the Double-Edged Sword of Data-Sharing in Ecosystems. Technology Innovation Management Review. Lauf, F., Scheider, S., Bartsch, J., Herrmann, P., Radic, M., Rebbert, M., Nemat, A., Langdon, C. S., Konrad, R., Sunyaev, A., & Meister, S. (2022). Linking Data Sovereignty and Data Economy: Arising Areas of Tension. Wirtschaftsinformatik 2022 Proceedings. https://aisel.aisnet.org/wi2022/it_for_development/it_for_development/19 Lis, D., & Otto, B. (2020). Data Governance in Data Ecosystems – Insights from Organizations. Loebbecke, C., van Fenema, P. C., & Powell, P. (2016). Managing inter-organizational knowledge sharing. The Journal of Strategic Information Systems, 25(1), 4–14. https://doi.org/10.1016/j.jsis.2015.12.002 Micheli, M., Farrel , E., Carbal a, S. B., Posada, S. M., Signorel i, S., & Vespe, M. (2023, August 24). Mapping the landscape of data intermediaries. JRC Publications Repository. https://doi.org/10.2760/261724 Moiso, C., & Minerva, R. (2012). Towards a user-centric personal data ecosystem The role of the bank of individuals’ data. 2012 16th International Conference on Intelligence in Next Generation Networks, 202–209. https://doi.org/10.1109/ICIN.2012.6376027 Morel, V., & Fischer-Hübner, S. (2023). Automating privacy decisions—Where to draw the line? Nickerson, R. C., Varshney, U., & Muntermann, J. (2013). A method for taxonomy development and its application in information systems. European Journal of Information Systems, 22(3), 336– 359. https://doi.org/10.1057/ejis.2012.26 Ojasalo, J., & Miskeljin, P. (2020). PROPOSING A PRELIMINARY CONCEPT FOR A PERSONAL DATA ECOSYSTEM (p. 7459). https://doi.org/10.21125/inted.2020.1992 Otto, B., & Teuscher, S. (2019). International Data Spaces Association—Reference Architecture Model (p. 118). International Data Spaces Association. https://internationaldataspaces.org/wp-content/uploads/IDS-Reference-Architecture- Model-3.0-2019.pdf Patton, M. (1999). Enhancing the quality and credibility of qualitative analysis. In BMC Health Serv Res (Vol. 34, p. 1208). Regulation (EU) 2022/868, 152 OJ L (2022). http://data.europa.eu/eli/reg/2022/868/oj/eng 258 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Runeson, P., & Höst, M. (2009). H??st, M.: Guidelines for Conducting and Reporting Case Study Research in Software Engineering. Empirical Software Engineering 14, 131-164. Empirical Software Engineering, 14, 131–164. https://doi.org/10.1007/s10664-008-9102-8 Sadowski, J., Viljoen, S., & Whittaker, M. (2021). Everyone should decide how their digital data are used—Not just tech companies. Nature, 595(7866), 169–171. https://doi.org/10.1038/d41586-021-01812-3 Scheider, S., Lauf, F., Möl er, F., & Otto, B. (2023). A Reference System Architecture with Data Sovereignty for Human-Centric Data Ecosystems. Business & Information Systems Engineering, 65(5), 577–595. https://doi.org/10.1007/s12599-023-00816-9 S. Oliveira, M. I., Barros Lima, G. de F., & Farias Lóscio, B. (2019). Investigations into Data Ecosystems: A systematic mapping study. Knowledge and Information Systems, 61(2), 589– 630. https://doi.org/10.1007/s10115-018-1323-6 Spiekermann, S., Acquisti, A., Böhme, R., & Hui, K.-L. (2015). The chal enges of personal data markets and privacy. Electronic Markets, 25(2), 161–167. https://doi.org/10.1007/s12525-015-0191-0 Tomi Dahlberg, & Nokkala, T. (2017). Willigness to share supply chain data in an ecosystem governed platform—An Interview Study. 32nd Bled eConference Humanizing Technology for a Sustainable Society, 32, 619–638. https://doi.org/10.18690/978-961-286-280-0 Van Kleek, M., & OHara, K. (2014). The Future of Social Is Personal: The Potential of the Personal Data Store. In D. Miorandi, V. Maltese, M. Rovatsos, A. Nijholt, & J. Stewart (Eds.), Social Collective Intelligence (pp. 125–158). Springer International Publishing. https://doi.org/10.1007/978-3-319-08681-1_7 Verner, J. M., Sampson, J., Tosic, V., Bakar, N. A. A., & Kitchenham, B. A. (2009). Guidelines for industrially-based multiple case studies in software engineering. 2009 Third International Conference on Research Chal enges in Information Science, 313–324. World Economic Forum. (2022). Advancing Digital Agency: The Power of Data Intermediaries. World Economic Forum. https://www.weforum.org/publications/advancing-digital-agency- the-power-of-data-intermediaries/ Yin, R. (2013). Case study Research and Applications (5th ed.). Sage Publishing. Zrenner, J., Möl er, F. O., Jung, C., Eitel, A., & Otto, B. (2019). Usage control architecture options for data sovereignty in business ecosystems. Journal of Enterprise Information Management, 32(3), 477–495. https://doi.org/10.1108/JEIM-03-2018-0058 Zuboff, S. (2015). Big other: Surveil ance Capitalism and the Prospects of an Information Civilization. Journal of Information Technology, 30(1), 75–89. https://doi.org/10.1057/jit.2015.5 ANALYSIS OF DEVELOPMENT AND USER ASPECTS OF EPRESCRIPTION IN SLOVENIA ŽIVA RANT,1 DALIBOR STANIMIROVIĆ2 1 National Institute of Public Health, Ljubljana, Slovenia ziva.rant@nijz.si 2 University of Ljubljana, Faculty of Public Administration, Ljubljana, Slovenia dalibor.stanimirovic@fu.uni-lj.si Slovenia has well-developed eHealth solutions. One of these is ePrescription, which enables the electronic prescription of medicines and their dispensing in pharmacies. With few exceptions, the everyday functioning of the healthcare system ground to a halt during the COVID-19 pandemic. New treatment protocols and the risk of infection meant that health professionals faced major challenges in their work. Patients tried to reduce the number of personal visits to healthcare institutions as much as possible, both out of fear of infection and in response to the changes introduced in healthcare provision and the various restrictions put in place. Because of the exceptional circumstances, other inherent factors and pressure from patients, health professionals began to make more intensive use of eHealth solutions. The ePrescription solution was already in Keywords: general use at this time and proved extremely important in ePrescription, ensuring the prescription of medicines. The paper wil describe electronic the ePrescription solution in detail, illustrate its development prescription, eHealth, and, with the help of data from statistics and business intel igence health modules, analyse its use. informatics DOI https://doi.org/10.18690/um.fov.4.2024.14 ISBN 978-961-286-871-0 260 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Slovenia has well-developed eHealth solutions. One of these is ePrescription, a national IT solution for the electronic prescription of medicines by healthcare providers and the dispensing of medicines in pharmacies. Despite the solution has been in use since 2015, there is very little written about it, especially in English. Our aim is to describe and analyse the ePrescription solution, provide data on its use over the years, and discuss this. Today ePrescription is one of the most recognised, effective and widely used solutions to have been developed in the context of Slovenian eHealth. Its effects have been evaluated by the Ministry of Public Administration in 2020 in the document “Evalvacija ukrepov iz enotne zbirke ukrepov, Vrednotenje učinkov implementacije projekta eZdravje: eRecept, eNaročanje” (Ministry of Public Administration, 2020). The DESI Report 2019 (European Commission, 2019) ranked Slovenia third in the EU in terms of the use of electric prescriptions. The ePresciption solution also won the "Informacijska jagoda" award for the best information society achievement in 2017. The aims of the ePrescription solution were the fol owing (Stanimirović & Matetić, 2018; National Institute of Public Health, 2020): • To increase patient safety and reduce errors due to the incorrect use of medicines through improved legibility of prescriptions and a smaller number of administrative errors. • To simplify procedures for patients in cases where, taking into account medical guidelines, a prescription may be issued in the absence of the patient, since in such cases the patient does not need to visit the prescriber in order to collect the prescription. • To ensure more efficient prescription and use of medicines, taking into account information on previously prescribed and dispensed medicines. • To increase the efficiency of the process of dispensing medicines within individual operators and between operators in healthcare, above al by reducing unnecessary contacts and journeys. Ž. Rant, D. Stanimirović: Analysis of Development and User Aspects of ePrescription in Slovenia 261 • To reduce administrative costs by eliminating the use of paper forms (purchase of forms, printing in doctor’s surgeries, filing systems in pharmacies). • To provide the data necessary for various analyses, including various types of studies. ePrescription use found an even bigger role during the COVID-19 pandemic. With few exceptions, the everyday functioning of the healthcare system ground to a halt during the COVID-19 pandemic. New treatment protocols and the risk of infection meant that health professionals faced major challenges in their work. Patients tried to reduce the number of personal visits to healthcare institutions as much as possible, both out of fear of infection and in response to the changes introduced in healthcare provision and the various restrictions put in place. Because of the exceptional circumstances, other inherent factors and pressure from patients, health professionals began to make more intensive use of eHealth solutions. During this period, prescribing medicines remotely without the patient being present made it possible for many patients to obtain medicines without interruption and greatly eased access to medicines during the pandemic. In this paper we describe and analyse the ePrescription solution, provide data on its use over the years, and discuss this. 2 Methodology The paper presents an analysis of the functionality and use of the ePrescription solution as part of Slovenian eHealth. The solution was introduced at the national level in 2015. During the COVID-19 pandemic it made it possible for large numbers of patients to access the medicines they needed, which indicates that this solution is important for Slovenian health system. Our research aimed to answer the question of what the ePrescription solution actual y consists of and to explore its development and use over the years. It was based on the case study research methodology (Kljajić Borštnar, 2021; Yin, 2018), which included an in-depth study of the field and its analysis. For data we used statistics from the administrative and business intelligence modules. 262 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The selection of this research methodology was driven by the distinct characteristics of the research topic, with the chosen method regarded as the most efficient for the study. Due to the exploratory nature of this research, quantitative empirical methods were found insufficient for producing satisfactory results or offering a credible evaluation of the field. The intricate landscape of healthcare digitalization in Slovenia is stil evolving, posing chal enges in ensuring the representativeness of the research sample. Thus, the case study framework, including exhaustive literature review and statistical data, was considered the most favourable methodological approach to enable a comprehensive exploration of the of the functionality and use of the ePrescription in Slovenia. Accordingly, the analysis was conducted on the one hand based on a review of the literature in this field (Rant et al., 2017; Rant et al., 2019; Stanimirović & Matetić, 2018; Stanimirović et al., 2022, Yang et al., 2022; Zidarn et al., 2018; Matetić et al., 2024) as wel as on the examination of project documentation, user instructions and the technical specifications of the solution. On the other hand, it based on observations and statistical usage data. We used statistics from the administrative and business intel igence modules in the analysis to compare the number of electronic prescriptions issued by year, their share comparing al prescriptions issued and compare number of electronic prescriptions issued by months in different years. We also compared data number of healthcare providers using this solution on the both side of it – prescribing prescriptions and dispensing medicines. We use data from the years 2016 to 2023. Analysis of the functionality and use of the ePrescription solution was carried out in December 2023 and January 2024. Statistical data were obtained from the business and administrative modules in January 2024. 3 Results 3.1 Description of the solution and analysis ePrescription is a national IT solution for the secure electronic prescribing and electronic dispensing of medicines. Electronic prescriptions are generated in the local IT systems of healthcare providers and stored in a central register of electronic prescriptions (Fig. 1). The latter serves as a source of information for pharmacies, Ž. Rant, D. Stanimirović: Analysis of Development and User Aspects of ePrescription in Slovenia 263 where medicines are dispensed to patients on the basis of ePrescriptions issued (Ministry of Health, 2013). Figure 1: Diagram of the ePrescription solution Source: (Stanimirović & Matetić, 2018; National Institute of Public Health, 2020) Even the development of the solution is an example of good practice, since seven software providers cooperated on its development as a consortium. The solution was introduced in November 2015 at the primary level throughout Slovenia, and then at the secondary and tertiary levels in February 2016. It is used on a daily basis by healthcare providers and in pharmacies. The solution consists of two components, namely the electronic prescribing and dispensing of medicinal products (Matetić et al., 2024). When prescribing a medicine, the doctor identifies themselves using their professional card and the patient’s health insurance card or health insurance number. The doctor examines the list of medication taken by the patient. Then doctor then chooses the medicine they wish to prescribe from the list of previously prescribed medicines or the Central Medicinal Products Database. They may also prescribe a magistral formula. At this stage they have the opportunity to check interactions and contraindications. The solution also allows doctors to check the suitability of a medicinal product for athletes (in the case of it having been placed on a list of banned 264 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION substances). The doctor thus prepares the prescription data and checks appropriateness. They then sign the bundle digital y and send it to the central register of electronic prescriptions (Matetić et al., 2024). At the pharmacy, the pharmacist uses the patient’s health insurance card and their own professional card to obtain patient data, select the prescription and select the medicine for dispensing. They then prepare and dispense the medicine and confirm dispensing in the central register of electronic prescriptions (Matetić et al., 2024). The patient can monitor al this information via the zVEM patient portal (National Institute of Public Health, 2024). Patients can see what medicines they have been prescribed, who prescribed them and when, medicines already dispensed and how many more times medicines may be dispensed in the case of repeat prescriptions. A patient can col ect medicines dispensed under the same repeat prescription from different pharmacies in Slovenia. The zVEM portal (National Institute of Public Health, 2024) also allows patients to see who has consulted their prescription data and when. The ePrescription solution al ows the doctor to view a patient’s electronic prescriptions, electronical y issue green and white prescriptions, issue structured prescriptions, prescribe a medicine from the Central Medicinal Products Database or a magistral formula, check for interactions and contraindications and the presence of substances that are banned in sport, sign the prescription bundle electronical y and send it to the central national register of electronic prescriptions. Meanwhile, in pharmacies it enables dispensing pharmacists to view electronic prescriptions in the central register, check for interactions and contraindications and the presence of substances that are banned in sport, and electronical y sign and forward dispensing data to the central register. ePrescription is an IT solution that has been in use since 2015. Despite this, we are constantly working to develop it. In 2023 we added the production and updating of the personal medication card (National Institute of Public Health, 2020b), which is done by specially trained pharmacists. These activities are actually out in another eHealth IT solution – cal ed zVEM plus – where, however, data on electronic prescriptions are used as a basis. Ž. Rant, D. Stanimirović: Analysis of Development and User Aspects of ePrescription in Slovenia 265 We are currently preparing a link to the Slovenian Anti-Doping Organisation (SLOADO). Checking medicines against the list of banned substances in sport wil be incorporated into the solution. We are also adding functionality that enables nurses to view electronic prescriptions. 3.2 Usage data Our research also looked at the use of the ePrescription solution. Data from the administrative module of the ePrescription solution show that the total number of electronic prescriptions issued did not change significantly between 2018 and 2020. Increases in numbers can be seen in 2021, 2022 and 2023. More than 15 million electronic prescriptions were issued in 2022, while in 2023 the total number of electronic prescriptions exceeded 15.6 million (Fig. 2). This is equivalent to every inhabitant of Slovenia receiving just under eight prescriptions per year. Figure 2: Number and share of electronic prescriptions issued by year Source: Own The ePrescription solution has been used by 1,500 healthcare providers. Electronic prescriptions have been issued in 1,100 organisations and medicines have been dispensed by 360 pharmacies. A review of the share of electronic prescriptions among al prescriptions shows that from 2020 onwards more than 96% of prescriptions issued were electronic. In 2023 this figure reached 97%. Our aim when launching ePrescription was to achieve a 266 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION share of 90%, a target we have significantly exceeded (Fig. 2). Existing rules mean that we wil never reach 100%. Some urgent prescriptions wil continue to be issued on paper, as wil prescriptions issued during home visits and personal prescriptions issued by doctors for their own use. It is also interesting to look at how the number of electronic prescriptions varies from month to month. There is a visible drop in the number of prescriptions in the summer months, while more prescriptions are issued in winter (Fig. 3). Figure 3: Number of electronic prescriptions issued by months in different years Source: Own 4 Discussion The independent report “Evalvacija ukrepov iz enotne zbirke ukrepov, Vrednotenje učinkov implementacije projekta eZdravje: eRecept, eNaročanje” (Ministry of Public Administration, 2020) lists the key positive changes for individual stakeholders. Key positive changes are understood as impacts of ePrescription, which have beneficial effects on stakeholders in the healthcare system, business dimensions, and other organizational, process and management aspects of the healthcare system. For doctors, the report emphasises better control and oversight of already issued prescriptions, more structured prescriptions, easier and faster checking of drug interactions as a result of additional functionalities, more effective control for individual risk groups, less administrative work and more accessible and faster data analysis. Positive changes for pharmacists include the possibility of checking drug interactions, elimination of the possibility of issuing the wrong medicine because of Ž. Rant, D. Stanimirović: Analysis of Development and User Aspects of ePrescription in Slovenia 267 a doctor’s illegible handwriting, less administrative work, better safety and more accessible and faster data analysis. The key positive changes for patients include streamlined prescribing and dispensing processes, a reduced chance of unwanted side effects from taking medicines, safe storage of data on prescribed medication, the ability to view prescription statuses, and options al owing the dispensing of medicines for ongoing treatment or remote prescribing. What's more, the report also cites significant systemic benefits from the ePrescription. Among the most important are certainly the financial savings, which range between €2 million and €3 million a year for 2016, 2017 and 2018. The solution also has other advantages: the option to issue prescriptions remotely without the physical presence of the patient in the doctor’s surgery; the elimination of errors resulting from incorrect notes or il egible handwriting and a consequent increase in patient safety; in the case of repeated prescriptions, the fact that patients no longer have to go the same pharmacy each time; the possibility of checking medicines for the presence of substances from the list of substances that are banned for athletes. Monitoring the consumption of medicines is also more simple and data reliability is improved. 5 Conclusions The ePrescription solution is an example of good practice in the digitalisation of healthcare in Slovenia. The solution was the first eHealth solution in Slovenia to be introduced at the national level. During the COVID-19 pandemic it made it possible for large numbers of patients to access the medicines they needed, by al owing doctors to issue prescriptions remotely and enabling patients to collect their prescriptions at any pharmacy in Slovenia using their health insurance card. More than 96% of prescriptions are issued electronically. These place Slovenia among the leading countries in the world in this sphere. The link to the zVEM patient portal, which al ows patients to monitor their own use of medicines, is also important. Although the ePrescription solution was developed and introduced in 2015, it still serves on a daily basis its purpose extremely wel and is constantly being upgraded in the light of new requirements and needs. Today it would be difficult to imagine Slovenia’s healthcare system without electronic prescribing. Both human resources and financial resources need to be provided to ensure the maintenance of the ePrescription solution and enable the necessary upgrades and enhancements. 268 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION References European Commission. (2019). Digital Economy and Society Index Report 2019; Digital Public Services. https://ec.europa.eu/digital-single-market/en/desi Kljajić Borštnar, M. (2021). Raziskovanje informacijskih sistemov. Matetić, V., Modic, L., Planinc, N., Škoro, H., Škuver, M., & Zupan, M. (2024). Funkcionalna specifikacija za eRecept. Ministry of Public Administration (2020). Evalvacija ukrepov iz enotne zbirke ukrepov. Vrednotenje učinkov implementacije projekta eZdravje: eRecept, eNaročanje. https://www.stopbirokraciji.gov.si/novice/razbremenitve-z-uvedbo-elektronskih-resitev- erecept-in-enarocanje-1 Ministry of Health (2013). Projekt eZdravje v praksi. Ministrstvo za zdravje Republike Slovenije, Sektor za eZdravje, Ljubljana, https://www.stat.si/doc/sosvet/Sosvet_01/Sos01_s1968-2013.pdf National Institute of Public Health (2020). eRecept – eZdravje. https://ezdrav.si/storitve/erecept/ National Institute of Public Health (2020b). Osebna kartica zdravil (OKZ). https://ezdrav.si/storitve/okz/ National Institute of Public Health (2024). Portal zVEM. https://zvem.ezdrav.si/portal. Rant, Ž., Stanimirović, D., Matetić, V., Indihar, S., Zidarn, J., Beštek, M., Tepej Jočić, L., Žlender, A., & Živa Rant, Dalibor Stanimirović, Vedrana Matetić, Simon Indihar, Janja Zidarn, Mate Beštek, Lucija Tepej-Jočić, A. Ž. (2017). eZdravje danes. Uporabna Informatika, 25(3), 169- 179. Rant, Ž., & Stanimirović, D. (2019). Analysis of e-Health solutions in Slovenia : a usage perspective. Uporabna Informatika, 27(4), 135-142. Stanimirović, D., & Matetić, V. (2018). eRecept - zastavljeni cilji in dosežki. NOVIS, 45(1/2), 13-15. https://issuu.com/zdrzz_novis/docs/novis_jan-feb_2018 Stanimirović, D., Drev, M., & Rant, Ž. (2022). Ekspanzija digitalnih rešitev eZdravja v času pandemije covida-19 v Sloveniji. Isis, 31(6), 34–37. http://online.pubhtml5.com/agha/mpic/#p=34 Yang, Y. F., Hoo, J. X., Tan, J. Y., & Lim, L. L. (2022). Multicomponent integrated care for patients with chronic heart failure: systematic review and meta-analysis. ESC Heart Failure. https://doi.org/10.1002/EHF2.14207 Yin, R. K. (2018). Case Study Research and applications, 6th edition. Paper Knowledge . Toward a Media History of Documents, 414. Zidarn, J., Stanimirović, D., & Indihar, S. (2018). eZdravje : eRecept in eNaročanje v obdobju 2017-2018 = eHealth. In T. Marčun (Ed.), 30 let izkušenj v podporo digitalizaciji zdravstva, 26-31. SDMI. DIGITAL PUBLIC SERVICE IMPROVEMENT IN CROSS-BORDER USE CASES HELEN RAAMAT,1 INNAR LIIV,2 SILVIA LIPS,3 ROZHA KAMAL AHMED,4,5 RAHUL SHARMA,4,6 DIRK DRAHEIM4 1 Information System Authority, Tallinn, Estonia helen.raamat@ria.ee 2 Tallinn University of Technology, Tallinn, Estonia innar.liiv@taltech.ee 3 Information System Authority, Tallinn, Estonia silvia.lips@taltech.ee 4 Tallinn University of Technology, Information Systems Group, Tallinn, Estonia rozha.ahmed@taltech.ee, rahul.sharma@taltech.ee, dirk.draheim@taltech.ee 5 Sulaimani Polytechnic University, Computer Science Institute, Sulaymaniyah, Iraq rozha.ahmed@spu.edu.iq 6 Ajay Kumar Garg Engineering College, Ghaziabad, India rahul.sharma@taltech.ee This research identifies the main legal and technical barriers connected to identity management and cross-border service provision. We also propose a solution that fits in the current state of play. We analyzed the existing documentation and conducted Keywords: interoperability, semi-structured interviews with digital public service providers cross-border and use the Estonia as a case study to map the current obstacles. digital To resolve the cross-border interoperability issues that digital public services, public services face, we explore the existing state of play for eIDAS, cross-border use cases through a process design and highlighting SDGR, the requirements for cross-border interoperability infrastructure. implementation challenges, As a result, we provide recommendations overcoming the electronic barriers that affect cross-border digital public service delivery. identity DOI https://doi.org/10.18690/um.fov.4.2024.15 ISBN 978-961-286-871-0 270 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The mobility of European Union (EU) citizens has grown in recent years, as wel as the demand and expectations to access cross-border digital public services (European Commission, 2021). The regulation on electronic identification and trust services for electronic transactions in the internal market (eIDAS) was established on July 23rd, 2014, in the EU to support the objectives of the digital single market and digital economy. 1 eIDAS aims to facilitate access to cross-border digital services by creating trust in the digital world, like in the physical world. According to the regulation, all the public and private sector authorities providing digital public services in the EU must mutual y recognize the notified eID means. For the implementation of eIDAS, the European Commission’s (EC) Connecting Europe Facility (CEF) has created an eID building block that provides a framework and a software platform for cross-border interoperability – eIDAS-Node. 2 As of 2020, most EU member states have already implemented eIDAS-Node in their national eID infrastructure. Although the eIDAS-Node software platform enables functionality for cross-border identification in EU digital public services, the accessibility to cross-border digital services under the eIDAS framework remains low. The implementation of the Single Digital Gateway Regulation (SDGR)3 foresees the increased use of electronic identification (eID) transactions across the EU (Kalvet et al., 2018). Therefore, it is necessary to specify what the cross-border eID infrastructure must provide to meet the needs and expectations of the Single Digital Gateway initiatives. During the research, we identify the main legal and technical barriers connected to identity management and service provision that prevent the cross-border use of digital public service procedures and provide a solution for changes that can fit in the current state of play. We aim to answer the following research questions: RQ 1. What are the key barriers that prevent seamless digital service delivery of (Estonian) public services in cross-border use cases by the means of EU member state notified eID? 1 eIDAS regulation. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv%3AOJ.L_.2014.257.01.0073.01.ENG 2 eIDAS-Node integration package. Available: https://ec.europa.eu/digital-building- blocks/sites/display/DIGITAL/eIDAS-Node+Integration+Package 3 SDGR regulation. Available: https://eur-lex.europa.eu/eli/reg/2018/1724/oj H. Raamat et al.: Digital Public Service Improvement in Cross-Border Use Cases 271 SRQ 1.1 How do the barriers affect the seamless delivery of (Estonian) digital public services in cross-border use cases on the legal, organizational, technical, and operational levels? RQ 2. How should the cross-border infrastructure be improved for seamless digital public service delivery? SRQ 2.1. What are the key requirements for successfully implementing a fully digital cross-border public service? The research is based on an Estonian case study. However, the outcomes of this study could be adopted as an example by other countries with similar e-government infrastructure. More specifical y, we focus on a cross-border scenario where an alien with an eID from one of the EU Member States wants to access one of the Estonian digital public service procedures. To map the current existing obstacles and a state of play, we rely on document analysis and semi-structured interviews conducted with digital public service providers in Estonia. As a result, we provide recommendations on how the barriers that affect cross-border digital public service delivery in Estonia could be overcome. This paper is organized as follows. In Sect. 2, we provide an overview of the EU's current interoperability framework. In Sect. 3, we present our research design and methodology. Sect. 4 gives an overview of the main research findings. In Sect. 5, we propose a solution for identity matching and make recommendations. Final y, we conclude the paper in Sect. 6 with the future research perspective. 2 Interoperability in the European Union In the need for specific common guidance on creating interoperable and high-quality digital public services, on 23 March 2017, the European Commission adopted the European Interoperability Framework (EIF) (Kalogirou & Charalabidis, 2019). The framework covers 12 underlying principles of European public services: subsidiarity and proportionality (1), openness (2), transparency (3), reusability (4), technological neutrality and data portability (5), user-centricity (6), inclusion and accessibility (7), security and privacy (8), multilingualism (9), administrative simplification (10), 272 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION preservation of information (11), assessment of effectiveness and efficiency (12) (European Commission, 2017). The EIF presents an interoperability model, where the interoperability is classified into four layers, containing legal, organizational, semantic, and technical interoperability (European Commission, 2017). The goal of adopting the eIDAS on the 23rd of July 2014 was to provide an EU- wide legal framework that enables secure and seamless electronic interactions between businesses, citizens, and public authorities (Lips et al., 2020). To support the interoperability of eIDs, the European Commission (EC) created the eID and eSignature building blocks to help member states’ public administrations and digital service providers extend the existing infrastructure for a secure cross-border service delivery. 4 The eIDAS regulation supports the secure mutual recognition of cross-border eIDs, which is backed with a respective framework and a technical system of eIDAS- Node. 5 The goal of the eIDAS-Node solution is to provide al Member States with an EU-compliant reference platform that enables interoperability between different eID protocols and standards. To establish cross-border recognition using eIDAS- Node software, the Member State must configure the software in its national infrastructure and implement an interface between the national eID ecosystem and the eIDAS network. eIDAS-Node supports two main cross-border scenarios: requesting and providing cross-border authentication. Figure 1 explains how the interoperability in the eIDAS Network is approached using the eIDAS-Nodes. The eIDAS-Node consists of three components: • eIDAS-Proxy-Service: a component that provides authentication data. • eIDAS-Connector: a component that requests cross-border authentication. • eIDAS-Middleware-Service: a component that provides authentication data and is being provided by the sending Member State and operated by a receiving member State. 6 4 https://ec.europa.eu/digital-building-blocks/sites/display/DIGITAL/ 5 https://digital-strategy.ec.europa.eu/en/policies/electronic-identification 6https://ec.europa.eu/digital-building-blocks/sites/display/DIGITAL/eIDAS-Node+version+2.0 H. Raamat et al.: Digital Public Service Improvement in Cross-Border Use Cases 273 Figure 1: The overview of the interoperability components in eIDAS Network Source: https://ec.europa.eu/digital-building-blocks/sites/display/DIGITAL/Proxy+to+proxy Estonia has implemented the eIDAS-compliant authentication gateway service (also known as TARA). 7, 8 Estonian eID ecosystem is described in detail in various other research papers (Lips et al., 2019; Saputro et al., 2020; Lips et al., 2023). The SDGR Article 14 sets requirements to establish a technical system for the cross-border automated data and evidence exchange between competent authorities in different member states and the application of the Once-Only Principle (OOP) for the online procedures. The Once Only Technical System is an EU-wide technical system currently under development, supervised, and provided by the European Commission. The OOP system aims to eliminate the administrative burden for citizens, public services, and businesses in the EU. It allows the sharing and reuse of data in real-time across borders, facilitating access to public cross-border online procedures and providing an automated exchange of evidence (Tepandi et al., 2021). 7 https://github.com/e-gov/eIDAS-SpecificProxyService 8 https://e-gov.github.io/TARA-Doku/TechnicalSpecification 274 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3 Research Design and Methodology The research fol ows the case study research methodology (Yin, 2018). We used multiple data sources during the research, including documents, archival records, and qualitative interviews (Creswell, 2016). We used descriptive case study methodology to understand the impact of the requirements on the implementation of cross-border digital services in compliance with SDGR and strengthen the existing theoretical knowledge. During the research, we mapped the current state of play of the cross-border service provision in Estonia and provided an improved process design based on the SDGR online procedures. To improve the validity of the outcomes in this study, we analyze the two observed cases using the cross-case synthesis method. The ideas for the proposed solution were based on the input from the qualitative data analysis, as wel as on the practical experience in the field of cross-border services and electronic identification. The research results were validated through the process design and official documentation. During the research, we conducted semi-structured interviews with the key stakeholders of the online electronic procedures specified in Annex II of the SDGR. We contacted 24 experts from 14 Estonian public sector bodies via e-mail and phone. In total, 14 interviewees responded with an interest in contributing. A list of the interviewees is provided in Table 1. The interview participants were selected based on the SDGR Annex II procedures and their relation to the service providers at the national level. During the data col ection phase, six expert interviews were conducted with various experts: four individual and two group interviews. The group interview form was chosen due to the data's richness and high quality (Flick, 2022). Although group interviews are typical y structured in their form, we chose to keep the semi- structured format throughout al the interviews as most of them were conducted in a semi-structured form. The interviews were conducted in Estonian and recorded using voice recording applications, Voice Memos by Apple, and Skype cal recording tool for transcript writing and further analysis. H. Raamat et al.: Digital Public Service Improvement in Cross-Border Use Cases 275 Table 1: List of interviewees Government body Interviewee Duration 2 experts from the Population Facts Ministry of the Interior 53:43 Department Estonian Social Insurance Board 1 expert from the Benefits Department 49:56 7 experts from the E-services and Estonian Road Administration 55:11 Information Technology Department Health and Welfare Information Systems architect Systems Centre 51:29 Estonian Tax and Customs Board 1 expert from the Tax Department 38:05 2 experts from the Public Services Estonian Tax and Customs Board 31:26 Department The data analysis in this research relies on three qualitative analysis techniques that were used to identify patterns, themes, and sequences of the data that had been previously col ected. Documenting at each step in the research analysis al ows trustworthy and valid conclusions to be achieved that explain the chain of evidence to the readers (Runeson et al., 2012). We use document analysis and thematic analysis following the guidelines provided by Braun and Clarke (Braun & Clarke, 2006). We also use triangulation of multiple data sources to strengthen the validity of the research results (Salkind, 2010). As a result, we propose a solution for improving the existing cross-border service provision process based on the theoretical framework and existing processes. We use the UML diagrams to interpret, assess, and validate the research results. 4 Research Findings During the thematic analysis, based on the qualitative interviews, we identified four fol owing themes: accessibility (A), data exchange (DE), identity management (IM), and interoperability (IO). These themes highlight the key findings of the current state of cross-border service provision based on the SDGR services in Estonia, the main obstacles, and the requirements for a successful cross-border service delivery. Figure 2 gives a detailed overview of the themes and codes. 276 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 2: Overview of the themes and codes Table 2: Key requirements for cross-border digital service provision ID Requirement Description Theme One of the most highlighted requirements by the current service provision concept is having the Estonian PIC. Without the latter, only a few digital public services are available for EU eID users. The data of the authenticated users is requested from and verified against the base registries, such as the Estonian population register, using the R-1 Existing X-Road data exchange platform. A, IO Estonian PIC9 It was also pointed out by more than half of the interviewees that currently, there are not many SDGR Annex II online procedures available in Estonia for EU eID users who do not have an existing event in the population register nor have issued an Estonian PIC. Existing events in the base Al the interviewees highlighted that one of the first and primary A, DE, R-2 registries for sources for the current identity mapping procedures relies on the data IM, IO identity requested from the Estonian population register. verification 9 Personal Identification Code H. Raamat et al.: Digital Public Service Improvement in Cross-Border Use Cases 277 ID Requirement Description Theme Since the X-Road, which already permits the automated exchange of evidence, has been widely adopted in e-Estonia, Estonian digital service providers would prefer using this platform for automated Unified cross- cross-border data exchange between EU countries. border platform Problem: The key requirement here is that there must be an existing R-3 for automated DE, IO record in the population register to perform a secure identity cross-border mapping procedure. Since no standardized identity mapping data exchange procedure is currently in use for the base registries, the data exchange of a new incoming EU eID user’s attributes is blocked over the X- Road, which directs us back to the accessibility issue. The main cross-border automated data exchange obstacle mentioned throughout the interviews was that many countries do not provide a joint identifier for natural persons, similar to Estonia. For example, DE, R-4 Persistent PIC each government authority assigns different identifiers to their across EU IM, IO citizens in Germany. That, in turn, brings along the mapping procedure. If the latter could be automatized, the cross-border data exchange could significantly raise the data quality. Since identity mapping highly relies on the existing records of a Automatized person in the base registries, the online verification of EU eID users can only be reliably performed with cross-border automated data R-5 identity IM mapping exchange. According to the interviewees, the central issue they hope procedure to resolve in the future is automating the identity mapping procedures. All the interviewees highlighted that one of the first and primary sources for the current identity mapping procedures relies on the data Standardized requested from the Estonian population register. It was pointed out R-6 identity that to automatize the process, there should be internationally agreed IM mapping standards for semantically describing the identities across registries. procedure Therefore, semantic interoperability could bring identity mapping to the next level. Sufficient The efficient and reliable identity mapping should highly rely on the provision of additional attributes that are sufficient and persistent as possible in R-7 attributes for time, including PIC, first name(s), family name(s), date of birth, IM identity country (as a prefix), current address, gender, nationality, place of mapping birth. Semantically Semantically interoperable attributes are one of the key enablers for R-8 interoperable IM seamless data exchange and data management. attributes Based on the interview results, we created a list of high-level requirements for cross- border digital service provision. Table 2 summarizes the key requirements and related themes based on the case of Estonia. 278 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The documentation and thematic analysis identified a common issue of the current cross-border service provision concept across al the life events of the SDGR analysed in this research – a lack of common understanding and non-existent standards for identity management. Moreover, we identified the fol owing barriers and factors that affect the cross-border interoperability of public services: 1) The complexity of the eID notification process slows down the recognition and, thereby, the accessibility to digital services and procedures for EU citizens. 2) The lack of unique and persistent identifiers on the access to digital services. 3) Due to the limitations in national policies and law in many EU countries, the cross-border exchange of data and evidence a citizen can be limited, which affects the overal success of the EU-wide adoption of the eIDAS and OOP in SDGR. 4) The lack of clear and standardized interoperability profile and reliable identity attributes in the EU on how to semantical y describe the identities across registries. 5) Missing standardized and automatized approach on identity matching at the EU and national level. Since identity matching relies on the existing records of a person in the base registries, the verification of EU eID users cannot be reliably performed without cross-border automated data exchange. 6) Different levels of assurance of eID means in the EU affect the availability of cross-border services. 7) The e-government systems and national service providers cannot handle the format of foreign identifiers. Therefore, the national identifier (PIC in Estonia) is a prerequisite for access to public services. 5 Proposed Solution for Identity Matching and Recommendations Based on the key requirements presented in Table 2 and analysing two Estonian cross-border evidence exchange cases, requesting proof of residence and requesting proof of registration of birth, we propose a solution for identity matching presented in Figure 3. H. Raamat et al.: Digital Public Service Improvement in Cross-Border Use Cases 279 Figure 3: Proposed solution for authentication of EU citizens Source: Own The cross-border availability of attributes highly affects the reliability of identity- matching mechanisms. In the UML scheme, an assumption is made that when a matching identity has been found, an existing Estonian PIC is already assigned to the EU citizen. The centralized approach to an identity-matching system is recommended to reduce the burden for the service providers and increase the quality and reliability of the identity-matching system. Assigning a national identifier, such as Estonian PIC, to the eIDAS eID users helps provide a seamless user experience and enables automatic enrolment. In order to request additional evidence and attributes from authoritative sources in cross-border use case scenarios, the OOP technical system and eIDAS-Node could facilitate access to cross-border data. As the eIDAS regulation is currently under revision, we make the following suggestions: 1) The expansion of the mandatory eIDAS minimum data set attributes to improve reliable matching of identities. Ideally, the mandatory data set should consist of the attributes that are sufficient and persistent as possible in time, including PIC, first name(s), family name(s), date of birth, country (as a prefix), current address, gender, nationality, place of birth. 2) The eIDAS eID notification procedure should emphasize the importance of unique identity attributes for cross-border use and, where possible, the unique identifier should ideally be the same for digital and physical eID to improve reliability. 3) Al EU countries should consider notifying at least one eID scheme that meets the highest level of assurance to improve the accessibility and availability of cross-border public services. 280 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Reusing the attribute information from base registries is essential for efficient and user-centric cross-border service delivery. Technical, legal, and semantical aspects must support the exchange of cross-border attributes. The centralized approach to the identity matching system is recommended to reduce the burden for the service providers and increase the quality and reliability of the identity matching system. As presented in the cross-case synthesis, the Estonian PIC can be used as a workaround for enabling access to foreign eID to Estonian public services and online procedures. Assigning a national identifier, such as Estonian PIC, to the eIDAS eID users helps provide a seamless user experience and enables automatic enrolment. However, some limitations apply to issuing Estonian PIC to foreign identities. The specific structure of Estonian PIC includes information about a person that cannot always be provided with the current minimum data set of eIDAS eID, such as gender. However, it can be retrieved if the eID country provides the additional attributes. 1 6 Limitations and Future Work The main legal and organizational barriers identified in this study refer to the limitations in national policies and law in many EU countries, where the cross- border exchange of identity attributes remains limited or low. This affects the overal success of the EU-wide adoption of the eIDAS and OOP in SDGR. As a result of the research, we propose a possible solution for identity matching based on the Estonian example and further recommendations. However, it is essential to continue the research and develop the proposed identity matching framework further, especially in the context of eIDAS 2.0, which will be adopted and implemented very soon. 10 It is also possible to continue the research at the more theoretical level. Since the SDGR implementing regulation has yet to be adopted and the OOP technical system has yet to be released, the author sees a further need to analyse the cross-border service provision improvement in Estonia. In recent years, the demand for cross-border services has increased in the EU and outside the EU borders. Therefore, the analysis of the EU interoperability frameworks, such as 10 eIDAS 2.0. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0281 H. Raamat et al.: Digital Public Service Improvement in Cross-Border Use Cases 281 eIDAS and SDGR, could be extended and analysed in the context of third countries. 7 Conclusion This research has identified that cross-border digital service accessibility highly relies on the following factors: 1) secure identification, 2) cross-border functional and secure attribute exchange, 3) automatized identity matching based on sufficient attributes, and 4) cross-border evidence exchange for specific procedures based on the OOP technical system. To sum up, we visualized the research findings and answers to the research questions in Figure 4. Figure 4: Summary of the research findings and answers to the research questions Source: Own 282 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION References Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2), 77-101. Creswell, J. W., & Poth, C. N. (2016). Qualitative inquiry and research design: Choosing among five approaches. Sage publications. European Commission. Directorate General for Informatics. (2017). New European interoperability framework: promoting seamless services and data flows for European public administrations. Publications Office. https://doi.org/10.2799/78681 European Commission. (2021). Impact Assessment Report Accompanying the document Proposal for a Regulation of the European Parliament and of the Council amending Regulation (EU) n° 910/2014 as regards establishing a framework for a European Digital Identity. European Commission. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021SC0124 Flick, U. (2022). An introduction to qualitative research. An introduction to qualitative research, 1100. Kalogirou, V., & Charalabidis, Y. (2019). The European union landscape on interoperability standardisation: status of European and national interoperability frameworks. In Enterprise Interoperability VIII: Smart Services and Business Impact of Enterprise Interoperability (pp. 359-368). Springer International Publishing. Kalvet, T., Toots, M., van Veenstra, A. F., & Krimmer, R. (2018, April). Cross-border e-government services in Europe: expected benefits, barriers and drivers of the once-only principle. In Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance (pp. 69-72). Lips, S., Aas, K., Pappel, I., & Draheim, D. (2019). Designing an Effective Long-Term Identity Management Strategy for a Mature e-State. In Electronic Government and the Information Systems Perspective (pp. 221–234). Springer International Publishing. https://doi.org/10.1007/978-3-030-27523-5_16 Lips, S., Bharosa, N., & Draheim, D. (2020, November). eIDAS implementation chal enges: the case of Estonia and the Netherlands. In International conference on electronic governance and open society: chal enges in Eurasia (pp. 75-89). Cham: Springer International Publishing. Lips, S., Tsap, V., Bharosa, N., Krimmer, R., Tammet, T., & Draheim, D. (2023). Management of National eID Infrastructure as a State-Critical Asset and Public-private Partnership: Learning from the Case of Estonia. Information Systems Frontiers, 1-18. Runeson, P., Host, M., Rainer, A., & Regnel , B. (2012). Case study research in software engineering: Guidelines and examples. John Wiley & Sons. Salkind, N. J. (Ed.). (2010). Encyclopedia of research design (Vol. 1). sage. Saputro, R., Pappel, I., Vainsalu, H., Lips, S., & Draheim, D. (2020, April). Prerequisites for the adoption of the X-Road interoperability and data exchange framework: a comparative study. In 2020 Seventh International Conference on eDemocracy & eGovernment (ICEDEG) (pp. 216-222). IEEE. Tepandi, J., Rotuna, C., Sellitto, G. P., Fieten, S., & Prentza, A. (2021). The Technical Challenges in OOP Application Across the European Union and the TOOP OOP Architecture. In The Once-Only Principle: The TOOP Project (pp. 141-163). Cham: Springer International Publishing. Yin, R. K. (2018). Case study research and Applications Design and methods. DESIGNING AND EVALUATING AN LLM-BASED HEALTH AI RESEARCH ASSISTANT FOR HYPERTENSION SELF-MANAGEMENT; USING HEALTH CLAIMS METADATA CRITERIA LUUK P.A. SIMONS, PRADEEP K. MURUKANNAIAH, MARK A. NEERINCX Delft University of Technology, Faculty of EEMCS, Delft, Netherland l.p.a.simons@tudelft.nl, P.K.Murukannaiah@tudelft.nl, M.A.Neerincx@tudelft.nl Hypertension is a condition affecting most people over 45 years old. Health Self-Management offers many opportunities for prevention and cure. However, most scientific health literature is unknown by health professionals and/or patients. Per year about 200.000 new scientific papers on cardiovascular health appear, which is too much for a human to read. Hence, an LLM-based Keywords: Health AI research assistant is developed for mining scientific hypertension, literature on blood pressure and food. A user evaluation was Self-Management support, conducted with n=8 participants who just completed an decision intensive lifestyle intervention for blood pressure self- support, management. They highlighted several chal enges and eHea lth, AI, opportunities for a Health AI, especially regarding claim LLM, transparency, data quality and risks of hallucinations. In the claims discussion we propose seven criteria using metadata and analysis, metadata, information characteristics to help evaluate ambiguous or public conflicting health science claims. health DOI https://doi.org/10.18690/um.fov.4.2024.16 ISBN 978-961-286-871-0 284 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction When searching Google Scholar for ‘cardiovascular health’ more than 6.000.000 studies show up. In the years 2023 and 2022 the number of scientific publications on this topic referenced by Google Scholar are 249.000 and 307.000 respectively, per year. These numbers are staggering and impossible to keep up with for human intel igence. Hence, we are looking at AI (Artificial Intel igence) tools for assistance. Our research goal is to design a Health Research Assistant AI tool (or short “Health AI”) based on LLMs (Large Language Models) with added metadata analysis tools. The use case domain for this paper is ‘hypertension and food.’ One the one hand, hypertension affects roughly half the people above the age of 45 years in the developed world (Ostchega, 2020, Zhou, 2021, Carey & Whelton, 2020), but hypertension can be significantly reduced with healthy food choices (Roberts & Barnard, 2005, Franzini, 2012, Rodriguez-Leyva, 2013, Kapil, 2015, Dickinson, 2014, Siervo, 2015). On the other hand, we had the opportunity to conduct a Health AI user evaluation with a group of participants who had just completed the first two intensive weeks of a hypertension health behaviour self-management Chal enge, giving them ample experience with hypertension information needs and various alternative sources of information, for us to investigate the added value of a Health AI concept. Since this is a design paper, we first col ect user feedback on a Health AI concept (section 4. Results), fol owed by a suggestion for seven metadata criteria that may be used for additional tools to help evaluate conflicting claims (section 5. Discussion). Hence the focus of the user evaluation in this paper is: ‘For users with hypertension health self-management experience, what are perceived usefulness and intention to use for a Health AI, compared to their other health information sources?’ The Research Questions for the user evaluation are: 1. In users’ solution space, what are their information needs and priorities? What would they most want to ask the Health AI tool? 2. How do they use and value other information sources (besides the Health AI)? 3. What is their ‘Technology Ac eptance’ evaluation and intention to use the Health AI? L. P.A. Simons, P. K. Murukannaiah, M. A. Neerincx: Designing and Evaluating an LLM-based Health AI Research Assistant for Hypertension Self-Management; Using Health Claims Metadata 285 Criteria 2 Related work In this section we briefly address three topics: LLMs for health, claims analysis in AI (Guo, 2022), and the role of competing alternatives when designing and evaluating the added value of new tools. In recent years multiple papers have been published on using LLMs for healthcare. Some using a review on opportunities and risks from mostly editorials (Sallam, 2023) or testing several use cases with health professionals (Cascella, 2023) or interviewing health professionals versus surveying the general public on their ChatGPT use in health (Raina, 2024). Some of the benefits that are relevant to our research question and general y mentioned in these papers are: utility in health research and benefits for health care practice (improving health literacy and efficiency in reviewing the literature). Risks that are often mentioned are: lack of transparency, risk of bias, incorrect citations, and risk of hal ucinations. Other papers focusing more on the technology address privacy, security, and data architecture issues (Montagna, 2023) or training and evaluating specialised LLMs to increase natural language qualities like perceived helpfulness, logic and empathic phrasing (Lai, 2023). Overall, given that LLMs can be described as ‘probable-word generators’ (Shah, 2023), it is not so surprising that health care professionals describe their capabilities as lacking depth and argumentation in health expertise and lacking understanding of complex relationships between personal-, health- and behaviour-aspects (Raina, 2024). However, we hold the view that from a technology perspective it is not enough to explicate risks of misinformation or lack of transparency of health claims. We must also think about the next steps forward: How to design and enhance generative AI tools such that these risks can be better managed? For example, when faced with conflicting claims from literature, it is not enough to just be transparent about the references used. To aid the user groups (see Method) we need to use metadata and develop additional tools that explicate how various sources and their claims can be weighed against each other. While using those tools, interpretation by human domain experts may likely be useful, hence creating a ‘hybrid intelligence’ (Simons, 2021, 2022a) combining the strengths of artificial and human intelligence. The specific domain of food and health has many conflicting claims (and conflicting 286 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION interests of scientific authors). So, an important question is how to use metadata, information characteristics and assessment criteria to help evaluate claims. The task of analysing claims is studied under the umbrella of automated fact checking (Guo, 2022) in the AI (specifically, Natural Language Processing) literature. Automated fact checking typically involves four subtasks: (1) Claim detection involves identifying claims for verification. An important aspect here is identifying claims that are check-worthy (i.e., claims whose truthfulness the public is interested in). (2) Evidence retrieval involves retrieving information which can be used to evaluate the veracity of the claim. (3) Verdict prediction involves determining the veracity of the claim by synthesising the pieces of evidence retrieved. (4) Justification production involves generating a justification for why a certain claim was ruled true or not true (or somewhere in between). This is an important and chal enging task, considering the black-box nature of the AI tools. The main chal enge for us is to formulate these tasks for the domain of our interest in a systematic manner. Finally, we must borrow some value evaluation principles from the field of new product design. This paper reports on a user evaluation of a Health AI concept (see its description in section 3. Methods and Materials). Besides the general frameworks of TAM (Technology Acceptance Model, Venkatesh, 2000) and UTAUT (Unified Theory of Acceptance and Use of Technology, Venkatesh, 2003) looking at concepts like perceived usefulness and ease of use, product design aims to specify and design these qualities in detail (Rondini, 2016). Moreover, the added value of those new qualities should also be considered in comparison to competing alternatives (Herzwurm & Shockert, 2003, Rondini, 2016). Hence, in our user evaluation, besides asking feedback on perceived usefulness of various Health AI functions, we will also ask which other information sources are used and/or preferred. Previous research gave some indications for two alternative hypertension information sources (Simons, 2021). Firstly, when using Google Scholar to search on health interventions or disease causes, the results listed are overspecialized and too diverse. Plus, they are not very user-centred or action-oriented for health self-management. Secondly, when visiting sites of main health institutes, the latest science cannot be found and health advise is ‘watered down’ to not scare away the less health-conscious target groups (Simons, 2021, 2022a). L. P.A. Simons, P. K. Murukannaiah, M. A. Neerincx: Designing and Evaluating an LLM-based Health AI Research Assistant for Hypertension Self-Management; Using Health Claims Metadata 287 Criteria 3 Methods and Materials Research Design: In a design research approach (Vaishnavi & Kuechler, 2004, Verschuren & Hartog, 2005) we developed a high-level Health AI concept (see below) and col ected feedback from n = 8 participants who had very recent experience with a healthy lifestyle intervention to reduce their hypertension, see section 4, Results. We used a mixed method approach for col ecting user inputs and design suggestions: quantified surveys (see below), open questions and action research (during the intervention as wel as the user evaluation) in the sense that we have a high level of 'access' to participants1 while at the same time helping them navigate the information diversity they encounter. The user evaluations showed support needs for interpreting (sometimes ambiguous) claims. In section 5, Discussion, we propose several AI tooling options to support these user needs. The Health AI concept: • The initial scope is limited to the domain of food and blood pressure. • Its training will include all (>100.000) recent scientific publications in this domain. • It’s precise details are yet to be determined, but its base is: • For the user, the Health AI resembles ChatGPT, Bing and Bard, with the addition that it is specifically trained to help interpret recent studies. • It works with questions and answers in plain Dutch. You can ask fol ow up questions on previous answers. You can see source publications which were used for the answers. Participants: In the first week of February 2024, we col ected feedback from n=8 Dutch participants which had started on January 15th with an intensive healthy lifestyle intervention to reduce their hypertension. All of them provided written consent. Details of this intervention are described elsewhere (Simons, 2022b, 2023a, 2024). Similar to previously published results, average blood pressures were successfully reduced from 140/87 to 122/77 in 12 days, thanks to many food- and other lifestyle improvements. The participants were al university employees: two of 1 By supporting individuals during hypertension lifestyle interventions, as wel as providing 6 months of healthy lifestyle coaching (Simons et al., 2010, 2017) for literally thousands of participants and caregivers in these domains, over the course of the past 10 years. 288 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION them scientists and six were supporting personnel. Half were male, half female. Their average age was 45, ranging from 29 to 58. All of them had experience with LLMs: two limited, three average and three of them a lot. All of them used multiple sources of information during the hypertension chal enge. User evaluation & data analysis: When going through health improvement iterations, participants cycle through three design spaces: ‘problem’-, ‘solution’- and ‘evaluation space’ (Simons, 2023b). This evaluation focusses on information usage for the ‘solution space’ (What are my most effective and attractive health behaviour options?) and the potential added value of the Health AI. In Table 1 the evaluation topics are listed. As explained in section 2, Related work, we need insights into their general information preferences (topic 1), which functionalities and support they would prefer from the Health AI (topic 2), their use of information sources during lifestyle changes, which are potential y competing options for the Health AI tool (topic 3), plus technology acceptance feedback (topic 4). For the latter, we used TAM (Venkatesh, 2000) and UTAUT (Venkatesh, 2003) for user evaluation concepts (perceived usefulness, ease-of-use, ability, trust, feeling, support, intention to use), with a focus more towards individual usage preferences, than on UTAUT’s organisational technology adoption processes (Carlsson 2006). Regarding data collection and analysis, for each topic we used questionnaire items for quantified evaluation. We also asked users for additional inputs per topic on what they valued and why, given our design evaluation focus. Table 1: User Evaluation topics Topics information use and Health AI added value 1. Information usefulness, in general 2. 'Voice of the user' Health AI preferences 3. Use of other information sources (during self-management) 4. Technology acceptance’ aspects for the Health AI L. P.A. Simons, P. K. Murukannaiah, M. A. Neerincx: Designing and Evaluating an LLM-based Health AI Research Assistant for Hypertension Self-Management; Using Health Claims Metadata 289 Criteria 4 Results from the User Evaluation on the Health AI tool concept To start our results section, we list the Research Questions (RQs) and the Tables containing the user evaluation summaries: 1. In users’ solution space, what are their information needs and priorities? What would they most want to ask the Health AI tool? (Table 2) 2. How do they use and value other information sources (besides the Health AI)? (Table 3) 3. What is their ‘Technology Ac eptance’ evaluation and intention to use the Health AI? (Table 4) In Table 2, addressing the first RQ, we list user responses for information usefulness (7-point Likert scale). The first part of the table addresses their general information opinions, whereas the second part specifical y addresses Health AI tool usefulness. We label ed the top 3 highest scores with green, for each question set, to highlight user preferences. Regarding general information usefulness, from the top 3 we can see that participants’ priorities are learning which health behaviours help best for health/hypertension and how to make those changes easy. Question 1 was exceptional in that it gained maximum scores from everyone. Other useful information sources mentioned (part 1 of Table 2, open question): • The conversations with the coach were most useful. I would hope the AI could have a similar conversation with us. • The context given during the Chal enge in relation to healthy choices was very useful, like for example “how sugar- and saturated-fat-spikes heighten artery systemic inflammation”. • During the Chal enge workshops we heard many things that you would never think of yourself, like for example the blood pressure lowering effect of seeds like flaxseeds. • I was happy to hear about the updated hypertension guidelines from the AHA (American Heart Association), this is new for the Dutch context and I will include this in my conversations with my family physician. • It’s nice to see food intervention studies and effect sizes on hypertension. 290 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 2: Information use & Health AI preferences (7-point (dis)agree, n=8, Avg=Average) I find the fol owing (general) information useful: Avg Score 1. Connections between blood pressure, health and behaviour 7.0 2. Most effective behaviour changes for hypertension 6.4 3. Knowing blood pressure effect sizes of behaviour changes 6.0 4. Tips for making behaviour changes easy 6.6 5. Tips for making behaviour changes successful 6.1 The Health AI tool would be useful for: 1. Comparing blood pressure effects of foods 5.9 2. Getting health feedback on a specific (supermarket) product 5.8 3. Learning the optimum dosage of a food product 5.0 4. Learning the broader health effects of a food 6.0 5. Comparing effect sizes of foods with other health behaviours 4.9 6. Practical tips on how to increase daily intake of health foods 5.8 7. Tips how to replace or avoid unhealthy foods 6.1 8. Tips how to deal with pitfalls/difficult moments 5.8 Part two of Table 2 shows the most useful applications of the Health AI tool, in the opinion of the participants. The top 3 scores are for learning blood pressure effects and broader health effects of food, plus practical tips on avoiding unhealthy foods. Just below the top 3 are three items each scoring 5.8 which al have a practical focus: daily eating patterns for increasing healthy foods, for dealing with pitfalls/difficult moments and aiding healthy choices when buying products in the supermarket. Interestingly, opinions varied on the practical advice items: some participants prefer to hear those practical tips from other participants (including usage/adoption context). As one of them stated: “By interacting with the others about what works and why, our conversations real y are part of our usage intention. The purpose is applying things yourself. Hence, the conversation is part of your own behaviour change , instead of just information gathering.” Stil , others prefer the AI tool for practicality, versus preferring the coach for learning ‘the bigger health picture and its relevant connections’. L. P.A. Simons, P. K. Murukannaiah, M. A. Neerincx: Designing and Evaluating an LLM-based Health AI Research Assistant for Hypertension Self-Management; Using Health Claims Metadata 291 Criteria Other Health AI usefulness mentioned (part 2 of Table 2, open question): • It would be nice if the Health AI could filter information based on aspects like gender, age, weight, sports background, vegetarianism, etc, to increase relevance for my own situation. • I would like to input my existing breakfast etc (which I like) and ask for health improvement suggestions. • If certain foods are useful for my blood pressure, please show me the links to the original studies, so I can read them for myself. (See also Table 4) • If the blood pressure food advice is distinct from the advice from my dietitian or weight watchers, can the Health AI explain why this may be so? • I want to ask questions on other topics like aspirin or sauna: do they also influence my blood pressure? In answer to RQ 2, Table 3 lists the extent of information source usage by participants during the Chal enge period of two weeks. These can be viewed as alternative sources, potential y ‘competing’ with the Health AI we plan to introduce. Everybody indicated having used the coach inputs regularly and all participants except one said that the inputs from other participants were useful. The third most used source was formed by official health institutes. Regarding the fourth (= personal network), most indicated that this was more about bringing/sharing than about getting information; although information inputs were received from their network on the practicalities of implementing healthy lifestyle behaviours. Other Internet sources were explicitly label ed by most as containing too much confusing or low-quality information. When asked what information was most useful (open question) al participants said that the Chal enge workshops were most useful (including materials, PowerPoints, references, online portal with health information, plus the explanations provided). Reasons stated: provided a good summary; value of the practical tips; a mirror to my own behaviours; the specific links and literature created a focused way for me to fol ow up on information; the summary and tips are saving me time; I don’t feel the need to do my own research because this was good enough for me. 292 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 3: Use of other information sources (Number of times, n=8, Avg Nr=Average Number of times) Number of times during Chal enge (of 2 weeks) Avg Nr 1. My personal network (family, friends, etc) 2 1.7 2. My physician or other health professionals 0.4 3. Sites/info from official health institutes 2.3 4. Other Internet sources 0.5 5. Google Scholar, PubMed or similar 0.3 6. Individual scientific papers 0.9 7. Inputs/remarks from other Chal enge participants 5.8 8. Inputs from Chal enge coach 7.6 Regarding Research Question 3, Table 4 shows answers to various Technology Acceptance aspects. Since three of the highest scoring items have the same score (6.1) we label ed a top four of items green. From these it can be seen that on the one hand the Health AI is found interesting and there is an intention to use it. On the other hand it was clear (from items 2, 4 and 6, as wel from the open answers) that al participants were wary about the risk of receiving unreliable answers from LLM tools like the Health AI. This is expressed in two of the top four items: 5. (‘it will gain my trust, following the degree of clarity of its sources’) and 7. (‘I find it useful to discuss the Health AI outputs with the coach’). Being able to second-guess and interpret Health AI answers, especial y using a human expert and hence creating a form of ‘hybrid intelligence’, is deemed a valuable way to use the Health AI. For this goal, interpretations of other participants (which have less expert knowledge in this domain) were deemed less useful. On a practical level of anticipated future Health AI use, preferences varied (in line with the variation in Table 2 answers): (1) some would prefer to get an introduction and practice session on how to (not) use it, whereas others would prefer to use it on their own, 2 One of the participants was an outlier with score 15, hence excluded from this item average. Moreover, al participants said it was more about sharing information than receiving information, except for practical tips/discussions on how to implement health behaviours. L. P.A. Simons, P. K. Murukannaiah, M. A. Neerincx: Designing and Evaluating an LLM-based Health AI Research Assistant for Hypertension Self-Management; Using Health Claims Metadata 293 Criteria (2) some would like to be able to ask al the health, food and blood pressure questions they also asked in the workshops, others would focus on science mining, and stil others would mainly want the Health AI for practical tips on daily health patterns (while bouncing these suggestions off others during workshop sessions). Table 4: Technology acceptance factors (7-point (dis)agree, n=8, Avg=Average) The Health AI tool..: Avg Score 1. is interesting 6.1 2. is useful for insights on improving my health 5.5 3. is easy to use for asking questions 6.0 4. is easy to interpret when presenting conflicting articles 4.9 5. wil gain my trust, fol owing the degree of clarity of its sources 6.1 6. I find it useful to discuss its outputs with other Chal enge 5.5 participants 7. I find it useful to discuss its outputs with the Chal enge coach 6.5 8. I find it useful to practice its use in Chal enge workshops 5.8 9. I would certainly use the Health AI 6.1 5 Discussion & AI Tooling implications A first limitation of this study is its explorative nature, with only n=8 participants. Still, for reaching input saturation at this design stage this appears sufficient; sometimes even five, six or seven users are enough (Faulkner, 2003). Second, the Health AI is only evaluated in concept. A next step in our research is to test a real prototype. Still, also on a concept level, user inputs are useful especially given their recent experience in dealing with ambiguous or conflicting claims from food and hypertension literature. Below we explore seven claim evaluation options, where metadata analysis tools can aid interpretation. Criteria and information characteristics to evaluate ambiguous claims: Besides analysing claims structures in scientific literature, there are other metadata that can be used to help evaluate the reliability of claims. Table 5 lists several criteria against which (possibly contradicting) claims be evaluated. 294 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 5: Claims evaluation criteria Evaluation criteria & interpretation examples from literature: 1. Time evolution of claims: Tools that show claim changes over time can be useful. Dr Neal Barnard (2018) eloquently explains how claims on cardiac health of eggs have (incorrectly) become more positive in the past decades, exactly because the previous decades had been so complete and conclusive about the negative cardiac health effects. In short, 'serious research' moved elsewhere, leaving a void fil ed by the egg industry to create 'recent studies' with doubtful claims. 2. Body of evidence: As one of the most-cited scientists in the domain of health behaviours and health risks shows in an overview article (Wil ett, 2012), it is important to assess the extent of a body of evidence. For over a century, causal relationships between saturated fats, blood cholesterol and CVD (CardioVascular Disease) have been show by a broad array of (large scale) studies: from animal studies to prospective human migration studies across the world and large human RCTs (Randomized Controlled Trials). 3. Consistency of claims: An example of consistency is provided by al the studies showing health benefits of fruits and vegetables. Consistency in those findings is enormous. Stil , some people (intervention participants, Internet sources, or sometimes even dietitians) discourage consuming more than two portions of fruits per day, claiming that their sugar content is bad for you. But even though refined sugars may be bad for you, when focusing on the overarching claims consistency (from studies) on fruits and their health effects, the consistency of positive health effects is clear and should prevail. 4. Burden of proof: Sometimes new claims go against ‘prevailing wisdom.’ This can either be a knowledge breakthrough (see next criterium) or a mistake. A famous example of the latter is the tobacco industry arguing that smoking is healthy since it reduces Parkinson’s disease (Greger & Stone, 2016, p.265). Even though it is true that tobacco (and tomato) plants contain substances that infer Parkinson’s protection, this is not enough. Burden of proof says that if there is massive proof pointing left (smoking kil s you), then you need to carry a very heavy burden of proof for the opposite (smoking is healthy). 5. Explicit arguments and proof for conflicting claims: If you introduce a claim that goes against an existing Body of evidence, the Burden of proof is on you to give an explicit argument and/or proof why the new claim is valid, in the face of al other evidence. Soy health for humans is an example of this. In the past, we were faced with multiple animal studies showing cancer risks from high volume soy consumption (even though this was inconsistent with Asian populations consuming a lot of soy in good health). Final y, studies showed that the rodents in those animal studies metabolize soy differently from humans, and explained how the previous conflict in claims was resolved (Setchel , 2011). L. P.A. Simons, P. K. Murukannaiah, M. A. Neerincx: Designing and Evaluating an LLM-based Health AI Research Assistant for Hypertension Self-Management; Using Health Claims Metadata 295 Criteria Evaluation criteria & interpretation examples from literature: 6. Weighing claims for type of study: The previous soy example also il ustrates an important fact in health: claims from a large scale, double-blind RCT in people carry much more evidence than animal studies (or observation studies). Even if this is obvious for some of us, it is helpful if a Health AI clarifies and uses this. 7. Claimer & industry af iliation analysis: Especial y in the food sciences it is scary to see how many studies and scientists have industry affiliations and conflicting interests. For example, even in the US Dietary Guidelines Advisory Committee, where objectivity should be a priority, it turns out that 19 out of 20 members have clear industry affiliations and conflicting interests (Mialon, 2022). Hence, a metadata analysis on claimer identity & industry affiliations can provide useful insights to claim validity. Hybrid intel igence for ambiguity ‘Rationale capturing’: In conclusion, our user evaluation confirmed the importance of information quality and science for healthy lifestyle choices. Especially regarding ambiguous or conflicting claims, participants expressed concern. They said they real y valued support for interpreting those claims and al of them wanted to consult a human expert. Given the explanations we heard from these users, this finding appears to have external validity for other health topics as wel . In summary, using expert opinion to provide a ‘rationale’ behind confusing claims is deemed very valuable. This helps answer the user requirements: ‘How to interpret claims?’ And ‘Is there an underlying story to explain the ambiguity?’ Hence a ‘hybrid intel igence' solution appears useful. In this paradigm the AI tools help reduce the information overload on experts, but the final advice is based on human (expert) explanation for the main user questions on claims confusion in the food and hypertension domain. Acknowledgements This research was (partly) funded by the https://www.hybrid-intel igence-centre.nl/ a 10-year programme funded the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, grant number 024.004.022 and by EU H2020 ICT48 project ``Humane AI Net'' under contract $\# $ 952026. 296 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION References Barnard R. J. (2018). How the Egg Industry Skews Science. YouTube, accessed 26-2-2024: https://www.youtube.com/watch?v=FyG8wr0gWIA. Carey, R. M., & Whelton, P. K. (2020). Evidence for the universal blood pressure goal of< 130/80 mm Hg is strong: controversies in hypertension-pro side of the argument. Hypertension, 76(5), 1384-1390. Carlsson, C., Carlsson, J., Hyvonen, K., Puhakainen, J., & Walden, P. (2006). Adoption of mobile devices/services—searching for answers with the UTAUT. In System Sciences, 2006. HICSS'06. Proceedings of the 39th Annual Hawai International Conference on (Vol. 6, pp. 132a-132a). IEEE. Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023). Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. Journal of Medical Systems, 47(1), 33. Dickinson, K. M., Clifton, P. M., Burrel , L. M., Barrett, P. H. R., & Keogh, J. B. (2014). Postprandial effects of a high salt meal on serum sodium, arterial stiffness, markers of nitric oxide production and markers of endothelial function. Atherosclerosis, 232(1), 211-216. Franzini, L., Ardigo, D., Valtuena, S., Pel egrini, N., Del Rio, D., Bianchi, M. A., . . & Zavaroni, I. (2012). Food selection based on high total antioxidant capacity improves endothelial function in a low cardiovascular risk population. Nutrition, Metabolism and Cardiovascular Diseases, 22(1), 50-57. Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35, 379-383. Greger, M., & Stone, G. (2016). How not to die: discover the foods scientifical y proven to prevent and reverse disease. Pan Macmillan, New York City. Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022). A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10, 178-206. Herzwurm, G., & Schockert, S. (2003). The leading edge in QFD for software and electronic business. International Journal of Quality & Reliability Management, 20(1), 36-55. Kapil, V., Khambata, R. S., Robertson, A., Caulfield, M. J., & Ahluwalia, A. (2015). Dietary nitrate provides sustained blood pressure lowering in hypertensive patients: a randomized, phase 2, double-blind, placebo-control ed study. Hypertension, 65(2), 320-327. Lai, T., Shi, Y., Du, Z., Wu, J., Fu, K., Dou, Y., & Wang, Z. (2023). Psy-llm: Scaling up global mental health psychological services with ai-based large language models. arXiv preprint arXiv:2307.11991. Mialon, M., Serodio, P., Crosbie, E., Teicholz, N., Naik, A., & Carriedo, A. (2022). Conflicts of interest for members of the US 2020 Dietary Guidelines Advisory Committee. Public Health Nutrition, 1-28. Montagna, S., Ferretti, S., Klopfenstein, L. C., Florio, A., & Pengo, M. F. (2023). Data decentralisation of LLM-based chatbot systems in chronic disease self-management. In Proceedings of the 2023 ACM Conference on Information Technology for Social Good (pp. 205-212). Ostchega, Y., Fryar, C. D., Nwankwo, T., & Nguyen, D. T. (2020). Hypertension prevalence among adults aged 18 and over: United States, 2017–2018, NCHS, National Health and Nutrition Examination Survey, 2017–2018: https://stacks.cdc.gov/view/cdc/87559. Raina, A., Mishra, P., & Kumar, D. (2024). AI as a Medical Al y: Evaluating ChatGPT's Usage and Impact in Indian Healthcare. arXiv preprint arXiv:2401.15605. Roberts, C. K., & Barnard, R. J. (2005). Effects of exercise and diet on chronic disease. Journal of applied physiology, 98(1), 3-30. Rodriguez-Leyva, D., Weighel , W., Edel, A. L., LaVal ee, R., Dibrov, E., Pinneker, R., . . & Pierce, G. N. (2013). Potent antihypertensive action of dietary flaxseed in hypertensive patients. Hypertension, 62(6), 1081-1089. L. P.A. Simons, P. K. Murukannaiah, M. A. Neerincx: Designing and Evaluating an LLM-based Health AI Research Assistant for Hypertension Self-Management; Using Health Claims Metadata 297 Criteria Rondini, A., Pezzotta, G., Pirola, F., Rossi, M., & Pina, P. (2016). How to design and evaluate early PSS concepts: the Product Service Concept Tree. Procedia CIRP, 50, 366-371. Sal am, M. (2023). ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare MDPI, Vol. 11, No. 6, p. 887. Setchel , K. D., Brown, N. M., Zhao, X., Lindley, S. L., Heubi, J. E., King, E. C., & Messina, M. J. (2011). Soy isoflavone phase II metabolism differs between rodents and humans: implications for the effect on breast cancer risk. The American journal of clinical nutrition, 94(5), 1284-1294. Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023). Creation and adoption of large language models in medicine. Jama, 330(9), 866-869. Siervo, M., Lara, J., Chowdhury, S., Ashor, A., Oggioni, C., & Mathers, J. C. (2015). Effects of the Dietary Approach to Stop Hypertension (DASH) diet on cardiovascular risk factors: a systematic review and meta-analysis. British Journal of Nutrition, 113(1), 1-15. Simons, LPA, Neerincx MA, Jonker CM (2021). Health Literature Hybrid AI for Health Improvement; A Design Analysis for Diabetes & Hypertension, pp. 184-197, 34th Bled eConference. June 27-30, Bled, Slovenia, Proceedings retrieval from www.bledconference.org. ISBN-13: 978-961-286-385-9, DOI: https://doi.org/10.18690/978-961-286-385-9 . Simons, LPA, Neerincx MA, Jonker CM (2022a). Is Google Making us Smart? Health Self- Management for High Performance Employees & Organisations, International Journal of Networking and Virtual Organisations, Vol 27, No 3, pp.200-216. DOI: 10.1504/IJNVO.2022.10053605 Simons, LPA, Gerritsen, B, Wielaard, B, Neerincx MA (2022b). Health Self-Management Support with Microlearning to Improve Hypertension, pp. 511-524, 35th Bled eConference. June 26-29, Bled, Slovenia, Proceedings retrieval from www.bledconference.org. ISBN-13: 978-961-286-616-7, DOI: 10.18690/um/fov.4.2022 Simons, LPA, Gerritsen, B, Wielaard, B, Neerincx MA (2023a). Hypertension Self-Management Success in 2 weeks; 3 Pilot Studies, pp.19-34, 36th Bled eConference. June 25-28, Bled, Slovenia, Proceedings retrieval www.bledconference.org. ISBN-13: 978-961-286-751-5, DOI: 10.18690/um.feri.6.2023 Simons, LPA, (2023b). Health 2050: Faster Cure via Bioinformatics & Quantified Self; A Design Analysis, International Journal of Networking and Virtual Organisations, Vol 28, No 1, pp.36-52. DOI:https://doi.org/10.1504/IJNVO.2023.130957 Simons, LPA, Gerritsen, B, Wielaard, B, Neerincx MA (2024). Employee Hypertension Self- Management Support with Microlearning and Social Learning. International Journal of Networking and Virtual Organisations, Accepted. Vaishnavi, V and Kuechler, W. 2004. Design Research in Information Systems. Last updated August 16, 2009 from http://desrist.org/design-research-in-information-systems Venkatesh, V. and Davis, F.D. (2000). "A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies," Management Science, 46, 186-204. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS quarterly, pp. 425-478. Verschuren, P., & Hartog, R. (2005). Evaluation in design-oriented research. Quality and Quantity, 39, 733-762. Wil ett, W. C. (2012). Dietary fats and coronary heart disease. J of internal medicine, 272(1), 13-24. Zhou, B., Perel, P., Mensah, G. A., & Ezzati, M. (2021). Global epidemiology, health burden and effective interventions for elevated blood pressure and hypertension. Nature Reviews Cardiology, 18(11), 785-802. 298 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION FUZZY SET QUALITATIVE COMPARATIVE ANALYSIS AS A TOOL FOR INDIVUDUAL AND ORGANIZATIONAL DECISION SUPPORT IN TECHNOLOGY ADOPTION: REVEALING THE POTENTIAL MAHINDA MAILAGAHA KUMBURE, PASI LUUKKA Business School, LUT University, Lappeenranta, Finland mahinda.mailagaha.kumbure@lut.fi, pasi.luukka@lut.fi As the digital economy and society rapidly grow, individual and organizational adaptation to technology has gained substantial concern across various sectors. However, this process involves many challenges, including uncertainty and complexity arising from factors such as the reliability, feasibility, and compatibility of technologies. Based on evidence from existing literature, this study proposes applying the fuzzy set qualitative comparative analysis (fsQCA) approach as a valuable tool in investigating associated chal enges and complex configurations of influential factors within the context of individual and organizational technology decision-making in technology adoption. The fsQCA has emerged as a popular tool in qualitative analysis, particularly in recent years, where its use has grown substantial y. This paper conducts a systematic literature review of journal articles published between 2015 and 2023 using fsQCA, focusing on Keywords: digital transformation, AI, IoT, e- and m-commerce applications, consumer digital assistants, business analytics, sustainable development, behavior, decision and machine learning. This study offers a detailed review of analytics, related research, the implications of the identified trends, and the fsQCA, potential for future research utilizing fsQCA to explore literature review, performance and human behavior in technology adoption and technology organizational technology decision-making contexts. adoption DOI https://doi.org/10.18690/um.fov.4.2024.17 ISBN 978-961-286-871-0 300 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Today's fast-changing digital world has increased interest in using artificial intelligence (AI) and innovative technological tools across various sectors. However, this surge in technological interest has also introduced numerous chal enges that make the decision-making process complex, particularly regarding how individuals and organizations interact with and adopt these new technologies. At the individual level, a person's decisions and actions are key to their intentions to adopt and persist with a specific system or technology (Granić, 2023). At the organizational level, the emergence of advanced technologies, for example, AI, big data analytics, and Internet of Things (IoT) has enabled organizations to automate and optimize their process and improve innovation capabilities (Fan et al., 2023; Costa-Climent et al., 2023). However, the successful implementation of new technology depends not solely on the technology itself but on how well it aligns with other factors (e.g., behavior of employees, top management support) within the organization and society (Haber & Carmeli, 2023). These emphasize the importance of understanding complex relationships between consumer preferences and perceptions related to technology products and services, as wel as examining how organizations navigate the complexities of technology adoption and integration into their operations. This is where fuzzy-set qualitative comparative analysis (fsQCA) can be valuable. fsQCA (Ragin, 2000) is a methodology that al ows researchers to analyze how different combinations of variables (i.e., causal relationships) contribute to specific outcomes (Hew et al., 2023; Chen & Ye, 2023), even in the presence of uncertainty. fsQCA has emerged as a promising method for handling the complexities inherent in individual and organizational decision-making contexts. By combining qualitative and quantitative elements, fsQCA offers a holistic approach (Ragin, 2008) to analyzing complex data sets and identifying key drivers of desired outcomes. fsQCA involves a systematic process of identifying and representing the key variables in a research problem and supports identifying the causal relationships among the variables. This approach al ows researchers to examine the causal relationships among variables more holistical y than traditional regression or correlation analysis (Kraus et al., 2018; Pappas & Woodside, 2021), which assumes linear relationships between variables and does not account for non-linear or interactive effects. Due to its potential and ability to handle complex data, fsQCA has been applied in various research contexts (Kraus et al., 2018), including consumer behavior studies (Diwanji, M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 301 2023), organizational decision-making (e.g., Fiss, 2011; Kumbure et al., 2020), information systems and marketing (Pappas & Woodside, 2021), education (e.g., Plewa et al., 2016), and online business (e.g., Pappas et al., 2016), to mention a few. As such, it has become an increasingly crucial methodological tool for researchers seeking an in-depth understanding of complex social phenomena (Kumar et al., 2022). The primary goal of this review study is to explore the application of fsQCA as a valuable tool for investigating the key factors and configurations that impact individual and organizational decision-making when considering the adoption of technology. Focusing on insights from various domains such as digitalization, big data analytics, visual analytics, machine learning, explainable AI, business analytics, Internet of Things (IoT), knowledge collaboration, information systems, and sustainability development, this paper seeks to reveal the applicability and effectiveness of fsQCA across a wide range of consumer behavior and organizational contexts. Accordingly, the primary research question for this systematic literature review is set as fol ows: What is the current state of fsQCA research in emerging technology, specifical y in consumer behavior with technology adoption and organizational technology decision-making? Consumer behavior and technology adoption research focuses on understanding how individuals interact with and adopt new technologies. It examines factors influencing consumers' decisions to accept or reject technology and their usage patterns and attitudes toward technology products or services. In contrast, organization technology decision- making research investigates how organizations make decisions regarding the adoption, implementation, and management of technology. It analyzes the organizational processes, structures, and dynamics that influence technology adoption, such as decision-making frameworks, resource al ocation, and organizational culture. Both research areas aim to provide insights into the complex interactions between individuals, organizations, and technology, helping offer strategies for successful technology adoption and implementation. With this research question, we focus on fsQCA research to deliver valuable insights regarding the influences of various factors and their connections to behavior intentions and organizational decision-making processes. 302 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Several review studies have already addressed fsQCA research across various domains, particularly in business and management (Wagemann et al., 2016), entrepreneurship and innovation (Kraus et al., 2018), and bibliometric analysis in business and consumer research (Kumar et al., 2022; Diwanji, 2023). However, our review paper distinguishes itself from existing reviews by focusing specifically on using fsQCA in the context of technology adoption at the individual and organizational levels. By revealing insights into fsQCA's application and efficacy within this context, we aim to bridge the gap in understanding the utilization of fsQCA in research on the digital era through a detailed review of relevant studies. 2 Fuzzy-set qualitative comparative analysis (fsQCA) Qualitative comparative analysis (QCA) is a set-theoretic methodology (Chuah et al., 2021) used to analyze the relationships between conditions and outcomes. fsQCA is a variant of QCA, introduced by Ragin (2000, 2008), based on fuzzy set theory (Zadeh, 1965), which examines the complex relationships between causal conditions and outcomes while dealing with associated relationship uncertainties. A fuzzy set is defined on a non-empty set (universal set) U by a mapping 𝜇𝜇𝐴𝐴: 𝑈𝑈 → [0, 1], where 𝜇𝜇𝐴𝐴 is cal ed a membership function of A. For any x ∈ U, the value 𝜇𝜇𝐴𝐴(𝑥𝑥) = 𝐴𝐴(𝑥𝑥) is cal ed a degree of membership of x to fuzzy set A. In fsQCA, we use evidence from cases and theoretical knowledge to calibrate the states of variables represented by fuzzy sets. The approach relies on two key concepts, consistency and coverage, to assess causal relationships between variables (Chuah et al., 2021), specifical y the relationship A⟹ B, depending on available data. Consistency measures the extent to which the data supports the investigated claim, A ⊆ B, by examining the proportion of cases where A coincides with B among al cases where A occurs in the data (Ragin, 2008; Kumbure et al., 2022a). A low consistency score for a given causal configuration indicates weak empirical evidence to support its existence. Coverage, on the other hand, indicates how much of the outcome variable B is explained by A (Ragin, 2008; Kumbure et al., 2022a). It measures the proportion of cases in which B is present among those cases where A is present (Ragin, 2008). A high coverage score suggests that A is an important factor in explaining the occurrence of B. The fsQCA is primarily based on these two concepts - their definitions can be presented according to Kumbure et al. (2022a) as follows: M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 303 𝑛𝑛 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶(𝐴𝐴 ⟹ 𝐵𝐵) = 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 (𝐴𝐴∩𝐵𝐵) = ∑ min (𝐴𝐴(𝑥𝑥 𝑖𝑖=1 𝑖𝑖),𝐵𝐵(𝑥𝑥𝑖𝑖)) (1) 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶(𝐴𝐴) ∑𝑛𝑛 𝐴𝐴(𝑥𝑥 𝑖𝑖=1 𝑖𝑖) 𝑛𝑛 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶(𝐴𝐴 ⟹ 𝐵𝐵) = 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 (𝐴𝐴∩𝐵𝐵) = ∑ min (𝐴𝐴(𝑥𝑥 𝑖𝑖=1 𝑖𝑖),𝐵𝐵(𝑥𝑥𝑖𝑖)) (2) 𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶(𝐵𝐵) ∑𝑛𝑛 𝐵𝐵(𝑥𝑥 𝑖𝑖=1 𝑖𝑖) Where A and B are fuzzy sets on a universal set U = {x1, x2,. ., xn}. Here, we assume that Card (A) ≠ 0 and Card (B) ≠ 0 with respect to the relationship A⟹ B. When A ⊆ B (i.e., A ∩ B = A), the consistency is 1(ideal consistency), indicating that A almost always leads to B ( B is fully consistent with A). Besides, the coverage = 1 expresses that only A is connected with the outcome B in the data ( B ⊆ A). If some other conditions affect B, the coverage value would be lower than 1. Overal , we seek to achieve a balanced combination of consistency and coverage for a particular situation to ensure the theoretical and empirical robustness of the outcomes (Kumbure et al., 2022a). When a relationship has very high consistency but low coverage, it fails to describe many cases, suggesting a potential y weak relationship. Contrarily, when a relationship has very high coverage but low consistency, it also signifies a weak relationship due to insufficient evidence from the data (Elliot, 2013). 3 Methodology The systematic review presented in this study attempted to identify the applications of fsQCA approaches in research in decision support and adoption of new technology. The methodology fol owed in this systematic literature review was based on the review protocols and guidelines introduced by Kichenham (2004) and Synder (2019) and techniques demonstrated by Kumbure et al. (2022b). The steps of the review methodology are discussed in detail next. 3.1 Search strategy To identify relevant studies for this systematic literature review, we performed an automated search using the Web of Science (WoS) database, which is a widely utilized database (Shukla et al., 2019) that provides access to a vast collection of 304 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION scholarly literature across various disciplines. To generate the search strategy, we used keywords ‘ fuzzy set qualitative comparative analysis,’ ‘ fsQCA,’ ‘ fuzzy set- QCA,’ ‘ fuzzy-set QCA,’ and application-related keywords, ‘ artificial intelligence,’ ‘ AI,’ ‘ internet of things,’ ‘ IoT,’ ‘ digitalization,’ ‘ explainable AI,’ ‘ machine learning,’ ‘ deep learning,’ ‘ big data,’ ‘ visualization,’ ‘ business analytics,’ ‘ sustainable development,’ and ‘ knowledge collaboration.’ The search string was finalized using these keywords, which yielded a set of relevant studies. Articles were searched using the search string in the titles and abstracts. 3.2 Inclusion & exclusion criteria and study selection The inclusion and exclusion criteria were applied as follows: only journal articles were considered, while conference papers and book chapters were excluded. This is because journal articles typical y undergo a rigorous peer review and focus on in- depth analysis and discussion, ensuring the quality and reliability of research. Articles had to be written in English and published between 2015 and 2023. In the next stage, journal articles with full-text availability were considered, and finally, relevant papers were selected by scanning abstracts. This selection process focused on topics such as AI, machine learning, digitalization, e-commerce, m-commerce, information systems, sustainable development, big data, IoT, and business analytics, aligning with the scope of this study. With only fsQCA-related keywords, the initial result of the automated search was 1835 articles. Subsequently, by employing a search strategy defined with both fsQCA- and application-related keywords and applying the filtering criteria, we identified a total of 20 articles related to the scope of this review. This indicates that although fsQCA research has broadened vastly in various disciplines, it is still a new topic of technology-adapted decision-making applications. After excluding irrelevant studies based on the context, 14 articles were included in the final set. 3.3 Data extraction and data synthesis In the data extraction, we primarily considered i) the focus of each paper using fsQCA, ii) the factors/variables examined, iii) the nature of the empirical data sample used, iv) other techniques used with fsQCA, and v) the key findings regarding the application context. In the data synthesis, we aimed to examine and summarize the M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 305 insights from the selected articles to address the research questions defined in our study. Focusing on a thematic synthesis (Braun, 2006) using qualitative data, we performed a descriptive analysis to provide evidence from the related research in favor of the objectives and research questions defined in this review study. 4 Findings 4.1 An overview of the reviewed articles The findings of the present review article are based on 14 articles. A summary of al reviewed articles in our study can be seen in Table 1, in which each row summarizes each study, its practical application (under categories: digital transformation, digital assistant, IoT, AI adoption, e-commerce, business analytics and information modeling, and machine learning), context (whether it is organization decision- making or consumer behavior study), methods (hybrid models are given with ‘ +’ ), underlying theory (i.e., guiding principles for research), and data sample (used in the empirical analysis). Table 1: An overview of reviewed articles Study Application Context Methods Underlying theory Data sample Data from 236 Zhang et Digital fsQCA, TOE Organizational construction enterprise al. (2023) transformation PLS-SEM framework managers Data from 80 Fan et al. Digital NCL + respondents from Organizational TAM (2023) transformation fsQCA petrochemical industry in China Data from annual Song et Digital TOE Organizational fsQCA reports of 388 listed al. (2023) transformation framework companies in China Data from multiple sources (e.g., Perinorm, Liu et al. Digital TOE Organizational fsQCA Korean standards (2023) transformation framework database, and Derwent database) Sharma Digital fsQCA, Survey data from a Consumer et al. assistant ANN, PLS- TAM selected group of 345 behavior (2022) (Chatbot) SEM millennials. 306 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Study Application Context Methods Underlying theory Data sample Al- Digital Consumer fsQCA, Survey data from 447 Emran et assistant TAM behavior PLS-SEM respondents al. (2023) (Chatbot) Chen Consumer fsQCA, Survey data focusing on and Ye IoT TAM behavior PLS-SEM the Chinese market (2023) AI adoption Chuah et Consumer Survey data from 566 (Service fsQCA TAM al. (2021) behavior Taiwanese consumers robots) Survey data from 224 Bawack AI adoption Consumer fsQCA, voice-shopping et al. (Voice TAM behavior PLS-SEM experienced citizens in (2021) shopping) US Mustafa AI adoption Consumer fsQCA, Survey data from 830 et al. (5G TAM behavior PLS-SEM respondents in China (2022) technology) Survey data from 1155 Hew et Consumer fsQCA + E-commerce TAM Malaysian mobile al. (2023) behavior ANN consumers Survey data from 450 Hayajneh Business fsQCA, ROT respondents from large et al. Organizational analytics PLS-SEM framework firms and SMEs in (2022) Saudi Arabia Li et al. Information fsQCA, Survey data from 192 Organizational TAM (2022) modeling NCL managers in SMEs Costa- A data set from 122 Climent Machine Organizational fsQCA BMT European start-ups et al. learning (Crunchbase, tweets) (2023) Table 1 shows that seven of the reviewed articles based on fsQCA are from an organizational context and seven from a consumer behavior context. The underlying theories of consumer behavior in technology adoption and organizational technology decision-making are often based on the technology acceptance model (TAM) and the technological-organizational-environmental framework (TOE). TAM focuses on understanding the factors influencing consumers' acceptance and adoption of new technologies (Chen and Ye, 2023). Besides, the TOE framework considers broader contextual factors influencing organizational technology adoption and implementation (Song et al., 2023). It considers not only technological factors but also organizational and environmental ones. However, two studies were based on different theoretical perspectives as mentioned in their studies; one was based on business model theory (BMT) (Costa-Climent et al., 2023), and the other was on M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 307 resource orchestration theory (ROT) (Hayajneh et al., 2022). ROT examines how companies select, design, and configure resources and capacities to succeed in business (Hayajneh et al., 2022), whereas BMT focuses on how an organization develops and offers value to customers (Costa-Climent et al., 2023). The following subsections discuss more details of the empirical part and key findings of each study in consumer behavior and organizational contexts. 4.2 Consumer behavior towards technology adoption Table 2 summarizes the factors considered in the empirical part and key results obtained using fsQCA (and other methods) in consumer behavior and technology adoption research. Note that al of these articles were based on the TAM theory. Table 2: Factors examined and key findings of fsQCA research in consumer behavior Study Tech solution Factors considered Key findings Human-likeness PLS-SEM, ANN Perceived interactivity has AI-based (perceived the strongest effect on the purchasing Chatbot anthropomorphism, intentions. Sharma et adoption for perceived social al. (2022) fsQCA Perceived interactivity, perceived online presence, perceived anthropomorphism, perceived social presence purchasing interactivity) and are core conditions for influencing purchasing information quality intentions. Performance AI-based expectancy, effort PLS-SEMSocial influence, facilitating Al-Emran Chatbot expectancy, social conditions, and hedonic motivation have no et al. adoption for influence, facilitating impact on the chatbot use. (2023) knowledge conditions, hedonic sharing fsQCAAl factors might have an impact. motivation, habit PLS-SEM Al FUN, EXP, AES influence PEOU, EXP does not significantly affect PU, Perceived usefulness PU and PEOU positively affect ATTs, and (PU), perceived ease of Smart PU and ATTs positively influence purchase Chen and use (PEOU), attitudes clothing intentions. Ye (2023) (ATTs), functionality adoption (FUN), expressiveness fsQCA The combination of FUN, AES, (EXP), aesthetics (AES) PU, and ATTs could be the best option to achieve a high level of smart clothing purchase intentions. Human-likeness None of these factors were found to be either (anthropomorphism, necessary or sufficient in achieving high-level Chuah et Service robot perceived intelligence), of behavioral intention. Instead, four causal al. (2021) adoption technology-likeness combinations serve as sufficient to boost (performance consumers' intention to use service robots. 308 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Study Tech solution Factors considered Key findings expectancy, hedonic motivation, privacy risks), personalities (extraversion, openness) Personality traits (extroversion, PLS-SEM Trust and privacy concerns as agreeableness, mediators between personality traits and voice conscientiousness, Voice shoppers’ perceptions of customer experience. Bawack et emotional instability, shopping al. (2021) intellect), privacy fsQCA key combinations of these factors adoption concerns, trust smart that lead to high customer experience speaker manufacturer, performance. prior experience with smart speakers Barriers (image, risk, All resistant barriers matter but are not equally Mobile- tradition, usage, value), important in causing resistant behavior. Hew et al. commerce resistant behavior Image, usage, value, and risk barriers were (2023) applications (rejection, oppositions, found to be core conditions, but tradition postponement) barrier was a peripheral condition. PLS-SEMEconomic factors are not Intrinsic factors, statistically significant to 5G technology psychological factors, 5G adoption. Mustafa et social factors, economic technology al. (2022) factors, health fsQCAEconomic factors are vital for 5G adoption consciousness, technology adoption. Also, six configurations behavioral intention to achieve high 5G adoption were identified. Conversational commerce is becoming increasingly popular as a powerful communication method to improve online purchasing (Sharma et al., 2022). According to Tables 1 and 2, Sharma et al. (2022) assessed the effects of digital assistant (e.g., AI-based chatbot) attributes on purchasing intentions using a combined approach of fsQCA, artificial neural networks (ANN), and partial least squares-structural equation modeling (PLS-SEM). Among the factors investigated, the result with fsQCA demonstrated that perceived interactivity, perceived anthropomorphism, and social presence are the core conditions for influencing the purchasing intentions of millennials. Al-Emran et al. (2024) also used a combined approach using fsQCA and PLS-SEM to examine factors influencing chatbots for knowledge sharing in education. The empirical analysis evaluated the positive effects of the factors considered. fsQCA showed that al factors might impact chatbot use, while PLS-SEM identified social influence, facilitating conditions, and hedonic as having no impact on chatbot adoption. M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 309 A recent study by Chen & Ye (2023) used both fsQCA and PLS-SEM to explore key factors influencing consumers’ intentions in purchasing smart clothing. The factors, PU, PEOU, ATTs, and three external factors, FUN, EXP, and AES, were considered effects for consumers’ purchasing behavior of smart clothing. The empirical study, using the fsQCA approach, examined six different configurations of those factors. The fsQCA results revealed that combining FUN, AES, PU, and ATTs could be the most suitable option for achieving high smart clothing purchase intentions. Moreover, Chuah et al. (2021) studied how factors such as human- likeness, technology-based, and consumer personalities interact as causal configurations to influence the intention to use robotic services. The fsQCA results showed that individual factors alone are not enough for high intention levels, but four combinations were identified as sufficient to improve consumers' intention to use service robots. Bawack et al. (2021) examined how personality traits, privacy concerns, trust, and prior experiences affect customer experience performance in voice shopping. They used both PLS-SEM and fsQCA methods to explore how the combinations of these factors could contribute to high customer experience performance. PLS-SEM results depicted privacy concerns and trust as mediators between personality traits and customer experience perceptions. fsQCA results identified key combinations of these factors leading to high customer experience performance. Furthermore, a recent attempt by Hew et al. (2023) used a hybrid approach based on fsQCA and ANN to examine three different forms of resistance behavior considering five different barriers exhibited by mobile consumers towards using mobile-commerce applications. The results showed that al active innovation resistance barriers are relevant but not equally significant in triggering resistance behaviors. Mustafa et al. (2022) investigated various factors influencing consumers’ decision-making in 5G technology adoption using a fsQCA-based approach. This study examined direct and indirect influences from intrinsic factors (perceived performance, perceived functional value, perceived value), psychological factors (satisfaction, habit, hedonic motivation, curiosity), social factors (social influence, environmental knowledge, and environmental awareness), economic factors (cost value, facilitating conditions), health consciousness, and behavioral intention to 5G adoption. The findings with PLS-SEM indicated that economic factors are not 310 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION significant in 5G technology adoption, contrasting with fsQCA, which identified that economic factors are vital and suggested six key combinations of those factors to achieve high 5G technology adoption. These studies are great examples of fsQCA as a suitable approach for examining complex and nonlinear relationships among variables/factors in consumer behavior and technology adoption research. 4.3 Organizational technology decision-making Table 3 summarizes fsQCA-based research studies in organizational technology decision-making, focusing on the application context, the factors considered in the empirical part, and key results. Note that studies by Zhang et al. (2023), Song et al. (2023), and Liu et al. (2023) were originated from the TOE framework, while those by Fan et al. (2023) and Li et al. (2022) from TAM. The studies of Hayajneh et al. (2022) and Costa-Climent et al. (2023) aligned with ROT and BMT frameworks, respectively. In addition, some studies, such as Li et al. (2022), Hayajneh et al. (2022), and Costa-Climent et al. (2023), focused on the individual firm level, while the rest focused on the organizational level. Table 3: Factors examined and key findings of fsQCA research in organizational context Study Application context Factors considered Key findings PLS-SEMSeven factors that Use of digital technology, significantly affect on digital digital employees, relative transformation. Digital advantage, competitive Zhang et transformation pressure, partner pressure, fsQCAThree configuration paths that can achieve high-level digital al. (2023) in construction policy support, digital cost, transformation. industry organizational readiness, digital transformation strategy, BothTop management support top management support and policy support are key factors in the dual effect. A combination of three driving paths was recognized to achieve high Digital Digital transformation of Fan et al. sustainable innovation capability and transformation product, organizational, (2023) four conditional configurations were in manufacturing service, process, and model found to lead to non-high sustainable innovation capability. Song et Digital R&D investment, high-level Three and two types of al. (2023) innovation in talents, organizational size, top configurations from organizational, enterprises management team and environmental conditions are M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 311 Study Application context Factors considered Key findings heterogeneity, industrial found to drive high digital innovation development speed, regional intention and performance, digitalization level respectively. NCAhigh perceived resource availability and high-performance expectancy (PE) are necessary BIM adoption in Perceived business value of conditions for high BIM adoption architecture, BIM, perceived benefits, Li et al. intention. engineering, and perceived costs, perceived (2022) construction risks, perceived resource fsQCAhigh PE is the single core industry availability condition for high AI. Also three configurations of managers' psychological factors were identified as influencial. fsQCA BA and 𝜋𝜋-shaped skil s are sufficient but not necessary for high Hayajneh innovative performance. Use of business Business analytics capability, et al. analytics (BA) PLS-SEM BA and 𝜋𝜋-shaped skil s (2022) 𝜋𝜋-shaped skil s are relevant antecedents for innovative performance but their combination did not hold. Costa- Start-ups focusing on both efficiency Use of machine Climent Total funding, novelty, utility, and novelty in their ML technology learning et al. performance, efficiency are more likely to create and technology (2023) appropriate value. Technological innovation Technology ability, academic research Four configurations that lead to high- standard intensity, government Liu et al. TSC. Among them, high academic competitiveness responsiveness, organizational (2023) research intensity and high market (TSC) in the AI participation, international size are the key factors. industry competitive pressure, market size Digitalization drives substantial industry changes and is recognized globally as crucial for traditional competitiveness (Zhang et al., 2023). Accordingly, extensive research already exists on the impact mechanism of digital transformation across diverse fields. As Tables 1 and 2 display, using a hybrid model that combines fsQCA and PLS-SEM, Zhang et al. (2023) studied TOE factors affecting digital transformation in construction enterprises from a dual-effect perspective. The fsQCA result revealed three configuration paths leading to high-level digital transformation activity, while both methods suggested that top management and policy support are the key factors from a dual perspective. 312 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Fan et al. (2023) also analyzed the different types of digital transformation in improving sustainable innovation ability in manufacturing firms by using a mixture model of NCL (also referred to as necessary condition analysis) and fsQCA methods. This analysis with fsQCA + NCL highlighted that integrating three types of digital transformation can effectively boost sustainable innovation in manufacturing firms: pure product-focused, model + organizational, and comprehensive digital transformation paths. Son et al. (2023) investigated the influence of six factors from the TOE framework on enterprise digital innovation using the fsQCA approach. The results with fsQCA identified different configuration paths based on technology-organization-environment types, which drive high digital innovation intentions and performance. Moreover, Li et al. (2022) examined how managers' psychological factors influence their decisions regarding adopting building information modeling (BIM) from their perspectives. By using survey data from 192 managers in SMEs, the fsQCA analysis identified high- performance expectancy (PE) as the primary condition for high BIM adoption intention (BIM-AI), while NCL indicated that both high perceived resource availability and PE are necessary conditions for high BIM-AI. Additionally, three configurations of managers' psychological factors, reflecting their decision preferences: loss aversion, benefit preference, and risk avoidance, were revealed by fsQCA analysis. Several authors have applied fsQCA approaches to assess the impact of data analytics and ML on firms' innovative and improved performance. Using fsQCA and PLS-SEM methods, Hayajneh et al. (2022) examined how business analytics and 𝜋𝜋-shaped skills impact a firm’s innovative performance while exploring the moderating influence (in terms of big data and analytics) of 𝜋𝜋-shaped1 skills. The results from fsQCA indicated that though business analytics and π-shaped capabilities contribute to high innovative performance, they are not necessary conditions. With a focus on how start-ups apply ML technology for success, Costa- Climent et al. (2023) utilized a fsQCA-based model to examine how a firm's business model design interacts with different factors to illustrate value creation and appropriation using ML. The empirical analysis offered a holistic view from multiple theoretical perspectives on how start-ups can strategical y leverage ML technologies 1 𝜋𝜋-shaped skil s represent agility and adaptability, enabling value creation and a competitive edge (Hayajneh et al., 2022). M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 313 for value creation and capture. Additionally, one of the key findings was that start- ups equipped with both funding and a high degree of novelty in their ML technology are better positioned to create value than start-ups with only one type of these factors. Lastly, Liu et al. (2023) applied the fsQCA approach to study the complex factors influencing technology standard competitiveness (TSC) within the AI industry and to identify multiple equivalent paths that jointly support TSC. The findings revealed no necessary conditions, but four different configuration paths sufficiently led to achieving high TSC. Among them, high academic research intensity and high market size were core factors yielding national high TSC. 4.4 Other methods used with fsQCA Looking at Table 1, we can observe that a few other methods, such as ANN, NCL, and PLS-SEM, have also been employed with fsQCA to enrich their empirical analysis and findings. Among them, fsQCA with PLS-SEM has been frequently combined, indicating the importance of integrating PLS-SEM with fsQCA in analyzing complex cases. Let us present some implications given by each study regarding the value added by fsQCA when used with the PLS-SEM method. PLS-SEM is a wel -known statistical approach that can model relationships from the independent variables to the desired outcome and identify key influential variables (Sharma et al., 2022). The PLS-SEM approach is considered symmetric (Al-Emran et al., 2023; Hew et al., 2023), indicating the ability to reveal linear relationships. As Chen and Ye (2023) stated, symmetric methods can analyze individual and net effects of the variables on the dependent variable but not non- linear relationships. Given this, Mustafa et al. (2022) and Sharma et al. (2022) showed the necessity of using a second stage of analysis when PLS-SEM is applied because PLS-SEM analysis may examine only linear relationships within the given research design and may not sufficiently predict outcomes of a complex decision-making process. They performed fsQCA-based second-stage analysis to mitigate the issues of PLS-SEM and achieve improved performance by analyzing non-linear relationships. This is because the fsQCA approach is considered an asymmetric approach (Al-Emran et al., 2023; Hew et al., 2023; Chen and Ye, 2023), allowing for identifying complex configurations of factors leading to specific outcomes. Similarly, Al-Emran et al., 2023 used fsQCA as the second stage method and demonstrated the value of the use of asymmetric analysis (fsQCA) and the danger of based only 314 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION on symmetrical analysis (PLS-SEM). Bawack et al. (2021) highlighted that fsQCA can identify unique and new insights considering complex situations by addressing the issue of the overly simplistic nature of hypotheses tested using traditional methods. Therefore, they have also employed fsQCA to explore crucial relationships between consumers’ perceptions of voice shopping adoption, which cannot be identified using the PLS-SEM approach. Chen and Yu (2023) has applied both PLS- SEM (to investigate individual effects of each antecedent) and fsQCA (to examine the effect of causal configurations) to deliver more accurate insights in the analysis of complex factors considering consumers' purchasing intentions. 4.5 Novel insights and unique properties offered by fsQCA Applying fsQCA in various studies has provided novel insights into complex causal relationships that traditional regression-based methods may fail to reveal. For example, in Hayajneh et al. (2022), while PLS-SEM did not support the theory of interaction between business analytics capabilities and 𝜋𝜋-skills, fsQCA revealed that combining them is crucial for innovation. The incorporation of complexity theory and configuration analysis, as illustrated by Li et al. (2022), further demonstrated the efficacy of fsQCA in uncovering complex patterns of causal interrelationships between managers' psychological factors and decision-making. The ability of fsQCA to identify effects caused by unobserved heterogeneity, as highlighted by Hayajneh et al. (2022), makes it more valuable than traditional methods, especially in handling continuous data types and interval scale variables, as noted by Song et al. (2023), Chuah et al. (2021), and Bawack et al. (2021). Liu et al., 2023 stated that fsQCA is not affected by variable deviations or outliers because it does not rely on underlying hypotheses, specific explanatory variables, or correlation analyses. Moreover, unlike symmetric methods, Zhang et al. (2023), Liu et al. (2023), and Sharma et al. (2022) emphasized the efficacy of fsQCA in revealing multiple conditions/paths instead of relying on a single factor/relation. Mustafa et al. (2021) argued that while traditional variance-based methods capture major effects, fsQCA drives deeper into complex, asymmetric relations, revealing influential combinations of pychological, social, economic, and health consciousness factors on 5G technology adoption. Particularly in consumer research, where asymmetric relationships and non-linearities are prevalent, Hew et al. (2023), Al-Emran et al. (2023), and Chen and Yu (2023) further demonstrated the practical use of fsQCA. Overall, the fsQCA could be a valuable M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 315 tool for researchers seeking to explore complex causal mechanisms and uncover valuable insights that traditional methods may overlook. 5 Discussion and future implications This review study covered two specific application areas of fsQCA: consumer behavior and technology adoption, as well as organizational technology decision- making from a technological perspective. It is worth noting that the most common underlying theory for each study was either the TAM or the TOE framework, indicating their widespread use in the field. It was evident that, by employing fsQCA, researchers can analyze various factors influencing adoption decisions, such as perceived usefulness, perceived ease of use, attitudes, social influence, and economic considerations, particularly in consumer behavior research. More importantly, fsQCA has allowed for the identification of multiple causal configurations that lead to achieving high performance of desired outcomes. It has especial y identified different configurations of TOE factors in organizational decision-making studies, which lead to high or low levels of adoption intentions and reveal valuable insights. This reflects the efficacy of the fsQCA method in capturing the complexity inherent in the context studied and supports identifying the nonlinear relationships and interactions among these factors, providing a more holistic understanding of consumer behavior and organization technology decision-making processes. Furthermore, a combination of fsQCA with PLS-SEM has been frequently applied, highlighting the importance of integrating PLS-SEM with fsQCA in analyzing complex cases. It is also worth mentioning that the empirical analysis of most studies was based on survey data. This is because the survey data al ows for direct insights from stakeholders, such as consumers and industry professionals, providing the relevance and applicability of research findings in real-world contexts directly. Another interesting finding was that the publication year of most related studies was 2022 or 2023, indicating that this field of research using fsQCA is still a relatively new topic of interest but wil likely continue to be researched in the upcoming years. Future research could explore novel applications of fsQCA in emerging fields, such as AI ethics, cybersecurity, and digital governance, as wel as generative AI and e- business platforms, to address complex chal enges and inform evidence-based policymaking and organizational strategies. Additional y, methodological advances 316 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION and hybrid models can further enhance the application of fsQCA as a robust analytical tool for identifying the complexities of digitalization and facilitating sustainable development. 6 Conclusions and limitations In this study, we investigated the applications of fsQCA in previous studies that focused on examining performance and human behavior regarding technology adoption, and organizational technology decision-making contexts. Based on the review results of the 14 selected articles, it is evident that empirical studies utilizing fsQCA have successfully provided significant insights into each practical application. Additional y, the fsQCA has demonstrated its capability to identify necessary and sufficient conditions for each case evaluated. One limitation of this study was the dependence on a single database (WoS) for article search, which might have left some related studies out. Also, relevant research might have been missed due to the need for specific keywords fitting the article's focus. However, given the scope of this study, we believe that we have compiled a reasonably comprehensive set of articles for this review. Nevertheless, this review paper il ustrates how fsQCA has been a valuable tool, particularly in technology transition and decision analytics studies. Hopefully, this paper could inspire further research into investigating the factors influencing various types of technology adoption and decision-making processes using fsQCA approaches. References Al-Emran, M., AlQudah, A. A., Abbasi, G. A., Al-Sharafi, M. A., & Iranmanesh, M. (2023). Determinants of Using AI-Based Chatbots for Knowledge Sharing: Evidence From PLS- SEM and Fuzzy Sets (fsQCA). IEEE Transactions on Engineering Management, 71, 1-15. Bawack, R. E., Wamba, S. F., & Caril o, K. D. (2021). Exploring the role of personality, trust, and privacy in customer experience performance during voice shopping: Evidence from SEM and fuzzy set qualitative comparative analysis. nternational Journal of Information Management, 58. Braun, V. (2006). Using thematic analysis in psychology. Qualitative Research In Psychology, 3, 77–101. Chen, S., & Ye, J. (2023). Understanding consumers' intentions to purchase smart clothing using PLS-SEM and fsQCA. PLoS ONE, 1-25. Chuah, S. H., Aw, E. C., & Yee, D. (2021). Unveiling the complexity of consumers’ intention to use service robots: An fsQCA approach. Computers in Human Behavior, 123, 106870. Costa-Climent, R., Navarrete, S. R., Haftor, D. M., & Staniewski, M. W. (2023). Value creation and appropriation from the use of machine learning: a study of start-ups using fuzzy-set qualitative comparative analysis. nternational Entrepreneurship and Management Journal. Diwanji, V. S. (2023). Fuzzy-set qualitative comparative analysis in consumer research: A systematic literature review. International Journal of Consumer Studies, 47, 2767–2789. M. Mailagaha Kumbure, P. Luukka: Fuzzy Set Qualitative Comparative Analysis as a Tool for Indivudual and Organizational Decision Support in Technology Adoption: Revealing the Potential 317 Elliot, T. (2013). Fuzzy-set/qualitative comparative analysis 2.0. 1-6. Fan, X., Wang, Y., & Lu, X. (2023). Digital Transformation Drives Sustainable Innovation Capability Improvement in Manufacturing Enterprises: Based on FsQCA and NCA Approaches. Sustainability. Fiss, P. C. (2011). Building better causal theories: A fuzzy set approach to typologies in organization research. The Academy of Management Journal, 54, 393–420. Granić, A. (2023) Technology adoption at individual level: toward an integrated overview. Universal Access in the Information Society. Haber, L. & Carmeli, A. (2023). Leading the chal enges of implementing new technoloies. Technology in Society, 74, 102300. Hayajneh, J. A., Elayan, M. B., Abdel atif, M. A., & Abubakar, A. M. (2022). Impact of business analytics and π-shaped skills on innovative performance: Findings from PLS-SEM and fsQCA. Technology in Society, 68, 101914. Hew, J. J., Lee, V. H., & Leong, L. Y. (2023). Why do mobile consumers resist mobile commerce applications? A hybrid fsQCA-ANN analysis. Journal of Retailing and Consumer Services, 75, 103526. Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele University, UK and National ICT Australia. Kraus, S., Ribeiro-Soriano, D., & Schüssler, M. (n.d.). Fuzzy-set qualitative comparative analysis (fsQCA) in entrepreneurship and innovation research – the rise of a method. International Entrepreneurship and Management Journal, 14, 15-33. Kumar, S., Sahoo, S. L., Kraus, S., & Bamel, U. (2022). Fuzzy-set qualitative comparative analysis (fsQCA) in business and management research: A contemporary overview. Technological Forecasting and Social Change, 121599. Kumbure, M. M., Luukka, P., Tarkiainen, A., Stoklasa, J., & Jantunen, A. (2022a). An Investigation of Hidden Shared Linkages Among Perceived Causal Relationships in Cognitive Maps. In P. S. Luukka, Intel igent Systems and Applications in Business and Finance. Springer. Kumbure, M. M., Lohrmann, C., Luukka, P. & Porras, J. (2022b). Machine learning techniques and data for stock market forecasting: A literature review. Expert Systems with Applications, 197, 116659. Kumbure, M. M., Tarkiainen, A., Luukka, P., Stoklasa, J., & Jantunen, A. (2020). Relation between managerial cognition and industrial performance: An assessment with strategic cognitive maps using fuzzy-set qualitative comparative analysis. Journal of Business Research, 160-172. Liu, S., Zhou, L., & Yang, J. (2023). Exploring the Formation Mechanism of Technology Standard Competitiveness in Artificial Intelligence Industry: a Fuzzy-Set Qualitative Comparative Analysis. Journal of Business Economics and Management, 24, 653-675. Mustafa, S., Zhang, W., Shehzad, M. U., Anwar, A., & Rubakula, G. (2022). Does Health Consciousness Matter to Adopt New Technology? An Integrated Model of UTAUT2 With SEM-fsQCA Approach. Frontiers in Psychology, 13, 1-19. Pappas, I. O., & Woodside, A. G. (2021). Fuzzy-set Qualitative Comparative Analysis (fsQCA): Guidelines for research practice in Information Systems and marketing. International Journal of Information Management, 58, 102310. Pappas, I. O., Kourouthanassis, P. E., Giannakos, M. N., & Chrissikopoulos, V. (2016). Explaining online shopping behavior with fsQCA: The role of cognitive and affective perceptions. Journal of Business Research, 69, 794–803. Plewa, C., Ho, J., Conduit, J., & Karpen, I. O. (2016). Reputation in higher education: A fuzzy set analysis of resource configurations. Journal of Business Research, 69, 3087-3095. Ragin, C. C. (2000). Fuzzy-set social science. University of Chicago Press. Ragin, C. C. (2008). Redesigning social inquiry: Fuzzy sets and beyond (Vol. 240). Wiley Online Library. Sharma, M., Joshi, S., Luthra, S., & Kumar, A. (2022). Impact of Digital Assistant Attributes on Millennials’ Purchasing Intentions: A Multi-Group Analysis using PLS-SEM, Artificial Neural Network and fsQCA. Information Systems Frontiers. 318 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Shukla, A. K., Janmaijaya, M., Abraham, A., & Muhuri, P. K. (2019). Engineering applications of artificial intelligence: A bibliometric analysis of 30 years (1988–2018). Engineering Applications of Artificial Intel igence, 85, 517-532. Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 104, 333-339. Song, Q., Chen, X., & Gu, H. (2023). How technological, organizational, and environmental factors drive enterprise digital innovation: Analysis based on the dynamic fsQCA approach. Sustainability, 15, 12248. Wagemann, C., Buche, J., & Siewert, M. B. (2016). QCA and business research: Work in progress or a consolidated agenda? Journal of Business Research, 69, 2531-2540. Zadeh, L. (1965). Fuzzy Sets. Zhang, G., Wang, T., Wang, Y., Zhang, S., Lin, W., Dou, Z. & Du, H. (2023). Study on the Influencing Factors of Digital Transformation of Construction Enterprises from the Perspective of Dual Effects—A Hybrid Approach Based on PLS-SEM and fsQCA Sustainability, 15. DIGITAL TWIN TECHNOLOGY IN THE RAIL INDUSTRY: A DUTCH QUALITATIVE CASE STUDY ON SUCCESS FACTORS, CHALLENGES, AND FUTURE USE CASES KOEN SMIT,1 JORIS MENS,2 CHAIM DE GELDER,1 JOHN VAN MEERTEN1 1 HU University of Applied Sciences Utrecht, Digital Ethics research chair, Ultrecht, Netherland koen.smit@hu.nl, chaim.degelder@hu.nl, john.vanmeerten@hu.nl 2 Railorg, Netherland jorism@gmail.com Assets in the rail industry are progressively being equipped with Internet of Things (IoT) technology. Digital Twins (DT) are increasingly being applied to manage those (critical) assets and the data they generate. One main problem area to which DTs could contribute is that of station management. However, few implementations are studied in-depth and empirical y reported upon. This study focuses on qualitative exploratory research to uncover success factors, chal enges, and future use cases regarding a DT implementation of a large station operated by a rail operator in the Netherlands. Results show that, in this case, Keywords: most success factors and chal enges are considered non- digital twin, technical, i.e., most focus on internal and external collaboration success within the project. We also identified consensus about how a DT factors, would elevate station management maturity in the future, challenges. implementation, featuring (critical) asset monitoring, maintenance, crowd control, case and safety management. study DOI https://doi.org/10.18690/um.fov.4.2024.18 ISBN 978-961-286-871-0 320 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The term Digital Twin (DT) emerged in the early 2010s, but its concepts have roots dating back to the 1960s at NASA (Rajamurugu & Karthik, 2022). A DT is a virtual representation of an object or system throughout its lifecycle (Singh et al., 2021), leveraging real-time data, simulation, machine learning, and reasoning to inform decision-making (Goodwin et al., 2024). It can also influence the physical world based on changes in the virtual model. While there is no consensus on a definition of a DT, most research views it as a cyber-physical system that shares the concepts of a physical entity, a virtual model, and connections between both (Liu et al., 2022). This study aligns with the definition of a DT as " a set of linked operation data artifacts and (simulation) models, which are of suitable granularity for their intended purpose and stakeholders and evolve throughout the product life-cycle" (Boschert & Rosen, 2016). Nevertheless, this definition falls short of a true DT's requirement for bi-directional communication. (Fuller et al., 2020; Liu et al., 2022). Digital Twin Technology (DTT) has expanded into various industries, including manufacturing, aerospace, aviation, and healthcare, due to its value in lifecycle data management, control, and monitoring (Uhlenkamp et al., 2022). In the rail sector, DTs can potential y aid in planning, design and engineering, construction and commissioning, and operations and maintenance. In this sector, we can differentiate between DTT for rolling stock (vehicles), rail infrastructure, and train stations. DTT promises to leverage innovation to improve design, enhance collaboration by visualizing (conflicting) stakeholder interests, and increase both asset reliability and performance (Botín-Sanabria et al., 2022). As Internet of Things (IoT) is increasingly implemented in the rail industry, there is an emerging need for infrastructure data management combined with proper approaches for visualization (Ghaboura et al., 2023). Also, because of IoT implementations, organizations in the rail industry are chal enged with the integration of data from various sources, and processing of large volumes of data (Kaewunruen et al., 2021). One of the primary purposes for using DTs in the rail industry is asset management. While asset management can relate to either rol ing stock, rail infrastructure, or train stations, this study focuses solely on train stations. A relevant case study based on Kings Cross station in London (UK) focused on sustainability evaluation (Kaewunruen & Xu, 2018), although it relies heavily on Building Information K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 321 Modelling (BIM) instead of a true DT of a railway station. Dirnfeld et al. (2024) conducted a structured literature review of DTs in the rail industry which uncovered technical chal enges applicable to DTs in practice, thus having a scoped focus on the technical perspective. The Asset Information Requirements (AIRs) studied by (Johnson et al., 2021) provide an inclusive approach to requirements gathering for the design and application of DTs in the rail industry. Still, recent literature suggests there is a lack of studies that focus on DTT for station management in the rail industry, which is also substantiated in the work of (Doubell et al., 2022; Ghaboura et al., 2023), for example stating: ' Considering digital twin adoption for public infrastructure, the rail industry is stil at an early stage with few recorded implementations. '' (Doubell et al., 2022). Adoption is a broad term here as it refers to technical and non-technical aspects (Leso et al., 2023). As train stations are critical assets in the rail industry, we argue that railway organizations could benefit from DTT (and relevant studies) to assist in the lifecycle management of these assets, especial y if such studies present applicable knowledge about success factors and chal enges to consider. Therefore, this paper aims to answer the following research question: Which success factors and chal enges regarding Digital Twin Technology for rail station management are relevant in the context of the Dutch Rail Industry? The Dutch rail industry was selected based on an existing DT implementation for a major train station in The Netherlands. The selection of this case is further defined in the research method. This paper is structured as follows: In section two, we provide an overview of the body of knowledge on DT implementations in the rail industry, with a focus on asset management. Within this scope, we identify success factors and chal enges from the literature. This is followed by the research method used for this study in section three. Sections four and five describe the data collection and analysis applied in this study. The results follow in section six. We conclude the paper with a discussion and future research directions in section seven and conclusions in section eight. 2 Background and Related Work Asset Lifecycle Management (ALM) in the rail industry, particularly for train stations, involves a systematic approach to managing station assets from planning, design, and construction to operation, maintenance, and eventual decommissioning. It 322 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION comprises al systems, methods, procedures, and tools to optimize costs, performance, and risks for the complete rail infrastructure lifecycle (Liljenström et al., 2022). DTs are increasingly applied to aid Application Lifecycle Manamagement (ALM) (Wilke, 2022), as is the case in the rail industry (Ghaboura et al., 2023). The recent contribution of Ghaboura et al. (2023) focused on providing a state-of-the-art overview of how DTs contribute towards digital transformation in the rail industry. Use cases most often discussed in the previous years in the body of knowledge are that of 1) maintenance and condition monitoring, 2) asset management, 3) damage detection, 4) predictive maintenance, 5) simulation modeling, and 6) optimization. These are all related to different or overlapping goals such as sustainability and safety. Figure 1: Digital Twins for Railways technology taxonomy Source: (Ghaboura et al., 2023) Furthermore, Ghaboura et al. (2023) identify key technologies regarding DTs in the rail industry from the extant body of knowledge, as summarized in Figure 1. These results encompass key technologies for all assets within the rail industry, of which train stations are a subset. We therefore include these key technologies in our gathering of empirical evidence for the case study. Studies in the body of knowledge describe chal enges regarding the design and implementation of DTs in the rail industry. For example, Dirnfeld et al. (2022) discuss the problem space of DTs and AI, concluding interoperability to be the most K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 323 discussed chalenge. Other papers (da Silva Mendonça et al., 2022; Dirnfeld et al., 2022; Zayed et al., 2023) with similar foci al mostly report on chal enges from an IT perspective, and most do so from a meta-perspective without access to empirical data. Our study focuses on the identification of success factors and chal enges in a more open setting similar to grounded theory (Packer-Muti, 2016), but more lenient in terms of mechanisms for protocol guidance. This case study attempts to identify success factors and chal enges in a more holistic sense by not focusing solely on technical aspects or theoretical perspectives. To provide a holistic overview of empirical evidence we zoom in on success factors that contribute to the adoption of technology at organizations, and the chal enges that are faced. To ground the discussion about (critical) success factors, a definition of a success factor is provided, which is used in this paper and during the analysis: “those few things that must go wel to ensure success for a manager or an organization, and, therefore, they represent those managerial or enterprise areas that must be given special and continual attention to bring about high performance. ” (Boynton & Zmud, 1984). In the context of this definition, we explore organizational chal enges and success factors in the rail industry. We emphasize the importance of examining both chal enges and success factors, as solutions to chal enges can reveal essential factors for success. This approach provides a more comprehensive understanding and aligns with in-depth case study methods. 3 Research Method The goal of this study is to identify success factors and chal enges regarding a DT initiative designed and implemented in the Dutch rail industry. As described earlier in this paper, the maturity of this research domain is limited; there is a lack of studies that address success factors and chal enges derived from real DT cases in the rail industry. In fields with limited maturity, an appropriate focus involves identifying new constructs and establishing relationships between identified constructs (Edmondson & Mcmanus, 2007). Examples are domain-specific concepts, processes, technologies, and cultural aspects, which are constructed, if defined, by Subject Matter Experts (SME’s) but when they are not defined, they could be discovered as patterns in data and validated with SME’s. Many researchers use explorative 324 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION qualitative research methods to do so. We therefore conduct a qualitative study, using case study data col ection and analysis to gather empirical evidence on success factors and chal enges. A case study approach helps us develop context-based descriptions of the phenomenon studied (Myers, 1997). 4 Data Collection and Analysis Data for this study was col ected for two months; from January to February 2024. The case study features a dual-method approach, composed of 1) secondary data collection and analysis and 2) semi-structured interviews. The selection of the participants in the case study should be based on the group of individuals, organizations, information technology, or communities that best represent the phenomenon studied (Packer-Muti, 2016). For this study, the phenomenon studied is represented by organizations and individuals who were involved with DT design and implementation for railway station management in the Dutch rail industry. A large organization (hereafter referred to as Railorg) in the Dutch rail industry was selected for this research. Over the past three years, Railorg invested resources to design and prototype a DT for the largest train station in the Netherlands. These characteristics provided the best fit for selecting a DT project in the Dutch rail industry setting. At Railorg, a case was defined as a single project that focused on the design and prototyping of a DT of a large railway station, with the goal of discovering whether DTT could help in providing comprehensive and complete 3D insights into the interior and exterior of the station in one integrated view. The interior and exterior were mapped using drone-mounted cameras and Lidar technology. Furthermore, the goal was to connect the 3D model of the station with real-time IoT data streams from elevators and escalators throughout the station. A screenshot of the DT is presented in Figure 2. Based on this project, a single-case design case study was adhered to (Yin, 2013). K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 325 Figure 2: Railorg's largest train station in a DT Source: Own 4.1 Secondary data col ection Secondary data collection was used as a supportive approach for the researchers to prepare the semi-structured interviews and provide the context of the selected case. Secondary data encompassed the documentation produced during the design and prototyping of the DT. The documents analyzed were mainly technical reports, advisory reports, and presentations that focused on the progression of the development during the project. The documents used in this study are produced by Railorg team members of the team responsible for the DT project, from different roles. 4.2 Semi-structured interviews The main data collection consisted of semi-structured interviews. In total, seven interviews were conducted. Six interviewees represent distinct roles and departments within Railorg. One interviewee is an external DT consultant from an IT consultancy firm in the Netherlands that supported the Railorg team during the project. The following roles were interviewed: 1) IT Manager 2) Business Consultant 3) Data Management Product Owner 4) External DT Consultant 5) Station Manager 6) Real Estate Project Manager 7) Maintenance Manager. Each interview was conducted for one hour. Three researchers were present for each interview, with two researchers 326 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION interviewing according to protocol and the third researcher taking notes. The interviews were conducted via MS Teams and recorded for reference purposes. 1. Role in the organization and relation to the DT project; 2. Examples of what did and did not work. What to repeat or avoid in the future; 3. Future of the project and how it should support Railorg; 4. Perspectives other than maintenance (e.g., sustainability, safety) the DT can support; 5. Which technologies could affect the future development of the DT in Railorg, and; 6. Other relevant sources to analyze or people to interview regarding this study. 4.3 Data analysis The secondary data sources along with the transcriptions of the semi-structured interviews were coded using thematic coding. Thematic coding was done using the following coding categories: 1) success factors, 2) challenges, and 3) potential future applications of DTs and how technological advancements would impact these DTs at Railorg in the future. The first two categories refer to the empirical experiences of the interviewees. The third category refers to the perceived usefulness of DTs for other areas of application based on the empirical experience gained during the project. We applied the Toulminian’s analysis approach of coding using claim- ground-warrant statements (Hitchcock, 2003). This al ows us to deconstruct the practical reasoning in the context of the project to capture, analyze, and elicit patterns within the arguments the participants made about success factors and chal enges. In total, there were 57 claims-grounds combinations relating to success factors, 79 combinations relating to chal enges, and 81 combinations relating to potential future applications and technological developments affecting the future of DTs at Railorg. The data was coded by four separate research team members of which one coded al interviews redundantly. Final y, an assessment of the intercoder agreement was made (average of 70,56% across al coders) and codes were merged. Two coders partook in a session where the codes were discussed, and consensus was reached K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 327 where codes were different amongst the independent coders, also described by Campbell as a "negotiated agreement" (Campbell et al., 2013). Two researchers sorted the codes into three categories. An example of a claim-ground-warrant combination from our data is as follows: Category- Chal enge, Claim- Adding all information to our Digital Twin poses security chal enges, Ground- Train stations have logistical hal ways that should not be visible to al stakeholders because of security risks, Warrant- Expert source. The secondary data analysis was conducted on-site at Railorg due to security constraints. With open coding, one researcher coded secondary data related to the project, using the same approach as applied to the interview data. 5 Results Data analysis was done according to the three categories: success factors, chalenges, and potential future use cases. The success factors and chal enges are clustered according to the topics they represent, starting with the topics that were most mentioned by the interviewees. This is fol owed by a description of potential future use cases as identified in the interviews. 5.1 Stakeholder collaboration Stakeholder collaboration is complex in this case study because of the variety of roles and departments involved, as well as the necessary data, skills, and technologies required to realize a digital twin. On top of this, train stations are characterized by a fragmented ownership structure that includes the Railorg itself, municipalities, and local governments, as well as infrastructure owners. This resulted in the following success factors (SF) and chal enges (C): 328 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Success factors: SF1. Involving only necessary stakeholders (in our case: IT and maintenance management) to reduce complexity and increase project velocity, while maintaining a balance with the necessary knowledge and skil s to complete the project. SF2. When including external partners, ensure that they have ample experience with DT technology from earlier projects. SF3. A combination of agile and waterfal project management methodologies to suit the different phases of the project. For example, data col ection using drones and point cloud processing was more suitable to a waterfal approach due to longer lead times. SF4. In phases where agile project management is most suitable, deliver tangible results in clearly defined sprints to keep project sponsorship healthy. Chal enges: C1. Stakeholders who were not initially involved in the project felt overlooked when they found out about the project at a later stage and were less wil ing to col aborate as a result. C2. The fragmented ownership structure of train stations made it difficult to identify parties (fiscal y) responsible for specific parts of the train station. C3. Organizations involved in station ownership have differing organizational structures and siloed IT systems without a single point of truth for all assets. Inter-organizational collaboration was further hampered because organizations were reluctant to share data that would result in exposing these issues. 5.2 Business Involvement The project was initiated by Railorg’s IT department as a technical proof of concept, with limited involvement from business stakeholders or a clearly defined business case. It was performed mostly to gain experience in the technical aspects of developing a DT. This resulted in the following success factors and chal enges. K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 329 Success factors: SF5. The limited scope and number of stakeholders involved in the project al owed for a quick exploration of DTT and its application in a practical domain. This resulted in a short design phase. Chal enges: C4. By not having al business stakeholders on board before initiating the project, alignment with their business goals was lacking and this obstructed the implementation and adoption of the DT by the business departments. C5. Because of its technological approach, some stakeholders deemed the DT ‘a solution seeking a problem’ which reduced their support of the project. C6. The project did not have a clear business case, making it more difficult to clearly measure its success and value. C7. The ful capabilities of DTT could not be explored because certain business stakeholders who owned data necessary to enrich the DT were not involved. While there was a complete 3D model of the station, it was only enriched with a limited set of sensor data. C8. The innovative nature of the project was seen as misaligned with the current organizational strategy to prioritize its critical business systems, resulting in reduced management support and negative sentiment towards investing resources into DTT. 5.3 3D visualization The DT was delivered as a 3D model of the interior and exterior of the train station. The interviews in this case study resulted in discussions on the effectiveness and usefulness of a 3D model vs. a 2D model or other types of visual representation. The fol owing success factors and chal enges regarding this topic were identified: Success factors: SF6. A 3D model can more easily convey information regarding dimensions and proportionality compared to other visualization methods. It also makes it easier to gain insight into multi-floored stations. 330 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION SF7. A 3D model is innovative and visually impressive and aids in marketing the project to stakeholders and project sponsors. Chal enges: C9. A 3D model is sometimes considered superfluous when the information presented to the end user is just as easily conveyed through a 2D map or other dashboard, for example, binary status information of an object (in operation / out of order). C10. The differing information requirements of departments within Railorg make it difficult to reach a consensus on the type of visualization, thereby obstructing collaboration between departments and the adoption of the DT. C11. The design, maintenance, and usage of a 3D model may necessitate the use of more powerful and costly hardware when compared to other visualization methods. C12. Current business processes rely heavily on on-site inspections and do not yet integrate dashboards or 3D models, which may chal enge the adoption of DTT. C13. Using LIDAR and drone technology to create a 3D model of the train station does not identify areas or installations that are obscured from view (hidden being ceilings or wal s). The 3D model therefore does not satisfy the information needs of some stakeholders. 5.4 IT and data maturity Interviewees mentioned that IT and data maturity (or lack thereof) influences the successful implementation of DT. We identify the fol owing success factors and chal enges for this topic. Success factors: SF8. The presence of sensors in the physical train station that can provide data to the DT model. SF9. Real-time and continuous availability of such data via an IoT platform to enrich the DT with information, enabling it to be used in an operational environment. K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 331 Chalenges: C14. IoT data is currently not centralized and sometimes managed by external vendors, making it difficult to gather and present al such data in the DT. C15. Experience with and maturity of IoT is limited, meaning that more IoT systems and data streams should be implemented before the insights provided by such systems can be leveraged in the DT. C16. Building information management is implemented only to a limited extent within Railorg. Because BIM can function as an important data source for a DT, the lack of BIM makes it more difficult to achieve an accurate and up-to- date Digital Twin. Departments within Railorg are not yet familiar with using BIM in their processes and suppliers/contractors are unable to sufficiently supply such information, except for a few technical installations. C17. Railorg is dealing with an aging workforce, negatively affecting the adoption of new technologies. IT-savviness of personnel is deemed insufficient by the interviewees. Processes that could be (partly) performed remotely through a DT instead rely on physical, on-site inspection. Automation sometimes does not extend beyond the use of basic office applications. The conservative way of working makes it difficult for the organization to get up to speed with new and advanced technologies such as DT. C18. The lack of a ‘Single Point of Truth’ in regard to assets within the train station that are managed by different organizations makes it difficult to pinpoint a leading data source as input for the DT, while also complicating the inter-organizational exchange of data and keeping the data systems consistently up-to-date with changes in the physical environment. 5.5 Potential use cases (and technologies leveraging the implementation and use) of DTs at Railorg Besides success factors and chal enges, the interviews explored other potential use cases for DTT in the organization. The following were mentioned consistently among the interviewees: 332 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION DTs for (remote) monitoring: Unlike station managers responsible for large train stations who mostly work onsite, station managers in rural areas often manage several train stations that are geographically spread apart. Using DTT a station manager could get remote information based on sensor data at those stations. This information should be translated into actionable messages about what is happening and show predictions for the next best actions. DTs for maintenance & asset management: Sensor data from (critical) assets within train stations could be used to further improve the maintenance process. Such assets include elevators, escalators, HVAC installations, and energy meters. Enabling predictive maintenance of these assets is desired by Railorg. Also, monitoring their performance helps to underpin contractual obligations between Railorg and external suppliers. Additional y, a DT could help direct external contractors to the relevant assets for repairs, cleaning, or resupplying activities. Calculations for the work to be carried out can be made beforehand, for example by measuring the total surface area of windows to be cleaned within the DT. This wil speed up the process of directing external contractors. To properly execute maintenance and asset management within the DT, Railorg as wel as external organizations responsible for (architectural) changes within the changes should commit these changes to the DT, possibly via BIM. Finally, a DT may help Railorg keep track of sustainability goals by measuring real-time energy consumption. DTs for crowd control & safety management. Major train stations deal with large crowds, especial y during peak hours or events. a DT could be used to monitor passenger flows in real time or al ow station managers to simulate passenger flows during maintenance or disruptions. The impact of changes within the train station can then be tested before the work is carried out, or different configurations of assets can be compared before rol out. Another possible use case mentioned is social safety, where sensors could identify brewing trouble based on images and sounds and convey this to the DT so that action can be taken to mitigate potential security risks. K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 333 6 Discussion Our study has limitations that should be noted. Firstly, the small size of the team constructing the Digital Twin (DT) at Railorg may affect the generalizability of the results to larger teams with different compositions. Collaboration is important as it was identified as both the most influential success factor and the biggest chal enge. However, the smal sample size makes the study more susceptible to cognitive biases such as social desirability, false consensus, and response bias. Additionally, confidentiality restrictions limited one researcher to review relevant documents on- site only, meaning that these documents cannot be reproduced for validation purposes. The project's focus on collaboration suggests that the cultural characteristics of the project team, such as its autonomy, and its task-oriented and individualistic approach, may affect its generalizability to other teams in different cultural settings. While our case description supports the generalizability to similar settings, it's important to note that train stations (and the organizations managing them) vary greatly in scope and complexity, both within the Netherlands and global y. This project's limited scope suggests that future research should explore relating success factors and chal enges of DT design and implementation in the rail industry to IS/IT frameworks. Using ontological foundations such as the extended information systems framework (Strong and Volkoff, 2010) could help identify hot spots and best practices, as our study identified success factors and chal enges using a more explorative approach. It is also important to gather more evidence from other cases and use quantitative data collection to improve generalizability and effective analysis of success factors and chal enges. Furthermore, we identified the most significant success factors and chal enges in this study, however, an important direction for future research would be to establish how these factors and chal enges relate to each other. Our study involved interviews with stakeholders from various domains including IT, maintenance, asset management, and innovation. Despite the varying terminologies used by the respondents, there is a significant degree of consensus within the organization regarding the desired attributes of digital twin technology. This suggests that future research should include these different viewpoints and preserve the 334 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION varying terminologies where possible, to gain a broad and diverse understanding of the topic. 7 Conclusion To conclude this paper, we revisit our main research question: Which success factors and challenges regarding Digital Twin Technology for rail station management are relevant in the context of the Dutch Rail Industry? Based on an in-depth case study featuring the design and implementation of a DT at Railorg, we identified (contextual) success factors, chal enges, and future use cases of DTs relevant to the rail industry. While we identified several DT and general IS/IT-related success factors and chal enges, most are related to the organizational culture, organizational structure, responsibility management, and how different stakeholders collaborate and inform each other, which is in line with other studies in the IS/IT research field (Leso et al., 2023). This paper provides valuable empirical insights into a real Digital Twin (DT) case, adding to the body of knowledge especial y in the context of the rail industry. While the findings may have limited generalizability, they contribute to understanding the importance of human factors in DT projects, opening avenues for further research. From a practical standpoint, the paper helps organizations, such as Railorg, in avoiding common pitfal s in DT projects and raising awareness of organizational culture and col aboration. The results are readily applicable in practice and can improve the efficiency and effectiveness of future DT projects. Acknowledgments We would like to thank Railorg, and al employees involved for their valuable contributions to this study. References Boschert, S., & Rosen, R. (2016). Digital Twin—The Simulation Aspect. In Mechatronic Futures (pp. 59–74). Springer International Publishing. https://doi.org/10.1007/978-3-319-32156-1_5 Botín-Sanabria, D. M., Mihaita, A.-S., Peimbert-García, R. E., Ramírez-Moreno, M. A., Ramírez-Mendoza, R. A., & Lozoya-Santos, J. de J. (2022). Digital Twin Technology Challenges and Applications: A Comprehensive Review. Remote Sensing, 14(6), 1335. https://doi.org/10.3390/rs14061335 Boynton, A. C., & Zmud, R. W. (1984). An assessment of critical success factors. Sloan Management Review, 25(4), 17–27. K. Smit et al.: Digital Twin Technology in the Rail Industry: A Dutch Qualitative Case Study on Success Factors, Chal enges, and Future Use Cases 335 Campbel, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013). Coding In-depth Semistructured Interviews. Sociological Methods & Research, 42(3), 294–320. https://doi.org/10.1177/0049124113500475 da Silva Mendonça, R., de Oliveira Lins, S., de Bessa, I. V., de Carvalho Ayres, F. A., de Medeiros, R. L. P., & de Lucena, V. F. (2022). Digital Twin Applications: A Survey of Recent Advances and Challenges. Processes, 10(4), 744. https://doi.org/10.3390/pr10040744 Dirnfeld, R., De Donato, L., Flammini, F., Azari, M. S., & Vittorini, V. (2022). Railway Digital Twins and Artificial Intelligence: Challenges and Design Guidelines (pp. 102–113). https://doi.org/10.1007/978-3-031-16245-9_8 Dirnfeld, R., De Donato, L., Somma, A., Azari, M. S., Marrone, S., Flammini, F., & Vittorini, V. (2024). Integrating AI and DTs: chal enges and opportunities in railway maintenance application and beyond. SIMULATION. https://doi.org/10.1177/00375497241229756 Doubel , G. C., Kruger, K., Basson, A. H., & Conradie, P. (2022). The Potential for Digital Twin Applications in Railway Infrastructure Management (pp. 241–249). https://doi.org/10.1007/978-3-030-96794-9_22 Edmondson, A. C., & Mcmanus, S. E. (2007). Methodological fit in management field research. Academy of Management Review, 32(4), 1246–1264. https://doi.org/10.5465/amr.2007.26586086 Fuller, A., Fan, Z., Day, C., & Barlowc C. (2020). Digital Twin: Enabling Technologies, Challenges and Open Research. IEEE Access, 8, 952–971. Ghaboura, S., Ferdousi, R., Laamarti, F., Yang, C., & Saddik, A. El. (2023). Digital Twin for Railway: A Comprehensive Survey. IEEE Access, 11, 120237–120257. https://doi.org/10.1109/ACCESS.2023.3327042 Goodwin, T., Xu, J., Celik, N., & Chen, C.-H. (2024). Real-time digital twin-based optimization with predictive simulation learning. Journal of Simulation, 18(1), 47–64. https://doi.org/10.1080/17477778.2022.2046520 Hitchcock, D. (2003). Toulmin’s Warrants (pp. 69–82). https://doi.org/10.1007/978-94-007-1078-8_6 Johnson, A., Heaton, J., Yule, S., Luke, S., Pocock, D., Parlikad, A. K., & Schooling, J. (2021). Informing the information requirements of a digital twin: a rail industry case study. Proceedings of the Institution of Civil Engineers - Smart Infrastructure and Construction, 174(2), 33–45. https://doi.org/10.1680/jsmic.20.00017 Kaewunruen, S., Sresakoolchai, J., & Lin, Y. (2021). Digital twins for managing railway maintenance and resilience. Open Research Europe, 1, 91. https://doi.org/10.12688/openreseurope.13806.2 Kaewunruen, S., & Xu, N. (2018). Digital Twin for Sustainability Evaluation of Railway Station Buildings. Frontiers in Built Environment, 4. https://doi.org/10.3389/fbuil.2018.00077 Leso, B. H., Cortimiglia, M. N., & Ghezzi, A. (2023). The contribution of organizational culture, structure, and leadership factors in the digital transformation of SMEs: a mixed-methods approach. Cognition, Technology & Work, 25(1), 151–179. https://doi.org/10.1007/s10111- 022-00714-2 Liljenström, C., Björklund, A., & Toller, S. (2022). Including maintenance in life cycle assessment of road and rail infrastructure—a literature review. The International Journal of Life Cycle Assessment, 27(2), 316–341. https://doi.org/10.1007/s11367-021-02012-x Liu, Y. K., Ong, S. K., & Nee, A. Y. C. (2022). State-of-the-art survey on digital twin implementations. Advances in Manufacturing, 10(1), 1–23. Myers, M. D. (1997). Qualitative Research in Information Systems. MIS Quarterly, 21(2), 241. https://doi.org/10.2307/249422 Packer-Muti, B. (2016). A Review of Corbin and Strauss’ Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. The Qualitative Report. https://doi.org/10.46743/2160-3715/2009.2838 Rajamurugu, N., & Karthik, M. K. (2022). Introduction, History, and Concept of Digital Twin. In Digital Twin Technology (pp. 19–32). Wiley. https://doi.org/10.1002/9781119842316.ch2 336 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Singh, M., Fuenmayor, E., Hinchy, E., Qiao, Y., Murray, N., & Devine, D. (2021). Digital Twin: Origin to Future. Applied System Innovation, 4(2), 36. https://doi.org/10.3390/asi4020036 Uhlenkamp, J.-F., Hauge, J. B., Broda, E., Lutjen, M., Freitag, M., & Thoben, K.-D. (2022). Digital Twins: A Maturity Model for Their Classification and Evaluation. IEEE Access, 10, 69605– 69635. https://doi.org/10.1109/ACCESS.2022.3186353 Wilke, D. N. (2022). Digital Twins for Physical Asset Lifecycle Management. In Digital Twins: Basics and Applications (pp. 13–26). Springer International Publishing. https://doi.org/10.1007/978-3-031-11401-4_3 Yin, R. K. (2013). Case study research: Design and methods. Sage publications. Zayed, S. M., Attiya, G. M., El-Sayed, A., & Hemdan, E. E.-D. (2023). A review study on digital twins with artificial intelligence and internet of things: concepts, opportunities, chal enges, tools and future scope. Multimedia Tools and Applications, 82(30), 47081–47107. https://doi.org/10.1007/s11042-023-15611-7 RESPONSIBLE IMPLEMENTATION OF AI IN HIGHER EDUCATION: ETHICAL FACTORS GUIDING DUTCH IT TEACHERS JOHN VAN MEERTEN, KOEN SMIT, MARLIES VAN STEENBERGEN, MATTHIJS BERKHOUT HU University of Applied Sciences Utrecht, Digital Ethics research chair, Utrecht, Netherland john.vanmeerten@hu.nl, koen.smit@hu.nl, marlies.vansteenbergen@hu.nl, matthijs.berkhout@hu.nl This study explores the ethical factors that influence teachers in the use of artificial intelligence (AI) in higher education. Employing a mixed methods approach, which includes a Systematic Literature Review (SLR), two focus groups involving IT teachers, a survey, and four interviews, a total of 37 ethical factors were identified through the SLR & focus groups. The ethical factors identified from the literature review and focus groups highlight the nuanced perspectives surrounding the use of AI implementation. The results from the survey and interviews provide an initial step toward further exploration and generalization of the research findings. The findings contribute to a refined understanding of ethical considerations in AI use for Keywords: artificial teachers, offering valuable insights for higher education intelligence, stakeholders. The study not only enhances ethical knowledge in AI, AI implementation but also underscores the importance of higher education, diverse perspectives in shaping ethical decision-making within teachers, the higher education landscape. bachelor IT DOI https://doi.org/10.18690/um.fov.4.2024.19 ISBN 978-961-286-871-0 338 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The rapid growth of artificial intel igence (AI) technology has led to new possibilities and chal enges in various domains, including higher education (Raman & Rathakrishnan, 2019). AI's evolving impact on society, particularly in higher education, shows potential e.g. improving learning experiences, administrative efficiency, and educational outcomes (Adiguzel et al., 2023; Chen et al., 2020). However, ethical considerations in AI integration require careful examination to ensure meaningful use (Bonini, 2020). AI is widely used in education for applications like automated essay scoring, dropout prediction, graduate admissions, and knowledge inference (Baker & Hawn, 2022; Ramineni & Williamson, 2013; San Pedro & Baker, 2021; Waters & Miikkulainen, 2014). An example is Jill Watson, an AI bot aiding students as a teaching assistant (McFarland, 2016). However, AI, including machine learning models, may exhibit biases and unfairness (Binns, 2018; Warner & Sloan, 2023). Instances of unfavorable outcomes in education include the University of Texas at Austin discontinuing a biased machine learning system in 2020 (Burke, 2020). Some argue that AI's impact on education quality can negatively affect learning outcomes (Horton, 2023; Ka et al., 2023; Korn & Kelly, 2023; Zhai, 2022). Artificial Intelligence in Education (AIED) technologies aim to enhance education, emphasizing the importance of ethical actions and processes (Hwang et al., 2020; Roll & Wylie, 2016). Teachers must make pedagogical choices mindful of ethics, considering potential (unintended) consequences. Ethical considerations in AI extend beyond technical capabilities to encompass fundamental values and principles in education. Understanding teachers' perceptions of ethical factors in AI adoption in higher education is crucial. AI holds promise in higher education but has diverse ethical implications (Brendel et al., 2021; Pavaloiu & Kose, 2017). AIED may worsen student inequality or commercialize education (Reiss, 2021). Ethical concerns include teachers fearing job loss due to AI automation (Shonubi, 2023). Globally, UNESCO outlines challenges in AI education (Pedro et al., 2019). Even though ethical concerns regarding the use of AI in education are becoming more widespread, research specifical y dedicated to higher education is stil ongoing. Numerous studies have been conducted to investigate ethical issues associated with the use of AI in higher education (Alexander et al., 2019; Bates et al., 2020; Köbis & Mehner, 2021; Ma & Siau, 2018). J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 339 Only a few studies are looking at what teachers think about the meaningful use of AI in education (Amhag et al., 2019; Celik, 2023; Chounta et al., 2022; Lindner & Romeike, 2019; Popenici & Kerr, 2017). There is a lack of research that specifical y addresses the ethical concerns related with the use of AI in higher education by Bachelor IT teachers, even though the ethical implications of AI have been extensively studied in a variety of industries (Aoun, 2018; Cox et al., 2019; Holzinger et al., 2019; Loureiro et al., 2021; Verma et al., 2021; Yu et al., 2018). Understanding the attitudes and concerns of teachers towards the use of AI in education is crucial since, for example, they significantly impact how students learn (Lindner & Romeike, 2019). Additional y, these teachers are teaching future IT professionals who wil engage with (future) AI technology. Based on the knowledge gap discussed, the fol owing research question is addressed in this paper: What ethical factors impact the meaningful utilization of (future) AI technology in higher education, as perceived by teachers within the Bachelor IT program? In this study, "meaningful" refers to the value it adds to education. "Future" encompasses potential applications that are currently unknown. 2 Methodology To address the research question on ethical factors impacting the use of (future) AI technology in higher education among bachelor IT program teachers, a mixed- methods research design was chosen. This approach, combining qualitative and quantitative methods, provides a comprehensive understanding and validation of results (Brannen, 2017; Leech et al., 2009; Tashakkori & Creswell, 2007). The research commences with a systematic literature review to establish a knowledge base and identify gaps. These gaps are then explored through focus groups and surveys, offering both qualitative and quantitative insights, as wel as understanding teacher dynamics. After the focus group sessions, surveys were administered directly after to validate empirical evidence from the literature review. Subsequently, interviews are conducted to delve deeper into teachers' opinions on identified factors discussed in focus groups. Moreover, participants prioritize these factors using q- methodology. Figure 1 provides an overview of the research steps and methods employed. 340 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 1: Research process overview Source: Own 2.1 Systematic literature review To gain a comprehensive understanding of the existing literature on the research question, a systematic literature review (SLR) was conducted (Nightingale, 2009; Xiao & Watson, 2019). Due to the large amount of available literature, the choice was made to use two search strings (Hao, 2019; Smit et al., 2020; Smit & Van Meerten, 2021). SRST I: ethical factors Ai artificial intel igence “higher education” teachers and SRSTII: Artificial Intel igence in Education AIED OR Explainable Artificial Intel igence in education XAI-ED OR TPACK model Technological Pedagogical Content Knowledge. The search period was set between January 2018 and April 2023. The SLR process was facilitated using the application Publish or Perish and Google Scholar was chosen as the search engine for its broader coverage compared to other search engines (Franceschet, 2010; Harzing & Alakangas, 2016; Jean-François et al., 2013; Wildgaard, 2015). Figure 2 shows the PRISMA flow diagram of both search strings. J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 341 Figure 2: PRISMA flow diagram Source: Own With a total of 5819 papers to be screened from SRSTI, ASReview was used for automated screening and selection in the systematic review (ASReview, 2023). To manage the large dataset, two stop criteria were adopted (Callaghan & Müller- Hansen, 2020), meeting either of which would halt the process: (1) surpassing 16 hours analyzing titles or (2) marking more than 50 consecutive titles as irrelevant. A total of 1712 papers were not screened because the time-based stopping criteria (16 hours) was met (Callaghan & Müller-Hansen, 2020). SRSTII didn't require ASReview as the total number of papers to be screened was 803; instead, findings were manual y analyzed. Results are categorized into empirical and non-empirical findings. As the study focuses on teachers' opinions, only empirical results, derived from experiments or studies, are considered. This led to a remaining total of 12 papers. Of the 12 papers, eight originated from SRST 1 and four from SRST 2. 2.2 Focus groups & survey After conducting systematic literature reviews, focus group discussions were held. Two focus group sessions were conducted in June 2023, with 7 and 8 participants each, from a Dutch University of Applied Sciences. The sample included 5 females and 10 males, aged 25 to 66, with work experience of 1 to over 21 years. Moreover, 5 participants had prior AI research experience. At the beginning of each focus group, participants were provided with a brief case study to facilitate the start of the 342 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION discussions. To gather quantitative data for statistical analysis, a survey was administered immediately after the focus group session, ensuring survey reliability. The survey consisted of 20 sections. The first 19 sections (ethical factors from the SLR) each contained two questions: the first question assessed participants' views on the importance of ethical considerations in AI use in higher education. The second question measured their agreement with the provided explanation) related to ethical factors in AI use in higher education, using a 5-point Likert scale. The last section included an open-ended question for written comments. 2.3 Interviews & Q-Methodology To generalize data from focus groups, semi-structured interviews were conducted with four IT lecturers from other higher education institutions, three from a University of Applied Sciences, and one from a university. This method al ows for a more thorough exploration of topics and enables in-depth probing, providing interviewees the freedom to express opinions without constraints (DiCicco-Bloom & Crabtree, 2006; Fontana, A. & Frey, 2000). Participants were selected based on a comprehensive perspective on education and a strong technical background. Interviews were conducted in June 2023. An interview protocol was developed by one author and validated by a second author. The interviews consisted of two parts: examining and analyzing ethical factors derived from focus groups and prioritizing these factors. 20 ethical factors were presented to interviewees, who were asked if they considered each a relevant ethical factor and to assess the provided explanation/definition. Participants could also share comments and reflections. In the second part of the interviews, ethical factors were prioritized using the Q- methodology (Brown, 1996), which combines qualitative and quantitative data for a deeper understanding of participants' perspectives. Participants were asked to place the 20 ethical factors from focus groups into specific positions within the pyramid, creating a hierarchy. The comments made by participants during this process were recorded by the interviewer and incorporated into the research analysis. J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 343 2.4 Data Analysis Each paper from the SLR underwent an analysis to identify and document the presence of ethical factors. Papers lacking these factors were stil analyzed for other pertinent information. Ethical factors identified in the literature review were categorized based on whether they originated from empirical research. This classification determined inclusion in either the empirical evidence or non-empirical evidence list. A final consolidated list of ethical factors from empirical research was created, combining similar factors and refining descriptions through a validation process with one of the other authors. The developed descriptions were translated into Dutch and reviewed by another author, considering the Dutch-speaking participants. The results from both focus groups, documented on the whiteboard by the participants during the sessions, were transcribed, and combined. The researcher's annotations and audio/video data were integrated for enrichment. Identical ethical factors from both groups underwent consolidation, including synthesized explanations. A secondary review with another author refined the ethical factors further. In the interviews, the interviewer transcribed participant comments for each ethical factor. These transcriptions were then adjacent for each factor, enabling a comparative analysis of participant comments. The data from Q- methodology underwent various analyses. Initially, an overview of all four pyramids was generated for a comprehensive view, aiding in pattern identification. Following this, a table detailing the frequency of ethical factors based on associated scores was created, highlighting frequently encountered factors within each score category. Lastly, a separate table with individual participant scores per ethical factor, including descriptive statistics, was developed. 3 Results The ethical factors found from both the SLR as wel as the focus groups can be found in Table 1. 344 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 1: Ethical factors derived from SLR & Focus Group sessions Ethical factors Description (SLR or FG) It is possible that a student who relies solely on the results of an AI model may be led astray, as there are enough factors within the Accuracy (FG) university that play a role in providing study advice. The accuracy of this advice depends on various aspects. However, how can we determine if the advice is reliable? In the context of AI in education, adoption refers to the active integration and approval of AI applications within the university environment, encompassing automation, process acceleration, Adoption (SLR) teaching enhancement, and the establishment of trust through reliability, transparency, and explanatory capabilities. (Bucea-Manea- Țoniş et al., 2022; Chatterjee & Bhattacharjee, 2020; Guàrdia et al., 2021; Kel er et al., 2019) Auditability (FG) The properties of the AI system must be controllable. Authorization Each role must have the appropriate authorization linked to specific (FG) tasks and responsibilities. In the context of AI in education, available refers to the accessibility and usability of AI systems in universities. These systems encompass Availability chatbots and learning analytics, serving various domains like (SLR) teaching, administration, and research, with future plans for expansion and the inclusion of multi-language support (Kel er et al., 2019). It is essential that the AI application is free from bias. While the Bias (FG) model can be trained based on teacher feedback, this must be done careful y. Furthermore, management can use the model to assess the performance of teachers. Communication of Outcomes The communication of an AI model should be objective and (FG) sensitive. In the context of AI in education, cost refers to the financial considerations, evaluation of benefits and risks, and the overal Cost (SLR) investment required for implementing AI. This also includes the lengthy funding process and the limited availability of funds, both of which can influence the cost of AI implementation (Kel er et al., 2019). In the context of AI in education, data refers to responsible data management and ethical handling in AI systems, encompassing risks, Data (SLR) ethical implications, discriminatory data, and unfair predictions, with the involvement of data protection officers to ensure proper usage and transparency (Kel er et al., 2019; Mâtâ & Boghian, 2019). Data origin Information sources for students can also come from platforms (FG) such as Steam or the UWV. J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 345 Ethical factors Description (SLR or FG) In the context of AI in education, decision-making refers to the act of assessing, analyzing, and considering various factors, such as risks, Decision- impacts, and performance, in order to make an informed judgment making (SLR) or decision. It encompasses the process of critical y examining the different aspects related to the use of AI and determining its potential effects and consequences (Bucea-Manea-Țoniş et al., 2022; Kel er et al., 2019; Sangapu, 2018). Discretionary When the outcome of an AI model conflicts with professional Authority (FG) judgment, it is important to consider how to handle it. Education In the context of AI in education, education refers to the (SLR) importance of introducing courses or topics on the ethical use of AI in the academic environment (Mâtâ & Boghian, 2019). In the context of AI in education, explainability refers to the quality exhibited by automated prediction systems in offering dependable Explainability explanations for their decisions. This quality ensures a lucid (SLR) communication and comprehension of the decision-making processes. Explainability directly tackles apprehensions about potential adverse effects, user acceptance, and the mitigation of incomprehensible "black box" systems (Kel er et al., 2019). In the context of AI in education, feedback refers to the offering of information, warnings, and risk indicators to students and teachers Feedback based on their learning behaviors. This approach emphasizes the (SLR) enhancement of performance, personalized support, and the elevation of teaching quality. Effective feedback systems, while taking into account the costs, benefits, and risks of AI, play a pivotal role in the advancement of education (Keller et al., 2019). Freedom of Students should have the freedom to decide for themselves whether Choice (FG) they want to be assessed by an AI model or not. Goal determination The goal of the model should be clearly established in advance. (FG) In the context of AI in education, human-machine collaboration Human- refers to the collaborative utilization of AI systems in universities to Machine enhance administrative processes, provide support in teaching, and Collaboration assist existing staff members, while recognizing the importance of (SLR) human expertise and maintaining a complementary role for AI technology (Kel er et al., 2019). Inclusivity (FG) A student is more than just the data they produce. In the context of AI in education, justice refers to the assurance that AI systems generate impartial predictions and decisions devoid of Justice (SLR) discriminatory factors. This entails addressing potential data misuse, taking into account the desirability of tasks, and integrating ethical principles such as fairness, transparency, and trustworthiness (Kel er et al., 2019). 346 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Ethical factors Description (SLR or FG) In the context of AI in education, learning refers to the process of acquiring knowledge and skil s with the support of artificial intelligence. It involves leveraging AI technologies to enhance Learning (SLR) communication, production, col aboration, content delivery, assessment, and teacher support, ultimately enabling students to engage in meaningful and effective learning experiences. Quality of learning (Keller et al., 2019; Torres-Rivera et al., 2021). In the context of AI in education, mental wel -being refers to the psychological dimension in IT ethics, which encompasses human Mental well- behavior, cognition, and emotions in ethical decision-making and being (SLR) technology use. Additional y, social bonding in education and AI pertains to meaningful connections and genuine interactions between individuals (Mâtâ & Boghian, 2019; Tsivitanidou & Ioannou, 2021). In the context of AI in education, misuse refers to the unethical use of computer programs or multimedia resources. This misuse encompasses unauthorized utilization, plagiarism, intellectual Misuse (SLR) property violations, and potential threats to human intel ect and copyright infringement, with negative implications for individuals, society, and intel ectual property rights (Celik, 2023; Mâtâ & Boghian, 2019; Sangapu, 2018). Open Access Who determines the sharing of a trained model with other Strategy (FG) universities? Ownership and Responsibilities Who is responsible for what? Data supply, aggregation, processing, (FG) storage, communication, etc Prediction In the context of AI in education, predicting refers to forecasting (SLR) (final) student outcomes (Kel er et al., 2019; Popkhadze, 2021). It is essential that an AI application handles sensitive data carefully Privacy (FG) and securely. The data should only be accessible to the students themselves and should not be shared with third parties. In the context of AI in education, professional development refers to initiatives aimed at enhancing skills and knowledge, emphasizing Professional continuous learning to keep up with advancements like AI-based Development tools. Addressing concerns about job substitution in universities due (SLR) to AI involves utilizing AI systems in a supportive rather than substitutive manner, which in turn necessitates further training to adapt to new roles (Celik, 2023; Kel er et al., 2019; Mâtâ & Boghian, 2019; Sangapu, 2018; Torres-Rivera et al., 2021). Role of AI (FG) The AI application should have a supportive role. An AI application can use multiple data points, al owing it to Scope of Data provide advice on more than just study guidance, such as fields of Collection (FG) study, internships, or career choices. This can lead to a different perception of students by teachers. Caution is required when J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 347 Ethical factors Description (SLR or FG) determining which data is included and which is not, including external sources. Security (FG) It is essential that the data and model are not hacked, as this also poses a risk to the integrity of a university. In the context of AI in education, teaching refers to the concerns expressed by teachers regarding technical issues and access to software, equipment, audio-video tools, and platforms during teaching. However, teachers also perceive AI as a means to enhance Teaching (SLR) teaching methodology and foster increased student engagement. They emphasize the importance of a balanced and limited use of AI to preserve students' critical thinking abilities (Celik, 2023; Joshi et al., 2021; Kel er et al., 2019; Mâtâ & Boghian, 2019; Sangapu, 2018; Torres-Rivera et al., 2021). SLR: In the context of AI in education, transparency refers to the imperative of openness and lucidity concerning the utilization of data and the decision-making procedures of AI systems. These address ethical apprehensions associated with data misuse and potential risks of discrimination. The evaluation of task desirability Transparency on an individual basis and the embodiment of European principles (SLR& FG) such as fairness, transparency, and trustworthiness assume pivotal roles. The importance of transparent AI decision-making takes center stage, particularly in domains like university admissions (Kel er et al., 2019). FG: It is important that the basis on which the AI model makes its choices is clear. The university should be able to assess whether the advice a student receives from the AI model is indeed meaningful. In the context of AI in education, trust refers to the ethical integrity and reliability of systems, taking into account data risks and aligning Trust (SLR) with European values. It involves evaluating tasks on an individual basis and placing emphasis on principles such as fairness, transparency, and the establishment of trustworthy AI (Kel er et al., 2019). SLR: In the context of AI in education, usefulness refers to the growing awareness and acceptance of AI-based tools among Usability (SLR teachers, which wil drive their increased integration into teaching & FG) practices. This integration reflects the recognition of AI's value and the evolving landscape of higher education (Celik, 2023; Gocen & Aydemir, 2021; Joshi et al., 2021; Sangapu, 2018). FG: It is important that an AI application is usable for teachers. Validation (FG) The models used must be validated for accuracy. Value of AI An AI application can provide insight into the learning process of (FG) students, while students can also learn about themselves at the same time. 348 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3.1 Survey During the second phase of the focus groups, the participants were asked to fil in a survey1. The results of the survey (n=15) show a significant skew towards the responses of "strongly agree" or "agree" for al questions. Furthermore, several ethical factors display "strongly agree" scores constituting equal to or exceeding 80% of the responses, namely data (93%), transparency (87%), trust (80%), and explainability (80%). Among the ethical factors, "prediction" exhibits the highest degree of spread along the consideration axis, with a mean (M) of 3.13 and a standard deviation (SD) of 1.51. Examining the ethical factor "teaching," it is observed that the respondents, who are teachers, have responded in a notably neutral manner to the explanation axis (47%), in contrast to their responses to other ethical factors along the same axis. Participant feedback at the survey's end was noted and integrated into a general comments category by the researcher. 3.2 Interviews The most notable results and contradictions from the interviews about the ethical factors that originated from the focus groups are described in this section, see Table 2. The Roman numerals represent participant IDs. Table 2: Results and observations from the interviews Ethical factors Notable observations There's a contradiction in how accuracy should be approached, with Participant I advocating for testing AI advice on a smal scale, while Accuracy Participant III questions the definition of accuracy itself. Participant II emphasizes user responsibility for providing accurate information, and Participant IV stresses the importance of accuracy in student assessments. Participant I stresses continuous monitoring, whereas Participant II Auditability vs. is uncertain about auditability and leans towards transparency. Transparency Participant IV demands transparency without delving into auditability, showing an inconsistency between the need for auditability and the preference for transparency. Al participants acknowledge bias but differ in their approaches. Bias Participant I emphasizes minimizing bias through transparency, while Participant II focuses on weighing risks and monitoring 1 Full results of the survey: https://osf.io/52up4/?view_only=b09296356217455ea491017b7c6418d3 J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 349 behaviour. Participant III prioritizes addressing bias and fairness, citing literature, and Participant IV points out the presence of bias in both students and teachers, urging clarity in identifying AI model biases. There's a general agreement on the importance of authorization, but Authorization Participant IV introduces a new perspective by linking authorization to competence in teaching without specifying parameters, suggesting a more nuanced view that considers context. Participants show varied opinions, from Participant I supporting Freedom of opt-in/opt-out options to Participant IV strongly opposing student Choice control over curriculum or AI assessment. This highlights a contradiction in the level of control and choice students should have. While al participants agree on the importance of clear objectives, Goal Participant III introduces a new perspective by suggesting input Determination from a "meta expert" for ethical considerations, indicating a divergence in how goals should be determined and by whom. Participant I advocates for collaboration and public ownership, Open Access while Participant IV is sceptical about the feasibility of idealistic Strategy model sharing, pointing to a new perspective in views on how open AI systems should be. There's a general consensus on the importance of privacy and Privacy GDPR compliance, but Participant IV suggests non-exclusive access for research, introducing a potential contradiction with the emphasis on data minimization and secure handling. Al participants agree on the importance of transparency, but there's a variance in how it's approached. Participant I emphasizes the Transparency chal enge of AI explainability, while Participant III is skeptical about the transparency facade and the chal enges of understanding algorithms. Participants differ in their views on the value of AI in education, from focusing on administrative tasks and information provision Value of AI (Participant II) to advocating for students learning from AI (Participant IV). This contradiction reflects differing perspectives on AI's primary role and value in education. 3.2.1 Q-methodology Figure 3 il ustrates how participants positioned the ethical factors from the focus groups on the Q-methodology pyramid from most important to least important. The pyramid shows that most ethical factors are consistently valued similarly by nearly all participants. 350 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 3: Q-Methodology results Source: Own 4 Discussion, Limitations & Outlook This study sets out to explore the ethical factors influencing the meaningful use of AI in higher education, with a focus on teachers within the Bachelor IT program. The analysis of results from the SLR focus groups, surveys, interviews, and q- methodology highlighted significant differences in the nature and depth of information gathered. The SLR, grounded in scientific articles, provided a broad, global perspective, contrasting sharply with the localized, in detail results captured through focus groups and interviews. This distinction was further seen by the granularity of ethical factors identified. Given this variation, integrating the findings from the different methodologies proved to be complex. The decision to present the research results separately was driven by the realization that combining them would not enhance their value due to the distinct contexts and levels of detail they encompass. This approach ensures transparency in presenting diverse insights into the ethical considerations surrounding AI's integration into education. Acknowledging the potential of AI in education, the study underscores the need for more in-depth, comprehensive research. This call for further investigation is not merely to bridge the gaps identified between the global insights of the SLR and the localized perspectives of the focus groups, surveys, and interviews but also to navigate the complexities of integrating these varied findings into a cohesive understanding of AI's ethical implications in higher education. Also, AI has many different possible application areas in higher education, each possibly with its own unique ethical implications. Future research should acknowledge these application areas and adopt a more detailed approach to AI compared to this study. J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 351 There are also some limitations to this study. The systematic literature review faced chal enges in search criteria (e.g. the substantial growing volume of publications in the field of artificial intelligence), potentially affecting the initial paper selection, and a single researcher's thematic coding raised concerns about bias. In the focus groups, efforts were made to enhance diversity, acknowledging an inherent lack of complete diversity within the population. The small sample size and exclusive affiliation of participants with one university of applied sciences introduced potential biases, but despite these limitations, the focus group data served as a valuable starting point for further exploration, contributing to the enrichment of the existing identified ethical factors. The recruitment method for interviews raised concerns about confirmation and homophily biases, but a careful selection process aimed to mitigate these biases. The study acknowledged potential biases in the translation process, addressing them through consultation with a senior researcher. 5 Conclusion In conclusion, addressing the research question of what ethical factors impact the meaningful utilization of future AI technology in higher education, as perceived by teachers within the Dutch bachelor IT program, highlights the need for a holistic approach to understanding the complexities of AI ethics in education, providing a foundation for developing ethical frameworks and informing policy. The study bridges ethics factors and educational technology, advocating for a broader consideration of stakeholder perspectives in AI integration, including teachers and students. This study examines the ethical factors that are relevant for bachelor IT- teachers when implementing AI in education. The results from this study can be used to set up a broad research project that includes the perspectives of al stakeholders to obtain a complete understanding of what AI in higher education wil entail. The study's theoretical contributions extend existing knowledge on ethical decision-making in higher education from a lecturer’s perspective by providing a list of ethical factors derived from literature and focus groups. References Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3), ep429. https://doi.org/10.30935/CEDTECH/13152 352 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Alexander, B., Ashford-Rowe, K., Barajas-Murphy, N., Dobbin, G., Knott, J., Mccormack, M., Pomerantz, J., Seilhamer, R., & Weber, N. (2019). EDUCAUSE Horizon Report 2019 Higher Education Edition. https://www.educause.edu/horizonreport. Amhag, L., Hel ström, L., & Stigmar, M. (2019). Teacher Educators’ Use of Digital Tools and Needs for Digital Competence in Higher Education. 35(4), 203–220. https://doi.org/10.1080/21532974.2019.1646169 Aoun, J. E. (2018). Robot-Proof - Higher Education in the Age of Artificial Intel igence. The MIT Press. ASReview. (2023). ASReview: AI-aided Open Source Systematic Review Software. https://asreview.nl/ Baker, R. S., & Hawn, A. (2022). Algorithmic Bias in Education. International Journal of Artificial Intel igence in Education, 32(4), 1052–1092. https://doi.org/10.1007/S40593-021-00285-9/METRICS Barrett, D., & Heale, R. (2020). What are Delphi studies? Evidence-Based Nursing, 23(3), 68–69. https://doi.org/10.1136/EBNURS-2020-103303 Bates, T., Cobo, C., Mariño, O., & Wheeler, S. (2020). Can artificial intel igence transform higher education? International Journal of Educational Technology in Higher Education, 17(1), 42. https://doi.org/10.1186/s41239-020-00218-x Binns, R. (2018). Fairness in Machine Learning: Lessons from Political Philosophy. In Proceedings of Machine Learning Research (Vol. 81, pp. 149–159). PMLR. https://proceedings.mlr.press/v81/binns18a.html Bloodgood, M., & Vijay-Shanker, K. (2014). A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping. CoNLL 2009 - Proceedings of the Thirteenth Conference on Computational Natural Language Learning, 39–47. https://doi.org/10.3115/1596374.1596384 Bonini, P. (2020). When Tomorrow Comes: Technology and the Future of Sustainability Learning in Higher Education. Environment: Science and Policy for Sustainable Development, 62(4), 39–48. https://doi.org/10.1080/00139157.2020.1764300 Brannen, J. (2017). Combining qualitative and quantitative approaches: an overview. Mixing Methods: Qualitative and Quantitative Research, 3–37. https://doi.org/10.4324/9781315248813-1 Brendel, A. B., Mirbabaie, M., Lembcke, T. B., & Hofeditz, L. (2021). Ethical Management of Artificial Intelligence. Sustainability 2021, Vol. 13, Page 1974, 13(4), 1974. https://doi.org/10.3390/SU13041974 Brown, S. R. (1996). Q Methodology and Qualitative Research. Ht p://Dx.Doi.Org/10.1177/104973239600600408, 6(4), 561–567. https://doi.org/10.1177/104973239600600408 Bucea-Manea-Țoniş, R., Kuleto, V., Gudei, S. C. D., Lianu, C., & . . (2022). Artificial intelligence potential in higher education institutions enhanced learning environment in Romania and Serbia. Sustainability. https://www.mdpi.com/1626874 Burke, L. (2020). The Death and Life of an Admissions Algorithm. https://www.insidehighered.com/admissions/article/2020/12/14/u-texas-will-stop-using- controversial-algorithm-evaluate-phd Celik, I. (2023). Towards Intelligent-TPACK: An empirical study on teachers’ professional knowledge to ethical y integrate artificial intel igence (AI)-based tools into education. Computers in Human Behavior, 138, 107468. https://doi.org/10.1016/j.chb.2022.107468 Chatterjee, S., & Bhattacharjee, K. K. (2020). Adoption of artificial intelligence in higher education: A quantitative analysis using structural equation modelling. Education and Information Technologies. https://doi.org/10.1007/s10639-020-10159-7 Chen, L., Chen, P., & Lin, Z. (2020). Artificial Intelligence in Education: A Review. IEEE Access, 8, 75264–75278. https://doi.org/10.1109/ACCESS.2020.2988510 Chounta, I. A., Bardone, E., Raudsep, A., & Pedaste, M. (2022). Exploring Teachers’ Perceptions of Artificial Intelligence as a Tool to Support their Practice in Estonian K-12 Education. International Journal of Artificial Intel igence in Education, 32(3), 725–755. https://doi.org/10.1007/S40593-021-00243-5/TABLES/7 J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 353 Cox, A. M., Pinfield, S., & Rutter, S. (2019). The intelligent library: Thought leaders’ views on the likely impact of artificial intelligence on academic libraries. Library Hi Tech, 37(3), 418–435. https://doi.org/10.1108/LHT-08-2018-0105/FULL/XML DiCicco-Bloom, B., & Crabtree, B. F. (2006). The qualitative research interview. Medical Education, 40(4), 314–321. https://doi.org/10.1111/J.1365-2929.2006.02418.X Fontana, A. & Frey, J. H. (2000). The interview: From structured questions to negotiated text. In Handbook of qualitative research. Second Edition. (pp. 645–672). Franceschet, M. (2010). A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar. Scientometrics, 83(1), 243–258. https://doi.org/10.1007/S11192-009-0021-2/METRICS Gocen, A., & Aydemir, F. (2021). Artificial intelligence in education and schools. Research on Education and Media. https://doi.org/10.2478/rem-2020-0003 Guàrdia, L., Clougher, D., Anderson, T., & .. (2021). IDEAS for transforming higher education: an overview of ongoing trends and chal enges. International Review of …. https://www.erudit.org/en/journals/irrodl/1900-v1-n1-irrodl06128/1078405ar/abstract/ Hao, K. (2019). We analyzed 16,625 papers to figure out where AI is headed next | MIT Technology Review. https://www.technologyreview.com/2019/01/25/1436/we-analyzed-16625-papers-to- figure-out-where-ai-is-headed-next/ Harzing, A. W., & Alakangas, S. (2016). Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787–804. https://doi.org/10.1007/S11192-015-1798-9/METRICS Harzing, A.-W. (2023). Publish or Perish. https://harzing.com/resources/publish-or-perish Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Mül er, H. (2019). Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(4), e1312. https://doi.org/10.1002/WIDM.1312 Horton, A. (2023). John Oliver on new AI programs: ‘The potential and the peril here are huge.’ https://www.theguardian.com/tv-and-radio/2023/feb/27/john-oliver-new-ai-programs- potential-peril Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, chal enges, roles and research issues of Artificial Intel igence in Education. Computers and Education: Artificial Intel igence, 1, 100001. https://doi.org/10.1016/J.CAEAI.2020.100001 Jean-François, G., Laetitia, R., & Stefan, D. (2013). Is the coverage of google scholar enough to be used alone for systematic reviews. BMC Medical Informatics and Decision Making, 13(1), 1–5. https://doi.org/10.1186/1472-6947-13-7/COMMENTS Joshi, S., Rambola, R. K., & Churi, P. (2021). Evaluating artificial intelligence in education for next generation. Journal of Physics: Conference …. https://doi.org/10.1088/1742-6596/1714/1/012039 Ka, C., Chan, Y., & Lee, K. K. W. (2023). The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and Mil ennial Generation teachers? https://arxiv.org/abs/2305.02878v1 Kel er, B., Baleis, J., Starke, C., & . . (2019). Machine learning and artificial intelligence in higher education: a state-of-the-art report on the German University landscape. In Heinrich-Heine …. sozwiss.hhu.de. https://www.sozwiss.hhu.de/fileadmin/redaktion/Fakultaeten/Philosophische_Fakultaet/S ozialwissenschaften/Kommunikations- _und_Medienwissenschaft_I/Dateien/Kel er_et_al.__2019__- _AI_in_Higher_Education.pdf Köbis, L., & Mehner, C. (2021). Ethical Questions Raised by AI-Supported Mentoring in Higher Education. Frontiers in Artificial Intel igence, 4, 21. https://doi.org/10.3389/FRAI.2021.624050/BIBTEX Korn, J., & Kel y, S. (2023). New York City public schools ban access to AI tool that could help students cheat . https://edition.cnn.com/2023/01/05/tech/chatgpt-nyc-school-ban/index.html 354 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Leech, N. L., Delinger, A. B., Brannagan, K. B., & Tanaka, H. (2009). Evaluating Mixed Research Studies: A Mixed Methods Approach. 4(1), 17–31. https://doi.org/10.1177/1558689809345262 Lindner, A., & Romeike, R. (2019). Teachers’ Perspectives on Artificial Intel igence. https://www.researchgate.net/publication/337716601 Loureiro, S. M. C., Guerreiro, J., & Tussyadiah, I. (2021). Artificial intelligence in business: State of the art and future research agenda. Journal of Business Research, 129, 911–926. https://doi.org/10.1016/J.JBUSRES.2020.11.001 Ma, Y., & Siau, K. L. (2018). Artificial Intelligence Impacts on Higher Education. MWAIS. Mâtâ, L., & Boghian, I. (2019). Perception of Teachers in Higher Education towards Ethical Issues of Information Technology Use. Revista Romaneasca Pentru Educatie Multidimensionala, 11(4Sup1), 156–169. https://doi.org/10.18662/RREM/183 McFarland, M. (2016). What happened when a professor built a chatbot to be his teaching assistant - The Washington Post. https://www.washingtonpost.com/news/innovations/wp/2016/05/11/this-professor- stunned-his-students-when-he-revealed-the-secret-identity-of-his-teaching-assistant/ Nightingale, A. (2009). A guide to systematic literature reviews. Surgery (Oxford), 27(9), 381–384. https://doi.org/10.1016/J.MPSUR.2009.07.005 Nilforoshan, H., Gaebler, J. D., Shroff, R., & Goel, S. (2022). Causal Conceptions of Fairness and their Consequences (pp. 16848–16887). PMLR. https://proceedings.mlr.press/v162/nilforoshan22a.html Olsson, F., & Tomanek, K. (2009). An Intrinsic Stopping Criterion for Committee-Based Active Learning (pp. 138–146). https://aclanthology.org/W09-1118 Pavaloiu, A., & Kose, U. (2017). Ethical Artificial Intelligence - An Open Question. Journal of Multidisciplinary Developments, 2(2), 15–27. https://arxiv.org/abs/1706.03021v1 Pedro, F., Subosa, M., Rivas, A., & Valverde, P. (2019). Artificial intelligence in education : chal enges and opportunities for sustainable development. MINISTERIO DE EDUCACIÓN. https://en.unesco.org/themes/education-policy- Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12(1), 1–13. https://doi.org/10.1186/S41039-017-0062-8/METRICS Popkhadze, N. (2021). The Good, The Bad and The Ugly: AI in the higher education. Hal innon Tutkimus. https://journal.fi/hallinnontutkimus/article/view/107896 Raman, A., & Rathakrishnan, M. (2019). Blended Learning in Higher Education 4.0. In Redesigning Higher Education Initiatives for Industry 4.0 (pp. 70–84). https://doi.org/10.4018/978-1-5225-7832-1.ch005 Ramineni, C., & Williamson, D. M. (2013). Automated essay scoring: Psychometric guidelines and practices. Assessing Writing, 18(1), 25–39. https://doi.org/10.1016/J.ASW.2012.10.004 Randal , D. M., & Fernandes, M. F. (1991). The social desirability response bias in ethics research. Journal of Business Ethics, 10(11), 805–817. https://doi.org/10.1007/BF00383696/METRICS Roll, I., & Wylie, R. (2016). Evolution and Revolution in Artificial Intelligence in Education. International Journal of Artificial Intel igence in Education, 26(2), 582–599. https://doi.org/10.1007/S40593-016-0110-3/TABLES/8 San Pedro, M. O. Z., & Baker, R. S. (2021). Knowledge Inference Models Used in Adaptive Learning. Methodology of Educational Measurement and Assessment, 61–77. https://doi.org/10.1007/978-3030-74394-9_5 Sangapu, I. (2018). Artificial intelligence in education-from a teacher and a student perspective. Available at SSRN 3372914. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3372914 Shonubi, O. (2023). AI In The Classroom: Pros, Cons And The Role Of EdTech Companies. https://www.forbes.com/sites/theyec/2023/02/21/ai-in-the-classroom-pros-cons-and-the- role-of-edtech-companies/?sh=65b4b3dcfeb4 J. van Meerten et al.: Responsible Implementation of AI in Higher Education: Ethical Factors Guiding Dutch IT Teachers 355 Smit, K., & Van Meerten, J. (2021). A Narrative Exploration of Improper AI Design and Execution and Possible Effects on Human Values. ACM International Conference Proceeding Series, 149–153. https://doi.org/10.1145/3507485.3507508 Smit, K., Zoet, M., & Van Meerten, J. (2020). A Review of AI Principles in Practice. Association for Information Systems, 6–22. https://aisel.aisnet.org/pacis2020 Tashakkori, A., & Creswel , J. W. (2007). Editorial: The New Era of Mixed Methods. Journal of Mixed Methods Research, 1(1), 3–7. https://doi.org/10.1177/2345678906293042 Torres-Rivera, A. D., Díaz-Torres, L. A., & .. (2021). Applications of artificial intelligence in the higher education. ICERI2021 …. https://library.iated.org/view/TORRESRIVERA2021APP Tsivitanidou, O., & Ioannou, A. (2021). Envisioned pedagogical uses of chatbots in higher education and perceived benefits and chal enges. … : Games and Virtual Environments for Learning …. https://doi.org/10.1007/978-3-030-77943-6_15 Vereniging Hogescholen. (2023). https://www.vereniginghogescholen.nl/kennisbank/feiten-en-cijfers Verma, A., Kumar, Y., & Kohli, R. (2021). Study of AI techniques in quality educations: chal enges and recent progress. SN Computer Science. https://doi.org/10.1007/s42979-021-00635-3 Vlachos, A. (2008). A stopping criterion for active learning. Computer Speech & Language, 22(3), 295– 312. https://doi.org/10.1016/J.CSL.2007.12.001 Warner, R., & Sloan, R. H. (2023). How AI Unfairly Tilts the Playing Field: Privacy, Fairness, and Risk Shadows. SSRN Electronic Journal. https://doi.org/10.2139/SSRN.4518510 Waters, A., & Mi kkulainen, R. (2014). GRADE: Machine Learning Support for Graduate Admissions. AI Magazine, 35(1), 64–64. https://doi.org/10.1609/AIMAG.V35I1.2504 Wildgaard, L. (2015). A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar. Scientometrics, 104(3), 873–906. https://doi.org/10.1007/S11192-015-1608-4/METRICS Xiao, Y., & Watson, M. (2019). Guidance on Conducting a Systematic Literature Review. Journal of Planning Education and Research, 39(1), 93–112. https://doi.org/10.1177/0739456X17723971/ Yu, K., Gong, R., Jiang, C., Hu, S., Sun, L., & .. (2018). A study on the effect of continuing education with digital technology on professional growth and job satisfaction of librarians. EURASIA Journal of …. https://www.ejmste.com/article/a-study-on-the-effect-of-continuing-education-with-digital-technology-on-professional-growth-and-job-5510 Zhai, X. (2022). ChatGPT User Experience: Implications for Education. SSRN Electronic Journal. https://doi.org/10.2139/SSRN.4312418 356 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION ETHNICAL ANTHROPOMORPHISM IN HUMAN- ROBOT INTERACTION: PERSONALIZED ROBOT TUTORS ANNEMAE VAN DE HOEF, SAM LEEWIS, KOEN SMIT, ŁUKASZ MIĘTKIEWICZ, EMIRHAN TALA, FÁBIO SEIJI SAKAI IWASHIMA, BERK TAHIR KILIÇ, MIRJAM DE HAAS HU University of Applied Sciences Utrecht, Digital Ethics, Utrecht, Netherland annemae.vandehoef@hu.nl, sam.leewis@hu.nl, koen.smit@hu.nl, lukasz.mietkiewicz@hu.nl, emirhan.atal@hu.nl, fabio.sakaiiwashima@hu.nl, berk.kilic@hu.nl, mirjam.dehaas@hu.nl Social robots are becoming increasingly relevant in education, for example, by using them as tutors. To create a more empathetic and engaging learning environment, it is important to consider the anthropomorphism of these social robots. However, an ethnic perspective on the use of anthropomorphization is still lacking when it comes to improving learning gains. Therefore, this research focuses on whether personalized, ethnicity-based anthropomorphization of a robot can enhance learning gains. To this end, history lessons were tutored with a Furhat robot, with groups of participants interacting with a Furhat whose face matched the ethnicity of the participants, in an experimental setting. Our results showed that participants who interacted with Keywords: the robot displaying the personalized, ethnicity-based human- robot anthropomorphization learned more than participants interaction, interacting with a robot displaying a robotic appearance. These social findings highlight the importance of incorporating cultural robot, personalized, diversity into educational technologies to foster more effective education, and inclusive learning environments. ethnicity DOI https://doi.org/10.18690/um.fov.4.2024.20 ISBN 978-961-286-871-0 358 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction In today’s world, technological advances offer a plethora of opportunities to improve many aspects of our lives (Bello et al., 2021; Nordrum, 2023). Using advanced technologies in various fields reflects a larger movement toward digital transformation (Baker, 2014). Among these technologies, social robots have the potential due to their ability to interact and engage with humans, creating a richer learning environment (Ayoko, 2021). Social robots have the potential to adapt themselves to each individual, capable of performing various educational tasks, from tutoring in specific subjects to facilitating language learning (e.g., Belpaeme, Kennedy, et al., 2018; Vogt et al., 2019). Moreover, they can provide opportunities for personalized learning, where robots adjust their teaching strategies to cater individual needs and learning styles of each student, thereby creating a shift toward digital, inclusive, and student-centered education (Belpaeme, Kennedy, et al., 2018; Cailloce, 2017). To potential y improve the robot’s performance, anthropomorphism (i.e., how much the robot’s attributes resemble a human) is applied in their design (Alves- Oliveira et al., 2016; Belpaeme, Kennedy, et al., 2018; Liew et al., 2022). For example, Mohd Tuah et al. (2016), propose an anthropomorphism design to guide better human-computer interactions. In addition, Eysell & Kuchenbrandt (2012) used anthropomorphism to investigate whether a robot’s ethnicity affects the participants’ social categorization. Furthermore, concerning ethnicity, Makatchev et al. (2013) investigated how a robot can be more ethnical y accurate not in terms of appearance but in verbal and non-verbal communication. Little research has been found, however, on the impact of anthropomorphism on learning gains when applying personalized ethnicity to a tutoring social robot. Indeed, research indicates that students often achieve higher learning gains when taught by teachers of the same ethnicity (Gottfried et al., 2023; Redding, 2019), suggesting a similar potential effect between students and robot tutors. Moreover, research showed that when humans interact with robots, homophily (i.e., having something in common) correlates with building trust in human-robot interaction (HRI) (Salek Shahrezaie et al., 2021). Therefore, this research paper describes a study that examines whether you can improve student performance by ethnical y anthropomorphizing a social robot. Accordingly, the following research question is raised: “To what extent does a A. van de Hoef et al.: Ethnical Anthropomorphism in Human-Robot Interaction: Personalized Robot Tutors 359 tutoring robot enhance the performance of its learners when using personalized and ethnicity-based anthropomorphization? ” 2 Background and Related Work Research has shown that social robots can have many benefits in the education field (Alves-Oliveira et al., 2016; Belpaeme, Kennedy, et al., 2018; Belpaeme, Vogt, et al., 2018; Donnermann et al., 2022; Gleason & Greenhow, 2017; Ramachandran et al., 2016; Rosenberg-Kima et al., 2020; Smakman et al., 2020; van den Berghe et al., 2019; Vincent et al., 2015). For example, social robot tutors can be beneficial, as suggested by research in which social robots were used to assist children in learning a second language (e.g., Vogt et al., 2019) or solving fraction problems (Ramachandran et al., 2016). Here, the robot acted as a language tutor, providing personalized vocabulary lessons and feedback. The results of this research showed that children who interacted with the robot showed improvement compared to those who did not receive vocabulary lessons. However, the effect of the robot on their performace was not clear (Vogt et al., 2019). On the other hand, gestures by a robot seem to have a positive effect on children's engagement (de Haas et al., 2020). Most research involving robots in an educational setting has concentrated on one- on-one interactions. Nevertheless, group settings could also impact learning gains positively. For example, the use of social robots in small group activities helped manage learning by introducing tasks, ensuring proper time management, and encouraging group discussions between students (Rosenberg-Kima et al., 2020). In addition, another study has implemented an adaptive robot tutor to support students with exam preparation (Donnermann et al., 2022). They found that students interacting with a robot with a personalized and more human-like behavior scored higher on the exam and had an increase in intrinsic motivation related to the course content in general compared to students who interacted with a robot that did not adapt itself to the participants. Other research came to a similar conclusion that designing the educational robots as more anthropomorphic, or at least fitting their demographic (i.e., children), results in better learning rates and positive social interactions (Belpaeme, Vogt, et al., 2018; van den Berghe et al., 2019; Vincent et al., 2015). Furthermore, some papers have shown results that an anthropomorphic social robot encourages responses that are beneficial for learning because it invites social interaction with the robot (Belpaeme, Kennedy, et al., 2018). A more general 360 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION example is that human-robot interaction puts a great emphasis on psychology and behavior for a more engaging and communicative interaction (Alves-Oliveira et al., 2016). Beyond the traditional education roles of social robots, there is an interesting implementation of robots to enhance the benefits of social robots: robots featuring ethnical anthropomorphism. For example, the same ethnicity between students and teachers results in students receiving fewer negative behavior ratings and being perceived as more favorable in terms of academic ability, especial y among Black and Latino/Latina students (Redding, 2019). Furthermore, students achieved higher scores in, e.g., math and reading (Gottfried et al., 2023) when taught by teachers of the same ethnicity. In contrast, some studies found no significant positive relationship between ethnic similarity and learning gains (Driessen, 2015; Hughes et al., 2005; Tom & Cronan, 1998). A review by Driessen (2015) reports that the results of 24 quantitative studies were mixed, and the article suggests that even if the studies found some positive results, these were more related to subjective teacher evaluations than to objective achievement outcome measures. While there are mixed opinions on the benefits of having the same ethnicity as a teacher on student performance, the effect of students’ interaction with a personalized ethnic social tutoring robot on students’ learning gains has not yet been studied. However, research has been conducted on ethnic anthropomorphism and robots. For example, Eysell & Kuchenbrandt (2011) concluded that German participants found a robot that represented the majority (in this case, a German ethnicity) more favorable (e.g., felt closer to or received more warmth from) than a robot that represented the minority (in this case, a Turkish ethnicity). Another angle explored how verbal and non-verbal communication can be used to represent ethnicity through a robot rather than a potential y offensive ethnic appearance (Makatchev et al., 2013). In addition, Mohd Tuah et al. (2016) proposed an anthropomorphism design scale, from anthropomorphism to animism, to guide a better understanding of how anthropomorphism can be used. Given the research on ethnicity and education and the current state of (ethnic) anthropomorphisms in research, it is suggested that an effect observed in human- human interactions may differ when similar approaches are applied in human- robot interactions. Therefore, this research aims to investigate whether social robots that implement ethnic anthropomorphism increase the history knowledge of students at an University of A. van de Hoef et al.: Ethnical Anthropomorphism in Human-Robot Interaction: Personalized Robot Tutors 361 Applied Sciences. We hypothesize that a robot that adapts its appearance to participants’ ethnicity improves history knowledge in a setting where it tutors history lessons to groups of participants (H1). 3 Research Method The experiment is conducted with the Furhat robot made by Furhat Robotics and students at an University of Applied Sciences. The robot’s face is projected onto a facial mask, making it possible to adapt its face based on the participants’ ethnicity. The faces used to represent certain ethnicities are pre-made by Furhat Robotics (as shown in Figure 1). Figure 1: Furhat Robotics Pre-made Faces Source: Adapted from Furhat Robotics: Furhat Robotics. (2024). Furhat SDK (Version 2.7.2) [Software] Researchers assigned faces to the robot based on the ethnic population the participants wanted to represent themselves with. The participants identified themselves as Dutch, American, Brazilian, European, Turkish, German, Syrian, Finnish, Tunisian, Arabic, Surinamese, Austrian, Afghan, Caribbean, French, and/or Angolan. The participants were divided into two groups: one with a robotic face (control) and the other with an appearance based on the participant’s ethnic identity (treatment), as shown in Figure 2. The control and treatment groups were then 362 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION divided into smaller groups of five participants. These groups of five were formed by the researchers based on random sampling, and congruent ethnicity and gender. If there were groups of participants with mixed ethnicities, they were placed in the control group. The participants in the treatment group were grouped based on the ethnicity they had specified beforehand and according to the available faces as shown in Figure 1. This way, the face of the robot matched the ethnicity of each participant in the treatment group. Figure 2: Experiment Design Visualization Source: Own The experiment took place at HU University of Applied Sciences Utrecht where the control group consisted of 31 participants and the treatment group of 30 participants. The robot automatical y changed its attention from one participant to another during the tutoring session, with each participant receiving equal attention. The robot in the control group did not make human-like head movements, had no human-like features, and had a robotic face. In contrast, the robot in the treatment group did make human-like head movements, wore a hat to appear more human, and changed its voice (e.g., male or female) and face to match the participants’ ethnic identity. Both before and after a tutoring session, each participant is asked about their confidence on the topic from 0 (No confidence) to 10 (Ful y confident) (e.g., How confident are you about your knowledge of the Osman Pasha who fought in A. van de Hoef et al.: Ethnical Anthropomorphism in Human-Robot Interaction: Personalized Robot Tutors 363 Plevna?). In this manner, participants who were too knowledgeable about this topic were excluded in advance and it was clear afterward whether participants understood the topic. The session consisted of a monologue by the Furhat robot about the history of the Battle of Plevna and a test with 10 multiple-choice questions (e.g., Why did Russian troops want to capture Plevna? (A) Because it was rich in resources (B) It was a key place to move towards Istanbul (C) Gazi Osman Pasha was born there (D) It was the capital of the Ottoman Empire). No pre-test is conducted for the 10 multiple-choice questions as this could affect the post-test results and the little knowledge of the topic during the pre-test might demotivate the participants. The topic of the experiment is chosen due to the researchers’ familiarity with it and the low chances of participants having prior knowledge of it. The variable used to represent the learning gains is the total number of correct answers scored by each participant on the test. Upon arrival, participants signed a consent form and were asked two questions about their confidence level in the subject before the tutoring session (mean ≈ 0.1, median = 0, range = 1 for the treatment group, mean ≈ 0.4, median = 0, range = 5 for the control group). The researchers then made the participants sit in chairs in front of the Furhat robot, which stood on top of a table so that it could see al the participants’ faces. While participants waited for the tutoring session to start, the Furhat robot uttered its idle lines every 15-20 seconds, e.g., “Just waiting till everyone is set” or “Take your time.” When the session started, the robot greeted the participants, introduced itself, and showed the participants that it could change its face, voice, and language. From here, the Furhat robot’s script changed based on the control and treatment conditions. The Furhat robot operator then changed the robot’s face and voice to match the ethnicity the participants identified themselves with beforehand. Then, participants in both conditions were informed by the Furhat robot that they would be tested regarding the information they would receive during the 30-minute tutoring session. The Furhat robot would then give information about Gazi Osman Paşa and the Plevna’s war. At the end of the tutoring session, it thanked everyone for participating in the experiment and informed the participants that they were not al owed to cheat or choose random answers. Participants were asked to leave the answer blank if they did not know the answer. After answering two questions about their confidence in the subject after the tutoring session (mean ≈ 5.6, median = 6, range = 9 for the treatment group, mean ≈ 5.2, median = 5, range = 6 for the control group), the participants took the multiple-choice test. 364 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4 Data Analysis and Results Data analysis was performed on the provided answers to the multiple-choice test. In addition, the furhat robot was programmed using Python 3 where the following modules were used: Scipy, Pandas, Seaborn, Pylab, and Matplotlib. A t-test was considered, but two assumptions could not be met. The Barlett’s test confirmed that there were no significant differences between the variances of the test results of the two groups (p-value ≈ 0.64). Furthermore, the data were not normal y distributed. Therefore, it was chosen to conduct a non-parametric test Mann-Withney U test. Data analysis results revealed that participants scored higher in the treatment condition (median = 7, range = 10) than in the control condition (median = 5, range = 7). As Figure 2 shows, the result of the Mann-Whitney U-test (W = 601, effect size = 0.29, p-value = 0.048) indicated a statistically significant positive effect of the treatment variable, meaning that participants scored higher when interacting with the robot adapting its appearance to the participants’ ethnicity. Figure 3: Difference between the control and treatment group Source: Own Moreover, the groups consisted of approximately 65% males and 35% females. However, their performance (mean ≈ 6.5, median = 7, range = 10 for males, mean ≈ 6.3, median = 7, range = 8 for females) did not reveal significant results. A. van de Hoef et al.: Ethnical Anthropomorphism in Human-Robot Interaction: Personalized Robot Tutors 365 6 Discussion and Future Research This research aimed to check for the difference in knowledge about the history of the Battle of Pevna between participants interacting with an ethnical y personalized face robot and a robotic face robot, hypothesizing that an ethnical y personalized robot could enhance knowledge by improving the educational environment. The results revealed that participants who interacted with the ethnical y personalized robot performed significantly better than participants in the control condition. The results may indicate that the effects observed in previous research, where students achieve higher scores in, e.g., math and reading (Gottfried et al., 2023), also occur when students interact with robots with an ethnical y personalized face. The same holds for more positive academic skills among Black and Latino/Latina students when taught by teachers of the same ethnicity. In other words, it could mean that personalizing a robotic tutor’s appearance to match students’ ethnic identity leads to better learning outcomes. However, the effect could also be attributed to the treatment robot’s more anthropomorphic appearance. It is possible that would, in turn, indicate that anthropomorphism influenced the result (Alves-Oliveira et al., 2016; Belpaeme, Kennedy, et al., 2018; Liew et al., 2022) more than ethnic personalization. Several limitations were observed in the study. Firstly, a few participants reported being more focused on the robot’s head movements than on its speech. While it may have caused some of them to score lower, the issue was unlikely to influence the between-group difference as it occurred in both groups. Secondly, we placed participants with different ethnicities in the control group, which could have influenced the dynamic of these groups and could have influenced the results. Furthermore, the quality of Furhat’s faces and voices left a lot to be desired. Making the faces more detailed as wel as improving the quality of the voices could increase the treatment variable’s impact or show other effects. Finally, the multiple-choice test consisted of 10 questions. Moreover, the results of this study are consistent with positive results on higher learning gains as stated by Gottfried et al. (2023) and Redding (2019), however are not consistent with the mixed results on learning gains as stated by Driessen (2015). A more comprehensive test should be conducted in the future to further validate the results. Moreover, future research could expand on the results of this research in many ways. Analyzing the outcomes of such research would help assess whether the difference in the target variable was caused more by 366 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION anthropomorphization than ethnic-personalization or vice versa. Furthermore, focusing on participants’ ethnicity instead of identity could lead to completely different findings and mitigate difficulty in interpreting their answers. By considering a participant’s ethnicity (e.g., urbanicity (Jang, 2020) or social characteristics), a deeper understanding is gained of how such a social robot can be personalized even more to further improve learning gains. Finally, it would also be beneficial to experiment with an alternative subject of teaching. For example, how a more visual subject, such as art or mathematics, affects students’ learning gains when taught by a social robot. We argue that extending the research on the use of a social robot to tutor multiple subjects contributes to a more general solution and knowledge about the advanced customization of social robots in practice. 7 Conclusion The purpose of this research was to examine the difference in history knowledge between people interacting with a Furhat tutor with a robot face and a Furhat tutor with an ethnical y personalized face, which raised the fol owing research question: “To what extent does a tutoring robot enhance the performance of its learners when using personalized and ethnicity-based anthropomorphization? ” An experiment was conducted where a history tutoring session with the Furhat was given to a treatment and control group. Here, the treatment condition would interact with a more human-like Furhat with an ethnical y personalized face and the control condition would interact with a more robot-like Furhat. After the tutoring session, the groups would take a multiple- choice test on the subject of teaching and submit the results, from which their knowledge could be assessed. The results revealed that participants interacting with an ethnical y personalized appearance during the tutoring session did receive better outcomes than those interacting with a robot with a robotic appearance. The result suggests that, in this study’s context, personalized ethnical anthropomorphization implemented in robotic tutors could increase the knowledge about the subject of their users. Furthermore, the results revealed that biological sex does not seem to significantly influence the outcome, in the context of this study. From a theoretical point of view, this research contributes to the body of knowledge of social robotics, as well as that it provides initial insights into the use of ethnic anthropomorphization in social robots used in education. From a practical point of view, this research contributes to the practical application of social robots in education and A. van de Hoef et al.: Ethnical Anthropomorphism in Human-Robot Interaction: Personalized Robot Tutors 367 demonstrates that social robots can be personalized towards someone’s ethnicity to boost knowledge as compared to social robots without ethnicity personalization. References Alves-Oliveira, P., Küster, D., Karpas, A., & Paiva, A. (2016). Psychological Science in HRI: Striving for a More Integrated Field of Research. https://doi.org/http://dx.doi.org/10.13140/RG.2.2.21549.49129 Ayoko, O. B. (2021). Digital Transformation, Robotics, Artificial Intelligence, and Innovation. Journal of Management & Organization, 27(5), 831–835. https://doi.org/10.1017/jmo.2021.64 Baker, M. (2014). Digital Transformation (4th ed.). Createspace Independent Publishing Platform. Bel o, S. A., Oyedele, L. O., Akinade, O. O., Bilal, M., Davila Delgado, J. M., Akanbi, L. A., Ajayi, A. O., & Owolabi, H. A. (2021). Cloud computing in construction industry: Use cases, benefits and challenges. Automation in Construction, 122, 103441. https://doi.org/10.1016/j.autcon.2020.103441 Belpaeme, T., Kennedy, J., Ramachandran, A., Scassel ati, B., & Tanaka, F. (2018). Social robots for education: A review. Science Robotics, 3(21). https://doi.org/10.1126/scirobotics.aat5954 Belpaeme, T., Vogt, P., van den Berghe, R., Bergmann, K., Göksun, T., de Haas, M., Kanero, J., Kennedy, J., Küntay, A. C., Oudgenoeg-Paz, O., Papadopoulos, F., Schodde, T., Verhagen, J., Wal bridge, C. D., Wil emsen, B., de Wit, J., Geçkin, V., Hoffmann, L., Kopp, S., … Pandey, A. K. (2018). Guidelines for Designing Social Robots as Second Language Tutors. International Journal of Social Robotics, 10(3), 325–341. https://doi.org/10.1007/s12369- 018-0467-6 Cailloce, L. (2017, January 31). The Challenges of Social Robotics. CNRS News. https://news.cnrs.fr/articles/the-chal enges-of-social-robotics Donnermann, M., Schaper, P., & Lugrin, B. (2022). Social Robots in Applied Settings: A Long-Term Study on Adaptive Robotic Tutors in Higher Education. Frontiers in Robotics and AI, 9. https://doi.org/10.3389/frobt.2022.831633 Driessen, G. (2015). Teacher ethnicity, student ethnicity, and student outcomes. Intercultural Education, 26(3), 179–191. https://doi.org/10.1080/14675986.2015.1048049 Eyssel, F., & Kuchenbrandt, D. (2012). Social categorization of social robots: Anthropomorphism as a function of robot group membership. British Journal of Social Psychology, 51(4), 724–731. https://doi.org/10.1111/j.2044-8309.2011.02082.x Gleason, B. W., & Greenhow, C. (2017). Hybrid Education: The Potential of Teaching and Learning with Robot-Mediated Communication. Online Learning, 21(4). https://doi.org/10.24059/olj.v21i4.1276 Gottfried, M., Little, M., & Ansari, A. (2023). Student-Teacher Ethnoracial Matching in the Earliest Grades: Benefits for Executive Function Skil s? Early Education and Development, 34(8), 1799–1815. https://doi.org/10.1080/10409289.2023.2172674 Hughes, J. N., Gleason, K. A., & Zhang, D. (2005). Relationship influences on teachers’ perceptions of academic competence in academical y at-risk minority and majority first grade students. Journal of School Psychology, 43(4), 303–320. https://doi.org/10.1016/j.jsp.2005.07.001 Jang, S. T. (2020). Contextualized Effects of Racial/Ethnic Matching Between Students and Teachers in Urban, Suburban, and Rural High Schools. Teachers Col ege Record: The Voice of Scholarship in Education, 122(11), 1–34. https://doi.org/10.1177/016146812012201110 Liew, T. W., Pang, W. M., Leow, M. C., & Tan, S.-M. (2022). Anthropomorphizing malware, bots, and servers with human-like images and dialogues: the emotional design effects in a multimedia learning environment. Smart Learning Environments, 9(1), 5. https://doi.org/10.1186/s40561-022-00187-w 368 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Makatchev, M., Simmons, R., Sakr, M., & Ziadee, M. (2013). Expressing ethnicity through behaviors of a robot character. 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 357–364. https://doi.org/10.1109/HRI.2013.6483610 Markowitz, A. J., Bassok, D., & Grissom, J. A. (2020). Teacher-Child Racial/Ethnic Match and Parental Engagement With Head Start. American Educational Research Journal, 57(5), 2132– 2174. https://doi.org/10.3102/0002831219899356 Mohd Tuah, N., Wil s, G., & Ranchhod, A. (2016, March). The characteristics and application of anthropomorphic interface: a design spectrum. Nordrum, A. (2023, December 8). 5 things we didn’t put on our 2024 list of 10 Breakthrough Technologies. MIT Technology Review. https://www.technologyreview.com/2023/12/08/1084728/leftovers-2024-list-10- breakthrough-technologies/ Ramachandran, A., Litoiu, A., & Scassellati, B. (2016). Shaping productive help-seeking behavior during robot-child tutoring interactions. 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 247–254. https://doi.org/10.1109/HRI.2016.7451759 Redding, C. (2019). A Teacher Like Me: A Review of the Effect of Student–Teacher Racial/Ethnic Matching on Teacher Perceptions of Students and Student Academic and Behavioral Outcomes. Review of Educational Research, 89(4), 499–535. https://doi.org/10.3102/0034654319853545 Rosenberg-Kima, R. B., Koren, Y., & Gordon, G. (2020). Robot-Supported Col aborative Learning (RSCL): Social Robots as Teaching Assistants for Higher Education Smal Group Facilitation. Frontiers in Robotics and AI, 6. https://doi.org/10.3389/frobt.2019.00148 Salek Shahrezaie, R., Anima, B. A., & Feil-Seifer, D. (2021). Birds of a Feather Flock Together: A Study of Status Homophily in HRI (pp. 281–291). https://doi.org/10.1007/978-3-030-90525-5_24 Smakman, M., Berket, J., & Konijn, E. A. (2020). The Impact of Social Robots in Education: Moral Considerations of Dutch Educational Policymakers. 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 647–652. https://doi.org/10.1109/RO-MAN47096.2020.9223582 Tom, T. L., & Cronan, T. A. (1998). The effects of ethnic similarity on tutor–tutee interactions. Journal of Community Psychology, 26(2), 119–129. https://doi.org/10.1002/(SICI)1520- 6629(199803)26:2<119::AID-JCOP2>3.0.CO;2-S van den Berghe, R., Verhagen, J., Oudgenoeg-Paz, O., van der Ven, S., & Leseman, P. (2019). Social Robots for Language Learning: A Review. Review of Educational Research, 89(2), 259–295. https://doi.org/10.3102/0034654318821286 Vincent, J., Taipale, S., Sapio, B., Lugano, G., & Fortunati, L. (Eds.). (2015). Social Robots from a Human Perspective. Springer International Publishing. https://doi.org/10.1007/978-3-319-15672-9 Vogt, P., van den Berghe, R., de Haas, M., Hoffman, L., Kanero, J., Mamus, E., Montanier, J.-M., Oranç, C., Oudgenoeg-Paz, O., García, D. H., Papadopoulos, F., Schodde, T., Verhagen, J., Wal bridgel , C. D., Wil emsen, B., de Wit, J., Belpaeme, T., Göksun, T., Kopp, S., … Pandey, A. K. (2019). Second Language Tutoring Using Social Robots: A Large-Scale Study. 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 497–505. https://doi.org/10.1109/HRI.2019.8673077 DIGITAL SUSTAINABILITY PRACTICES: A RESEARCH AGENDA IRENE R. JONKERS, XIAO PENG, WILLEMIJN VAN HAEFTEN University of Applied Sciences Utrecht, Utrecht, Netherland irene.jonkers@hu.nl, xiao.peng@hu.nl, wil emijn.vanhaeften@hu.nl Expectations are high for digital technologies to address sustainability related challenges. While research into such applications and the twin transformation is growing rapidly, insights in the actual daily practices of digital sustainability within organizations is lacking. This is problematic as the contributions of digital tools to sustainability goals gain shape in organizational practices. To bridge this gap, we develop a theoretical Keywords: perspective on digital sustainability practices based on practice digital theory, with an emphasis on the concept of sociomateriality. We sustainability, argue that connecting meanings related to sustainability with twin transformation, digital technologies is essential to establish beneficial practices. practice Next, we contend that the meaning of sustainability is context- theory, specific, which calls for a local meaning making process. Based sociomateriality, sustainability, on our theoretical exploration we develop an empirical research digital agenda. technologies DOI https://doi.org/10.18690/um.fov.4.2024.21 ISBN 978-961-286-871-0 370 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction As reflected in a frequently cited statement of Dutch transition scientist Rotmans “we aren’t living in an era of change, but in a change of era” 1, it is broadly recognized that our societies are quickly evolving due to several major trends. Two of those trends concern the rapid advancement of digital technologies, and the growing acknowledgment of the urgency of the sustainability agenda. Both trends trigger transformation of organizations and societies at large. In academia, the concept of ‘twin transformation’ (e.g. Fouquet & Hippe, 2022; Graf-Drasch et al., 2023) or ‘dual transformation’ (e.g. Kürpick, Kühn, et al., 2023; Kürpick, Rasor, et al., 2023) is gaining ground, denoting an intertwined connection between the digital and sustainability transformation. Whilst the label attached to the concept implies that there are two transformation processes that develop simultaneously, it is often used to refer to one organizational transformation process in which digital technologies are implemented to advance the sustainability agenda (e.g., Graf-Drasch et al., 2023). Or, in other words, a process in which the enabling properties of the digital technologies are connected to the meanings and goals of the sustainability agenda (Kürpick, Rasor, et al., 2023). Alternately, the term ‘ digital sustainability’ is applied, which refers to “the organizational activities that seek to advance the sustainable development goals through creative deployment of technologies that create, use, transmit, or source electronic data” (George et al., 2021, p. 1000). To emphasize the intended outcome of the organizational change process, in this paper we prefer the latter term. In digital sustainability on the one hand digital technologies are employed to help a transformation towards a sustainable organization and societies (e.g., Chatzistamoulou, 2023; Feroz et al., 2021), and on the other hand the value driven sustainability agenda can help move to a more human driven digitalization (e.g., Nahavandi, 2019). In the upcoming literature, attention is mostly paid to possible applications of digital tools for sustainability (Carvalho & da Silva, 2021). However, knowing what tools can do to advance sustainability, does not suffice to understand if and how connections with sustainability are meaningfully made in daily organizational practices. To understand digital sustainability in organizations, we therefore cal for research from a practice approach. Practice theory refers to a range 1 See: https://janrotmans.nl/, statement translated from Dutch by authors. [last accessed 6 May 2024] I. R. Jonkers, X. Peng, W. van Haeften: Digital Sustainability Practices: A Research Agenda 371 of sociological theoretical perspectives that give ontological primacy to everyday sayings and doings of people, by zooming in on the interconnections between agents, rules and expectations (explicit and implicit), and material elements. In this paper we relate primarily to the perspective of Shove et al. (2012), in which they present an analytical framework of the building blocks of practices: competences, tools, and meanings. Further emphasizing the inseparable relationships between practitioners and digital technologies, we relate to the concept of ‘sociomateriality’ which emphasizes that technologies gain shape in practices (Orlikowski & Scott, 2008). In this paper we argue that a deeper understanding of the connections between practitioners, sustainability related meanings, and digital technologies in organizational practices is a starting point to understand the potential synergy between reactions to the mega two trends. The overal aim is to propel further research in this area. Without disregarding the relevance of developing digital sustainability competences to form digital sustainability practices, here we focus on the connection of digital technologies and sustainability related meanings. After elaborating on practice theory and sociomateriality, we advance our argument by demonstrating the lack of a uniform understanding of the concept of sustainability. We distill starting points available in literature, and finally claim that the meaning of sustainability should be context specific, which calls for local meaning making processes. We develop our ideas theoretical y and end with a research agenda for empirical substantiation. 2 Twin transformation Focusing on digital technologies and sustainability, we interact with the terms ‘twin transformation’ and ‘twin transition’. These originate from the policy domain (Diodato et al., 2023), and more specifical y from the EU at which level the requirement of digitalization to achieve the decarbonization goals was acknowledged (Fouquet & Hippe, 2022). Guandalini (2022) concluded that manifest attention to the topic in the management literature is lacking, which is surprising as it is acknowledged that both the sustainability transformation (ST) (e.g., Millar et al., 2012; Sancak, 2023) and the digital transformation (DT) require fundamental organizational changes (e.g., Gong & Ribiere, 2020; Hanelt et al., 2021). Combining the DT with ST, Graf-Drasch et al. (2023) define twin transformation as: “a 372 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION fundamental organisational change process that enables organisations to address digital and societal chal enges synergistical y by harnessing the power of DT to enable ST and leveraging ST to redesign DT” (p. 4). The focus in this definition is on the way that digital and sustainability related changes are addressed simultaneously and synergistically on the organizational level. The required knowledge for the twin transformation hence goes far beyond examples of application of specific technological tools for sustainability, and rather requires insights in the embedding of such solutions in organizational practices, as wel as the competences needed for this embedding process. To establish alignment to existing and ongoing work, in this paper we do accord to the term ‘twin transformation’ to refer to the change process that is involved in applying digital technologies to address sustainability related issues. We prefer the term transformation over transition due to underlying notions of the two terms. From an etymological perspective transition relates to the process of moving from one state to another. Transformation, on the other hand, means a change in shape (Hölscher et al., 2018). We hold the perspective that we find ourselves in fundamental changes in organizations, economies and societies at large of which the ‘new’ is stil ‘becoming’, and use the term transformation to refer to this process of deep change “that requires new ways of thinking and behaving, (. ) is major in scope, discontinuous with the past and generally irreversible” (Quinn, 1996, p.3). In what follows, we dive into the concepts of ‘digital sustainability’ and ‘practices of digital sustainability’. Looking in more detail on how DT and ST (can) complement each other, our thesis in this paper aligns with the argument of Lock and Seele (2017) that sustainability should form the normative core of digitalization efforts. According to a study of Kürpick et al. (2023) business leaders also tend to perceive DT from an ‘enabler perspective’, as opposed to the ST which is perceived from a ‘target perspective’. To further elaborate this connection, we continue with introducing our theoretical perspective. I. R. Jonkers, X. Peng, W. van Haeften: Digital Sustainability Practices: A Research Agenda 373 3 Conceptualizing digital sustainability 3.1 Relational approach to understand people and technology We relate to theories of practice to conceptualize the linkage between digital developments and the sustainability agenda. Theories of practice are based on a relational ontology and consider social reality being made up of ‘a bundle of practices’ (Schatzki, 2012). Practices can be defined as “shared, routinized, ordinary ways of doings and sayings, enacted by knowledgeable and capable human agents who – while interacting with the material elements that co-constitute the practice – know what to do next in a non-discursive, practical manner” (Spaargaren et al., 2019, p. 8). Practice theories in general consider actions performed by people as inherently social, or as culturally and historically embedded (Reckwitz, 2002). People that are part of a ‘community’ hold shared understandings and norms regarding what sayings and doings are expected in a given setting. Based on these, actions (sayings and/or doings) are (repeatedly) performed in interaction, to achieve a certain outcome. So, practices are goal-oriented. And, the knowledge that practitioners have available regarding the practice is not considered from a cognitive stance, but is of a practical or embodied nature. Practice theory “connects 'knowing' with 'doing'” (Gherardi, 2001, p. 136). Where the earlier theories of practice focused mostly on connecting doings to shared norms and meanings, interest in the role of materiality rose later (Shove et al., 2012). The concept of ‘sociomateriality’ indicates that the social and material are not separate entities but rather that the relations between humans and materials are enacted in practice (Orlikowski & Scott, 2008). Looking at digital developments, new technologies should not be considered as an exogenous force that impact societies. Rather, following Orlikowski’s reasoning, they gain their relevance and meaning when they are employed and become part of our daily doings. Applying the sociomateriality lens to working with data, Mathiassen et al. (2023) cal for a deeper understanding of how distributed organizational actors use digital tools to transfer and consume information. Data are not neutral, rather actors translate information to meanings, and then transform those into action. This involves complex processes of producing, transferring and consuming information; processes which are embedded in practices. Based on a similar framing, Bähr and Fliaster (2023) illustrate that certain digital technological frames lead to more sustainable value propositions. 374 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION In reference to work of Orlikowski and Gash (1994), the authors argue for a distinction between three domains of frames: the nature of the technology (‘what is it?’), the technology strategy (‘why should it be implemented?’), and technology in use (‘how it is used on a daily base?’). To be able to research how practices arise, develop and dissolve, Shove et al. (2012) propose an analytical model that highlights three core elements of practices: (shared) meanings, tools and/or materials, and competencies (the practical know-how to perform a practice). Only when these three elements are meaningful y and iteratively interlinked by practitioners, a practice is formed and sustained. The availability of new digital tools and technologies from this reasoning is not a sufficient condition, but does open possibilities for practices of digital sustainability arising. To conclude, we focus on the question how new technologies and digital tools can be integrated in our daily activities in a way that helps to advance goals related to the sustainability agenda. This is essential to alter the ways things are done and give shape to the transformation. Combining elements of definitions of practice theory (Spaargaren et al., 2019) and digital sustainability (George et al., 2021), we propose the following working definition of digital sustainability practices in turn as: ‘the daily shared, routinized and ordinary ways of interacting of practitioners with digital technologies that seek to advance a transformation towards a system which is both ecologically and socially sustainable in the long term’. We further elaborate on this definition in the upcoming sections. 3.2 Sustainability related meanings Schatzki (2001) talks about the teleoaffective structure which is central to the organizing of practices, indicating that practices are performed to achieve something. In cultural historical activity theory this goal is referred to as the ‘outcome’ of a practice (Engeström, 1987/2019). For Shove et al. (2012) this element is captured in the category ‘meanings’ which refers to a broad category of “symbolic meanings, ideas and aspirations” (p. 14). The authors hold that practices exist when meanings are connected to materials and competences. Existing meanings can be connected to new tools, and meanings are “extended or eroded as a result of dynamic processes of association” (p. 55). When considering digital sustainability I. R. Jonkers, X. Peng, W. van Haeften: Digital Sustainability Practices: A Research Agenda 375 practices, it hence is important to further dive into the meanings associated with sustainability. The term sustainability originates from the policy domain (Kuhlman & Farrington, 2010) in which attention was drawn to intergenerational equity and a focus on welfare. In the often cited Club of Rome report ‘Limits to growth’ (Meadows et al., 1972) five major global trends are investigated (i.e., accelerating industrialization, rapid population growth, widespread malnutrition, depletion of nonrenewable resources, and a deteriorating environment), and the conclusion is drawn that with no action the limits of growth wil be reached within the century, leading to “a rather sudden and uncontrollable decline in both population and industrial capacity” (p. 23). The authors cal upon people to prepare for “the transition from growth to global equilibrium” (p. 24). This equilibrium denotes a condition of ecological and economic stability which is sustainable into the far future. Within this equilibrium “the basic material needs of each person on earth are satisfied and each person has an equal opportunity to realize his individual human potential” (p. 24). In this sense sustainable means transforming the system for long-term maintenance. Next, it is associated with issues like equality and realizing human potential. In 1987 a UN report (Brundtland, 1987) follows in which sustainable development is depicted as development that “meets the needs of the present without compromising the ability of future generations to meet their own needs” (section 3, no. 27). Again, the term sustainable by itself refers to a state that can be maintained; factors that endanger this state (or present limits to growth) are a downward spiral of poverty, environmental degradation, and inequality. From these foundational reports, we take that sustainability refers to a transformation to reach ‘a state that can be maintained on the long term, both environmentally and socially’. As the term ‘sustainability’ got traction in the academic debate (Salas‐Zapata & Ortiz‐Muñoz, 2019), the policy domain (Leach et al., 2010), and business domains, its meaning diffused. A range of thematic guidelines have been developed to assist organizations in their ST (e.g., the Sustainable Development Goals, the Global Reporting Initiative, the European Sustainability Reporting Standards). Whilst helpful in translating an abstract idea to organizational practice, there are downsides of a thematic approach. With an almost infinite list of issues to ‘pick and choose’ from, the original focus on a fundamental transformation towards a sustainable state is not integrated in al understandings of sustainability (cf., Johnson et al., 2018). 376 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Elkington came to a similar conclusion. In 1994 his triple bottom line brought sustainability to the business world. His framework examines a company’s social, environmental, and economic impact. Elkington (2018) afterwards retracted his framework as he noted that it reverted into an accounting tool based on a trade-off mentality. This, while with his framework he intended to “provoke deeper thinking about capitalism and its future” (p. 2). Similarly Leach et al. (2010) observe that also institutions often view sustainability conservatively, focusing on maintenance rather than transformation. Resilience thinking is one of the approaches that instead gives insight in underlying principles of a sustainable system. A resilient system is one that has the capacity to “absorb change and disturbances, and stil retain its basic structure and function” (Walker & Salt, 2006, p. 113). It is important to not cross thresholds that shifts the current system into a next one, of which it is uncertain what services it wil deliver. Walker and Salt (2006) highlight how we are al part of the system – as ecological and social systems are inextricably linked. In Western societies we tend to live in a paradigm in which humans and nature are separated, and nature is even seen as something that can be exploited (Mazzocchi, 2020). Acknowledging that nature and culture are not to be understood as two separate domains, but instead as one concept (nature/culture) divided in two parts (Latour, 2017) is crucial. As Walker and Salt (2006) further elaborate, systems are complex, consisting of many linkages and feedback-loops. It is this last point, that shows why a ‘trade-off’ mindset to sustainability is unwanted. One of the dangers of the current economic paradigm is a sole focus on efficiency, which reduces variety and flexibility. As Kennedy and Linnenluecke (2022) for instance argue, a sole focus on efficiency can help reduce material usage, but often also lowers costs which in turn increases sales nullifying the environmental benefits. These authors hence, point out the complexity underlying the needed transformation and the dangers of ignoring interconnections with other aspects of the system. Whilst the transference of the resilience concept from the ecological to the social domain is not uncontested (Keck & Sakdapolrak, 2013), it does resonate for instance with ideas about the importance for organizations to be embedded in strong networks of stakeholders (Busch et al., 2018). Reasoning from the original ideas of Brundtland (1987) and Meadows et al. (1972), inequality and poverty can be seen as a danger for the sustainability of a new system. Also here complexity of systems should be taken into account. For example, I. R. Jonkers, X. Peng, W. van Haeften: Digital Sustainability Practices: A Research Agenda 377 removing production processes from the global south, reduces the risk of child labor, but it can have tremendous negative effects on local communities. So where does this leave us in terms of meanings associated with sustainability? It firstly is important to keep in scope the element of transforming the current system into one that can be sustained in the long term in environmental and social terms. This means that interconnectedness, variety and flexibility need to be nurtured. However, the original policy reports (i.e., Brundtland, 1987; Meadows et al., 1972), nor resilience thinking (Walker & Salt, 2006), offer guidance of what kind of sustainable system we want; this is a normative question. The thematic guidelines available can offer guidance. But it is important to acknowledge that normative and value driven choices need to be made. And these choices are always context- bounded (Leach et al., 2010). Moving back to our practice theory framing, it is important to acknowledge that ‘meanings’ are not purely cognitive concepts, but actually are established ‘in use’ (Ramsey, 2015, in reference to the body of work of Wittgenstein). Meanings emerge in practice (Ramsey, 2015), but can be prompted with a collective meaning making process (Jonkers, 2022). As Jonkers (2022) argues, such a meaning making process is of a reflexive and iterative nature and involves steps like specifying, diversifying, connecting to existing meanings, placing it in a broader historical and societal context, and balancing it with other practices and goals. Al of this is necessary to integrate the emergent practice in the total configuration of organizational practices. Relating to the considerations above, in this meaning-making-process, reflections on the system and its interconnections would be paramount to develop practices adding up to a sustainable situation. 3.3 Digital technologies In Shove et al.’s (2012) elemental approach of practices, ‘materials’ is a broad category referring to “objects, infrastructures, tools, hardware and the body itself” (p. 23). In this paper we refer to tools that are based on digital technologies. The digital landscape is quickly evolving, with technologies like Artificial Intelligence, machine learning, the Internet of Things, blockchains, cloud computing, and Augmented or Virtual Reality. Many of the new tools already impact sectors like 378 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION healthcare, finance, and transportation and are reshaping industries. Meanwhile, the pace of technological integration continues to grow. Technologies can be used for good, but also can have dark sides (Trittin-Ulbrich et al., 2021). Nahvandi (2019) indicates that in Industry 4.0 the main focus is on creating efficiency of processes, while ignoring human costs. Efficiency itself can have a negative effect on the resilience of the system we live in (Walker & Salt, 2006). Nahavandi (2019) proposes that sustainability is embraced in the development of Industry 5.0. Karneborgen et al. (2023) emphasize that current leaders “transform their internal organizations while navigating the broader ecosystem simultaneously” (p. 78) to unlock the value of digital technology. Both in public, policy, and academic debates the interest in technological solutions for the sustainability agenda is rising. For instance, a number of digital tools are developed by entrepreneurs to overcome managerial problems that can hinder a transformation to sustainability (George et al., 2021b). Examples of such managerial problems relate to communication towards e.g., customers of sustainable products or the costs associated with coordination across supply chains. In a conference proceeding, Kürpick, Kühn, et al. (2023) describe nine possible applications of technology for sustainability. Examples are data-based life cycle assessment, digital product passports or smart factory infrastructures. These applications are considered to have “a basical y positive impact on sustainability” (p. 179). Two important requirements of applications are also addressed to prevent negative impacts of the solutions: the need for trustful and fair analytics and concerns about energy consumption and e-waste related to the technological applications. Whilst the potential of technologies receives ample attention in the literature, insight in the actual embedding of technologies in practices of digital sustainability is still lacking. Also in other domains, there is limited research available on how data-related technologies are used in work practices (Mathiassen et al., 2023). Porto de Albuquerque et al. (2021) propose a critical research agenda regarding the generation, circulation, and usage of data specifically in the transformation towards sustainability. Questions they pose are for instance: I. R. Jonkers, X. Peng, W. van Haeften: Digital Sustainability Practices: A Research Agenda 379 Who defines which data is being produced and how? Is the data generation building new capacities and critical consciousness or contributing to reduce inequalities? (…) Who defines what counts as data and which data is important? (…) What are the social and material processes for building trust in data and how this shapes decision- making in practice? (p. 160). With formulating such questions, they argue for a more detailed understanding of how data are embedded in socio-material practices. Only from such a perspective, it becomes clear how they actual y intervene in decision-making processes. We concur with this argument, and broaden the added value of a socio-material perspective to the role that digital technologies in general play in a transformation to sustainability. 4 Digital sustainability practices and a research agenda Reflecting on the argument made above, we propose taking a practice approach to deepen our understanding of to digital sustainability, or the twin transformation. By focusing on the interrelation of actors and digital technologies (the sociomateriality) in everyday sayings and doings, we can achieve deeper insight in how technological tools gain shape in the daily organizational realities. Based on the analytical framework of Shove et al. (2012) we further examined the meanings associated with sustainability and conveyed the need of context-bounded meaning making. Additionally, we explored the potential and possible risks associated with technologies available. Based on these considerations we formulated our working definition of digital sustainability practices which focuses on the daily interplay between practitioners and digital technologies in the process of transformation towards a sustainable state. In this last section we develop a research agenda resulting from this framing, to increase our understanding of digital sustainability practices. The agenda is summarized in Table 1. Currently, there is little empirical evidence to build upon. An exception is the study Bähr and Fliaster (2023), revealing that how digital technologies are framed shape both the business digitalization strategies at the firm level, and the contributions to the sustainability transformation. These findings support our cal for further research into the association between sustainability related meanings and digital technologies. In developing our suggestions we follow the others in relating to the distinction of Orlikowski and Gash (1994) between the questions regarding the nature of the 380 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION technology ('what is it?'), the technology strategy ('why should it be implemented?'), and technology in use ('how it is used on a daily base?'). The current literature on digital sustainability/twin transformation mainly focuses on the nature of the technology or what it could do. We shift the focus to the other two levels. We start with elaborating on the strategic aspects. This choice is made based on the limited insight in the actual connections currently made between the ‘what’ of digital technologies and the ‘why’ of sustainability. This is also a pragmatic choice. Researching the daily practices in use, would favor a methodology which includes shadowing of practitioners (e.g., Nicolini, 2012). Such studies are time-consuming and access can be chal enging. To prepare for such studies, we suggest to start building the empirical base for digital sustainability practices with alternative and less invasive approaches. We firstly suggest to investigate how organizations currently link the application of digital technologies with sustainability related meanings. Organizational documents or websites can be taken as a proxy for the connection made between digital technologies and sustainability goals. Larger companies that are committed to sustainability (e.g. B-Corps) can provide a starting point. It is most likely that the digital tools these companies use, are tied to sustainability associated meanings. While documents are not ‘pure’ mirrors of organizational practices, they are linguistic manifestations of discursive practices associated with strategy formation (Nicolini, 2012). By adding an intertextual lens, which ideal y includes a historical perspective, deeper insights into these processes can be gained. Building upon this first research suggestion, it is important to acknowledge that documents also are artefacts that co-constitute practices. Documents are a result of production processes and are the starting points for consumption processes by readers (Prior, 2003). From the latter perspective, strategic documents are assumed to play a role in the technology usage (cf., Bähr & Fliaster, 2023). Research based on document analysis, can be further enriched with interviews. To zoom in on the production of the document, people involved in both the digital and the sustainability strategy can be interviewed. This could shed light on the digital sustainability strategy formation cross the boundaries of digital and sustainability departments. On the other hand, interviews with internal readers of these documents can help to understand their perception of digital sustainability and I. R. Jonkers, X. Peng, W. van Haeften: Digital Sustainability Practices: A Research Agenda 381 potential cals for action they take from the document. It would be interesting to learn if and how the application of digital tools is adjusted based on strategic documents. Studying technology in use would require sufficient time within an organization, likely resulting in case studies. Mapping the interlinked practices associated with a tool or application, could provide a starting point. Next, observing practitioners in each of the steps, while constantly asking ‘why’ questions would result in fine- grained data regarding the interlinkage between tools and meanings in practice. Alternatively, applying the ‘instructions to the double’ interview technique (Nicolini, 2009) could serve as a proxy for observing the practices while executed. Another avenue for further research, which could fit in both categories, is looking at tools based on digital technologies that are available for organizations to adopt. This could be general tools that can be applied to achieve sustainability related goals, or tools that were developed specifical y to help companies in their digital sustainability. Investigating with which meanings these tools are associated upon implementation in context-bounded organizational practices can empirical y substantiate the importance of local meaning making. We conclude this paper by relating back to Ramsey (2015) who pleas for understanding ‘sustainability meanings in use’ based on analysis of organizational practices. Although such empirical insights are valuable, we chal enge the current depth of integration of digital technologies and sustainability related meanings. In the area of digital sustainability we therefore argue to complement more descriptive research, with a critical perspective (e.g., following Porto de Albuquerque et al., 2021). This way, we aim to ultimately encourage organizations to deepen the interconnectedness between the DT and ST. 382 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 1: Overview research agenda Digital Sustainability practices Technology strategy Technology in use Single case studies  Mapping linkages between practices Document analysis related to a specific tool connected to sustainability related meanings  Establishing linkage between meanings and digital technologies  Observation of practitioners involved in the interlinked  Intertextual analysis to track evolvement of meaning and practices, complemented with interconnections to broader probing on the ‘why’ of sayings and discourse doings  Alternative approach based on ‘Instructions to the double’ interviews Interview study  Meaning making and connection to digital technologies (across departments)  Internal readers of documents: meaning consumption and impact on practices Comparative case study, organizations with a similar tool References Bähr, K., & Fliaster, A. (2023). The twofold transition: Framing digital innovations and incumbents’ value propositions for sustainability. Business Strategy and the Environment, 32(2), 920–935. https://doi.org/10.1002/bse.3082 Brundtland, G. H. (1987). Our common future: Report of the World Commission on Environment and Development. United Nations. Busch, T., Hamprecht, J., & Waddock, S. (2018). Value(s) for whom? Creating value(s) for stakeholders. Organization and Environment, 31(3), 210–222. https://doi.org/10.1177/1086026618793962/FORMAT/EPUB Carvalho, R., & da Silva, A. R. (2021). Sustainability requirements of digital twin-based systems: A meta systematic literature review. Applied Sciences, 11(12), 5519. https://doi.org/10.3390/app11125519 Chatzistamoulou, N. (2023). Is digital transformation the Deus ex Machina towards sustainability transition of the European SMEs? Ecological Economics, 206. https://doi.org/10.1016/j.ecolecon.2023.107739 Diodato, D., Huergo, E., Moncada-Paternò-Castel o, P., Rentocchini, F., & Timmermans, B. (2023). Introduction to the special issue on “the twin (digital and green) transition: Handling the economic and social chal enges.” Industry and Innovation, 30(7), 755–765. https://doi.org/10.1080/13662716.2023.2254272 Elkington, J. (2018). 25 Years ago I coined the phrase “Triple Bottom Line.” Here’s why it’s time to rethink it. Harvard Business Review Digital Articles, 2–5. https://search-ebscohost- I. R. Jonkers, X. Peng, W. van Haeften: Digital Sustainability Practices: A Research Agenda 383 com.hu.idm.oclc.org/login.aspx?direct=true&db=buh&AN=130449047&site=ehost- live&scope=site Engeström (1987/2019). Learning by expanding: An activity-theoretical approach to developmental research. Orienta-Konsultit. Feroz, A. K., Zo, H., & Chiravuri, A. (2021). Digital transformation and environmental sustainability: A review and research agenda. Sustainability, 13(3), 1530. https://doi.org/10.3390/su13031530 Fouquet, R., & Hippe, R. (2022). Twin transitions of decarbonisation and digitalisation: A historical perspective on energy and information in European economies. Energy Research & Social Science, 91, 2214–6296. https://doi.org/10.1016/j.erss.2022.102736 George, G., Merrill, R. K., & Schillebeeckx, S. J. D. (2021). Digital Sustainability and Entrepreneurship: How Digital Innovations are helping tackle climate change and sustainable development. Entrepreneurship Theory and Practice, 45(5), 999–1027. https://doi.org/10.1177/1042258719899425 Gherardi, S. (2001). From oganizational learning to practice-based knowing. Human Relations, 54(1), 131–139. https://doi.org/10.1177/0018726701541016 Gong, C., & Ribiere, V. (2020). Developing a unified definition of digital transformation. https://doi.org/10.1016/j.technovation.2020.102217 Graf-Drasch, V., Kauffeld, L., Kempf, L., Maria Oberländer, A., Teuchert, A., & Maria, A. (2023). Driving twin transformation - The interplay of digital transformation and sustainability transformation. ECIS 2023 Research Papers, 5–11. https://aisel.aisnet.org/ecis2023_rp/255 Hanelt, A., Bohnsack, R., Marz, D., & Antunes Marante, C. (2021). A systematic review of the literature on digital transformation: Insights and implications for strategy and organizational change. Journal of Management Studies, 58(5), 1159–1197. https://doi.org/10.1111/joms.12639 Hölscher, K., Wittmayer, J. M., & Loorbach, D. (2018). Transition versus transformation: What’s the difference? Environmental Innovation and Societal Transitions, 27, 1–3. https://doi.org/10.1016/j.eist.2017.10.007 Johnson, J. L., Zanotti, L., Ma, Z., Yu, D. J., Johnson, D. R., Kirkham, A., & Carothers, C. (2018). Interplays of Sustainability, Resilience, Adaptation and Transformation (pp. 3–25). https://doi.org/10.1007/978-3-319-67122-2_1 Jonkers, I. R. (2022 ). How organizational actors social y innovate: An action learning research project in an established organization [PhD thesis Nyenrode Business Universiteit]. Nyenrode Business Universiteit. Can be retrieved from: https://www.nyenrode.nl/docs/default- source/faculty/promoties/irene-r.-jonkers.pdf?sfvrsn=11b27813_3 Karneborgen, P., Oberländer, A. M., & Rövekamp, P. (2023). ‘Unlocking digital value at the intersection of organizational digital transformation and digital business ecosystems’. In: M. K. Brohman, G. S. Dawson, & K. C. Desouza (eds.), Digitalization and sustainability: Advancing digital value (pp. 77-104). Edward Elgar Publishing. Keck, M., & Sakdapolrak, P. (2013). What is social resilience? Lessons learned and ways forward. Erdkunde, 67(1), 5–19. https://doi.org/10.3112/erdkunde.2013.01.02 Kennedy, S., & Linnenluecke, M. K. (2022). Circular economy and resilience: A research agenda. Business Strategy and the Environment, 31(6), 2754–2765. https://doi.org/10.1002/bse.3004 Kuhlman, T., & Farrington, J. (2010). What is Sustainability? Sustainability, 2(11), 3436–3448. https://doi.org/10.3390/su2113436 Kürpick, C., Kühn, A., Olszewski, L., & Dumitrescu, R. (2023). Framework for dual transformation: A systematic literature review on the interplays between digitalization and sustainability. 2023 IEEE Conference on Technologies for Sustainability (SusTech), 175–182. https://doi.org/10.1109/SusTech57309.2023.10129630 Kürpick, C., Rasor, A., Scholtysik, M., Kühn, A., Koldewey, C., & Dumitrescu, R. (2023). An integrative view of the transformations towards sustainability and digitalization: The case for a dual transformation. Procedia CIRP, 119, 614–619. https://doi.org/10.1016/j.procir.2023.02.155 Latour, B. (2017). Facing Gaia: Eight lectures on the new climatic regime. Polity Press 384 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Leach, M., Scoones, I., & Stirling, A. (2010). Dynamic Sustainabilities. Earthscan. Lock, I., & Seele, P. (2017). Theorizing stakeholders of sustainability in the digital age. Sustainability Science, 12(2), 235–245. https://doi.org/10.1007/s11625-016-0404-2 Mathiassen, L., Jonsson, K., & Holmstrom, J. (2023). Tensions in transfer, translation, and transformation of information: A sociomaterial perspective on heterogeneous work arrangements. Journal of Information Technology, 38(3), 334–350. https://doi.org/10.1177/02683962231164426 Mazzocchi, F. (2020). A deeper meaning of sustainability: Insights from indigenous knowledge. The Anthropocene Review, 7(1), 77–93. https://doi.org/10.1177/2053019619898888 Meadows, D. H., Meadows, D. L., Rgen, J. •, Wil iam, R., & Behrens Il , W. (1972). The limits to growth: A report for the Club of Rome’s project on the predicament of mankind. Universe Books. Millar, C., Hind, P., & Magala, S. (2012). Sustainability and the need for change: organisational change and transformational vision. Journal of Organizational Change Management, 25(4), 489–500. https://doi.org/10.1108/09534811211239272 Nahavandi, S. (2019). Industry 5.0-a human-centric solution. Sustainability (Switzerland), 11(16). https://doi.org/10.3390/su11164371 Nicolini, D. (2009). Articulating practice through the interview to the double. Management Learning, 40(2), 195-212. https://doi.org/10.1177/1350507608101230 Nicolini, D. (2012). Practice theory, work, and organization: An introduction. Oxford university press. Orlikowski, W. J., & Gash, D. C. (1994). Technological frames. ACM Transactions on Information Systems, 12(2), 174–207. https://doi.org/10.1145/196734.196745 Orlikowski, W. J., & Scott, S. V. (2008). 10 Sociomateriality: Challenging the Separation of Technology, Work and Organization. The Academy of Management Annals, 2(1), 433–474. https://doi.org/10.1080/19416520802211644 Porto de Albuquerque, J., Anderson, L., Calvil o, N., Coaffee, J., Cunha, M. A., Degrossi, L. C., Dolif, G., Horita, F., Klonner, C., Lima-Silva, F., Marchezini, V., Martins, M. H. da M., Pajarito-Grajales, D., Pitidis, V., Rudorff, C., Tkacz, N., Traijber, R., & Zipf, A. (2021). The role of data in transformations to sustainability: a critical research agenda. Current Opinion in Environmental Sustainability, 49, 153–163. https://doi.org/10.1016/J.COSUST.2021.06.009 Prior, L. (2003). Using documents in social research. Sage Publications. Ramsey, J. L. (2015). On not defining sustainability. Journal of Agricultural and Environmental Ethics, 28(6), 1075–1087. https://doi.org/10.1007/s10806-015-9578-3 Reckwitz, A. (2002). Toward a theory of social practices. European Journal of Social Theory, 5(2), 243– 263. https://doi.org/10.1177/13684310222225432 Salas‐Zapata, W. A., & Ortiz‐Muñoz, S. M. (2019). Analysis of meanings of the concept of sustainability. Sustainable Development, 27(1), 153–161. https://doi.org/10.1002/sd.1885 Sancak, I. E. (2023). Change management in sustainability transformation: A model for business organizations. Journal of Environmental Management, 330, 117165. https://doi.org/10.1016/j.jenvman.2022.117165 Schatzki, T. R. (2012). A primer on practices. In J. Higgs, R. Barnett, S. Bil et, M. Hutchings, & F. Trede (Eds.), Practice-based education: Perspectives and strategies (pp. 13-26). Sense publishers. Shove, E., Pantzar, M., & Watson, M. (2012). The dynamics of social practice: Everyday life and how it changes. Sage Publications. Spaargaren, G., Lamers, M., & Weenink, D. (2019). Introduction: Using practice theory to research social life. In G. Spaargaren, D. Weenink, & M. Lamers (Eds.), Practice theory and research: Exploring the dynamics of social life (pp. 3-27). Routledge. Trittin-Ulbrich, H., Scherer, A. G., Munro, I., & Whelan, G. (2021). Exploring the dark and unexpected sides of digitalization: Toward a critical agenda. Organization, 28(1), 8–25. https://doi.org/10.1177/1350508420968184 Walker, B., & Salt, D. (2016). Resilience thinking: Sustaining ecosystems and people in a changing world. IslandsPress. BRIDGING THE AI SKILLS GAP IN EUROPE: A DETAILED ANALYSIS OF AI SKILLS AND ROLES WILLEMIJN VAN HAEFTEN,1 RAN ZHANG,1 SABINE BOESEN - MARIANI,1 XANDER LUB,1 PASCAL RAVESTEIJN,1 PAUL AERTSEN2 1 University of Applied Sciences Utrecht, Utrecht, Netherland wil emijn.vanhaeften@hu.nl, ran.zhang@hu.nl, sabine.boesen-mariani@hu.nl, xander.lub@hu.nl, pascal.ravesteijn@hu.nl 2 Vrije Universiteit Amsterdam, Amsterdam, Netherland p.w.aertsen@vu.nl This study provides a comprehensive analysis of the AI-related skills and roles needed to bridge the AI skills gap in Europe. Using a mixed-method research approach, this study investigated the most in-demand AI expertise areas and roles by surveying 409 organizations in Europe, analyzing 2,563 AI-related job advertisements, and conducting 24 focus group sessions with 145 industry and policy experts. The findings underscore the importance of both general technical skil s in AI related to big data, machine learning and deep learning, cyber and data security, large language models as well as AI soft skills such as problem- Keywords: AI skil s, solving and effective communication. This study sets the AI roles, foundation for future research directions, emphasizing the Europe, importance of upskilling initiatives and the evolving nature of AI workforce development, skil s demand, contributing to an EU-wide strategy for future AI mixed-method skills development. research DOI https://doi.org/10.18690/um.fov.4.2024.22 ISBN 978-961-286-871-0 386 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The fast developing and evolving field of Artificial Intelligence (AI) is reshaping industries and altering the professional landscape at an unprecedented pace (Jan et al., 2023; ARISA, 2023). Characterized by rapid advancements in machine learning, big data, and computing power, AI technologies are now pivotal in driving economic growth, enhancing productivity, and fostering innovation across various sectors (Anagnostou et al., 2022). From revolutionizing medical diagnostics (Kamdar et al., 2020) to propel ing the development of autonomous vehicles (Namatherdhala et al., 2022), AI's integration into daily operations and strategic decision-making processes underscores its transformative potential. Recognizing this, the World Economic Forum's Future of Jobs report 2023 highlights AI and related roles among the fastest-growing job sectors, emphasizing the critical need for skills training in these areas (World Economic Forum, 2023). To ful y understand and accurately pinpoint what AI-related skil s are needed and what training programs ought to be set up in order to cultivate those skills the ARtificial Intelligence & Skills Alliance (ARISA) Consortium conducted a study with the aims to comprehensively analyze the current AI landscape, emphasizing the critical need for specialized AI roles and skil s within the context of the European Union. Drawing on a mixed-method research approach, including both quantitative and qualitative methods, the study explores the demand for AI professionals, the impact of AI on organizational and societal levels, and the pressing need for informed decision-making within this context. As such, the central research question this study seeks to address is as follows: What are key AI-related skil s and AI roles required by organizations in the European Union in the next five to ten years? By investigating this question, the consortium makes critical contributions to AI research, literature, and European policies by systematical y analyzing AI-related skil s and roles in the EU and uncovering the essential combinations and patterns of technical and soft skil s needed in AI-relevant sectors. Through a mixed-method approach, the findings of the study can provide an improved understanding of current and future demands for AI expertise. In sum, the key contribution of this study is in facilitating the closing of AI-related skills gap as well as setting the stage W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 387 for informed policy-making and strategic upskilling initiatives, thereby guiding the evolution of AI skills development within the EU. 2 Research Background This research aims to investigate the AI roles and the AI-related skil s needed for those roles in relation to the current professionals and skills available in the AI sector. This section wil clarify the scope of this research and specifies the foundation for the AI roles and skil s that are relevant and important. Lack of skil s hinders AI adoption and deployment: besides cost constraint, the lack of skills remains the biggest barrier in AI advancements (IBM, 2022). According to Deloitte’s (2022) survey findings, AI technology cannot deliver transformative results unless organizations reimagine operations and how work is structured and executed (i.e., rethinking operations including the business workflow, and within their IT and data science team processes). Therefore, in the current research, it is key to consider the roles and skil s for AI professionals with a need for further education in their specific areas of expertise. Moreover, this consideration is also important for non-AI professionals (e.g., decision-makers and managers) who need a basic understanding of AI and skills in implementing AI in organizations. AI also can and wil have a fundamental impact on society in general (Crafts et al. 2018; Deranty & Corbin, 2022; Stanford University, 2023). It is important that good, solidly underpinned decisions are made to make the best use of the possibilities of AI, while limiting the risks associated with it. To achieve these well-informed decisions, decision-makers within organizations and policymakers within (governmental) institutions need to possess AI knowledge and skil s. Therefore, this research wil also address these target audiences. Considering AI professionals, the consortium has defined a specific scope of relevant AI-related skills. Stanford University's classification of AI skills served as the foundational structure for this study (Stanford University, 2023). Stanford classification is based on the AI-related skills list and clusters defined by Lightcast (2022) built on extensive analysis of job vacancies and skills taxonomy. The clusters encompass areas such as Artificial Intel igence, Neural Networks, Autonomous Driving, Natural Language Processing, Machine Learning, Robotics, and Visual 388 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Image Recognition. The relevancy of the skills clusters outlined in Stanford University's classification were also verified by reports from OECD (Borgonovi et al., 2023), the World Economic Forum (2023), and by frameworks regarding the most important AI roles from Coursera (2023), Leeds University, (2023), McKinsey, (2022), and Springboard (2023). Five of the seven clusters proposed by Stanford align with the specific focus on AI skills of this research. Clusters related to specific AI-applications (such as autonomous driving), rather than the skil s needed for using these applications, are excluded from the study. However, the skill of processing Data in AI not present among the Stanford (2023) clusters, cannot be overlooked. Acknowledging the role of AI skills in data-related domains, the category 'Data' is therefore added in the classification scheme by the authors. The clusters of AI-related skills in scope of this research are, therefore, as follows: 1) Natural language processing, 2) Visual image recognition (computer vision), 3) Robotics, 4) Machine learning, 5) Neural networks (deep learning) and 6) Data processing and analysis. With the application of AI in businesses, new AI-related roles emerge as organizations create functions dedicated to the needed AI-skill clusters. The AI-roles related to the above identified AI-skills are included in this study. Namely: 1) Machine learning engineers, 2) Natural language processing engineers, 3) Computer vision engineers and 4) Data scientists, analysts, and engineers. The research design builds upon the above named AI-skills clusters and related AI- roles, this design is presented in the section below. 3 Research Methods This section outlines the methodology used to investigate the current and future demand for skil s and roles relating to AI. To better understand the fast-changing AI roles and skills, both quantitative and qualitative data were collected. Given the broad scope of the research topic, a mixed-method approach is particularly appropriate as this approach integrates various methodologies enhances the understanding of complex issues (Molina-Azorin, 2016). The study consisted of an EU-wide industry questionnaire, expert focus groups and a job vacancy analysis. More specifical y, the questionnaire was used to col ect data from industry on their W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 389 current needs for AI roles and skils as wel as in the future. The current need was further investigated by performing a job vacancy analysis using an AI-driven tool. To enrich and expand upon the insights derived from the industry questionnaire and the job vacancy analyses, expert focus group sessions were conducted. The integration of industry questionnaires, job vacancy analyses, and focus groups was strategical y employed to enhance the robustness and comprehensiveness of the research findings. This methodological approach, incorporating both quantitative and qualitative data and the use of multiple research methods, was designed to ensure a multidimensional understanding of the AI skills landscape, improved research reliability and validity, ultimately augmenting the study with diverse perspectives and deepening the analysis. An overview of the data col ected is shown in Table 1. Table 1: Research methods Data col ection in numbers Research method Data col ected Industry questionnaire 409 Responses, 12 EU countries, col ected from October to December 2022 Job Vacancy analysis 2563 Job advertisements, 11 EU countries, posted from November 2022 to May 19th 2023 145 Participants, 12 focus groups of policymakers Expert focus groups and 12 focus groups of industry experts, 12 EU countries, col ected from September 2022 to January 2023 Industry questionnaire The industry questionnaire was developed to gather data from industry stakeholders working in sectors which apply AI to their operations or at early stages of AI adoption. The main aim was to determine current, urgent and emergent demand for AI roles within industries adopting or planning to adopt AI, and whether these roles have a "business" focus or a "technology" focus. Therefore, the consortium formulated the industry questionnaire to capture both technical (e.g.: data management, machine learning, etc.) and transversal skills (e.g.: soft skills) that are perceived as important currently as wel as in the near future based on the organizations' projections. These foci and objectives guided the formulation of the specific questionnaire items used in the industry questionnaire. The questionnaire included items to measure the demand and motivation for considering AI adoption and usage (sample items: "rate the strategic importance of AI and Analytics to your 390 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION organizational growth and development"; "what is your motivation for considering AI within your business? ") and the focus for AI skil s within the organization (sample items: "where is your greatest demand for AI business skil s within your organization?" and "how would you rate the level of the following skills within your organization today?"). Based on the Advanced Technologies for Industry - AT Watch (Cattaneo et al., 2020), a list of sectors was chosen as target groups. Each ARISA consortium partner electronically distributed the questionnaire to the potential respondents fitting the recruitment criteria defined by the consortium (e.g., organizations who are actively adopting or supporting the adoption of AI within the commercial sector). A total of 409 organizations participated in the industry questionnaire between October to December 2022. A detailed overview of the participating organizations is shown in the Results section. Job vacancy analysis To complement the insights obtained from the industry questionnaire, a comprehensive job vacancy analysis was conducted, using an AI-driven tool. A commissioned and specialized organization used this tool to capture and analyze job vacancy advertisements across job platforms across 11 EU countries. Data sources included both National Public Employment services (e.g., Jobtech and Tyomarkkinatori) and Private Job Portals (e.g., Indeed and Hel owork). The extraction of the job vacancy data was done via authorized Application Programming Interfaces (APIs), fully complying with regulatory frameworks put in place by these data sources as wel as other relevant regulatory bodies. To identify the most in-demand AI skills and roles, a total of 756,076 job advertisements posted within the EU area between November 2022 and May 2023 were examined through the Headai Dynamic Ontology (Headai, 2024). Headai Dynamic Ontology (HDO) is a specialized analytical methodology anchored on Natural Language Processing (NLP) techniques and has been used by scientific researchers in the conduct and publication of multiple research studies (Aunimo et al., 2021; Okkonen et al., 2020). This HDO methodology was utilized in this study to scan and analyse job vacancies. Overall, this process involved identifying relevant job titles, extracting skills from job advertisements, mapping these skills to a standardized classification system, and filtering based on exact skil s to focus on the specific requirements of AI-related job vacancies. W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 391 Focus groups with experts Focus group sessions were conducted with policy and industry experts. These focus groups were categorized into two types: those involving experts from the policy domain and those involving experts from industry. Totaling 145 participants, 12 focus groups of policymakers and 12 focus groups of industry experts were held, across 12 EU countries. The focus groups took place from September 2022 to January 2023. The policymaker focus groups aimed to gather information specific to AI-related skills for policymakers. The aim of the policymaker focus groups was to col ect specific insights on AI-related skil s necessary for policymakers while the sessions with industry experts served to validate and supplement the findings from the industry questionnaire. The objective for both sets of focus groups -- policymakers and industry experts -- was to validate the information gathered through questionnaires and to enrich it with detailed qualitative insights from the focus group sessions. 4 Results Industry Questionnaire A total of 409 organizations completed the industry questionnaire. The geographical coverage was based on the consortium partner countries. Of the 409 organizations, 53% were categorized as large-scale entities with more than 250 employees; approximately one-fourth of the organizations represented in the sample employed fewer than 50 individuals; 21% employed between 50 and 249 individuals. As such, the sample's constitution ful y represented both large-scale organizations and small- to-medium-sized enterprises (SMEs). The results from the Industry Questionnaire provided significant insights into the evolving needs of organizations concerning AI roles. In terms of AI roles, Data Scientists, Machine Learning Engineers, and Data Engineers emerge as preeminently sought-after technical positions (see Figure 1). The prominence of these roles underscores their indispensable contribution to the efficacious use of AI technologies by organizations. Further analysis of the variances in demand for these technical roles between the current landscape and the future reveals discernible fluctuations. 392 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 1: Technical AI roles (now and future) Source. From AI skil s needs analysis: An insight into the AI roles and skil s needed for Europe. (ARISA, 2023, p. 22. CC 4.0) In terms of non-technical, managerial decision-making roles that necessitate proficiency in AI skil s, the results showed that Project Managers, Product Managers, and Business Unit Managers were identified by the sampled organizations as the positions demonstrating the most pronounced levels of demand and significance. Moreover, the roles of Financial Managers, Auditors, and C-level Executives (CxOs) have been observed to manifest a significant increase in reported demand. This trend shows a growing acknowledgment of the essential role these decision-making positions occupy in the incorporation of AI technologies into financial management, auditing practices, and executive-level strategic determinations (see Figure 2). Figure 2: Managerial decision-making roles requiring AI skil s (now and future) Source. From AI skil s needs analysis: An insight into the AI roles and skil s needed for Europe. (ARISA, 2023, p. 23. CC 4.0) W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 393 Concerning AI skils, the results of the Industry Questionnaire revealed that individuals possessing the competence to understand business opportunities arising from AI implementation are greatly needed. Recognizing and utilizing the advantages of AI technologies for organizational benefit is key to guiding strategic decisions and improving competitive positioning. Additional y, the importance of individuals who can source high-quality data is emphasized. The skills required to collect, manage, and organize data critical for the success of AI applications and models are vital for producing reliable and actionable intelligence. Lastly, there is an essential demand for securing AI knowledge and expertise. Attracting individuals with a thorough understanding and skill in AI technologies is essential for efficiently managing the chal enges associated with AI development, deployment, and upkeep. A detailed visualized breakdown of the above-mentioned AI skil s is shown in Figure 3. The Industry Questionnaire included a series of questions regarding the soft skills necessary for the effective implementation of AI within organizations. Analysis of the answers revealed the specific soft skil s considered crucial by organizations for managing the intricacies of AI deployment. Specifical y, problem-solving, critical thinking, effective communication, and an innovative mindset were identified as the key soft skil s by the sampled organizations (see Figure 4). Of these, problem-solving was rated as the most important, indicating its pivotal role in overcoming complex issues and crafting viable solutions in environments driven by AI. Additionally, critical thinking was also highly ranked by the respondents for its importance, reflecting the role critical thinking would play in fostering analytical thought and rational judgement in AI-related decision-making scenarios. Figure 3: AI skil requirements Source: From AI skil s needs analysis: An insight into the AI roles and skil s needed for Europe. (ARISA, 2023, p. 24. CC 4.0) 394 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 4: Soft skil s for AI success Source: From AI skil s needs analysis: An insight into the AI roles and skil s needed for Europe, by ARISA, 2023, p. 25. CC 4.0 Analysis of Job Vacancies Using the above mentioned HeadAI Dynamic Ontology (HDO) method, this research identified and analyzed, among the 756,076 job vacancies, 2,563 AI-specific job advertisements to deduce the skil s demanded within the sector. The search query was based on job roles that were sourced from policy papers and forecast reports (such as IBM, 2022), on exact skills (‘unsupervised machine learning') software industry experts and also ESCO classification system. HDO's innovative interactive interface facilitated the exploration of these skills, showcasing their interconnections and relevance to various roles within the AI domain. The skil s extracted from the job vacancies are grouped into five primary clusters: general technical skills, big data & data analytics, machine learning & deep learning, cyber and data security, and large language models. These clusters were generated based on the frequency and closeness of links between various skil s identified from the dataset. Using this methodology, the researchers were able to provide illustrative visualizations of the present and future skill demands in the AI field by grouping skills into relevant categories and showing the interactions and overlaps between these groupings (see Figure 5). W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 395 Figure 5: HDO interface showing AI roles and skil s Source: From AI skil s needs analysis: An insight into the AI roles and skil s needed for Europe, (ARISA, 2023, p. 27. CC 4.0) General technical skills The analysis underscored the importance of a broad skilset encompassing both generic and specific technical skills. High demand was observed for skills in programming languages such as Python, Java, JavaScript, and C++, indicating the multi-disciplinary nature of AI. Additionally, traditional ICT skills, including structured query language (SQL) and Big Data, continue to hold significant value in the job market. The findings suggest a need for comprehensive training that spans beyond current buzzwords to include a wide array of technical topics and soft skil s like teamwork. Big data and data analytics This cluster, closely associated with data science, machine learning, data management, and security, is central to AI. The analysis revealed a strong emphasis on the business application of data analytics, including business intelligence, systems architecture, process automation, and the integration of technology with business strategies. Additional y, the significance of security and renewable technology within this context was highlighted, pointing towards a broad scope of relevant skil s. 396 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Machine Learning and Deep Learning As the most expansive and interconnected skil cluster, machine learning and deep learning are deemed essential for understanding AI. These areas are fundamental to data literacy within AI, underscoring the necessity for these topics to be at the core of AI training and education across all job roles and industries. Cyber security and data security The importance of cyber and data security spans the entire ICT sector, with a specific emphasis on AI. The analysis stressed that understanding AI security involves more than just knowledge of protocols or certifications; it requires a comprehensive grasp of how security mechanisms are integrated within AI systems. Language Models The emergence of language models, particularly Large Language Models (LLMs), has introduced a rapidly growing demand for skills in natural language processing (NLP), deep learning, and neural networks. The dynamic nature of LLM applications necessitates focused attention in education and training, highlighting the fast-paced evolution in this area. Role-specific insights The analysis also provided insights into specific roles, such as data scientists and data engineers, illustrating the different skill emphases between closely related positions. For data scientists, a combination of technical skills and soft skills like team collaboration, business development, and project management was highlighted. In contrast, data engineers' skil sets were noted to focus more on technical aspects, such as mathematics, statistics, and machine learning algorithms. Focus Groups The results from the expert focus groups have yielded critical insights into the evolving landscape of AI skills requirements and policy-making strategies across the European Union. The following findings are derived from focus group discussions with 41 policymakers and 104 industry experts across various EU countries. W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 397 Policymakers' AI skil s needs • Roles and levels of policymakers The focus groups underscored the necessity for AI skils among policymakers at both national and European levels. It was identified that not only policymakers but also their advisors need to be equipped with relevant AI knowledge to make informed decisions impacting society at large. The discussions and discourse during the expert sessions highlighted the importance of upskil ing a wide array of policymakers, including those in non-technical fields such as education, environment, health, and the economy. A significant emphasis was placed on the roles of national chief information officers (CIOs) and government ministers, alongside their ministries and advisors, as crucial targets for AI skills enhancement. • AI skills and knowledge The discussions revealed an urgent need for basic AI knowledge among policymakers to address legal, ethical, and technical chal enges posed by AI technologies. Quotes from participants such as "We cannot wait. The future is now and af ects everybody" and "One of the problems we face in the near future is that policymakers know too lit le about how it real y works" underline the immediate need for upskilling in AI. The focus shifted towards the significance of comprehensive training in AI for policymakers, emphasizing the integration of legal experts, ethical advisors, and strategic technology advisors in policy formulation processes. Organizational AI skills needs • Skills for decision-makers The need for organizational decision-makers to adapt to AI developments through change management skills was particularly noted. Skills in understanding the relevance of data and data science, managing risks, and formulating digital strategies were deemed essential. The importance of fostering an organizational culture conducive to AI adoption was also highlighted, with quotes such as " fostering a culture of experimentation", and " culture is important for change". 398 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • Skills for AI professionals Technical AI skills such as data science, data analysis, machine learning, and deep learning emerged as highly valuable. The discussions highlighted the necessity of not only technical but also soft and business-related skills for AI professionals. Quotes like " There is a strong need for soft skil s" and " AI practitioners need skil s such as understanding business processes, ability to present and sel a product, customer relations, conflict management" illustrate the multifaceted skillsets required for AI roles. In sum, the expert focus groups have elucidated the critical areas of AI skils needs among policymakers and organizational decision-makers. The urgency for AI education and training, coupled with the necessity for a diverse set of skil s ranging from technical to soft skil s, underpins the recommendations for bridging the AI skills gap. The insights drawn from these discussions will be instrumental in shaping future policies and strategies for AI skil s development across the European Union. 5 Discussion, Conclusions, and Implications The objective of this research was to identify and pinpoint essential skil s and roles in the area of AI. Through analyzing job vacancies, conducting focus group sessions, and administering an industry questionnaire, this study uncovered significant insights into the prevailing needs and demands for these AI-related skills and roles in Europe. Results show that there is a strong demand for AI professionals -- data scientists, machine learning engineers, and data engineers, while the need for data analysts may be experiencing a potential decrease. Additionally, managerial roles including project managers, product managers and business unit managers have been identified as crucial positions requiring knowledge and skills relating to AI. These results are broadly in line with and further refine previous work (e.g., Bukartaite & Hooper, 2023; Cramarenco et al., 2023; Sofia et al., 2023) and highlight AI development directions in Europe for the future. The findings of this research highlight the complex landscape of AI related skil s demands. On the one hand, five main clusters of technical AI-related skills that are needed are underscored, namely 1) general technical skil s, 2) big data & data W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 399 analytics, 3) machine- and deep learning, 4) cyber- and data security and 5) language models. In these skil s, one can recognize the need for strong technical foundations. Additionally, soft skills such as effective communication and ethical considerations also emerged in this research, identified as necessary to effectively col aborate and essential to understand both end-user needs and to continuously consider possible downsides. On the other hand, this research places an emphasis on the critical importance of basic AI-knowledge and skills for policymakers and organizational decision-makers, ensuring AI-fundamentals are taken into account in the decision- making and policy-making process of organizations utilizing and regulating AI. The findings of this study have impactful implications for businesses and industries, governmental institutions, and society at large. On the governmental front, the creation of supportive policy frameworks is essential to facilitate the rapid development of AI skills. This includes foundational AI literacy for those in policymaking and decision-making positions to enable informed governance and strategy formulation. Investing in upskilling programs, particularly those addressing urgent skill gaps in AI, will be crucial for maintaining Europe's competitive edge in the global AI arena. Moreover, governments must craft regulations that ensure ethical AI usage, balancing the drive for innovation with considerations of privacy, security, and ethics. For citizens and society at large, adopting a mindset geared towards lifelong learning is imperative in the age of AI, with an emphasis on acquiring both technical and transversal AI skills. The ethical implications of AI necessitate broad engagement, advocating for technologies that are inclusive, equitable, and beneficial to all. Community engagement with educational institutions and industries is vital to ensure the equitable distribution of AI benefits and to make skill development opportunities accessible to a diverse population, thereby fostering an inclusive approach to AI advancement in Europe. In summary, this study casts light on the vital need for key AI-related skills and roles as well as a collaborative and adaptable approach to advancing AI skills in Europe, underscoring the importance of keeping pace with the rapid innovations in AI technology. Researchers are encouraged to integrate these findings into broader strategic efforts aimed at mitigating skill shortages through a variety of initiatives. Future research should prioritize the design and assessment of pilot projects that 400 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION directly tackle the identified skill shortages, alongside efforts to share knowledge and best practices to continue addressing the AI skills gap in Europe. This work will be instrumental in informing an EU-wide strategy for AI re-skilling and upskilling, reflecting a comprehensive approach to equipping the workforce with the necessary skil s to navigate the chal enges and opportunities presented by continued AI development. Acknowledgements This paper presents the findings of a European research initiative undertaken within the framework of the ARtificial Intelligence & Skills Alliance (ARISA) project's Needs Analysis Work Package. The individuals credited as authors of this paper have contributed in various capacities, including research, writing, and editorial roles, to the development of the ARISA Needs Analysis Report. It is important to recognize that the Needs Analysis Report and the research conducted also benefited greatly from the contributions of other researchers affiliated with the ARISA Project Consortium. The authorship and editorial teams of the Needs Analysis Report are ful y listed as follows: Authors: Paul Aertsen (Global Knowledge France), Carmel Somers (Digital Technology Skills Limited), Xander Lub (University of Applied Sciences Utrecht), Pascal Ravesteijn (University of Applied Sciences Utrecht), Ran Zhang (University of Applied Sciences Utrecht). Editors & Reviewers: Mateja Baebler (CCIS), Bálint Gyires-Tóth (Budapest University of Technology), Jose Martinez-Usero (DIGITALEUROPE), Kerli Saarelaid (BCS Koolitus), Ants Sild (BCS Koolitus), Katarzyna Udala (DIGITALEUROPE), Willemijn van Haeften (University of Applied Sciences Utrecht), Blaz Zupan (University of Ljubljana). The ARISA Needs Analysis Report has been made accessible to the public under the Creative Commons license 4.0. For further information and to view the report, please visit the official website of the ARISA Project at www.aiskil s.eu. The ARISA project has received funding from the European Union’s Erasmus+ Programme under grant agreement No 101056236. References Anagnostou, M., Karvounidou, O., Katritzidaki, C., Kechagia, C., Melidou, K., Mpeza, E., .. & Peristeras, V. (2022). Characteristics and chal enges in the industries towards responsible AI: a systematic literature review. Ethics and Information Technology, 24(3), 37. ARISA. (2023). AI skil s needs analysis: An insight into the AI roles and skil s needed for Europe. https://aiskills.eu/wp-content/uploads/2023/06/ARISA_AI-Skills-Needs-Analysis_V1.pdf Aunimo, L., Alamäki, A. V., & Ketamo, H. (2021). Big data governance in agile and data-driven software development: A market entry case in the educational game industry. In Research Anthology on Privatizing and Securing Data (pp. 344-364). IGI Global. Borgonovi, F., Calvino, F., Criscuolo, C., Samek, L., Seitz, H., Nania, J., . . & O’Kane, L. (2023). Emerging trends in AI skil demand across 14 OECD countries. W. van Haeften et al.: Bridging the AI Skills Gap in Europe: A Detailed Analysis of AI Skil s and Roles 401 Bukartaite, R., & Hooper, D. (2023). Automation, artificial intelligence and future skills needs: an Irish perspective. European Journal of Training and Development, 47(10), 163-185. Cattaneo, G., Siviero A., Micheletti G., Carosel a G. and Magnani I. (2020). Advanced Technologies for Industry – AT Watch: Technology Focus on Artificial Intel igence. https://nightingalehq.ai/pdfs/AT-Watch-AI.pdf Coursera (2023). Six Artificial Intel igence (AI) Jobs to Consider in 2023. https://www.coursera.org/articles/artificial-intelligence-jobs Crafts, N., Ghahramani, Z., McLean, A., & Wilson, A. (2018). The impact of artificial intel igence on work: an evidence synthesis. The Royal Society and British Academy. Cramarenco, R. E., Burcă-Voicu, M. I., & Dabija, D. C. (2023). The impact of artificial intelligence (AI) on employees’ skil s and well-being in global labor markets: A systematic review. Oeconomia Copernicana, 14(3), 731-767. Deloitte (2022). Becoming an AI-fueled organization, Deloitte’s State of AI in the Enterprise, 4th Edition. Deranty, J. P., & Corbin, T. (2022). Artificial intelligence and work: a critical review of recent research from the social sciences. AI & Society, 1-17. Headai (2024, February). Science. https://www.headai.com/science IBM (2022). IBM Global AI adoption index 2022. IBM Corporation. https://www.ibm.com/downloads/cas/GVAGA3JP Jan, Z., Ahamed, F., Mayer, W., Patel, N., Grossmann, G., Stumptner, M., & Kuusk, A. (2023). Artificial intelligence for industry 4.0: Systematic review of applications, challenges, and opportunities. Expert Systems with Applications, 216, 119456. Kamdar, J. H., Jeba Praba, J., & Georrge, J. J. (2020). Artificial intelligence in medical diagnosis: methods, algorithms and applications. Machine Learning with Health Care Perspective: Machine Learning and Healthcare, 27-37. Leeds University (2023). What are the top 5 jobs in AI? https://pg-online.leeds.ac.uk/blogs/top-5-jobs- in-ai/ Lightcast. (2022). Annual AI Report. https://lightcast.io/resources/research McKinsey (2022). The state of AI in 2022—and a half decade in review. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2022- and-a-half-decade-in-review Molina-Azorin, J.F. (2016). Mixed methods research: An opportunity to improve our studies and our research skil s. European Journal of Management and Business Economics, 25(2), 37-38. Namatherdhala, B., Mazher, N., & Sriram, G. K. (2022). Uses of Artificial Intelligence in Autonomous Driving and V2X communication. International Research Journal of Modernization in Engineering Technology and Science, 4(7), 1932-1936. Okkonen, J., Ketamo, H., Lindsten, H., Rauhala, T., & Viteli, J. (2020). Using AI to Decrease Demand and Supply Mismatch in ITC Labour Market. In Advances in Human Factors in Training, Education, and Learning Sciences: Proceedings of the AHFE 2020 Virtual Conference on Human Factors in Training, Education, and Learning Sciences, July 16-20, 2020, USA (pp. 310-316). Springer International Publishing. Sofia, M., Fraboni, F., De Angelis, M., Puzzo, G., Giusino, D., & Pietrantoni, L. (2023). The impact of artificial intelligence on workers’ skills: Upskilling and reskilling in organisations. Informing Science: The International Journal of an Emerging Transdiscipline, 26, 39-68. Springboard (2023). 10 Awesome & High-Paying AI Careers to Pursue in 2023. https://www.springboard.com/blog/data-science/careers-in-ai/ Stanford University (2023). Artificial Intel igence Index Report 2023. https://ai ndex.stanford.edu/report/ World Economic Forum. (2023). Future of Jobs Report 2023. https://www.weforum.org/reports/thefuture-ofjobs-report-2023/ 402 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION MARITIME TRANSPORT AND LOGISTICS DIGITAL SOLUTIONS OPTIMIZATION USING ADVANCED DATA SHARING PLATFORMS: EPICENTER PROJECTS' CASE ZDRAVKO PALADIN,1 MIRA ŠOROVIĆ,1 SANJA BAUK,2 FRANK KNOORS,3 JULIAN STEPHENS,4 NEXHAT KAPIDANI,1 ŽARKO LUKŠIĆ,1 RASIM MUJALOVIĆ5 1 Administration for Maritime Safety and Port Management of Montenegro, Bar, Montenegro zdravko.paladin@pomorstvo.me, mira.sorovic@pomorstvo.me, nexhat.kapidani@pomorstvo.me, zarko.luksic@pomorstvo.me 2 Estonian Maritime Academy, Tallinn University of Technology, Talinn, Estonia bsanjaster@gmail.com; sanja.bauk@taltech.ee 3 Transporeon (a Trimble Company), Antwerp, Belgium frank_knoors@trimble.com; 4 MJC2, Berkshire, United Kingdom jas@mjc2.co.uk; 5 JadroAgent, Bar, Montenegro rasi@t-com.me The maritime transport sector within the multimodal transport chain aims to increase overal efficiency by using advanced technological solutions for planning and management operations, including handling vast amounts of data like cargo, positions, trade flows, vessels, and terminals. Al these requires an integrated approach to data management is required in the form of multi-sectoral data-sharing platforms with process interoperability. Hence, this paper describes international legislation requirements related to maritime data sharing provisioned by International Maritime Organization (IMO)and United Nations (UN), along withsome recent technical solutions used in the maritime sector as Maritime Single Window (MSW), Port Community System (PCS), and smart containers on blockchain platforms. Also, the Keywords: paper provides an overview of the EU-H2020 project ePIcenter data focused on multimodal transportation chains efficiency. Some features sharing, maritime of high-level data integration, exchange, and decision support modules, logistic, such as Transporeon and Synchromodality optimization, are described transport, as well. ports DOI https://doi.org/10.18690/um.fov.4.2024.23 ISBN 978-961-286-871-0 404 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Contemporary management of multimodal transport flows, maritime routes scheduling, and big port capacities, requires the consistent, harmonized, and reliable networking environment among involved stakeholders. This can be achieved by utilization of safe, advanced, and optimized digital solutions for relevant information and data sharing. For this purpose, the maritime sector goes through a digital transition and adopts the latest technical developments to meet the efficiency requirements of international trade players and optimize the port cal s process (Lind et al., 2018; 2020). In general, data sharing could be considered as a computer-supported cooperative sharing of available resources on the network and through communication channels for establishing the interaction and coordinatedly use of shared databases. Data are organized, stored, analyzed, and retrieved in a systematic way through a Database Management System (DBMS) and shared over cross-platforms that comprise peer- to-peer systems, client-server platforms, and centralized or decentralized platforms (Ahmad, et al., 2011). New technologies, especial y those related to data management, already transformed the logistics and transport sector and various applications for data collection and advanced data analytics tools. This encompasses advanced systems and concepts such as the (Industrial) Internet of Things ((I)IoT), blockchain (distributed ledger platforms), Machine learning (ML) algorithms, artificial intelligence (AI)-based solutions, digital twins, etc., (Kirsten, 2018; Surucu et al., 2023). The purpose of this paper is to review the mandatory documentation and international regulations in maritime data exchange and reporting. It also aims to show the benefits of the latest Information Technology (IT) development trends in the form of data-sharing platforms, including the tools deployed for maritime transport and logistics within the EU H2020 ePIcenter project. The remaining part of the paper unfolds as follows: Chapter 2 presents the environmental scan of several important initiatives for maritime and transport data sharing platforms supported with Port Community System (PCS) and blockchain technologies for containers and dangerous cargo; Chapter 3 gives research approach; Chapter 4 analysis mandatory regulations, documents, and standards for data- Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 405 sharing provisions issued by IMO and UN. Within this chapter the MSW concept is reviewed; Chapter 5 provides case study based on ePIcenter project results related to digital transformation and smart containers in the maritime industry and logistics, comprising data sharing and optimization platforms used in the “ePI-Link” demonstrator. Discussions and concluding remarks are given in Chapter 6. 2 Environmental scan The following is an overview of some of the most recent platforms for the exchange of maritime data. One of the recently introduced data collection and sharing systems among the Adriatic sea authorities is the ADRIREP system. This is a platform for vessels with dangerous cargo reporting (Šorović et al., 2023a). Another important platform is the Sea Traffic Management(STM), launched in the Baltic region for optimization of safety, efficiency, and environmental aspects in maritime transport (Šorović et al., 2023b). Also, special attention should be paid to maritime transport chain data flow resilience in terms of ensuring cybersecurity. Maritime shipping companies and terminals were exposed to cyber attacks that caused prolonged outages or disruptions in timely freight distribution (Maatsch, Jović, et al., 2022; Ntshangase and Bauk, 2024). To achieve better interconnectivity and efficiency of maritime surveil ance, data sharing and safety response, the harmonization of Vessel Traffic System (VTS) services and training on these platforms is necessary for al maritime transport involved agencies (Šorović et al., 2023c). The relevant institution in the maritime regulations area is the International Port Community System Association (IPCSA), which plays a very important role in shipping and port logistics concerning the PCS of users. This organization is competent in developing and supporting initiatives for trade facilitation and greater transparency in supply chain and logistics flows. The IPCSA prioritizes new emerging technologies with maritime applications, such as Blockchain Bill of Lading, a logistic visibility task force, and Network of Trusted Networks(Paladin et al., 2024; Jaiman et al., 2022). In accordance with the mentioned, there was a justified need for the creation of a PCS, as a neutral platform for electronic data sharing and exchanging, related to cargo, amongst al parties involved in this process (Bezić et al., 2011). Thanks to the harmonization between PCS and MSW, the most important goal has been reached: the elimination of data duplication. In such a manner, once entered data is visible to all seaport system stakeholders, developing more structured 406 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION and functionalbusiness operations. Also, PCS aids port stakeholders in reducing logistics costs, facilitating faster cargo delivery, boosting economic growth, and reducing externalities like pollution and harmful emissions (Tijan et al., 2012; 2018). In the last decade1, the European Union (EU) has put an effort into creating the European MSW(EMSW) to completely harmonize the interfaces that ship operators can use to deliver necessary data throughout the EU. The EMSW aims to standardize the data required for port management to ensure al relevant stakeholders have access to the submitted information openly. The national MSW and EMSW need to be synchronized and completely unified with the PCS to enhance business processes. Al the parties engaged in running a certain port cluster share and aggregate data through the PCS. Many nations have highly developed PCS systems in place of their underdeveloped national MSW, which is quite varied. Unlike the national MSW, there isn't a universally applicable, standardized PCS model for all ports. Since each PCS is unique, because every nation has unique laws, each port community creates its own PCS based on its requirements. Every PCS should connect over the national MSW interface to prevent data duplication and expedite business procedures (Kapidani et al., 2015; Tijan et al., 2019; Jović et al., 2022). Another important concept are the smart containers on blockchain, the area that is still in its infancy and under-researched, but relevant for further development and ful introduction of blockchain in maritime container transport. The number of academic articles on the subject is quite limited. However, we have found several relevant manuscripts. The white paper (UNECE, n.d.) gives an overview of the perspectives of smart containers in the context of IoT, big data, data pipelines, and blockchain technology (Fig. 1). With regards to blockchain, Iakushkin et al. (2019) paper deals with the architecture of the system for tracking and tracing(T&T) smart containers using blockchain, while the data on the location and status of the smart containers are recorded on the 1 Since June 2015, parties engaged in maritime trade and transportation within the EU have been required by law to submit information and documents through an electronic SW to comply with reporting requirements. Individual data items should only be submitted once (Directive 2010/65/EU of the European Parliament and of the Council, 2010). The agreement between the EC and the European Parliament and Council about the implementation of the EMSW, which is anticipated to come into effect in 2025, was signed at the beginning of 2019, and wel received by the maritime sector, including the European maritime ports. Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 407 permissioned blockchain, Hyperledger Iroha, e.g. Elmay et al. (2022) present data and documentation flows in container transport in unimodal and multimodal logistics, using the InterPlanetary File System and the Ethereum. Baygin et al. (2022) demonstrate blockchain-based smart containers T&T by UHF RFID chips. In this case, Solidity language was used, and different experiments were performed with Ganache, Truffle, and Metamask blockchain platforms. Furthermore, the Meyer et al. (2019) article gives a framework for blockchain interoperability in T&T smart containers in the Physical Internet (PI) environment. Bauk et al. (2023) developed a prototype for T&T smart container based on Raspberry Pi and Python, and plan to extend this project towards data integration of smart container on the blockchain. The impediments on blockchain mainstream implementation in maritime, not only in cargo T&T, as a case study, are considered in Bauk and Ntshangase (2023). The structure of related data, events, document flow, and network participants’ relations wil be the subject of further research in this domain. Figure 1: Smart container technology, data, players, and benefits Source: Own research. 3 Research design This paper is based on exploratory desktop analyses of the IMO and UN regulations and standards for data exchange and trade facilitation. Based on the same research principles, the paper provides a comprehensive presentation of the unified MSW platform. This manuscript also includes a case study, which is based on the latest research results from the ePIcenter project. The case study is focused on Transporeon solution and Synchromodal optimization model. 408 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4 Mandatory IMO and UN regulations on data sharing The IMO as the world's top institution for the establishment of general regulations and provisions for every aspect of global maritime affairs, through its working groups and together with the UN institutions, constitutes, updates, discusses, adopts, and final y publishes the regulations and documents applied in every member state countries. One of these working groups is the Expert Group on Data Harmonization (EGDH), which significantly contributed to setting and advancing the system of data sharing and electronic business regulation. Its core documents is the “Compendium on Facilitation and Electronic Business”, a reference manual for creating and harmonizing the systems needed to support the transmission, receipt, and response of information required for the arrival, stay, and departure of the ship, persons, and cargo via electronic data exchange (IMO, EGDH 3/20/1, 2020). To include e-business solutions beyond those related to the “Convention on Facilitation of International Maritime Traffic” (FAL Convention), the formal document FAL 42 is extended to address port logistics operational data for digital exchange between the port and the ship (IMO, FAL 42/8, 2018). However, the major barriers to adopting FAL requirements for electronic data exchange are the following (IMO, Facilitation Committee, FAL 45/5, 2021): • Multi-stakeholder interests in port communities and established practices should enable the reuse and data sharing, to make electronic reporting and clearance of vessels, cargo, crew, and passengers efficient. • The legal framework is a barrier as it can depend on competing and/or overlapping public administrations. Herewith, the IMO Data Set (DS) is constructed to identify al the data elements for reporting information and the IMO Reference Data Model (RDM) for the underlying hierarchical data structure used in electronic data exchange. The IMO DS combined with the IMO RDM promotes harmonization used for electronic business from the World Customs Organization, the UN Economic Commission for Europe, and the International Organization for Standardization (ISO) TC 8. It is proved that data harmonization contributed to the MSW concept implementation, which is a high-level priority of the IMO. This process increased international trade efficiency, by simplifying communications among stakeholders in an electronic Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 409 information environment that promotes accountability, transparency, informed decision-making, and interoperability (ePIcenter D1.7, 2022). Apart from mandatory IMO regulations related to data sharing and reporting, there are multiple international data standards in use in maritime trade. Among many of these protocols, the two most relevant ones are UN/EDIFACT and Extensive Markup Language (XML). The top UN institution, the UN Centre for Trade Facilitation and Electronic Business (UN/CEFACT), is responsible for the launching, development, and governance of communication standards in the world trade business (UNECE/CEFACT, 2013; 2017). The UN/CEFACT improves the ability of business, trade, and administrative organizations to exchange products and relevant services effectively. The UN/EDIFACT and related standards, are used for establishing the MSW and other electronic platforms, containing the following prepositions (UNECE/CEFACT, 2011; 2020): 1. involvement of relevant trade and transport stakeholders; 2. standardized information and formalized documents; 3. single entry point in fulfilling regulatory requirements, and 4. single submission of the data. The XML is a commonly used in electronic messaging, particularly in administrations, but the UN/EDIFACT messages are stil widely used. The XML has extensive support in common office automation tools and off-the-shelf or public-domain computer software. The advantage is that an information system that adopts the XML format for Electronic Data Interchange (EDI) is relatively simple compared with traditional EDI systems that adopt a UN/EDIFACT format. However, creating new variants of XML has led to many different and partly competing standards, which apply to ship clearance, although the use of XML for this purpose is not widely implemented. Some relatively wel -known examples are PortNet in Finland; the Electronic Notice of Arrival/Departure (eNOA/D) system by the USA Coast Guard and SafeSeaNet (SSN) – EMSA in Europe (IMO, FAL 42/8). Regarding the national MSW, international standards for implementing the system interface are UN/EDIFACT, the WCO data model, and the ISO standard on 410 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION electronic port clearance - ISO 28005. To ensure information reporting regardless of the standards, the interoperability between the messaging systems implemented by the national MSWs is essential. The exchange of information through SSN requires that the digital format of the messages be used within national SafeSeaNet systems, following Directive 2002/59/EC. Also, EU member states shal comply with the harmonized XML messages and the technical standards developed for exchanging information through SSN (EC, DG MOVE, 2015). 4.1 Maritime Single Window To comply with national and international legislation and to complete ship reporting requirements for entering or leaving ports, parties who are participating in maritime transport now, must produce and submit a substantial amount of documentation. The document submission (paper and/or electronic form) to various authorities is required. This procedure takes a lot of time and places a significant administrative burden on the business community and governments (Economic Commission for Europe, 2005). Because of that, the idea of a MSW has been presented as an important concept in reducing trade costs and delays. Also, it is a platform for better cooperation and information sharing between many government departments engaged in foreign commerce, through a single-entry point, without data duplication (Kapidani and Kočan, 2015). Furthermore, under the specified level of authentication and authorization, it is ensured that the entered data is immediately visible in other systems (Tijan et al., 2018; Jović et al., 2022). In the maritime industry, this requirement for automated and standardized reporting procedures is critical, since it has affected the effectiveness and safety of maritime transport (Kapidani and Kočan, 2015). To tackle the aforementioned issue, the UN/CEFACT advises the implementation of a Single Window (SW), which is described as a “facility that allows parties involved in trade and transport to provide standardized information and documents through a single entry point to fulfill all import, export and transit-related regulatory requirements” (Ibid.). Hence, the IMO and more precisely, its FAL Convention plays a significant part in easier data exchange in maritime transport, as wel as among other issues related to the uniformity in formalities and precise procedures. Following that, the IMO arrived with standardized forms – IMO FAL Forms, to simplify the formalities, and procedures and integrate the documents that are Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 411 demanded to be sent to the authorities. Thus, every contracting country of the IMO FAL Convention is in the obligation to be involved in electronic information exchange between ports and ships. This mandatory requirement came into force on April 8th, 2019. In that manner, the Convention stimulates the data use of a SW to allow a submission of all required information related to the arrival, stay, and departure of ships, persons, and cargo, via a single portal, without data duplication (Kapidani et al., 2020). Figure 2: Model of MNSW environment Source: Kapidani et al., 2020. The MNSW (Fig. 2) is closely defined as a location, where al data about the maritime environment are entered once and made available to different parties (Tijan et al., 2018). Its emphasis is on the information related to vessels, rather than the information regarding trade and cargo. As the authority operating the SW for ship clearance, the national MSW shal , at the very least, handle the vessel's IMO FAL data, which contains general safety and security information on the cargo being transported. Additional y, the national MSW must be designed to handle reporting requirements arising from international legislation to which the particular country has acceded at both, the regional and global levels. 412 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The information on the ship clearance that is mandated by national law should also be covered by the national MSW (Kapidani et al., 2020). However, numerous entries of the same data, in several different applications, have caused errors that frequently happen during repetitive data input, which are resulting in complications in official procedures. Hereby, data duplication still occurs through electronic and paper methods. 5 The ePIcenter project recent results The latest developments of technological progress and global transport community requirements are directed towards more efficient transport digitalized solutions. Following the international regulations and initiatives, the ePIcenter project aims to create multilayered advanced platforms for transport effectiveness, increasing the optimization of resources, Physical Internet (PI) implementation, and development of eco-friendly and AI-supported toolsets and concepts in transport and logistics. In this regard, here has been conducted huge research considering the recent trends development of IT in providing sustainable, resilient and synchronized transport solutions. As a result of research made within the ePIcenter project, here are presented the most important features and tools for implementation of the “ePI- Link” demonstrator. This demonstrator shows the platforms for an integrated network of global transport partners and freight flows to efficiently connect ports, terminals, and shipping companies. 5.1 Transporeon: Multimodal visibility As part of the ePI-Link demonstrator results, the industrial partners AB InBev and Den Hartogh Logistics wanted to increase visibility in their multimodal transport flows. Transporeon (Trimble Company) further developed and integrated their multimodal visibility solution with the aim to obtain better estimated and actual data and be able to distribute that information based on a need-to-know basis such that the logistics management process can be triggered when needed. The development centered around three main achievements: Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 413 1. Consolidation of multiple data sources. In a context where individual data sources are fragmented (only covering part of the multimodal transport chain and part of the shipments) and prone to inaccuracies and delays, al available data sources need to be combined through an extensive process of data cleansing, normalization, and prioritization. Integrating deep sea terminal data was achieved within the project. As a result, we could eliminate blind spots and make information available at an earlier point in time. 2. Predicting estimated time of arrival (ETA). Shippers and 3PLs need to go through a constant cycle of planning, monitoring, and adjusting their shipments. To support this, it is essential to have reliable data on the estimated time of arrival (ETA) at a port of discharge and the final destination. Transporeon developed a layered model to predict such ETAs in which information was combined on multimodal combinations of transport services, transshipments, service loop structures, and real- time movements. Through smart combinations of transit time statistics and dynamic prioritization of data sources, an improvement was achieved in the completeness, accuracy, and latency of ETA information (Fig. 3). Figure 3: The ETA prediction module within Transporeon solution Source: Own research. 3. The SW interface for multimodality. All relevant information was integrated into a single control tower, combining main haulage (ocean) and pre-/on-carriage into a comprehensive view that supported both carrier haulage and merchant haulage 414 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION scenarios. All modalities and milestones were integrated into a seamless end-to-end process. In addition, key functionalities were added to support usability like monitoring of detention & demurrage charges, configurable notifications and widgets, integrated carbon footprint calculations, and management reporting (Fig. 4). The impact of these developments is situated mainly in higher transparency and more proactive information provisioning towards control towers and thereby enabling them to timely plan on-carriage transportation. This in turn creates potential savings on demurrage costs and the ability to better assess options for more sustainable non-road transport alternatives for on-carriage. Figure 4: The SW interface for multimodality within Transporeon solution Source: Own research. 5.2 MJC2: Synchromodality optimization Synchromodality refers to the ability to dynamicaly route and re-route freight through a multimodal network, automatical y changing and updating the route in response to disruptions, using sophisticated AI-based algorithms. As any transport planner will attest, the size and complexity of modern logistics operations mean that this is a very chal enging computational problem. The European TEN-T network itself is very large, as illustrated by the European Commission’s infographic (Fig. 5). Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 415 At each node, there are multiple modes, routes, and service providers to consider, each with its own constraints, costs, and benefits. Extending this picture to contemplate multiple international routes and options (ocean and rail) increases the scale and complexity. Optimizing freight routing in a network of this nature is extremely chal enging: in many cases it can be shown that there are mil ions of possible options for a single container, so for a large organization managing 1000s of such movements the optimization problem is vast. Figure 5: European TEN-T network components Source: transport.ec.europa.eu Within the results of the ePIcenter project, the participating MJC2 company has developed innovative multimodal optimization algorithms that enable synchromodal logistics planning in large complex networks and international freight transport operations. The core innovation is a very powerful optimization engine that can dynamically allocate and route 1000s of container movements in seconds. Apart from the intrinsic speed of the optimization process, a flexible multi-layered approach has been developed that can accurately model operational rules and constraints, ensuring that outputs are feasible and aligned with business processes and expectations. The research of new multimodal logistics algorithms to facilitate synchromodal planning, developing entirely new optimization techniques to tackle this very chal enging logistics scheduling problem, has been thoroughly conducted for this purpose. There are both short- and long-term benefits. The results and solutions developed, which are ahead of the current state-of-the-art, can be applied to today’s 416 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION logistics networks, creating major cost savings for the companies involved, while reducing congestion and GHG emissions. The approach has been proven in real-world scenarios, optimizing complex networks and transport operations, and showing how logistics operations can reduce fuel usage and driver workload through new optimization solutions. Longer-term, these innovations take a significant step toward the PI paradigm. The concept is based on an analogy with the way data flows through the Internet in the form of “packets”. In the proposed PI world freight would be moved around an open logistics network in intelligent π-containers, which are automatical y routed between nodes (warehouses, ports, terminals) by synchromodal algorithms. The ePIcenter synchromodal algorithms can be used for the automated decision-making needed at each node in the logistics chain. These algorithms also al ow companies to deal with uncertainty and disruption in the supply chain in a much faster and more efficient way. They absorb information received from real-time tracking and visibility solutions and automatically find new, optimized logistics plans and transport resource schedules. 5.3 Smart containers An important goal of the project ePIcenter is the research of introduction of smart containers in the operation and this investigation resulted in many positive findings. Statistics show that there are approximately 65 million cargo containers in use worldwide today (TheShip, n.d.). Only 5.6% of them are equipped with telematics devices to record and transmit their number, position, status, and the condition of the cargo inside in real-time. This percentage is increasing, and it is expected that by 2027, 30% of containers in use will be chipped (TGL, n.d.). These so-cal ed smart containers help manage the supply chain, provide up-to-the-minute T&T, and predict arrival times. They save time, reduce paperwork and human error, prevent loss and theft, increase visibility, accountability, and more (BOXPORT, n.d.; Kollman, n.d.). Freight container T&T is important for the optimal use of empty containers, storage, filling, (un)loading, manipulation, transport, transshipment, delivery, etc., but also for monitoring and controlling the conditions inside. This is extremely important when dealing with dangerous, perishable, and high-specific value cargoes, such as nuclear waste, pharmaceuticals, food, animals, plants, flowers, treasures, etc. In such circumstances, it is advisable to T&T each container Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 417 separately, in parallel with the physical conditions inside, including the condition of the cargo itself (Bauk et al. , 2023). However, this is much easier to achieve on land than at sea. There are many reasons for this: the Internet at sea is not as stable and fast as on land, due to the movement of the sea surface, occlusion by waves, usual y harsh weather conditions, the 'urban canyon' effect, multipath fading, etc. In addition, whereas overland containers are usual y lined up one after the other along the route, containers shipped by sea are loaded in huge blocks while being exposed to electromagnetic fields from electronic equipment and machinery on board. Containers transported by sea are placed side by side on several levels in the ship's hul and on board. This makes it difficult to access and col ect data. The situation is less complex, for example, in the maritime transport of radioactive cargo. Ships built for this purpose can carry only about twenty special y designed containers, called casks or drums (WNTI, 2021; PNTL, 2021). Although it is much easier to T&T containers in this arrangement, to the best of our knowledge, information on the status of the casks is sent manual y by the officer on watch (OoW) to the main control center on shore, rather than automatical y in real-time (Bauk, 2020). Further research is therefore needed. 6 Conclusion In the global environment, harmonization in data and information sharing with common structures, contributes to higher interoperability, creating communities with common values and shared goals. In maritime transport field, the stakeholders are mostly the agencies, administrative institutions, ports, or regional/international consortiums. Harmonization and interoperability tend to leverage technology, minimize administrative burdens, reduce paperwork, and facilitate the application of PCS, MSW, blockchain, smart containers and ETA prediction tools in the maritime sector. In general, ship reporting procedures include electronic transmission of sensitive, private, and proprietary information, ship location/destination, cargo types/amounts, passenger names and identity, and security/safety-related information (IMO, Facilitation Committee, FAL 44). Therefore, the paper gives a broad literature review and discusses the most important regulations from IMO/IPCSA/UN organizations that indicate mandatory documentation for ship 418 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION reporting, data harmonization, exchange, and facilitation of global maritime trade. These significantly shape the recent trends in the development of technical solutions for big data management and integration. Among these, MNSW and PCS play a crucial role together with other initiatives for the development of algorithm toolsets and AI-supported modules for transport data-sharing platforms optimization. The case study was conducted over the innovative research project ePIcenter, considering the platforms and modules features (Transporeon, Syncrhomodal optimization and smart containers on blockchain) deployed for trial Enhanced Physical Internet-enabled Global-European Network with aim to increase efficiency and reduce transport chal enges. The solutions resulted with provision of optimal technical capabilities and decision support for maritime and intermodal supply chain management participants. Furthermore, as final results are new resilience strategies for global operations and the ePIcenter technology. At least, this concept would al ow businesses to develop more robust operations and contingency strategies to mitigate the impact of future disruptions in maritime and intermodal transport. Acknowledgment This work has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No 883374 (project ePIcenter). This article reflects only the author’s views and the Research Executive Agency (REA) is not responsible for any use that may be made of the information it contains. References Ahmad, S., Abidin, S., Omar, N. (2011). Data Sharing in Networked Environments: Organization, Platforms and Issues. In Proc. 5th WSEAS int. conf. on ICT, pp. 207-213. Bauk, S. (2020). Modelling Radioactive Materials Tracking in Sea Transportation by RFID Technology. TransNav – International Journal on Maritime Navigation and Safety of Sea Transportation, Vol. 14, No. 4, pp. 1009-1014. Bauk, S., Ntshangase, L.H. (2023). Blockchain Implementation Barriers in Maritime: A Case Study based on ISM and MICMAC Techniques. Journal of Maritime Research. Vol. 20, No. 3, 2023, pp. 72-80; https://www.jmr.unican.es/index.php/jmr/article/view/736/766 Bauk, S., Radulovic, A., Dzankic, R. (2023). Physical Computing in a Freight Container Tracking: An Experiment. In the Proceedings of the 12th Mediterranean Conference on Embedded Computing (MECO’2023), Montenegro, doi: 10.1109/MECO58584.2023.10155008. Baygin, M., Yaman, O., Baygin, N., Karakose, M. (2022). A blockchain-based approach to smart cargo transportation using UHF RFID, Expert Systems with Applications, Vol 188, 116030, https://doi.org/10.1016/j.eswa.2021.116030. Bezić H., Tijan E., Aksentijević S. (2011). Port Community Systems – Economic Feasibility Evaluation . Review of Contemporary Entrepreneurship, Business, and Economic Issues. J. J. Strossmayer University of Osijek, No. 2/2011 pp. 247-256. Z. Paladin et al.: Maritime Transport And Logistics Digital Solutions Optimization Using Advanced Data Sharing Platforms: ePIcenter Projects' Case 419 BOXXPORT GmbH (n.d.). What are Smart Containers? Available online: https://www.youtube.com/watch?v=6YtNAYtHBlA (accessed on 01/03/2024). Directive 2010/65/EU of the European Parliament and Council on reporting formalities for ships arriving and departing from ports of the Member States, Strasbourg, 20/10/ 2010. Economic Commission for Europe, United Nations Centre for Trade Facilitation and Electronic Business (UN/CEFACT). (2005). Recommendation and Guidelines on establishing a Single Window to enhance the efficient exchange of information between trade and government. Recommendation No. 33, United Nations New York and Geneva. Elmay, F.K., et al. (2022). Blockchain-Based Traceability for Shipping Containers in Unimodal and Multimodal Logistics. IEEE Access, vol. 10, pp. 133539-133556, doi: 10.1109/ACCESS.2022.3231689. ePIcenter project deliverable. (2022). D1.7 Final Review of EU/Global Initiatives Policies and Standards Impacting ePIcenter. Public report within Project GA No. 861584. European Commission, DG MOVE, Directorate D - Logistics, maritime & land transport and passenger rights, D.1 - Maritime transport & logistics, National Single Window Guidelines, Final version, Bruxelles, 17/04/2015. Iakushkin, O., Selivanov, D., Pavlova, E., Korkhov, V. (2019). Architecture of a Smart Container Using Blockchain Technology. In: Misra, S., et al. Computational Science and Its Applications – ICCSA 2019. Lecture Notes in Computer Science, vol 11620. Springer, Cham. https://doi.org/10.1007/978-3-030-24296-1_42. IMO, Expert Group on Data Harmonization. (2020). EGDH 3/20/1, Any Other Business, Modelling progress update fol owing EGDH 2, Submitted by UNECE, WCO, ISO. IMO, Facilitation Committee. (2020). FAL 44/WP.5, Review and Revision of the IMO Compendium on Facilitation and Electronic Business (Item 6), Developing Guidance for Authentication, Integrity and Confidentiality of Content for Exchange via MSW (Item 7). IMO, FAL 42/8, Update the Guidelines for Setting up a Single Window System in Maritime Transport, 5th March 2018. Jaiman, V., Pernice, L., Urovi, V. (2022). User incentives for blockchain-based data sharing platforms. PLoS ONE Vol. 17, No.4: e0266624. https://doi.org/10.1371/journal.pone.0266624. Jović, M., Tijan, E., Brčić, D., Pucihar, A. (2022). Digitalization in Maritime Transport and Seaports: Bibliometric, Content and Thematic Analysis. J. Mar. Sci. Eng. Vol. 10, No. 486., pp. 1 -24. Kapidani, N., Kočan, E. (2015). Implementation of National Maritime Single Window in Montenegro. 23rd Telecommunications Forum Telfor (TELFOR). Kapidani, N., Tijan, E., Jović, M., Kočan, E. (2020). National Maritime Single Window – Cost-Benefit Analysis of Monetengro Case Study. Promet – Traf ic & Transportation, Vol. 32, No. 4, pp.543-557. Kirstein, L. (2018). Information Sharing for Efficient Maritime Logistics. OECD/The International Transport Forum. Kol man, J. (n.d.) Smart containers key to improved supply chains. Available online: https://www.ingwb.com/progress/insights-sustainable-transformation/smart-containers- key-to-improved-supply-chains (accessed on 01/03/2024). Lind, M., Michailides, M., Ward, R. Watson, R. (2021). Maritime Informatics. Springer, 2021. Lind, M., Ward, R., Bergmann, M., Haraldson, S., Zerem, A. (2020). Digitalizing the port cal process. Transport and Trade Facilitation Series No 13, UNCTAD. Maatsch, S., Jović, M., Jungen, H., Mül er, R. (2022). Position paper: Resilience of maritime transport chains. https://island.isl.org/sites/default/files/2023-09/Position_paper_ Resilience_of_maritime_transport _chains_0.pdf (accessed Dec. 12, 2023). Meyer, T., et al. (2019). Blockchain technology enabling the Physical Internet: A synergetic application framework. In Computers & Industrial Engineering, Vol.136, pp. 5-17, https://doi.org/10.1016/j.cie.2019.07.006. Ntshangase, L. H. and Bauk, S. (2024). Blockchain applications and cybersecurity threats: A review. In Proceedings of the 28th International Conference on Information Technology (IT), Zabljak, Montenegro; https://doi.org/10.1109/IT61232.2024.10475777 420 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Paladin, Z., Bauk, S., Mujalović, R., Kapidani, N., Lukšić, Ž. (2024). Blockchain Technology’s Effects on Big Data in Maritime Transportation. In Proceedings of 28th Information Technologies Conference (IT), Zabljak, Montenegro. PNTL (n.d.). PNTL Ship Tour. Available online: https://www.pntl.co.uk/our-fleet/pntl-ship-tour/ (accessed on 01/03/2024). Surucu-Balci, E., Çagatay, I., Balci, G. (2023). Digital information maritime supply chains with blockchain and cloud platforms: Supply chain capabilities, barriers, and research opportunities. Technological Forecasting & Social Change, No. 198, 122978, pp. 1-16. Šorović, M., Brčić, D., Frančić, V., Maričević, T., Đurović, Z., Kapidani, N., Lukšić, Ž. (2023a). Integrated information solution as a platform for data exchange in a modernized ADRIREP system. In 22nd International Symposium Infoteh-Jahorina (INFOTEH), pp. 1-6. Šorović, M., Kapidani, N., Lukšić, Ž., Maričević, T., Marušić, Š., Frančić, V., Brčić, D., Strabić, M., Đurović, Z. (2023b). Towards the Introduction of the Sea Traffic Management System in the Adriatic Sea. In Proc. 12th MECO Conference, pp. 610-615. Šorović, M., Kapidani, N., Lukšić, Ž., Frančić, V., Strabić, M., Brčić, D., Malovrh, A., Đurović, Z., Maričević, T. (2023c). Harmonisation of VTS Training Programs – A Case Study of the Adriatic-Ionian Region. In 3rd Internatıonal Conference of Marıtıme Scıence & Technology - Naše More 2023, Safety, Innovation, Resilience, Dubrovnik. TEN-T transport network statistical pocketbook (2021). https://transport.ec.europa.eu/facts- funding/studies-data/eu-transport-figures-statistical-pocketbook/statistical-pocketbook-2021_en (accessed on 6/12/2023) TGL -Think Global Logistics (n.d.). What are Smart Containers? The Rise of Smart Containers. Available online: https://www.youtube.com/watch?v=-Ry-0HermnE (accessed on 01/03/2024). TheShip (n.d.). Shipping Containers & Solutions. How Many Shipping Containers Are There in the World? Available online: https://theship.ai/blog/how-many-shipping-containers-are-there-in-the-world/#:~:text=One%20of%20the%20most%20popular,shipping%20containers%20in% 20use%20worldwid (accessed on 01/03/2024). Tijan, E., Agatić, A., Hlača, B. (2012). The Necessity of Port Community System Implementation in the Croatian Seaports. In Promet – Traf ic&Transportation, Vol. 24, No. 4, pp. 305-315. Tijan, E., Agatić, A., Jović, M., Aksentijević, S. (2019). Maritime National Single Window—A Prerequisite for Sustainable Seaport Business. In Sustainability 2019, Vol. 11, No. 17, p. 4570; https://doi.org/10.3390/su11174570 Tijan, E., Jardas, M., Aksentijević, S., Perić Hadžić, A. (2018). Integrating Maritime National Single Window with Port Community System – Case Study Croatia. 31st Bled eConference – Digital Transformation: Meeting the Chal enges Conference Proceedings, Bled, Slovenia. pp. 1-11. UNECE (n.d.). Real-time smart container data for supply-chain excel ence. Available online: https://unece.org/DAM/cefact/GuidanceMaterials/WhitePapers/WP- SmartContainers_Eng.pdf (accessed on 01/03/2024). UNECE, UN/CEFACT, Data Simplification and Standardization for International Trade, Recommendation No.34., NY and Geneva, 2013. UNECE, UN/CEFACT, Recommendation and Guidelines on Establishing a Single Window - To Enhance the Efficient Exchange of International Trade Information Between Trade and Government, Recommendation No.33., Geneva, 2020. UNECE, UN/CEFACT, Trade and Transport Facilitation Monitoring Mechanism, Recommendation No.42., New York and Geneva 2017. UNECE/TRADE/C/CEFACT/2011/4, Centre for Trade Facilitation and Electronic Business 17th session, Revision of Recommendation 12: Measures to facilitate maritime transport documents procedures, 2011. WNTI-World Nuclear Transport Institute (2021). Fact sheet: Quick facts on the transport of Nuclear Fuel Cycle Transport. Available online: https://www.wnti.co.uk/wp-content/uploads/2021/01/Quick-Facts-on-the-Transport-of-Nuclear-Fuel-Cycle-Transport.pdf (accessed on 01/03/2024). ENABLING CARDIAC REHABILITATION AT HOME: A WEB-BASED SOLUTION NILMINI WICKRAMASINGHE,1 NALIKA ULAPANE,1 SASAN ADIBI2 Keywords: 1 accessibility, La Trobe University, Melbourne, Australia n.wickramasinghe@latrobe.edu.au, nalika.ulapane@gmail.com adherence, 2 Deakin University, Melbourne, Australia cardiac sasan.adibi@deakin.edu.au rehabilitation (CR), cardiovascular Cardiovascular diseases are alarmingly prevalent, impacting diseases, design millions worldwide. Cardiac rehabilitation (CR) is a cornerstone science of recovery, encompassing exercise, education, and lifestyle research, modifications. However, despite its proven benefits, adherence digital health, to and completion of CR programs remains suboptimal. Barriers exercise such as cost, geographical constraints, and transportation regimens, difficulties hinder patient participation. In response, we pose the healthcare technology, research question: How might we design a digital health solution home-based to enhance accessibility and engagement in cardiac rehabilitation rehabilitation, patient-centric from the comfort of patients’ homes? Leveraging design science care, research principles, we have meticulously crafted a web-based patient prototype. Our solution integrates personalized exercise engagement, quality of regimens, educational modules, and progress tracking. By life, empowering patients to actively participate in their recovery remote journey, we aim to revolutionize CR delivery. This paper presents monitoring, telehealth, our ongoing progress, emphasizing the potential impact on web-based patient outcomes and quality of life. solution DOI https://doi.org/10.18690/um.fov.4.2024.24 ISBN 978-961-286-871-0 422 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Cardiovascular diseases (CVDs) pose a significant worldwide health chal enge, affecting millions of individuals and placing substantial burdens on healthcare systems. In Australia, heart, stroke, and vascular diseases collectively impact approximately 5.2% of the population, equivalent to 1.3 million people (ABS, 2022). These conditions contribute to a staggering 25% of al deaths in the country (AIHW, 2023). Internationally, the picture is equally concerning, with CVDs being the leading cause of death, accounting for 32% of worldwide mortality (WHO, 2021). The prevalence of coronary heart disease and heart failure is rising, necessitating effective strategies for prevention, management, and rehabilitation. Cardiac rehabilitation (CR) emerges as a pivotal component in the continuum of care for individuals with CVD. CR encompasses exercise training, risk factor management, and psychosocial counseling, aiming to optimize patient outcomes and enhance quality of life. Despite its proven benefits, CR remains underutilized, with participation rates commonly hovering below 50% (Beatty et al., 2023). Challenges such as geographical barriers, work commitments, and transportation difficulties contribute to this suboptimal adherence. Consequently, there is an urgent need to explore innovative approaches that enhance accessibility and engagement in CR. Considering these chal enges, our research question emerges: How might we design a digital health solution to enhance accessibility and engagement in cardiac rehabilitation from the comfort of patients’ homes? This question is not only timely but also critical for addressing the gaps in CR delivery. By leveraging design science research principles, we have developed a web-based prototype solution that aims to revolutionize CR. Our approach integrates personalized exercise regimens, educational modules, and progress tracking, all accessible from the comfort of patients’ homes. The potential impact of this solution extends beyond individual patients—it has the capacity to transform CR delivery, reduce disparities, and improve outcomes for people with CVD. Our prototype represents a paradigm shift, aligning with the evolving landscape of digital health. By harnessing technology, we empower patients to actively participate in their recovery journey. The patient portal offers tailored content, evidence-based guidance, and interactive features. As we embark on this transformative path, we N. Wickramasinghe, N. Ulapane, S. Adibi: Enabling Cardiac Rehabilitation at Home: A Web-Based Solution 423 emphasize that our solution is grounded in rigorous research methodologies. The principles of design science guide our every step, ensuring that evidence informs innovation. Through iterative development and user feedback, we strive to create a seamless, patient-centric experience that transcends geographical boundaries and fosters adherence. In summary, our research endeavors to bridge the gap between evidence and practice. As we introduce this web-based cardiac rehabilitation solution—a demonstrable prototype—we invite stakeholders, including patients, clinicians, technology experts, and policymakers, to join us in this journey toward improved cardiovascular health. Together, we can unlock the full potential of CR, making it accessible, engaging, and impactful for all. We present a pilot study in this paper. The likes of extensive validation, user-testing, economic analysis and so on are out of scope for this paper but wil be focused on in our future work. 2 Review of Relevant Works Few tools have been developed at commercial scale to deliver at-home cardiac rehabilitation, and in this section we attempt to gain insights from them. Developed by researchers at the Australian eHealth Research Centre in col aboration with Queensland Health, Cardihab™ (CSIRO, 2019) is an online cardiac rehabilitation platform. It provides convenient access to rehabilitation tools remotely. The platform includes a web portal for clinicians and a smartphone application for patients. Patients can engage in rehab from the comfort of their homes, reducing the need for frequent outpatient clinic visits. Cardihab™ delivers core components of cardiac rehab, such as education, behavior modification, and psychological counseling, tailored to patients’ clinical needs. Clinicians can remotely access patient data through the web portal, enhancing communication and outcomes. Notably, Cardihab™ was commercialized in 2017 after raising venture capital and has demonstrated comparable or better health outcomes compared to traditional rehabilitation programs (CSIRO, 2019). 424 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Carda Health (CardaHealth, 2023) offers an at-home virtual cardiac rehab program. It provides treatments similar to those expected from in-person cardiac rehab programs. Patients can participate in exercises and receive support remotely, making it a convenient option for home-based rehabilitation (CardaHealth, 2023). During the COVID-19 pandemic, novel approaches to cardiac rehabilitation, including hybrid models, have emerged (Dalal et al., 2021). These models combine virtual and in-person elements, al owing greater patient choice and potential y increasing uptake of cardiac rehab. While not specific commercial tools, hybrid models offer flexibility and adaptability for patients seeking home-based rehab (Dalal et al., 2021). It is also important to understand the limitations and gaps in the existing solutions for at-home cardiac rehabilitation. Some limitations and gaps can be identified as follows. Firstly, while home-based tools like Cardihab™ and Carda Health offer convenience, some patients may find it chal enging to engage consistently. Factors such as technological literacy, motivation, and adherence can impact the effectiveness of these tools. As such there is a need for interventions that address these engagement barriers to enhance long-term outcomes (Dalal et al., 2021). Secondly, commercial tools may not adequately account for cultural diversity (Cardihab, 2022). Patients from different backgrounds may have varying preferences, beliefs, and health practices. A weakness lies in the lack of cultural y tailored content, potential y affecting patient engagement and outcomes. As such, adopting more cultural y sensitive approaches in cardiac rehab tools would be beneficial. Thirdly, despite the promise of remote tools, geographic disparities persist (Cardihab, 2022). Patients in rural or underserved areas, in addition to challenges with transportation, may also face chal enges accessing reliable internet or mobile networks. It is therefore important to focus on equitable access as wel to cardiac rehab, perhaps with a multimodal approach comprising of downloadable text resources and phone consultations in addition to material like videos. N. Wickramasinghe, N. Ulapane, S. Adibi: Enabling Cardiac Rehabilitation at Home: A Web-Based Solution 425 Fourthly, while tools focus on physical aspects, they often overlook psychosocial support. Emotional well-being, anxiety, and depression play crucial roles in recovery (Dalal et al., 2021). Existing tools may lack comprehensive strategies for addressing these aspects. Therefore, adopting holistic approaches that integrate additional services as mental health support is also important. Lastly, cost implications pose a major barrier. Although home-based rehab is cost- effective compared to traditional center-based programs (Dalal et al., 2021), there are still cost implications. Some patients may struggle with out-of-pocket expenses related to technology, subscriptions, or equipment. As such, it is important to minimize financial barriers and promote affordability. Motivated by such gaps, we aimed to develop a web-based prototype enabling at- home cardiac rehab, that could potential y have more robustness and flexibility in terms of factors such as ease-of-use, cultural sensitivity, ease-of-access, and affordability. 3 Relevant Theories: Design Science Research Methodology (DSRM) This study involves designing artifacts, and therefore, the principles of Design Science Research Methodology (DSRM) (Baskerville et al., 2018) are fol owed. The process of DSRM is for systematically conceptualizing, designing, developing, and assessing artifacts so that the desirability of the artifacts can be maximized to meet the stakeholder needs. The process typical y includes six steps: (1) Problem identification and motivation; (2) Defining the objectives for a solution; (3) Design and development; (4) Demonstration, (5) Evaluation, and (6) Communication. Research can be integrated in each, or al of the first five steps. Research can aim at understanding and solving any issues to maximize the desirability of the artifacts. The landmark publications (Hevner et al., 2010; Hevner & Wickramasinghe, 2018; Peffers et al., 2007) are useful for more details. In summary, DSRM combines creativity, practicality, and scholarly rigor to produce actionable solutions for real-world chal enges. Through DSRM, researchers can engage with stakeholders, create artifacts, and contribute to both theory and practice. 426 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4 Methodology The DSRM-inspired methodology followed in this study is depicted in Figure 1. The participants of the design process are listed in Table 1. The subsections that follow are devoted for describing the various stages of the design process. Figure 1: DSRM-inspired methodology fol owed in this study Source: Own Table 1: Sample table Researcher’s Description about the code participant Role in the project C1 Senior Clinician (Doctor) Clinician consultant A1 Professor in Digital Health Principal Investigator Research Fel ow in Associate R1 Computer Science Investigator Research Fel ow in Associate R2 Computer Science Investigator 4.1 Problem Identification As the problem identification phase, researchers A1 and R1 met with C1, and discussed the needs for a digital health solution in the cardiac rehabilitation space. C1 highlighted the importance of cardiac rehabilitation. Then, C1 highlighted some trial digital health solutions they have attempted at his hospital. These solutions have been targeted at providing education and guidance to patients regarding certain hospital procedures they would undergo. Citing such trials, C1 emphasized the importance of similar solutions for cardiac rehabilitation. This discussion was N. Wickramasinghe, N. Ulapane, S. Adibi: Enabling Cardiac Rehabilitation at Home: A Web-Based Solution 427 inspirational for future work. The outcome of this discussion was a list of strengths, weaknesses, opportunities, and threats that face digital health solutions for cardiac rehab (given in Figure 2). 4.2 Defining Objectives Following problem identification, A1 sourced a seed grant for a 6-week mini project. It was decided to dedicate this funding to designing and developing a prototype digital health solution for cardiac rehab. Researchers R1 and R2 led this phase and planned suitable objectives to be delivered within 6 weeks (given in Figure 3). 4.3 Designing As shown in Figure 3, it was agreed to design: (a) a web-based clinician-facing frontend; (b) a web-based patient-facing frontend, and (c) a database backend to collect as much data as possible. In this phase, R1 in consultation with A1 drew skeletal illustrations for each of the segments. Some examples are provided in Figures 4 and 5. 4.4 Development In this phase, R2 developed the web-based front ends using HTML, following the designs planned in the previous phase. R1 developed the required database backends and also PHP interfaces to enable communications such as clinician and patient registration and login. Once developed, the frontend and backend were integrated, and the resulting website was hosted in a private hosting platform. 4.5 The Artifact The artifact was the website resulting following the ‘Development’ phase. As discussed before, this website had a web-based clinician facing end, a patient-facing end, and a backend database. The clinician facing end gave the ability to register new clinicians and patients, to provide material to patients, and view progress and feedback made by patients. The patient facing end gave the ability to login as a unique patient, and then view and follow the provided material, and also provide feedback. Some screenshots of the artifact are provided in Figures 6 to 10. 428 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.6 Demonstration A meeting was organized with an external expert in health, and the implemented website was demonstrated. Feedback was recorded along with the possible additions that might be possible going forward. A completed wishlist of possible additions to this solution recorded at the meeting is presented in Figure 11. 4.7 Feedback to be Taken Forward The notable outcome of the 'Demonstration' phase was the wishlist presented in Figure 11. This wishlist helps flesh out the cardiac rehab solution to a much richer solution that would help derive insights and offer personalized care. 5 Results Results are presented in Figures 2 to 11. These figures are the outcomes of the different phases of the design cycles discussed in Section 4. The figures are mostly self-explanatory. Further elaborations are avoided due to space restrictions. Figure 2: Strengths, weaknesses, opportunities and threats for digital health solutions in cardiac rehab, outcome of ‘Problem Identification’ phase Source: Own N. Wickramasinghe, N. Ulapane, S. Adibi: Enabling Cardiac Rehabilitation at Home: A Web-Based Solution 429 Figure 3: Objectives planned to be delivered within 6 weeks, outcome of ‘Defining Objectives’ phase Source: Own (a) interface to enable registering a new patient. (b) interface to enable monitoring patient progress. Figure 4: Some designs employed for the clinician-facing interface Source: Own 430 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 5: The design employed for the patient-facing interface to display videos Source: Own Figure 6: The welcome page to both clinicians and patients Source: Own Figure 7: The main page inside the clinician portal Source: Own N. Wickramasinghe, N. Ulapane, S. Adibi: Enabling Cardiac Rehabilitation at Home: A Web-Based Solution 431 Figure 8: Some interfaces inside the clinician portal Source: Own Figure 9: Interface in clinician’s portal to monitor patient interactions Source: Own 432 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 10: Some interfaces inside the patient portal Source: Own Figure 11: The wish list of data and notifications to be handled in future work Source: Own 6 Discussion and Conclusions The study presented the DSRM-inspired approach adopted to design and develop a web-based digital health platform to enable at-home cardiac rehab, in an attempt to answer the following research question: How might we design a digital health solution to enhance accessibility and engagement in cardiac rehabilitation from the comfort of patients’ homes? The methodology followed was detailed, and the outcomes of each design phase were outlined. The study made a twofold contribution: A contribution to practice and a contribution to theory. N. Wickramasinghe, N. Ulapane, S. Adibi: Enabling Cardiac Rehabilitation at Home: A Web-Based Solution 433 As a contribution to practice, our study developed a web-based prototype to enable cardiac rehab at home. Our solution includes a clinician-facing interface, a patient- facing interface, and a database backend to capture interactions. Snapshots of our development were presented in this paper, and the methodology followed was detailed. As such, our work would serve inspirational for technology design in many healthcare fronts. As a contribution to theory, our work presented a list of factors (Figure 11) that could contribute many a digital health solution to become a fleshed out tool that captures important information about patients to enable deriving insights and personalization. Limitations of this study include the limited scope and the limited time – this paper reports the outcomes of a 6-week project. Our future work would attempt to flesh out the current development through a more elaborate research project. Given the increasing prevalence of cardiac issues globally, CR is becoming increasingly important to ensure strong clinical outcomes, high patient satisfaction and high value care. The proposed digital health solution to support superior cardiac rehab, serves to support a healthcare value proposition of better quality of care, better access to care and that high value care ensues. Moreover, the approach adopted ensures that responsible healthcare delivery results. Taken together this ensures high patient satisfaction and strong clincial outcomes. Acknowledgements The authors acknowledge OPTUS and CERI (Care Economy Research Institute) at La Trobe University, Australia for their respective support for this research. References ABS. (2022). Heart, stroke and vascular disease. ht ps://www.abs.gov.au/statistics/health/health-conditions-and-risks/heart-stroke-and-vascular-disease/latest-release, Date of last access: March 7, 2024. AIHW. (2023). Heart, stroke and vascular disease: Australian facts. ht ps://www.aihw.gov.au/reports/heart-stroke-vascular-diseases/hsvd-facts, Date of last access: March 7, 2024. Baskervil e, R., Baiyere, A., Gregor, S., Hevner, A., & Rossi, M. (2018). Design science research contributions: Finding a balance between artifact and theory. Journal of the Association for Information Systems, 19(5), 3. 434 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Beatty, A. L., Beckie, T. M., Dodson, J., Goldstein, C. M., Hughes, J. W., Kraus, W. E., Martin, S. S., Olson, T. P., Pack, Q. R., & Stolp, H. (2023). A new era in cardiac rehabilitation delivery: research gaps, questions, strategies, and priorities. Circulation, 147(3), 254-266. CardaHealth. (2023). At-Home Cardiac Rehabilitation: 8 Basic Exercises. ht ps://www.cardahealth.com/post/home-cardiac-rehab, Date of last access: March 7, 2024. Cardihab. (2022). Clinical study finds that offering digital health platform Cardihab increases participation in cardiac rehabilitation. ht ps://cardihab.com/news/clinical-study-finds-that-of ering-digital-health-platform-cardihab-increases-participation-in-cardiac-rehabilitation/, Date of last access: March 7, 2024. CSIRO. (2019). A mobile platform for cardiac rehab. ht ps://www.csiro.au/en/research/health-medical/diseases/Cardiac-rehab-app, Date of last access: March 7, 2024. Dalal, H. M., Doherty, P., McDonagh, S. T., Paul, K., & Taylor, R. S. (2021). Virtual and in-person cardiac rehabilitation. bmj, 373. Hevner, A., Chatterjee, S., Hevner, A., & Chatterjee, S. (2010). Design science research in information systems. Design research in information systems: theory and practice, 9-22. Hevner, A. R., & Wickramasinghe, N. (2018). Design science research opportunities in health care. Theories to inform superior health informatics research and practice, 3-18. Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of management information systems, 24(3), 45-77. WHO. (2021). News Room. Fact Sheets. Cardiovascular Diseases. ht ps://www.who.int/news-room/factsheets/detail/cardiovascular-diseases-%28cvds%29, Date of last access: March 7, 2024. DECISION ANALYTICS—LESS EXPECTED MOTIVATORS FOR HEPA PROGRAMS CHRISTER CARLSSON Institute for Advanced Management Systems Research, Abo Akademi University, Abo, Finland christer.carlsson@abo.fi There is a need to raise the physical activity levels in the population as this will give significant savings in healthcare costs, both long- and short term. This has motivated numerous projects and campaigns during the last 15-20 years, which mostly have not produced any long-term, and not any significantly positive results. We need “some better ways”, which build on the design of programs for young elderly (our target group) that get adapted to and adopted for sustained use. Decision analytics could be a possible approach to find the wanted “better ways”. The DigitalWel s program is a first implementation of decision Keywords: analytics with (at least) partial answers to the complaint - “you HEPA, analytics, cannot be sure that time spent will actually give sufficient health digital effects” – a key reasons why exercise programs are discontinued. ecosystem, DigitalWells is a digital ecosystem, an effective and useful context young elderly, for advanced digital analytics, which offers better forms for user preventive guidance and support. healthcare DOI https://doi.org/10.18690/um.fov.4.2024.25 ISBN 978-961-286-871-0 436 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction In a recent study [18] the authors state as an established fact that low physical activity and high sedentary behaviour unquestionably have an impact on public health and wil also increase direct and indirect costs. One way to summarize the result is the estimate that the total costs of low physical activity in Finland [18] in 2017 was 3.2 B€ (direct costs 683 M€, indirect costs 2.5 B€) and costs of high sedentary behaviour roughly 1.5 B€. We will not work out the models that generated these estimates but accept the contention that low physical activity and high sedentary behaviour levels carry substantial societal costs; prior studies referred to in [18] state that physical inactivity cause about 0.3% - 4.6% of a nation’s healthcare costs. Thus, one conclusion that can be made is that raising the physical activity levels in the population will give significant savings in healthcare costs, both long-term and in the short term (cf. [13]). This insight has motivated projects and campaigns that have targeted various groups and segments of the population at various points during the last 15-20 years (if we restrict the discussion to more modern efforts), but it appears that these efforts have not produced any long-term, and not any sustainable, significant, and positive results. Projects and campaigns come and go (cf. [13], [18]) and it appears that col ection of facts or systematic databases, that can be reused as references and starting points for renewed studies have not been created and maintained. Our conceptual framework builds on the theory, history and systematic knowledge built in the constructs of information systems, which in turn is part of the ICT framework, and which wil offer some insight and some means to work out the problems with non-sustainable efforts to raise the physical activity levels in the population. The first decision we need to make is that we should not try to find or build generic solutions that could apply to al groups and segments of the population. We have a better possibility to find viable and useful solutions by working with definable problems for specific (but sufficiently large) groups (cf. [18]). We decided on working with the young elderly, the age group 60-75 years, which public policy more or less ignores in planning and programs for the ageing population – common wisdom finds that they are too healthy, too active, and with too good social networks to need any intervention and support from public resources; and they are too many C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 437 (about 1.3 million citizens in Finland, 23% of the population) – the last point is a sarcastic interpretation of prevailing political opinions. Nevertheless, it makes good sense to build preventive programs to counter the effects of ageing early enough – with a focus on young elderly – to get timely, sustainable, and long-term results for ageing citizens (cf. [18]). A second decision is to focus on what we mean by “raising the physical activity levels” and bearing in mind that we want to carry this out for the young elderly. HEPA is an acronym for health enhancing physical activity, which translates to physical activity (PA) of enough intensity and duration to give short- and long-term health effects. In health recommendations regular PA at moderate intensity for at least 150 minutes per week is expected to have positive health effects (cf. [14]). The European HEPA network claims that thirty minutes a day of moderate-intensity activity is enough to benefit health. This type of recommendations offers some guidelines but appear to be insufficient as motivators to raise PA levels – in field studies (cf. [10-12], [15-16]) we have seen comments like – “you cannot be sure that the time spent will actually give sufficient health effects”. The HEPA recommendations apply to healthy adults with individual differences in the effects of PA programs. There are variations when we focus on young elderly, in terms of female/male, age groups, BMI, socio-economic background, history of physical demands from work history and HEPA capacity (decided by PA history and physical shape). Work with young elderly (cf. [10-12], [15-16]) showed that PA programs offered in projects and campaigns over several years, (i) were not intensive enough, (ii) were not running for enough time (cf. [18]), and (iii) were not regular enough to be adopted and become sustainable habits for young elderly. Thus, we need to work out some better ways to design, introduce and sustain HEPA programs for young elderly. These two decisions form the context and the limitations of the paper, we do not aim for any generalizations – any general principles or theory. The methodology used in the DigitalWells program is a combination of building new, innovative artefacts combined with technical testing and verification of software solutions and functions, and empirical testing of usability and relevance with samples of young elderly users (cf. [4] for similar studies). 438 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The “some better ways” points to systematic thinking and rational decision-making, the key principles of which have been close to us for more than 50 years (cf. [25]). Following the axioms of decision analysis, the best decision to choose (e.g. the best PA program) is the one whose consequences have the maximum expected utility or offer the maximum probability of achieving a wanted aspiration level (e.g. getting better sustainable health). Since the early, rather theory-oriented days decision analysis has evolved into a mature professional discipline [4] which has developed series of methods that have been and are used to support business and public-policy decision-making, often in cases where the decisions aim to resolve large, complex, and critical problems (cf. [4]). One of the (Wikipedia) definitions states, “decision analysis (DA) is the discipline comprising the philosophy, methodology, and professional practice necessary to address important decisions in a formal manner”. Then why do we not come across DA to guide the composition and selection of HEPA programs? The guide to best decisions with DA models (Wikipedia) handle “uncertainties through subjective probabilities for which the decision maker’s attitude to risk is represented by utility functions and the attitude to trade-offs between conflicting aspirations (e.g. re composition of HEPA programs) are expressed with multi-attribute value or multi- attribute utility functions; utility functions can be replaced by the probability of achieving a wanted aspiration level. There is some doubt if this guide to best decisions, despite being theoretically and logically precise, is very useful for our context. The principles of decision analysis have been reinterpreted, enhanced, and adjusted over the years to meet the needs from growing complexities of large, multinational, dynamical y interdependent industries, and corporations that in ever growing competition adapt to dynamically evolving innovations. The reinterpretations introduced methods and algorithms that are mathematical y more advanced and more powerful to meet the chal enges from complex problems formed by large groups of dynamic, interactive elements, i.e. when we cannot grasp and tackle the interactions. The reinterpretations formed the theory and methodology of operational research, management science, multiple criteria decision making, etc. that have offered formal frameworks for important decisions from 1980’es through 2020’es. The DA theoreticians are, however, not overwhelmed by the thousands of success stories (Wikipedia) … “while there may occasionally be justification for such C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 439 methods in applications (e.g., based on ease of use), decision analysts would argue for multi-attribute utility theory as the gold standard to which other methods should be compared, based on its rigorous axiomatic basis”. In the 2020’es this conceptual animosity is more or less forgotten as it has been over- shadowed by the chal enges of big data, which refers to problem-solving that is complicated with huge amounts of structured, semi-structured or unstructured data made available from hundreds and thousands of data sources with the help of modern, advanced ICT technology which operates 24/7 to offer instant and constant access to data sources. The decision analysis evolved into a 2020’es version called decision analytics, which now has developed the methodology and tools to work in and deal with big data environments. Before we get into decision analytics there are a few lessons learned in the decision support systems (DSS) movement (about 1980-2010) that we could make use of for developing optimal HEPA programs for young elderly and seniors. DSS builders focused on the users’ priorities, they developed systems linked to key business activities and they viewed the quality of a system from the value it gives to the users rather than the level of technology applied. DSS reflected demand economics: service, fast delivery, ease of use, benefit focused more than cost, imprecision al owed for timely delivery and user control (cf. [4]). Analytics represents a shift in focus from both DA and DSS towards developing and delivering critical data, information and knowledge for management, decision- making, negotiations, planning, operations, (public, private sector) administration, etc. Analytics builds on theory and advanced algorithms as part of information systems, to which digital technology now is making inroads. Research used for decision analytics (cf. [5]) shows themes like big data, machine learning, business and service analytics, gamification, virtual and augmented reality, visual decision analytics, soft computing, logistics and supply chain management, explainable AI, etc. which are described as “hot topics” and which we could/should make use of for our present purposes (cf. Decision Analytics Track, HICSS). Decision analytics, in practical terms (cf. [7]), uses combinations of mathematics and statistics, data techniques and advanced algorithms to predict and quantify performance, risk, cost, and revenue with rich data visualization to communicate 440 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION valuable insights to key stakeholders and decision makers. This agenda is quite the same as for decision analysis but new technology and advances in algorithms have given decision analytics significant impact on real world problem-solving, planning and decision making. Analytics has gained in importance in business and industry over the last 10-12 years (cf. [6-9]), but the introduction of analytic theory and increasingly advanced algorithms also meets with resistance; senior managers and executives are not comfortable with elements of black boxes (i.e., advanced mathematics) as key parts of planning, problem solving and decision making. Russel Ackoff, one of the pioneers of Operations Research [1], found (already in 1974) that mathematical models tackle and solve mainly limited and abstract representations of actual problems and that these are mostly rather useless for handling real world issues. Lotfi Zadeh (in an HICSS keynote address) had a similar message and formulated it as “you can increase precision if you are willing to give up on relevance or you can increase relevance if you are willing to give up on precision, but you cannot do both at the same time”. In the 2020’s we tend to forget this lesson as we aim at getting more advanced (e.g., deep and machine learning) algorithms to deal with still larger and more complex problems for which planning, problem solving and decision making need to be fast moving, highly dynamic and mostly right (not optimal). Our quest is to find “some better ways to design, introduce and sustain HEPA programs for young elderly” than the traditional ad hoc projects and campaigns that seem not to produce any long-term, and not any sustainable, significant, and/or positive results. Our proposition is to build on systematic thinking and rational decision-making, the theory, and axioms of decision analytics, for the design and introduction of HEPA programs for young elderly. We collected more and new principles for decision support systems, decision technology, and algorithms in analytics to meet the chal enges of big data, and then collected and formalized in decision analytics. The state-of-the-art methods and technology promise to support HEPA program designs that quickly adapt to changes and support choices and decisions that are mostly right (i.e., not exactly optimal but “0.95 good enough”). C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 441 2 Decision Analytics for HEPA Programs – First Explorations The DigitalWells program run 2019-22 and collected over 294 000 PA entries in its database from more than 1000 participants in 24+ months. It is a first implementation of decision analytics – “combinations of statistics, data techniques and algorithms to quantify performance, with rich data visualization to communicate valuable insights to DW participants (decision makers)”. This was combined with cross-sectional and longitudinal studies with several samples of 100-250 participants at 4–6-month intervals to show, (i) the acceptance and adoption of the DW 3.0 application, (ii) the support of HEPA programs, and (iii) the sustainability of accepted HEPA programs. The DW 3.0 app for smart mobile phones (Android, iOS) went through several iterations with groups of users to improve its design and functionality. DW 3.0 composes and runs weekly PA programs and registers the actual activities (cf. fig.1). The logging of activities on the smart phone is done in the left part of the screen (cf. fig.1): (i) the user selects the activity (gym training), (i ) the intensity (moderate), (i i) the date from the calendar, (iv) the duration (hours, minutes) after which the app (v) calculates and shows the effect of the PA exercise (MET-min, kcal). A MET-minute is the amount of energy spent during a minute while at rest; CPA (cf. [2]) has calibrated more than 800 PA exercises in terms of MET (metabolic equivalent of task) to show the energy spent per time unit. The MET-min measure in DW 3.0 uses the CPA calibration to give a facts-based, standardized estimate of the physical activity level of the exercise. The most recent entries are col ected in the second column and produce reports on a user’s smart phone (the fourth column): (i) a specified PA report (weekly), (ii) the reported week, (i i) the PA as MET-minutes per week, and (iv) MET-minutes per day; graphical reports on MET-minutes per day, MET-minutes per activity and Minutes per activity are shown in the third column. The PA entry results update a secure, cloud-based database where the entries are stored with individual 8-digit pseudonyms for the users. The MET-min calculation is done with algorithms to decide the activity level of PA exercises (efforts and effects are functions of the user’s age, BMI and gender, and the type of a PA activity). This now offers (at least) partial answers to the complaint - “you cannot be sure that the time spent will actually give sufficient health effects”. 442 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 1: DW 3.0 – A first decision analytics application Source: Own The DW 3.0 uses the Wellmo platform, which offers updated interface solutions to most smart watches, and we used this feature to integrate DW 3.0 with Polar smart watches; users started to enter PA activities from smart watches, which measure heart rate and show a factual measure of the physical activity level (better than an assessment of “how demanding (1, 2, or 3) was the PA?”). With a smart watch it is also possible to measure and record several PA modules during one HEPA session. Wellmo is a multi-purpose, container platform that offers a wide range of functionalities that can be added to further versions of the DW 3.0 app. Users suggested that we include active support (“coaching”) to guide them to select effective HEPA exercises and advise them on how to make progress towards better health. In some research papers (cf. [5-9]) we have worked out how digital coaching [7] could be designed and implemented for use. 3 HEPA Programs – User Reactions and Comments In the DigitalWells program, Makkonen et al. [20] collected a sample of 115 young elderly who used a DW 3.0 PA logger to log, keep track of and get updates on their weekly PA exercises; daily activity data was analyzed with partial least square structural equation modelling (PLS-SEM) using the enhanced unified theory of acceptance and use of technology (UTAUT2) as the research model (cf. [28]): performance expectancy, hedonic motivation, and habit had positive and statistically C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 443 significant effects on behavioural intention to adopt and use the PA logger, which is seen as a first step to adopt HEPA routines. In two further, similar studies with different samples of young elderly participants [21], the focus was on how the adoption and use of the PA logger evolves after an initial acceptance. A longitudinal study captures “lapses” in the intention to use (and the use) for reasons which change and/or evolve over time. PA data was col ected in three subsequent surveys, after four months (T1), 12 months (T2) and 18 months (T3) of using the DW 3.0 PA logger. With the UTAUT2 (cf. [28]) hedonic motivation and habit, had positive and statistically significant effects on the adoption and use of the logger; performance expectancy had a positive and statistical y significant effect at T1 and T3, but not at T2; effort expectancy had a positive and statistically significant effect at T2, but not at T1 and T3. The results are interesting: (i) the construct scores stabilized over time, and (i ) declined quite strongly between T1 and T2, but less so between T2 and T3. A likely explanation is the novelty effect of the PA logger, as the scores for habit also declined strongly between T2 and T3; the effects of performance and effort expectancy appear to switch places, which could explain lapses in the use of the PA logger. It appears that the UTAUT2 constructs primarily explain intention to use a PA logger but not necessarily the adoption and use of HEPA programs. It can of course be argued that once a PA logger is adopted with an intention to use it, the user has started PA exercises and a HEPA program. Self-efficacy offers a conceptual framework for work on sustainable HEPA programs. Bandura [3] shows that self-efficacy beliefs affect the quality of human functioning through cognitive, motivational, affective, and decisional processes. Self-efficacy beliefs influence outcome expectations, and causal attributions for successes and failures. This quite well fits an intuitive understanding of what it would take to adapt to, adopt, and sustain HEPA programs. In the DigitalWells Kari et al [15] studied how effective the DW 3.0 PA logger is in promoting PA self-efficacy in several groups of young elderly that had been 12 months or more with the DigitalWel s program. The study traced changes in self- efficacy, at T1(+4 months), T2(+12 months) and at T3 (+18 months). A participant assesses his/her ability to exercise for 20 minutes three times per week and reports 444 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION his/her personal confidence on a [0, 10] scale relative to nine statements on obstacles; an overall self-efficacy TS [0, 90] is the sum of the nine statement measures. A group of 165 participants responded to al three self-efficacy questionnaires and formed the sample. At the construct level, the total score (TS) showed a statistical y significant change both at T1 and T2; the mean total score had increased from 56.0 (T1) to 62.0 (T2) and 61.5 (T3). The changes in self-efficacy were positive after 4 months and sustained after 12 months; the main explanation for the changes was found in improved mastery experience. Bandura suggested that self-efficacy could be raised (or lowered) by non- performance means (cf. [3]) as control variables: (i) age group, (ii) gender, (iii) education, (iv) experience with apps, (v) BMI, and (vi) residential environment. The effects of the background factors were tested (with a multifactor variance analysis) on changes in self-efficacy and actual MET-minutes with difference variables. This appears to work as there were three statistical y significant factors: education (p = 0.009), BMI (p = 0.018) and residential environment (p = 0.027); the increase between T1 and T2 is larger among university educated than for those with vocational education; the increase is smal er in the obese group than normal weight and overweight groups; the increase is larger in the big city group than in small or medium-sized city and countryside groups (cf. [15], [17], [26]). The increase in self-efficacy for PA exercise and HEPA programs is important for sustained HEPA (e.g., [21], [26-27]). Sustained improvement in self-efficacy supports sustained adoption of HEPA programs, which contributes to health benefits as sustained improvement secures long-term health effects. Self-efficacy wil not increase indefinitely, mastering PA tasks and the PA application is typical y accomplished in 1-2 months, after which the novelty wears off (and no further self-efficacy increase is expected). The self-efficacy conceptual framework, and the statistical models which are part of the decision analytics framework, points to possibilities for sustained HEPA programs. Regular health-enhancing physical activity can serve as preventive health care, which will improve and sustain quality of life and save health-care costs for an ageing population. Decision analytics models and tools gain support as they produce useful and important results. C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 445 4 Decision Analytics for HEPA Programs – Digital Ecosystems and Smart Systems Digital ecosystems are part of a wave of new theoretical constructs that form the era of the digital economy; typical digital ecosystems are Airbnb and Uber, also Amazon was originally a digital ecosystem but evolved into a multi-channel platform for goods and services. The term “digital ecosystem” is used in an inflationary manner without precise definitions and in literature it appears that many terms and concepts are digital ecosystems – platform ecosystems, business ecosystems, software ecosystems, platform economy, sharing economy, etc. – without pointing to specific constructs and features. Koch et al [19] worked out a set of seven key properties that we wil use to outline a digital ecosystem; five of these are relevant for our present study (DE represents Koch’s “digital ecosystem-like” construct). KP1–service focus. DEs establish business models in which the main revenue stream comes from the provision of combinations of digital services; services are “woven into software-based network fabrics”. KP2–network effects. The success of DEs is largely driven by network effects that describe the increase in value generated by an increasing user base; network effects are the primary drivers to “create and capture value”. KP3-–shift of value creation. DEs may generate value by connecting consumers and providers using a shared platform; such value creation is further enhanced as the number of users increases. KP5–openness. Openness is understood as facilitating the system’s accessibility in order to enable the “use, development, and commercialization of a technology”. KP6–collaboration. Col aboration may span industries, companies, or organizations. Competition, collaboration, or a mixed model of both increase productivity in DEs. Experience gained from work with the DigitalWel s program suggests that we need an extended and enhanced digital ecosystem to accommodate (i) much larger groups of users [KP2, KP3], (ii) enhanced PA Logger versions with statistics and graphics [KP5, KP6], (iii) support for service asset providers [KP1, KP2], (iv) support for service asset brokers [KP1, KP2], (v) analytics tools [KP5], (vi) knowledge base 446 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION support [KP5, KP6], (vii) digital coaching support [KP3], and (viii) digital personal trainer services [KP3]. The digital platform should also include tools for local and global ad hoc group support [KP2, KP5, KP6], and support tools for easy integration with public preventive health care programs. The key properties for a digital ecosystem that Koch et al [19] worked out are shown tentatively and should be tested and validated with actual constructs. Service asset providers could participate in HEPA programs as partners and subcontractors; service asset brokers could be implemented as intel igent agent services on the platform (cf. [23-24]). The digital ecosystem is a new and enhanced context that wil al ow the use of different model ing tools, more advanced algorithmic tools and software-based networking and interfacing instruments (for integrating data, information, and knowledge from a diversity of sources). This offers opportunities to develop new, enhanced versions of decision analytics that wil further remove it from the restrictions of decision analysis and its guide to best decisions with DA models that handle “uncertainties with multi-attribute value or multi-attribute utility functions”. In the digital ecosystem context “best decisions” are formed by service focus [KP1], shifts in value creation [KP3] and collaboration [KP6], al enabled with smart systems and intel igent technology. A first draft of such a digital ecosystem, with some interpretations of KP1-KP6, is worked out in fig. 2. Fig.2 is the DigitalWells user ecosystem as it takes form with the sports clock and smart phone platforms. The user works with DP1, the DW PA Logger registers the activities on DP3, and the user will (later on) get support from DP5 and DP6. C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 447 Figure 2: TE-DW Digital Ecosystem Source: Own The digital ecosystem builds on and supports six digital platforms - DP1 offers the HEPA program support with existing and new versions of the DW 3.0 PA Logger application; DP1 builds on the Wellmo platform. DP2 supports young elderly and senior HEPA program users with team and group support, inter- and intra-group collaboration, gaming and smal -scale competition, secure communication, etc. DP3 supports analytics models and tools that work on HEPA program outcomes, survey statistics and a HEPA knowledge base to be used for and support preventive health care programs. DP4 invites, activates, and supports HEPA service asset providers; the first invitees are sports institutes. DP5 is an open research and development platform for digital coaching. DP6 offers a research and development platform for digital support (e.g., digital personal trainers, and nutrition counselling); both DP5 and DP6 can support HEPA specialists. The service asset broker [B] can be a partner company or can be an intel igent agent-type system that (i) onboards HEPA program service assets with program users, (i ) records and (i i) charges for the use and monitors the payment routines. The [R] represents research groups with access to the platforms to carry out research work programs. The digital ecosystem includes an infrastructure for mobile and web-based payments of digital HEPA programs. Digital coaching is part of the digital ecosystem and is an interpretation of KP3, shift of value creation through guidance of the user. Work with digital coaching got started a few years ago (cf. [6-7]) to help human operators to master advanced 448 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION automated systems in complex, very large industrial process systems. Digital coaching works with data from digital devices, instruments, tools, monitoring systems, sensor systems, software systems, data and knowledge bases, big data sets, etc. Digital coaching requires transitions from data to information, and then on to knowledge (and vice versa), referred to as digital fusion (cf. [7-9], [22]). Data fusion harmonizes data from a variety of sources with different formats; information fusion builds syntheses of data to describe, explain and predict; knowledge fusion uses ontology to build and formalize insight from data and information fusion for computational intel igence methods, AI, machine learning, soft computing, approximate reasoning, etc. (cf. [6], [22-24]). Digital fusion appears to be a key component in models and algorithms that form modern versions of decision analytics and allows progress from e.g. multi-attribute utility functions. 5 Summary and Conclusions We stated the continuously recurring problem that there is a need to raise the physical activity levels in the population as this will give significant savings in healthcare costs, both long-term and in the short term. Numerous studies have supported and validated this observation. This has motivated and initiated numerous projects and campaigns during the last 15-20 years, which – however – have not produced any long-term, sustainable, significant, and positive results. Work with young elderly showed that PA programs offered in projects and campaigns (i) were not intensive enough, (ii) were not running for enough time, and (ii ) were not regular enough to be adopted and become sustainable habits for young elderly. The conclusion was to work out “some better ways”. We introduced the concept of HEPA programs (HEPA is health enhancing physical activity) and decided that the “some better ways” are to design and introduce HEPA programs for the young elderly that get adapted to and adopted for sustained use. We chose, in this paper, to show that decision analytics could be a possible approach to the “better ways”. The DigitalWells program is a first implementation of decision analytics – “combinations of statistics, data techniques and algorithms to quantify performance, with rich data visualization to communicate valuable insights to DW participants (decision makers)”. The MET-min calculation introduced is done with algorithms C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 449 to decide the activity level of PA exercises (efforts and effects are functions of the user’s age, BMI and gender, and the type of PA activity). This now offers (at least) partial answers to what young elderly could (or should) do to get potentially sufficient health effects for the time they spend on their PA exercises. One of the key reasons why PA exercise programs are discontinued is the uncertainty that they actual y wil give the wanted health effects. In the next step we introduced the TE-DW digital ecosystems with six platforms to guide and support users to effective HEPA programs; the digital ecosystem is also an effective and useful context for advanced forms of digital analytics that wil offer more and better forms for user guidance and support. The proposal to introduce and make use of decision analytics is part of an on-going research program that in the next phase wil expand to larger groups of users (first to 3000 and then to 10 000 users) in order to harvest larger numbers of PA events from the HEPA programs. This wil , in turn, al ow us to apply “big data” methods and algorithms to trace trends in PA behaviour among different groups of young elderly, to estimate “typical” choices of PA events, “typical” intensity of exercises and “typical” duration of PA events. Analytics will allow us to build HEPA standards and norms, i.e. what young elderly should choose as goals to make sure that they can resolve the initial issue - “you cannot be sure that time spent wil actual y give sufficient health effects”. References Ackoff, Russel L. (1974). Beyond Problem Solving, Wiley Ainsworth, B.E. et al. (2011). 2011 Compendium of Physical Activities, 0195-9131/11/4308-1575/0, Medicine & Science in Sports & Exercise, DOI: 10.1249/MSS.0b013e31821ece12 Bandura, A. (1977). Self-efficacy: Toward a Unifying Theory of Behavioral Change, Psychological Review, Vol. 84, No. 2, 191-215 Carlsson, C. and Fuller, R. (2011). Possibility for Decision, Studies in Fuzziness, Springer 270 Carlsson, C. (2012). Soft computing in analytics: handling imprecision and uncertainty in strategic decisions, Fuzzy Econ. Rev. XVII (2) (2012) 3–21. Carlsson, C., Heikkilä, M., Mezei, J. (2016). Fuzzy entropy used for predictive analytics, in: C. Kahraman, U. Kaymak, A. Yazici (Eds.), Fuzzy Logic in its 50th Year. New Developments, Direc-tions and Challenges, Studies in Fuzziness, Springer, 341, 2016, pp. 187–210. Carlsson, C. (2018). Decision Analytics Mobilised with Digital Coaching, Intelligent Systems in Accounting, Finance and Management, ISAF1421, January/March 2018, pp 3-17 DOI: 10.1002/isaf.1421 450 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Carlsson, C. (2019). Combining ANFIS and Digital Coaching for Good Decisions in Industrial Processes, IFSA/NAFIPS'2019 (Lafayette, Louisiana) Proceedings, Springer Verlag 2019, pp 190-200 Carlsson, C. (2019). Digital Coaching to Make Fuzzy Real Options Methods Viable for Invest-ment Decisions, FUZZ-IEEE 2019 Proceedings, New Orleans 2019, pp 406-411, 978-1-5386-1728-1/19 ©2019 IEEE Carlsson, C., Kari, T., Makkonen, M., Frank, L. and Walden, P. (2020). Sustained Adoption of Systematic Physical Activity Programs for Young Elderly – A Developed UTAUT Approach, in: A. Pucihar et al (eds.) Proceedings of the 33rd Bled eConference, 29.6.2020. Carlsson, C. and Walden. P (2022). Digital Support Activates Young Elderly to Health-Enhancing Physical Activity, in: Piet Kommers and Mario Macedo (eds) Proceedings of the MCCSIS 2022 Conference/E-Health 2022, July 19-22, 2022, Lisbon, pp 189-196, Carlsson, C. and Walden. P. (2023). Young Elderly DSS Users – Some Reasons for Sustained and Successful Adoption, ICDSST 2023 Proceedings, Springer Ding, D., Lawson, K.D., Kolbe-Alexander, T.L. (2016). The economic burden of physical inac-tivity: a global analysis of major non-communicable diseases, Lancet 2016; 388: 1311-24 Finne-Soveri, H., Jakovljevic, D., Mäkelä, M. (2018). Pain management of a severely memory- impaired elderly person is worse in an assisted living facility than an institution, Suomen Lääkärilehti 2018;73:1137- 42 Kari, T., Makkonen, M., Frank, L., Carlsson, J. and Sel , A. (2020). The Effects of Using a Mobile Wellness Application on Physical Activity Levels: A Four-Month Fol ow-Up Study Among Aged People, in: A. Pucihar et al (eds.) Proceedings of the 33rd Bled eConference, 29.6.2020. Kari, T., Makkonen, M. and Carlsson, C. (2022). Physical Activity Tracker Application in Pro-moting Physical Activity Behavior Among Older Adults: A 24-month Fol ow-up Study, Journal of Aging and Health, 2022; https://doi.org/10.1177/08982643221135812 Kettunen, E., Kari, T., Makkonen, M., Frank, L. and Critchley, W. (2020). Young Elderly and Digital Coaching: A Quantitative Intervention Study on Exercise Self-efficacy, in: A. Pucihar et al (eds.) Proceedings of the 33rd Bled eConference, 29.6.2020. Kolu, P., Kari, J.T., Raitanen, J., Sievänen, H., Tokola, K., Havas, E., Pehkonen, J., Tammelin, T.H., Pahkala, K., Hutri-Kähönen, N., Raitakari, O.T., Vasankari, T. (2022). Economic burden of low physical activity and high sedentary behaviour in Finland, J Epidemiol Community Health 2022; 76: 677-684 Koch, M., Krohmer, D., Naab, M., Rost, D., Trapp, M. (2022). A matter of definition: Criteria for digital ecosystems, Digital Business, Vol.2, Issue 2, 1-13 Makkonen, M., Kari, T. and Frank, L. (2020). Applying UTAUT2 to Explain the Use of Physi-cal Activity Logger Applications Among Young Elderly, in: A. Pucihar et al (eds.) Proceedings of the 33rd Bled eConference, 29.6.2020. Makkonen, M., Kari, T., and Frank L, (2020). Changes in the Use Intention of Digital Wellness Technologies and Its Antecedents Over Time: The Use of Physical Activity Logger Applications Among Young Elderly in Finland. Proceedings of the HICSS-54 Conference, pp 1262-1271. Mezei, J., Brunel i, M., Carlsson, C. (2017). A fuzzy approach to using expert knowledge for tuning paper machines, JORS 68 (6) (2017) 605–616. Morente-Molinera, J.A., Wikström, R., Carlsson, C., Viedma-Herrera, E. (2016) A linguistic mobile decision support system based on fuzzy ontology to facilitate knowledge mobilization, Decision Support Systems, 81 (2016) 66–75. Morente-Molinera, J.A., Mezei, J., Carlsson, C., Viedma-Herrera, E. (2016). Improving super-vised learning classification methods using multi-granular linguistic modelling and fuzzy entropy, Trans. Fuzzy Syst. 2016 (3) (2016) 250–260. Raiffa, Howard (1968). Decision Analysis, Longman Reyes-Mercado P. (2018). Adoption of fitness wearables. Insights from partial least squares and qualitative comparative analysis, Journal of Systems and Information Technology, Vol. 20 No. 1, 2018, pp. 103-127 C. Carlsson: Decision Analytics—Less Expected Motivators for HEPA Programs 451 Stiggelbout, M., Hopman-Rock, M., Tak, E., Lechner, L. and van Mechelen. W. (2005). Drop-out from exercise programs for seniors: a prospective cohort study, Journal of Aging and Physical Activity, 13, 409-421 Venkatesh, V., Thong, J. Y. L., and Xu, X. (2016). Unified Theory of Acceptance and Use of Technology: A synthesis and the Road Ahead. JAIS, Vol 17, Issue 5, 328-376 452 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION BALANCING PERSONALIZATION AND PRIVACY: TOWARDS PERSONALIZED SAVING EXPERIENCE IN BANKING APPS FOR YOUNG ADULTS ALEX WAHLROOS,1 LEONID FREIDOVICH,2 ULRIK SÖDERSTRÖM,1 THOMAS MEJTOFT1 1 Umeå University, Digital Media Lab, Umeå, Sweden alwa0043@ad.umu.se, ulrik.soderstrom@umu.se, thomas.mejtoft@umu.se 2 Umeå University, Umeå, Sweden leonid.freidovich@umu.se This pilot study investigates how personalized content can be used in banking applications to encourage shaping good saving habits and increase overal financial literacy among young people. The preliminary result are recommendations including providing clear and personalized saving goals, incorporating educational content on investing and financial planning, and implementing features for tracking and categorizing expenses. Some trade-offs Keywords: in usage that are presented are to minimize the use of personal personalization, data to what is necessary for personalization purposes, treat the online privacy, data on a group level or increase transparency of data usage. The personal pilot study concludes that personalization is likely to be beneficial data, for both banks and their customers, given personal data is banking apps, handled carefully and used in a sound financial consumer digital protection framework. nudging DOI https://doi.org/10.18690/um.fov.4.2024.26 ISBN 978-961-286-871-0 454 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Navigating the financial world can be a daunting task for young adults. Potential insecurities in financial matters, coupled with the burden of debts like student loans, contribute to financial stress among this group (Lindgren et al., 2023). The multi- faceted concept financial literacy relates to “a person’s competency for managing money” (Remund, 2010, p. 279) and is often overlooked by young adults, which might lead to inadequate financial literacy in the future (OECD, 2020). In today’s global economic instability (IMF, 2023), financial literacy and buffers are crucial. For example, in Sweden, 25% lack a safety net for unexpected expenses (SBAB, 2024). Having a savings account early in life is emphasized, serving as a steppingstone to current and future financial health (Friedline et al., 2014). Parents shape young adults’ financial decisions and including children in financial discussions influences their saving attitudes (Sinnewe & Nicholson, 2023; Te’eni-Harari, 2016). Generally, savings lead to asset ownership, psychological benefits, and social mobility (Heckman & Hanna, 2015). Furthermore, it offers financial security and young savers experience better wel -being and less distress (Helm et al., 2019; O’Neill, 2009). However, achieving financial goals requires clarity and emotional investment (Caceres-Santamaria, 2023). To promote savings, personalized services in banking apps can cater to users' individual financial needs. Large-scale personalization in personal finance requires effective, user-centered solutions and wel executed, banking apps can offer a cost- effective solution. AI plays a key role in meeting changing customer preferences for quick, personalized responses (Noreen et al., 2023) and robo-advice (Gomber et al., 2017), AI-based financial planning and investment services, are today used by banks for savings and investment plans. Targeting young adults who have the capacity to save but face obstacles, banking apps can use personalization to guide users towards informed financial decisions. This approach predicts customer behavior, fosters financial discussions, and builds loyalty (Brodski et al., 2019). According to Walstad & Wagner, 2023, financial education, whether mandatory or self-chosen, positively impacts saving behavior, with repeated exposure enhancing this effect. Furthermore, those receiving personality-tailored interventions by an automatic service are many times more likely to reach the savings goal (Matz et al., 2023). A. Wahlroos et al.: Balancing Personalization and Privacy: Towards Personalized Saving Experience in Banking Apps for Young Adults 455 This pilot study investigates the balance between personalization and user privacy in banking apps for young adults aged 18–30. The future aim of this research is to start a discussion on guidelines for designing apps that promote savings for young adults in a non-intrusive manner. The pilot study is based on a survey of 37 respondents (49% male and 51% female) with 56% students, 39% full-time employees, and 5% part-time employees. 2 Theoretical approach Nudging is a design approach where the users’ decisions is guided towards their intended direction and can be implemented using a variety of techniques (Kahneman et al., 1991; Mirsch et al., 2017; Schneider et al., 2018; Thaler & Sunstein, 2008). Nudges serve as a valuable instrument in shaping digital design, particularly when decision-making proves challenging (Mejtoft, Ristiniemi et al., 2019). Hansen and Jespersen (2013) describe different types of approaches to nudging in relation to the Automatic System and the Reflective System described by Thaler and Sunstein (2008) and the level of transparency of the nudge. According to the ethical guidelines by Meske and Amojo (2020), nudges need to be transparent to be justified. Non-transparent and automatic nudges should be clearly prompted, e.g., by working with intentional friction in the user interface, so-called design friction (Mejtoft, Hale et al., 2019; Mejtoft, Parsjö et al., 2023). Personalization in digital systems impose a possibility to increase the effect of nudges (Mills, 2022; Peer et al., 2020). A main concern regarding personalization and the opportunities it brings is that for content to be truly personal, a system must have extensive knowledge about the user. Aguirre et al. (2016) denotes this the personalization-privacy paradox and refer to the fact that high personalization can enhance the user experience as wel as diminish the engagement with a firm. Nudging should be considered in contrast to concepts such as deceptive design (Brignull, 2013), which are techniques that are created intentionally to deceive users by implementing a type of non-transparent nudges (cf. Hansen & Jespersen, 2013) that are not in the user’s best interest (Gray et al., 2018). In its simplest form this could be to, e.g., trick users to accept cookies by the design of cookie prompts (Mejtoft et al., 2021; 2023). 456 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3 Results and discussion Most of the respondents perceived the knowledge regarding their personal economy as average or above average (Figure 1) and they were confident in taking actions regarding financial planning (41%). However, regarding investing money, 46% of the respondents were unconfident or very unconfident. Figure 1: Confidence in acting regarding 1) financial planning, 2) saving money on a regular savings account, and 3) investing money (in mutual funds, stocks etc.) Source: Own Most respondents (76%) saved money on a regular basis. However, the remaining (24%) did not save money regularly, but when they feel like it. The most frequent reasons for saving were economic buffer and long-term goals. 3.1 Nudging towards saving goals Regarding application of personalization to support saving goals, the respondents’ answers could be divided into categories of personalization. The first category is financial planning and recommendations, consisting of answers that suggest a recommended savings amount based on fixed expenses. Other suggestions are prioritizing saving when the salary is deposited and providing personalized encouragement when saving goals are approached. The second category is expense tracking and budgeting. This category consists of answers that suggest that personalization can give a better understanding of where money is spent in order to give users better control over their expenses and cut costs of A. Wahlroos et al.: Balancing Personalization and Privacy: Towards Personalized Saving Experience in Banking Apps for Young Adults 457 unnecessary consumption, such as tobacco or energy drinks, is also suggested. Another suggestion is data visualization, e.g., expenses for different months can be overviewed and compared, providing clear milestones for financial goals, and forecasts for saving habits. With increased knowledge about spending patterns, digital nudging could support users by both making savings part of the automatic system as wel as spending part of the reflective system using design friction. The last category of answers consists of investment guidance and knowledge acquisition. Respondents state they feel insecure about investing in stocks or funds and believe a personalized way to get a better understanding of this topic could help them make better decisions. This applies to all current levels of knowledge, as personalization addresses individual needs. Since the respondents stated uncertainty towards e.g. financial investments, reflective nudging is necessary for a high transparency and gradual y increased knowledge around this topic. Many of the respondents agreed that a summary of personal spending is desired, and one suggestion was that it could be analyzed to find room for improvement. Other suggestions are to give recommendations of cheaper options for products or services that cost a lot of money, tips on investing to reach certain goals, and a visual representation of the balance between income and expenses. In terms of personalization, the results show that support is needed regarding financial planning, financial recommendations, expense tracking, and budgeting. Since the respondents were divided in their opinion, it should be taken as an indication of a slight resistance towards the use of personal data, meaning one solution could be to treat data on a group level instead of individually tailoring each user’s content. The trade-off here is not easily managed (cf. Noreen et al., 2023), as users demand quick responses with personalized content. By reducing the use of personal data, therefore potential y reducing the personalization opportunities in the banking app, users may not receive the support to be nudged towards a healthier economic structure and savings routine. The suggestion is therefore to use personalization when the financial literacy and awareness of the individual are sufficient. 458 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3.2 Privacy The respondents are not overly concerned about the use of their personal data (Figure 2). It should, however, be used with caution and care (cf. Aguirre et al., 2016). One reason for not being concerned might be the laws to protect personal data, e.g. the GDPR, that have been introduced over the last decade. Nevertheless, the respondents stated that they do not have good knowledge about the actual use of their data, and they believed that it is fairly important to minimize the use of personal data. Figure 2: Privacy assumptions among the respondents regarding 1) I have good knowledge of how my personal data is used, 2) I am worried about the use of my personal data, and 3) It is important to minimize the use of personal data Source: Own Risks in using personal data in banking apps fal into two categories. The first being privacy and security, with concerns about data leaks, account hacking, unauthorized access, and personal information being used for scams or frauds. Some respondents also object to banks knowing their spending habits and money sources. The second category involves bias and misuse. Respondents fear that personalization could narrow financial perspectives, limit diverse options, and promote banks' interests. They're also concerned about the banks’ use of their predicted spending patterns for targeted advertising. This is in line with the respondents being afraid of deceptive design in the applications. A. Wahlroos et al.: Balancing Personalization and Privacy: Towards Personalized Saving Experience in Banking Apps for Young Adults 459 4 Towards design recommendations The increased use of personal data in financial services can be positive for consumers if it takes place in a sound financial consumer protection framework and is matched by sufficient financial literacy and awareness. Digital nudging can be used to support young adults to increase their financial literacy and by combining automatic and reflective decisions. The following design recommendations for personalization of financial applications are proposed based on the results of this paper: • Provide the user with clear and personalized saving goals and recommendations. A common desire amongst the respondents was to get a better, more visual connection to the savings goal. • Include personalized educational content on investments and financial planning, as it can provide important tools to acquire a better savings routine. To make the nudging transparent it is important that the application gradually increases the financial literacy among the users and that nudges are constantly aligned with the users’ level of financial confidence. • Use features for tracking and categorizing expenses. This leads to a better overview of the personal economy and makes it easier to optimize savings. Transparent design in terms of data use is important to avoid users feeling deceived. References Aguirre, E., Roggeveen, A. L., Grewal, D., & Wetzels, M. (2016). The personalization-privacy paradox: Implications for new media. Journal of Consumer Marketing, 33(2), 98–110. https://doi.org/10.1108/jcm-06-2015-1458 Brignull, H. (2013, August 29). Dark patterns: Inside the interfaces designed to trick you. Retrieved from https://www.theverge.com/2013/8/29/4640308/dark-patterns-inside-the-interfaces-designed-to-trick-you Brodski, S., Desmangles, L., Fanfaril o, S., Khodabandeh, S., Palumbo, S., & Santinel i, M. (2019, March 12). What does personalization in banking really mean? Boston Consulting Group. Retrieved from https://www.bcg.com/publications/2019/what-does-personalization-banking-real y-mean Caceres-Santamaria, A. J. (2023, April). Why are we so impatient? A look into money and delayed gratification. Page One Economics. https://research.stlouisfed.org/publications/page1-econ/2023/04/03/why-are-we-so-impatient-a-look-into-money-and-delayed-gratification Friedline, T., Johnson, P., & Hughes, R. (2014). Toward healthy balance sheets: Are savings accounts a gateway to young adults’ asset diversification and accumulation? Federal Reserve Bank of St. Louis Review, 96(4), 359–389. 460 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Gomber, P., Koch, J.-A., & Siering, M. (2017). Digital finance and fintech: current research and future research directions. Journal of Business Economics, 87(5), 537–580. https://doi.org/10.1007/s11573-017-0852-x Gray, C. M., Kou, Y., Battles, B., Hoggatt, J., & Toombs, A. L. (2018). The dark (patterns) side of UX design. In Proceedings of the 2018 chi conference on human factors in computing systems, Article 534. New York: NY, USA: ACM. https://doi.org/10.1145/3173574.3174108 Hansen, P. G., & Jespersen, A. M. (2013). Nudge and the manipulation of choice: A framework for the responsible use of the nudge approach to behaviour change in public policy. European Journal of Risk Regulation, 4(1), 3–28. https://doi.org/10.1017/S1867299X00002762 Heckman, S., & Hanna, S. D. (2015). Individual and institutional factors related to low-income household saving behavior. Journal of Financial Counseling and Planning, 22(2). https://ssrn.com/abstract=2589976 Helm, S., Serido, J., Ahn, S. Y., Ligon, V., & Shim, S. (2019). Materialist values, financial and pro-environmental behaviors, and well-being. Young Consumers, 20(4), 264–284. https://doi.org/10.1108/YC-10-2018-0867 IMF. (2023, April). World economic outlook: A rockey recovery (Tech. Rep.). Retrieved from https://www.imf.org/-/media/Files/Publications/WEO/2023/April/English/text.ashx Kahneman, D., Knetsch, J. L., & Thaler, R. H. (1991). Anomalies: The endowment effect, loss aversion, and status quo bias. Journal of Economic Perspectives, 5(1), 193–206. https://doi.org/10.1257/jep.5.1.193 Lindgren, K. P., Tristao, T., & Neighbors, C. (2023). The association between student loan debt and perceived socioeconomic status and problematic drinking and mental health symptoms: A preliminary investigation. Addictive Behaviors, 139, 107576. https://doi.org/10.1016/j.addbeh.2022.107576 Matz, S. C., Gladstone, J. J., & Farrokhnia, R. A. (2023). Leveraging psychological fit to encourage saving behavior. American Psychologist, 78(7). https://doi.org/10.1037/amp0001128 Mejtoft, T., Frängsmyr, E., Söderström, U., & Norberg, O. (2021). Deceptive design: Cookie consent and manipulative patterns. In A. Pucihar, M. K. Borštnar, R. Bons, H. Cripps, A. Sheombar, & D. Vidmar (Eds.), 34th Bled eConference: Digital support from crisis to progressive change (pp. 397– 408). Maribor, Slovenia: University of Maribor University Press. https://doi.org/10.18690/978-961-286-485-9.29 Mejtoft, T., Hale, S., & Söderström, U. (2019). Design Friction: How intentionally added friction affect users level of satisfaction. In M. Mulvenna & R. Bond (Eds.), Proceedings of the 31st European Conference on Cognitive Ergonomics (pp. 41–44). New York, NY: ACM. https://doi.org/10.1145/3335082.3335106 Mejtoft, T., Parsjö, E., Norberg, O., & Söderström, U. (2023). Design friction and digital nudging: Impact on the human decision-making process. In Proceedings of the 2023 5th International Conference on Image, Video and Signal Processing (pp. 183–190). New York, NY, USA: ACM. https://doi.org/10.1145/3591156.3591183 Mejtoft, T., Ristiniemi, C., Söderström, U., & Mårel -Olsson, E. (2019). User experience design and digital nudging in a decision making process. In A. Pucihar, M. K. Borštnar, R. Bons, J. Seitz, H. Cripps, & D. Vidmar (Eds.), 32nd Bled eConference: Humanizing technology for a sustainable society: Conference proceedings (pp. 427–442). Maribor, Slovenia: University of Maribor University Press. https://doi.org/10.18690/978-961-286-280-0.23 Mejtoft, T., Vejbrink Starbrink, N., Roos Morales, C., Norberg, O., Andersson, M, & Söderström, U. (2023). Cookies and Trust: Trust in organizations and the design of cookie consent prompts. In A. Dix, I. Reppa, C. Westling, H. Witchel, S. Safin, G. van der Veer, J. MacInnes & R. Bond (Eds.), Proceedings of the European Conference on Cognitive Ergonomics 2023, Article 18. New York, NY: ACM. https://doi.org/10.1145/3605655.3605668 Meske, C., & Amojo, I. (2020). Ethical guidelines for the construction of digital nudges. In Proceedings of the 53rd Hawai international conference on system sciences (pp. 3928–3937). Honolulu: HI, USA: HICSS. https://doi.org/10.24251/HICSS.2020.480 A. Wahlroos et al.: Balancing Personalization and Privacy: Towards Personalized Saving Experience in Banking Apps for Young Adults 461 Mills, S. (2022). Personalized nudging. Behavioural Public Policy, 6(1), 150–159. https://doi.org/10.1017/bpp.2020.7 Mirsch, T., Lehrer, C., & Jung, R. (2017). Digital nudging: Altering user behavior in digital environments. In Proceedings of the 13th International Conference on Wirtschaftsinformatik (WI) 2017 (pp. 634–648). St. Gallen, Switzerland: University of St.Gallen. Noreen, U., Shafique, A., Ahmed, Z., & Ashfaq, M. (2023). Banking 4.0: Artificial intel igence (AI) in banking industry & consumer’s perspective. Sustainability, 15(4). https://doi.org/10.3390/su15043682 OECD. (2020). Pisa 2018 results (volume IV): Are students smart about money? (Tech. Rep.). https://doi.org/10.1787/48ebd1ba-en O’Neill, B. (2009, February). The benefits of saving money. Rutgers. Retrieved from https://njaes.rutgers.edu/sshw/message/message.php?p=Finance&m=122 Peer, E., Egelman, S., Harbach, M., Malkin, N., Mathur, A., & Frik, A. (2020). Nudge me right: Personalizing online security nudges to people’s decision-making styles. Computers in Human Behavior, 109, 106347. https://doi.org/10.1016/j.chb.2020.106347 Remund, D. L. (2010). Financial literacy explicated: The case for a clearer definition in an increasingly complex economy. Journal of Consumer Af airs, 44(2), 276–295. https://doi.org/10.1111/j.1745-6606.2010.01169.x SBAB. (2024, February 6). Var fjärde person saknar til räcklig buf ert för oförutsedda utgifter. Retrieved from https://www.sbab.se/download/18.43c35c8018d11ae303026d6/1707135423714/240206_Pr essmeddelande%20Spartempen.pdf Schneider, C., Weinmann, M., & Vom Brocke, J. (2018). Digital nudging: Guiding online user choices through interface design. Communications of the ACM, 61(7), 67–73. https://doi.org/10.1145/3213765 Sinnewe, E., & Nicholson, G. (2023). Healthy financial habits in young adults: An exploratory study of the relationship between subjective financial literacy, engagement with finances, and financial decision-making. Journal of Consumer Af airs, 57(1), 564–592. https://doi.org/10.1111/joca.12512 Te’eni-Harari, T. (2016). Financial literacy among children: the role of involvement in saving money. Young Consumers, 17(2), 197–208. https://doi.org/10.1108/YC-01-2016-00579 Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. New Haven, CT, USA: Yale University Press. Walstad, W. B., & Wagner, J. (2023). Required or voluntary financial education and saving behaviors. The Journal of Economic Education, 54(1), 17–37. https://doi.org/10.1080/00220485.2022.2144573 462 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION DECISION MINING IN THE RAIL INDUSTRY: A CASE STUDY IN THE CONTEXT OF AN INDUSTRIAL WHEELSET REVISION PROCESS BART HOORNSTRA,1 JORIS MENS,2 IRIS BEEREPOOT,1 INGE VAN DE WEERD1 1 Utrecht University, Faculty of Science, Utrecht, Netherland b.j.g.s.hoornstra@students.uu.nl i.m.beerepoot@uu.nl g.c.vandeweerd@uu.nl 2 N.V. Nederlandse Spoorwegen (Dutch Railways), Utrecht, Netherlands joris.mens@ns.nl Process mining has led to new avenues of analysis and better process understanding. However, the role of decisions within the modeling and analysis of processes is underexplored. Following design science, a methodology for integrated process and decision mining was developed, based on the synthesis of an established process mining project methodology and a systematic Keywords: decision literature review of existing decision mining approaches. The mining, methodology was applied and evaluated in a case study at the process Dutch national railway company. The results demonstrated that mining, process the addition of a decision perspective to process models al ows model for better process understanding. In addition, the evaluation enhancement, identified a new form of conformance checking that can be used conformance checking, to validate whether the process was executed correctly in rail accordance with the decisions taken. industry DOI https://doi.org/10.18690/um.fov.4.2024.27 ISBN 978-961-286-871-0 464 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Top-performing organizations typical y employ agile decision-making based on rigorous analysis and use these insights to improve their day-to-day operations as well as to guide future strategies (LaValle et al., 2010). However, the upfront understanding of organizational decision-making is paramount for successful business analytics implementations (Sharma et al., 2014). Therefore, these do not inherently create value, especial y since the technologies should merely be seen as tools — not drivers — that aid in dealing with information overload (Edmunds & Morris, 2000). The rapid advances in information technology have led to the paradoxical condition that, even though available information is abundant, it is more difficult to extract relevant and useful information when needed (Edmunds & Morris, 2000). Nevertheless, the potential value of improved decision-making enabled by inclusion of contextual process information justifies investments in new forms of data-driven analytics (Sharma et al., 2014). A promising research area in data-driven analytics is process mining (van der Aalst & Weijters, 2004. Process mining allows not only for the investigation of causal relations between activities but also additional data attributes that enable the investigation of performance (timestamps) and workload (resources) (van der Aalst & Weijters, 2004). With the abundance of data available, it becomes increasingly relevant to critically assess and evaluate event log quality (Kherbouche et al., 2016). While research has been carried out to address these latter aspects for event logs (Fischer et al., 2020; Suriadi et al., 2017; van Wensveen, 2020), limited attempts have been made to enhance event logs with data from the context of the process execution (Banham et al., 2022). In that respect, the field of decision mining recently gained more widespread attention within the scientific community (De Smedt, vanden Broucke, et al., 2017). This development is grounded in the idea that at least some separation of concerns between business logic (rules, decisions) and processes should be achieved for the appropriate balance between flexibility, compliance, efficiency, and effectiveness of supporting information systems (Vanthienen et al., 2013). While processes and decisions are intertwined by nature, there are several addressable issues observed at their intersection. Firstly, when a process model incorporates too detailed decision paths, it becomes more or less a decision tree B. Hoornstra et al.: Decision Mining in the Rail Industry: A Case Study in the Context of an Industrial Wheelset Revision Process 465 represented as a cluttered process model. These unnecessarily convoluted process models are difficult to reuse and maintain (De Smedt, vanden Broucke, et al., 2017). Secondly, in process models where business rules imperatively constrain the control- flow, the flexibility required for the high volatility of such rules might be impaired. Thirdly, decisions might be the driver behind the activities and workflows of al process stakeholders, and as such they should be modeled separately to accurately document the related knowledge and to al ow for reuse beyond a single process. Fourthly, a process might be the execution of a complex decision in itself, where the relationships between decisions should be explicitly modeled such that decision- making can be facilitated by an optimal process. Finally, processes that are highly dynamic, human-centric, and non-standardized could benefit from declarative process modeling where the principles are the same, but each case is genuinely distinct (Vanthienen et al., 2013). The aforementioned issues indicate that there does not exist a one-size-fits-al solution to integrate business logic with process knowledge and that knowledge on extending process mining with decision mining is lacking. Therefore, the research question for this paper is as follows: How to extend process mining with decision mining? An existing process mining project methodology is followed in the form of PM2 (Van Eck et al., 2015). The extended framework PM2xDM is developed using the DSRM (Peffers et al., 2007), and subsequently applied and empirically validated in an embedded, single-case study (Yin, 2018). The remainder of this paper elaborates on the aforementioned concepts and is structured as follows. First, the background is sketched in terms of fundamental concept definitions related to process and decision mining, before the context of the case study is further elaborated. Then, the research method is explained concerning the phases of the DSRM and the results of the case study are presented. Final y, the implications, contributions, chal enges, and limitations of this research are discussed, and an overall conclusion is drawn, complemented by an outlook on future research directions. 466 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 2 Background 2.1 Process mining Process mining aims to discover, monitor and improve real processes by extracting knowledge from event logs (van der Aalst, 2011). The smallest unit of examination is an event, where each event refers to an activity within the process (e.g. a single step that has been completed). Each event belongs to a particular case, which is one execution of the process, sometimes referred to as process instance. All events must be ordered sequential y, either by a numerical property or for example by a timestamp. In addition, each event could contain more information such as the resource involved with the activity or additional data attributes about conditions, the state or execution of the process. Al events from a set of process instances combined form an event log (van der Aalst, 2012). Three types of process mining activities are commonly identified: discovery, conformance checking and enhancement (or extension). Process discovery is the creation of a model solely based on the observed events. Conformance checking deals with verifying whether an event log complies with an (existing) process model, and the other way around. Contrary to conformance checking, process enhancement does not compare a model with reality (van der Aalst, 2012). Instead, it tries to change, correct, extend or enrich the already existing model. This can either be already accomplished by examining timestamps and calculating time differences to demonstrate service times, and to indicate possible bottlenecks. Additional y, one could include the resource attribute to for example identify underutilized resources, frequently execute related activities, or lead to specific or unwanted behavior. These different activities in turn correlate with four dominant analysis perspectives within the process mining paradigm (van der Aalst, 2016): control-flow, time, organizational/resource, and data. 2.2 From decision management and modeling to decision mining Decision management and modeling are critical components of organizational strategy, that comprises a suite of methodologies and technologies designed to automate and refine decision-making processes (Yates, 2003). Central to this tandem is the use of data analysis, where business rules and business logic are investigated B. Hoornstra et al.: Decision Mining in the Rail Industry: A Case Study in the Context of an Industrial Wheelset Revision Process 467 (Morgan, 2002; Von Halle & Goldberg, 2009). Business rules provide granular, formal guidelines for consistent, accurate, and legally compliant operations, while business logic offers a broader set of principles and processes that shape strategic decision-making and organizational operations, integrating goals, strategies, and operating principles with business rules, best practices, and industry standards (Morgan, 2002; Von Halle & Goldberg, 2009; Levina et al., 2010). Emerging from this complex decision-making landscape is decision mining, a discipline that extends the traditional focus of process mining by exploring the impact of data attributes on decision-making within processes (Beerepoot et al., 2023). Decision mining acknowledges the data perspective of process mining, examining the nuances of how data informs workflow choices and complements process mining analyses (De Smedt et al., 2019; de Jong et al., 2021). It challenges the notion that workflow data and control-flow must be correlated, recognizing that decisions can affect data attributes and activities throughout a workflow without altering the sequence of activity execution (De Smedt et al., 2019). The integration of decision mining techniques with traditional process mining tools offers the potential for a comprehensive approach to process improvement, aiming for an integrated decision and process model representation that can better capture the complexity of organizational decision-making in relation to process execution (De Smedt, vanden Broucke, et al., 2017). In sum, decision management, modeling, and mining can work in concert to enhance the organizational capacity for informed and strategic decision-making. By recognizing the distinctive but overlapping roles of these disciplines, organizations can harness a holistic approach to improve their capabilities for process analysis and improvement. 3 Research method The artifact that is being developed in this study is an extended methodological framework for the application of decision mining within a process mining project. Therefore, this project fol ows the design science research methodology (DSRM) proposed by Peffers et al. (2007). The steps are illustrated in Figure 1 and further described thereafter. 468 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 1: The DSRM and its implementation specific to this research project Source: adapted from Peffers et al. (2007). Problem identification and motivation. As there does not exist a methodology for a decision mining project, an existing process mining project methodology is used as a basis. Objectives of a solution. The objective is to design an extended methodology that integrates decision mining activities into a process mining project. The subsequent goal is to present an enhanced perspective on the process, where the integration of decision information into the process models al ows for a better understanding of the process and relevant analysis activities, such as conformance checking. Design and development. Based on a systematic review of the state-of-the-art literature, relevant activities and contextualized evaluation strategies are identified. These are subsequently integrated into the proposed methodological framework. Demonstration and evaluation. The initial framework is applied within a process mining project at the Dutch national railway company in the context of an industrial wheelset revision process. The evaluation of the artifacts and the resulting insights is carried out with the relevant stakeholders and experts through a focus group. Communication. The results are integral y presented to the stakeholders as part of the evaluation. Furthermore, the publication of this research report is an additional means of dissemination of the findings. 4 The initial methodological framework: PM2xDM The methodological framework is constructed as an adaptation and extension of the widely-used PM2 methodology by Van Eck et al. (2015). Figure 2 shows an overview of the initial framework. For each phase of PM2, one or more complementary B. Hoornstra et al.: Decision Mining in the Rail Industry: A Case Study in the Context of an Industrial Wheelset Revision Process 469 decision mining-related activities have been identified and assigned to those. The depicted steps are further il ustrated and described in the context of the case study in Section 6. Figure 2: An initial overview of PM2xDM’s decision-related activities Source: based on Van Eck et al. (2015) 5.1 Case study The case study is performed within the largest rail operator in the Netherlands. The organization employs around twenty thousand people and is responsible for the operation and maintenance of trains as wel as al train stations in The Netherlands. 5.2 Context Due to the size of the organization a diverse array of process domains is present. The current case study regarding wheelset revision is a subdomain of the maintenance organization. A previous study by Smit & Mens (2019) identified this process as having a high data and event log quality due to its automated production line. It also scored highly on process mining success factors identified by Mans et al. (2013), when compared to other processes in the organization. The quality and availability of necessary data, as well as stakeholder commitment, contributed to the suitability of this process for the case study. 470 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The wheelset revision process starts with preparation steps that involve cleaning, bearing removal, and gearbox inspection. Furthermore, a material plan is developed from pre-screening results to direct the treatment and routing of wheelsets and components. The actual wheelset revision fol ows, encompassing disassembly, axle decoating and inspection using non-destructive techniques, conservation with dual- layer coating, reassembly at the on-press station, and final measurements and adjustments. Non-gearbox axles undergo additional balance testing before final assembly and quality checks. The facility accommodates 24 wheelset types, each with a numerical identifier and specific to train models. Wheelsets are categorized into motor types, equipped with gearboxes and brake plates, and running types, which lack a direct drive connection. The treatment path for each wheelset type is predefined in a material plan based on its components, guiding the process flow upon factory entry. 5.3 Stage 1: Planning The revision process is managed by a Manufacturing Execution System (MES), ranging from measurement assessment, routing decisions, and control of equipment and machines. We identified the related information systems architecture supporting the process through document analysis and meetings with the MES system’s product owner. MES as orchestrator interfaces with a system for logistic tracking and financial reporting, a system for asset maintenance tracking, while an ERP system manages inventory. A configuration management system stores unstructured text documents related to work procedures, which is not interfaced with MES. MES has its own internal repository for routing logic and measurement criteria. 5.4 Stage 2: Extraction Event data for 2022 was provided as a CSV file with over 10 million rows and six columns, comprising an order ( case), workstation ( activity), and key-value pairs of activity-related attributes (e.g. text, numerical, timestamp). The data was reshaped into wide format using Python with Pandas in a Jupyter Notebook. This resulted in an event log with 510 attributes for decision mining. Decision data extraction focused on the MES’s descriptive attributes without seeking external sources. This phase aimed to understand routing decisions based on internal criteria, acknowledging the chal enges in extracting comprehensive decision data at this B. Hoornstra et al.: Decision Mining in the Rail Industry: A Case Study in the Context of an Industrial Wheelset Revision Process 471 stage. Knowledge transfer involved data reshaping and mapping to process mining concepts with domain expert involvement, streamlined into several interactive sessions and communications to minimize the expert burden. 5.5 Stage 3: Data processing This stage utilized three tools for data exploration, event log manipulation, and model generation: Fluxicon Disco 3.6.7 for exploration of the data sets and creation/manipulation of event logs, ProM 6.13 (Verbeek et al., 2011) for process model generation beyond Directly Follows Graphs (DFGs) and PM4Py 2.7.4 with Scikit-learn (Berti et al., 2023; Pedregosa et al., 2011) for Petri net generation and decision mining. Initial log analysis revealed a highly complex spaghetti-like process model. Further investigation and expert discussions identified discrepancies due to premature equipment start events. To address this, additional activities were added to the event log, ensuring a comprehensive analysis while maintaining data integrity and clarity. This process refinement led to a streamlined dataset that preserves all data attributes, conducive to identifying process variances and generating a readable model despite inherent complexity. 5.6 Stage 4: Mining and analysis 5.6.1 Decision point and model discovery An initial Directly-Follows Graph (DFG) for the wheelset revision process was generated using Disco, based on a Fuzzy miner approach (Gunther & van der Aalst, 2007). Despite technical challenges, such as Java errors in ProM due to the large feature space, adjustments to noise thresholds and filtering strategies enabled the creation of more interpretable models. Analysis in a Jupyter Notebook with Pandas and PM4Py facilitated the discovery of decision points and the examination of process variants and exceptions. By focusing on complete events and applying filters, issues related to loops were mitigated although this incurred some information loss. This highlighted the importance of considering both low-frequency paths for compliance and more frequent exceptions for pattern analysis. 472 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 5.6.2 Decision rule validation and model enhancement Conformance checking is aimed at aligning real-world behavior with the process model, focusing on fitness and appropriateness. To investigate the different rules and path associations, the paths should be at least present in the model. Therefore, the emphasis was on accommodating al traces and variants in the log and investigating exceptions through decision mining, even if this meant accepting certain exceptional cases to maintain a fitness level of 100% for an accurate decision mining analysis. The enhancement phase involved refining the process model with additional decision-related information, using a decision tree classifier for attribute analysis. This phase underscored the relevance of feature selection and the need to exclude non-explanatory attributes. Annotated decision points with guard expressions il ustrated how specific conditions could direct process flow, enhancing model accuracy and interpretability. Figure 3 presents an example of such an annotation, where the conditional routing was discovered. Figure 3: Example of an annotated decision point for an optional examination step in the Petri net for the most common wheelset type Source: Own 5.7 Stage 5: Evaluation A focus group, complemented by intermediate collaborative discussions, evaluated the results of the framework with industry experts, focusing on its application and improvement opportunities. The final focus group evaluation episode, including the researcher and three domain experts, fol owed a predefined protocol (Krueger & Casey, 2015, Saunders et al., 2009) and lasted slightly more than two hours, discussing the application and the results thematically. The emphasis was on the decision point discovery and validating the respective annotations in the model. More advanced activities from the framework such as decision-based process metric, B. Hoornstra et al.: Decision Mining in the Rail Industry: A Case Study in the Context of an Industrial Wheelset Revision Process 473 trend analysis, and predictive analysis were omitted due to feasibility reasons, either incurred by the available data or time constraints. 5.8 Stage 6: Process improvement and support Although actual decision logic refinement and automation was out of scope, the initial part of the evaluation revealed the adaptability of the process and the impact of its physical and logical architecture on the abstraction of event data. It was identified that physical constraints and logical configurability dictate process adaptability. Physically, some activities are time and location bound, due to an ordering constraint or factory layout. Nevertheless, the MES offers infinite logical configurations for extensive customization, influencing routing based on decision thresholds. The decision-making process is embedded in the software, with execution criteria evaluated at each step without making use of forecasting. An interesting notion was that revision processes like this reveal needs and information progressively, contrasting with predefined paths within a production process. The former trait is also seen in other types of processes, such as patient trajectories in healthcare, where diagnosis outcomes alter needs during execution. Furthermore, the evaluation underscored the importance of refining process models and decision criteria to obtain more accurate, applicable, and useful analysis results. Firstly, incorporating annotated decision points could improve process model accuracy and applicability, as one expert remarked that it is useful in that “we want to understand the process, not the physical stations.” Although not all validated decision attributes were necessarily correct or explanatory, the expert remarked that “I am cautiously a bit positive that you are already showing more than what I have seen so far in process mining by adding those decisions [in the model].” Secondly, future work on this particular case should therefore first focus on refined feature engineering and subsequently on decision criteria representation in other modeling paradigms, such as BPMN. Thirdly, it was also identified that process mining tools and artifacts need better support for handling deliberate loops and rework, as this was represented as an attribute. However, representing a repeated activity separately could lead to a less comprehensible model. Finally, the evaluation concluded with an outlook on future use of the presented concept. The experts indicated that it could be used to validate if the wheelsets have been revised according to the regulations, in what would entail decision-based conformance checking. In other words, the paths in the model 474 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION should align with the expected attribute values. This is especially relevant if a process exhibits more variation than expected. One expert il ustrated that by stating that "we apparently went through 262 dif erent processes to deliver a wheelset. So, how do we know that al 262 variations are valid and produced a sound product? How can you guarantee that? [. .] How can you adequately assess 262 dif erent variations? [.. ] I think this should be possible if your model is a bit more accurate." Another expert confirmed: "Yes, this [concept] could then definitely help with that." 6 Discussion This research has demonstrated the relevance and applicability of decision mining within a process mining project. Enhanced process models were produced using case and activity attribute data, building on limited initial semantic knowledge about the process. An analysis of the decision points within the process aided by such visualizations demonstrated an interesting starting point for further applications, such as richer process documentation that shows under which conditions certain paths are taken (De Smedt, Hasić, et al. , 2017). In addition, an enhanced form of conformance checking could be developed using the enhanced models. Validation of whether the production of assets has been performed in accordance with the required guidelines and regulations could be supported using these artifacts (Levina et al., 2010). This implies that, depending on the project goals, it is worthwhile to assess the suitability for decision-mining analysis. However, improvements should be made to the input data and the decision-mining algorithm. More elaborate feature engineering and reduction of the feature space are areas of optimization. Moreover, the attributes from nonlocal activities should be considered, e.g. by enriching activities with attributes from earlier activities or a symbolic link that states the attributes of which other activities should be considered at a certain decision point. Furthermore, we investigated what and how activities should be carried out and what they entail in terms of suitable process characteristics and data requirements to pursue a relevant and meaningful decision-mining analysis. A significant observation was that it should be possible to obtain a sufficiently readable process model at fitness levels greater than 80% to be able to perform a meaningful analysis. An argument for this is that if specific deviations are not present in the model, these wil also not be annotated with the conditions under which they occur. Therefore, this type of analysis is less applicable to processes that are only loosely structured or B. Hoornstra et al.: Decision Mining in the Rail Industry: A Case Study in the Context of an Industrial Wheelset Revision Process 475 exhibit an extreme degree of variation. This is in line with the analysis challenges posed by knowledge-intensive processes (Di Ciccio et al., 2015) or processes that accommodate a wide variety of different needs, such as healthcare processes (Munoz-Gama et al., 2022). 7.1 Contributions The scientific contributions of this research are twofold. First, this research explored a potential avenue for a more holistic integration between process and decision mining, as suggested by De Smedt, Hasić, et al. (2017). Although it was unfeasible with the present tools and techniques to discover a ful y integrated model of control flow and decisions, it supports the notion that the underutilized data perspective of process mining can provide relevant insights (Banham et al., 2022; van der Aalst, 2016). The methodology was implemented within a case study in a real-world context, and the insights were validated and evaluated within a focus group. Second, the foundational PM2 methodology (Van Eck et al., 2015) has been extended with a decision-mining component. The synthesis of the common activities based on the literature and the practical implementation helps to increase our common understanding of the intersection between process and decision-mining, and helps in shaping future research opportunities for the respective activities that have been defined. From a practical perspective, the proposed methodology can help practitioners systematical y execute decision mining within a process mining project. Furthermore, since it is based on and integrated with a generic process mining project methodology, it can be included in an existing project if it aligns with the project goals. This in turn helps optimize efficient resource usage, as it does not require the creation of a distinct project as is the case with classical data mining projects that serve similar purposes (Osei-Bryson, 2012). 7.2 Future research Future work could build on this research in several ways. First and foremost, the PM2xDM framework should be repeatedly applied in different environments and contexts to develop a more robust context-agnostic version. Such follow-up experiments could, in addition, contain a part that also pays special attention to the 476 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION execution of the methodology itself by process analysts. Second, research could focus on developing a toolkit that integrates several of the decision-mining assessment steps and activities of the framework into a single software package, for a more straightforward application within a process mining project. Furthermore, research could also focus on enabling additional interoperability between visualizations, such as the conversion of Petri nets with data into BPMN diagrams that retain these conditions. 8 Conclusion The methodological framework PM2xDM was developed based on the established process mining project methodology PM2. It al owed us to enrich a Petri net process model with conditions based on the event data attributes, converting it into a Petri net with data (DPN). This research has shown that visualization of decisions in process models can be useful to organizations implementing a process mining project. Additional y, it helps to present a more realistic perspective on the process during discovery, and it allows for enhanced activities, such as decision-based conformance checking. Acknowledgements This research was carried out as part of an MSc thesis internship project of the primary author. The authors thank the host organization (Dutch Railways) for their cooperation in this research. In particular, we thank the experts from the wheelset revision department (RLW) for their time investment and invaluable contributions that resulted in the successful execution of the different phases of this project. Al authors have seen and agree with the contents of this manuscript and this research was not funded; therefore, no conflicts of interest could be identified or reported. References Banham, A., Leemans, S. J., Wynn, M. T., & Andrews, R. (2022). xPM: a Framework for Process Mining with Exogenous Data. Process Mining Workshops: ICPM 2021 International Workshops, Eindhoven, The Netherlands, October 31–November 4, 2021, Revised Selected Papers, 85–97. Beerepoot, I., Barenholz, D., Beekhuis, S., Gulden, J., Lee, S., Lu, X., Overbeek, S., van de Weerd, I., van der Werf, J. M., & Reijers, H. A. (2023). A Window of Opportunity: Active Window Tracking for Mining Work Practices. 2023 5th International Conference on Process Mining (ICPM), 57–64. Berti, A., van Zelst, S., & Schuster, D. (2023). PM4Py: A process mining library for Python. Software Impacts, 17, 100556. https://doi.org/https://doi.org/10.1016/j.simpa.2023.100556 Chen, H., Chiang, R. H., & Storey, V. C. (2012). Business intel igence and analytics: From big data to big impact. MIS quarterly, 1165–1188. B. Hoornstra et al.: Decision Mining in the Rail Industry: A Case Study in the Context of an Industrial Wheelset Revision Process 477 De Smedt, J., Hasić, F., vanden Broucke, S. K., & Vanthienen, J. (2017). Towards a holistic discovery of decisions in process-aware information systems. Business Process Management: 15th International Conference, BPM 2017, Barcelona, Spain, September 10–15, 2017, Proceedings 15, 183–199. De Smedt, J., Hasić, F., vanden Broucke, S. K., & Vanthienen, J. (2019). Holistic discovery of decision models from process execution data. Knowledge-Based Systems, 183, 104866. De Smedt, J., vanden Broucke, S. K., Obregon, J., Kim, A., Jung, J.-Y., & Vanthienen, J. (2017). Decision mining in a broader context: An overview of the current landscape and future directions. Business Process Management Workshops: BPM 2016 International Workshops, Rio de Janeiro, Brazil, September 19, 2016, Revised Papers 14, 197–207. de Jong, R., Leewis, S., & Berkhout, M. (2021). Decision Mining versus Process Mining: a Comparison of Mining Methods. 2021 5th International Conference on Software and e-Business (ICSEB), 28–32. Di Ciccio, C., Marrel a, A., & Russo, A. (2015). Knowledge-intensive processes: Characteristics, requirements and analysis of contemporary approaches. Journal on Data Semantics, 4, 29–57. Edmunds, A., & Morris, A. (2000). The problem of information overload in business organisations: a review of the literature. International Journal of Information Management, 20(1), 17–28. Fischer, D. A., Goel, K., Andrews, R., van Dun, C. G. J., Wynn, M. T., & Röglinger, M. (2020). Enhancing event log quality: Detecting and quantifying timestamp imperfections. Business Process Management: 18th International Conference, BPM 2020, Seville, Spain, September 13–18, 2020, Proceedings 18, 309–326. Gunther, C. W., & van der Aalst, W. M. (2007). Fuzzy mining–adaptive process simplification based on multiperspective metrics. Business Process Management: 5th International Conference, BPM 2007, Brisbane, Australia, September 24-28, 2007, Proceedings, 4714, 328. Hammer, M., & Champy, J. (1993). Reengineering the corporation: A manifesto for business revolution. Harper Collins, New York. Kherbouche, M. O., Laga, N., & Masse, P.-A. (2016). Towards a better assessment of event logs quality. 2016 IEEE Symposium Series on Computational Intelligence (SSCI), 1–8. Krueger, R. A., & Casey, M. A. (2015). Focus group interviewing. Handbook of practical program evaluation, 506–534. LaVal e, S., Lesser, E., Shockley, R., Hopkins, M. S., & Kruschwitz, N. (2010). Big data, analytics and the path from insights to value. MIT Sloan Management Review. Levina, O., Holschke, O., & Rake-Revelant, J. (2010). Extracting business logic from business process models. 2010 2nd IEEE International Conference on Information Management and Engineering, 289–293. Mans, R. R., Reijers, H., Berends, H., Bandara, W., & Prince, R. (2013). Business process mining success. Proceedings of the 21st European Conference on Information Systems, 1–13. Morgan, T. (2002). Business rules and information systems: aligning IT with business goals. Addison-Wesley Professional. Munoz-Gama, J., Martin, N., Fernandez-Llatas, C., Johnson, O. A., Sepúlveda, M., Helm, E., Galvez-Yanjari, V., Rojas, E., Martinez-Millana, A., Aloini, D., et al. (2022). Process mining for healthcare: Characteristics and chal enges. Journal of Biomedical Informatics, 127, 103994. Osei-Bryson, K.-M. (2012). A context-aware data mining process model based framework for supporting evaluation of data mining results. Expert Systems with Applications, 39(1), 1156– 1164. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A design science research methodology for information systems research. Journal of Management Information Systems, 24(3), 45–77. 478 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Saunders, M., Lewis, P., & Thornhil, A. (2009). Research methods for business students. Pearson Education. Sharma, R., Mithas, S., & Kankanhal i, A. (2014). Transforming decision-making processes: a research agenda for understanding the impact of business analytics on organisations. European Journal of Information Systems, 23(4), 433–441. Smit, K. & Mens, J. (2019). Process mining in the rail industry: a qualitative analysis of success factors and remaining chal enges. Bled 2019 Proceedings, vol. 25 Suriadi, S., Andrews, R., ter Hofstede, A. H., & Wynn, M. T. (2017). Event log imperfection patterns for process mining: Towards a systematic approach to cleaning event logs. Information systems, 64, 132– 150. Van der Aalst, W., Weijters, T., & Maruster, L. (2004). Workflow mining: Discovering process models from event logs. IEEE transactions on knowledge and data engineering, 16(9), 1128– 1142. Van der Aalst, W. M. (1998). The application of Petri nets to workflow management. Journal of Circuits, Systems, and Computers, 8(01), 21–66. Van der Aalst, W. M., & Weijters, A. J. (2004). Process mining: a research agenda. Computers In Industry, 53(3), 231–244. van der Aalst, W. M. P. (2004). Business Process Management: A personal view. Business Process Management Journal, 10(2). van der Aalst, W. M. P. (2011). Process Mining: Discovery, Conformance and Enhancement of Business Processes (Vol. 2). Springer. van der Aalst, W. M. P. (2012). Process mining: Overview and opportunities. ACM Transactions on Management Information Systems (TMIS), 3(2), 1–17. van der Aalst, W. M. P. (2016). Process Mining: Data Science in Action (Vol. 2). Springer. van der Aalst, W. M. P., ter Hofstede, A. H. M., & Weske, M. (2003). Business Process Management: A Survey. Business Process Management: International Conference, BPM 2003, Eindhoven, The Netherlands, June 26–27, 2003 Proceedings 1, 1–12. van der Aalst, W. M. P., Van Dongen, B. F., Herbst, J., Maruster, L., Schimm, G., & Weijters, A. J. M. M. (2003). Workflow mining: A survey of issues and approaches. Data & Knowledge Engineering, 47(2), 237–267. Van Eck, M. L., Lu, X., Leemans, S. J., & Van Der Aalst, W. M. (2015). PM2: a process mining project methodology. Advanced Information Systems Engineering: 27th International Conference, CAiSE 2015, Stockholm, Sweden, June 8-12, 2015, Proceedings, 297–313. Vanthienen, J., Caron, F., & De Smedt, J. (2013). Business rules, decisions and processes: five reflections upon living apart together. Proceedings SIGBPS Workshop on Business Processes and Services (BPS’13), 76–81. van Wensveen, B. R. (2020). Estimation and analysis of the quality of event log samples for process discovery [Master’s thesis]. Verbeek, H., Buijs, J. C., Van Dongen, B. F., & Van Der Aalst, W. M. (2011). XES, XESame, and ProM 6. Information Systems Evolution: CAiSE Forum 2010, Hammamet, Tunisia, June 7-9, 2010, Selected Extended Papers 22, 60–75. Von Hal e, B., & Goldberg, L. (2009). The decision model: a business logic framework linking business and technology. CRC Press. Yates, J. F. (2003). Decision management: How to assure better decisions in your company. John Wiley & Sons. Yin, R. K. (2018). Case Study Research and Applications: Design and Methods (6th ed.). Sage. TOWARDS SOCIAL DIGITAL TWINS – AN INTEGRATED SOCIOTECHNICAL APPROACH FOR THE URBAN ENERGY TRANSITION STEVEN P. HAVEMAN,1 XIAO PENG,1 RAN ZHANG,1 TADEO-BALDIRI SALCEDO-RAHOLA2 1 HU University of Applied Sciences Utrecht, Lectorate Organisations in Digital Transition, Utrecht, Netherland steven.haveman@hu.nl, xiao.peng@hu.nl, ran.zhang@hu.nl 2 The Hague University of Applied Sciences, Faculty of Technology, Innovation and Society, The Hague, Netherland t.b.salcedorahola@hhs.nl The urban energy transition is crucial for a sustainable future. To support this transition, Digital Twins are employed in an increasing fashion, providing decision makers with data-driven insights from mainly technological perspectives. Based on a case study of a neighbourhood in a Dutch municipality, we argue the need to address social perspectives more explicitly while employing Digital Twins. To this end, we identify three potential strategies for an integrated socio-technological approach for Keywords: Digital Twins. These strategies are model ing social d igital characteristics at a macro-economic scale, involving stakeholders twin, urban in participatory approaches, and finally explicitly modelling digital stakeholder behaviour. Given its promise for our case study, we twin, elaborate this last strategy with a conceptual method that aims to energy transition, explicitly model citizens’ decision-making processes through an simulation, agent-based model ing approach. socio-technical DOI https://doi.org/10.18690/um.fov.4.2024.28 ISBN 978-961-286-871-0 480 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Aligned with the 2015 Paris Climate Agreement (United Nations, 2015), The Netherlands has set targets to reduce greenhouse gases by 49 per cent by 2030 relative to 1990, aiming for a carbon-neutral energy system by 2050 (Government of the Netherlands, 2019a). To achieve the set targets, the Dutch government has defined the general energy transition guidelines in the Climate Agreement (Government of the Netherlands, 2019a), emphasizing that a col ective effort from Dutch residents is needed to undergo an energy transition. The Dutch government has therefore chosen for a decentralized approach, where municipalities play the lead role in coordinating and supporting citizens within cities for the energy transition. Municipalities are therefore responsible to design and implement specific plans for their own city as defined in the Regional Energy Strategy plan (Government of the Netherlands, 2019b). 1.1 Urban Energy Transitions with Citizen Insights To facilitate urban energy transitions, municipalities are already taking actions, for example implementing subsidies to make the shift financial y manageable for energy users (Association of Dutch Municipalities (VNG), 2023). Despite these efforts, achieving public participation and engagement remains a chal enge. Numerous barriers impede the energy transition, including gaps in knowledge and understanding of the process, conflicting interests and preferences among stakeholders, and chal enges that extend beyond technological and financial constraints (Dorenbos et al., 2020). The sustainable energy transition involves a complex interplay of interests among various stakeholders, where their objectives may sometimes align, and other times, diverge. Among the key stakeholders, municipalities could exhibit interests that significantly differ from those of local citizens (Buana et al., 2023). Municipalities may prioritize societal economic development, infrastructure improvements, and broader environmental goals set by the government. Whereas citizens prioritize personal benefits, with minimal regulatory and administrative burdens (Deng et al., 2023), and direct impacts on their daily lives, such as energy costs and savings (Bellos, 2018), ease and comfort (Wang et al., 2020), and potential property value enhancement (Selvakkumaran & Ahlgren, 2019). S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 481 The mismatch of interests between municipalities and citizens could create chal enges in fostering engagement in sustainable energy initiatives. This underscores the imperative for local governments to incorporate a more nuanced understanding of the socio-psychological dimensions underpinning citizens' behaviour, ensuring a more holistic and effective approach to the sustainable energy transition process. 1.2 Digital Twins for Urban Energy Transitions Employing a Digital Twin approach to model a neighbourhood’s energy system (including buildings and technical instal ations) appears to be a promising strategy to support urban energy transitions. Digital Twins allow stakeholders to explore various scenarios with changing conditions and interventions in a digital world that is closely connected and linked to the real world including the physical living environment and people who live there, to final y support their decision-making processes. The concept of Digital Twins is broad (Lehtola et al., 2022) and requires a further definition. In many industries and domains, the concept of Digital Twins has been employed. For example, a building’s energy system (C. Li et al., 2023), the traffic situation on a highway (Saroj et al., 2021) or the status of a fleet of trains (Dimitrova & Tomov, 2021). In these situations, the concept is mainly employed to measure data from tangible real-world phenomena using various sensors to bring real-world data to a digital representation. Subsequently, the digital representation might also control actuators in the real-world system, to adjust or change its operations. With a Digital Twin approach, various scenarios towards a more sustainable energy system can be explored. However, in these types of transitions, stakeholders, such as citizens, play a much more prominent role compared to when a Digital Twin is employed to monitor e.g. a fleet of trains. In our view, this requires a digital representation of these actors, their beliefs, capabilities, and decision-making strategies. 482 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1.3 Research Objective Currently, it is still challenging to model a complex system, particularly one that incorporates human elements (Wan et al., 2019). We aim to provide a stepping stone for this chal enge in the context of urban energy transitions by exploring the following research question in a Dutch context: “How to incorporate social perspectives of citizens in Digital Twins that model the technical aspects of neighbourhood energy transitions to improve decision-making on urban energy transitions for e.g. municipalities and citizens?” In this paper, our objective is thus to outline relevant methods that can be used to capture, understand, and potentially steer social decision-making dynamics in sustainable energy transitions. We acknowledge the argument that in a most truthful implementation to the concept, a Digital Twin should rely on (near) real-time data and control (Jafari et al., 2023; Sepasgozar, 2021), although others do not view real-time data as mandatory (do Amaral et al., 2023). In this work, we employ the notion of Digital Twins with an aim to achieve an accurate and truthful digital representation of our system-of- interest. This representation also entails monitoring and control, but on a longer time scale of e.g. policy implementations and subsequent monitoring of developments in a neighbourhood. The focus of this work towards the Digital Twin concept is however aimed at the simulation aspects of the virtual model in a Digital Twin. We explore how to model and simulate both technical and social aspects in a single approach, expanding the potential application area for Digital Twins. Especially, this application provides a holistic view and supports strategy formulation, planning, and decision-making processes for energy transitions. 1.4 Paper Outline In section 2, we discuss our case study and research approach. Section 3 presents further background to the social perspectives that we intend to address. Section 4 dives into the state of the art of Digital Twin model ing in urban, energy and S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 483 transitional contexts. In section 5, we outline how these social perspectives can be included in Digital Twin approaches, resulting in “Social Digital Twins”. Finally, a discussion and conclusion are presented in section 6. 2 Methodology This section outlines our methodological approach. The context of the research through a case study is presented in section 2.1. Following this, our research approach is explained in section 2.2. 2.1 Case Study Background The specific area that is subject of the research project “Sustainable and Social Local Energy Systems” (Hogeschool Utrecht, n.d.) is the “Heuvel-Amstelwijk” district in the city of Leidschendam-Voorburg in the Netherlands. This neighbourhood can be characterized as a relatively low-income and multi-cultural area with a large share of high-rise apartment buildings (Alle Cijfers, 2024). On a strategic level, the general action plan of the city has been outlined in the document Transition Vision Heating (Municipality Leidschendam-Voorburg, 2021) as part of Regional Energy Strategy (Government of the Netherlands, 2019b). It has been identified that each neighbourhood could require different approaches and that the choice for each approach depends on several factors, among which technological, financial and organisational (Municipality Leidschendam-Voorburg, 2021). 2.2 Research Approach To uncover relevant social perspectives, which are presented in section 3, we employ a three-fold approach. This approach is based on both previous and forthcoming work in the context of our wider research project and aims to gives context for subsequent sections in this work. The first part of the approach consists of a qualitative literature review. Secondly, we have employed a coded-interview approach with various relevant stakeholder. The subsequent analysis and results are published in previous work (Peng, 2023) and have enriched the discussion presented in section 3. Lastly, we distributed surveys among residents. Data acquisition in the 484 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION survey is still ongoing but preliminary insights are used in a qualitative manner in this work. To assess the current state and usage of Digital Twins for urban energy transitions we have employed a literature review in section 4 with the fol owing keywords "Digital Twin" or "Digital Twinning" and “District” or “Urban” or “Neighbourhood” or “Neighbourhood” and “Energy System” or “Energy Transition”. We have compared various scientific databases from 2015 until now and determined that Google Scholar delivered the most extensive results (n = 293). We then performed a qualitative analysis of these results and present the most relevant results in section 4. Section 5 then discusses the inclusion of social perspectives based on a qualitative literature review using the terms “Socio-technical” or “Social” and “Digital Twin” and “Energy Transition”. 3 Social Perspectives for Digital Twins Financial considerations (Bellos, 2018) and environmental concerns (Selvakkumaran & Ahlgren, 2019) are commonly recognized factors influencing individuals' energy consumption decisions. Research from the fields of economics and psychology indicates that behavioural interventions are an effective tool that can significantly reduce the energy consumption of private households (Andor & Fels, 2018). Consequently, it is crucial to explore and elucidate how social behavioural factors impact residents' energy decisions. Research has revealed that in addition to individual personality traits (Selvakkumaran & Ahlgren, 2019; Tanveer et al., 2021), social factors such as peer effects (Palm, 2017; Wolske et al., 2020), social norms (Tanveer et al., 2021), social altruism (Selvakkumaran & Ahlgren, 2019), concern for own children and others (Ataei et al., 2021; Wang et al., 2020) could all potentially impact one's behaviour and decisions in sustainable energy adoption and usage. While these social factors may function as direct predictors and motivators for decision-making (Ataei et al., 2021; Selvakkumaran & Ahlgren, 2019), another perspective is that their influence goes beyond direct causation, serving instead as complex moderating and mediating variables in the decision-making process. It is conceivable that an individual's environmental concern might be influenced or moderated by social factors. In other words, a person's interest in environmental S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 485 issues and wil ingness to adopt sustainable energy solutions can be shaped by their social surroundings. To include these varied perspectives in a full approach for urban energy transitions, Kourtit et al. (2023) advocate the use of a five-dimensional model, which includes hardware, infoware, finware, socioware and software. Translating this to our case in the Heuvel-Amstelwijk, the hardware represents the building stock and energy systems. The infoware dimension captures the perceptions and preferences of inhabitants. The finware dimension addresses financial viability, which is a challenge in this generally low-income neighbourhood. The socioware dimension comprises social interactions such as common energy initiatives. This links for example to the complex decision making of owners’ associations in the many apartment buildings in the district. These decision-making processes have been recognized as a significant inhibitor of the energy transition in the Netherlands (RTL Nieuws, 2024). Final y, software relates to advanced tools to monitor and guide the urban energy transition, which in our case is the development of a Social Digital Twin. The related work of Nijkamp et al. (2023) presents a participatory concept with a diabolo model in which an energy broker acts a mediator between citizens and government. In this process, they argue that “the use of modern advanced statistical and digital research and visualisation tools seems to be indispensable for suc essful urban energy transitions” (Nijkamp et al., 2023, p. 14). 4 Current Advancements in Digital Twins Given the sentiment expressed by Nijkamp et al. (2023) in the previous section, it is a logical next step to outline the role of Digital Twins in this process. The use of Digital Twins is growing in the field of urban energy transitions (Strielkowski et al., 2022; Weil et al., 2023). In this work, we focus on applying Digital Twins to facilitate decision making in urban energy transitions. Based on this focus, we present our findings in three perspectives. These perspectives are Urban Digital Twins (4.1), Energy Digital Twins (4.2) and finally, Digital Twins for strategic decision-making (4.3). 486 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.1 Urban Digital Twins Weil et al. (2023) present a comprehensive literature review identifying a wide range of chal enges for Urban Digital Twins (UDTs). UDTs collect information from a wide array of sources to cover a multitude of relevant aspects in an urban context, such as transportation, energy, water management, crowd management, noise pollution and climatology (Alva et al., 2022; Lehtola et al., 2022). In certain developments, the aim is even to offer an open data ecosystem for urban data (Cureton & Hartley, 2023). Caprari et al. (2022) present several Digital Twin implementations that utilize urban Digital Twins in the context of urban planning and conclude among others that although participatory approaches are employed, further work is needed for a ful representation. Weil et al. (2023) also highlight that UDTs are not only for decision-making support but also for fostering trust between public and governance. A main and prevalent concern is ongoing lack of proven effectiveness in decision-making, particularly for long-term planning and decisions in the context of dynamic changes. In this sense, one could argue that a neighbourhood energy transition fal s in this category of being both long-term and dynamic. Finally, Nochta et al. (2021) raise the notion that UDTs represent a “paradigm shift in urban model ing”, since more abstract models are exchanged for exact mirrors of the physical system. 4.2 Energy Digital Twins The energy system is one of the key infrastructure elements of a smart city and often the singular focus of Energy Digital Twin (EDT) implementations (Martinelli, 2023). Ghenai et al. (2022) identify a growing interest into Digital Twins in the energy sector. Current EDTs are often used to monitor systems and determine optimization strategies (Bortolini et al., 2022; Ghenai et al., 2022; B. Li & Tan, 2023). B. Li & Tan (2023) conclude that a real-time Digital Twin approach enables faster and more accurate prediction of system performance. The increased availability of smart meter data further accelerates this process (Bayer & Pruckner, 2023; Martinelli, 2023). A systematic literature review of EDTs presented in the work of do Amaral et al. (2023) identifies the most common approaches to model the system as numerical S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 487 methods and data-driven (machine learning) approaches. They also identify several advantages, issues, and opportunities in the application of EDTs. One of the opportunities is “Handle with multi-objective problems, considering the divergent stakeholders’ interests” (do Amaral et al., 2023, p. 12). However, the work does not delve into specific examples of these divergent interests. Bortolini et al. (2022) have conducted a review for EDTs focusing on building energy efficiency and identified a relatively small number of publications, indicating that this field is still novel. Their findings highlight four distinctly different application goals of EDTs, being design optimization, occupant comfort, building operation and maintenance and energy consumption simulation. Specific implementations of EDTs are for example developed for a combination of PV installations, EV charging systems, battery storage systems, heat pumps, heating grids or smart meter data (Agostinelli et al., 2022; Bayer & Pruckner, 2023; Zinsmeister & Perić, 2022). EDTs with a focus on optimal energy price discovery or energy trading are also being explored (Andriopoulos et al., 2023; Dulaimi et al., 2022; Fathy et al., 2021; Tsado et al., 2022). Another angle is to uncover energy demand, e.g. at district level (Huang et al., 2022; Rovers et al., 2022), to provide smart grid control (Mourtzis et al., 2022) or to assess the sustainability of an area (Calabuig-Moreno et al., 2022). 4.3 Digital Twins for Energy Transitions Lesnyak et al. (2023) categorize the application scale of Digital Twins in urban energy transitions into three levels: building, campus, or urban. In this work, we focus on a neighbourhood level, which can be characterized as a large campus. Lesnyak et al. (2023) also discuss that for energy transitions and heating transitions several chal enges remain. At the building level the objective is to address the heterogeneity of individual housing units, questioning the feasibility of a one-fits-all approach. At a campus or neighbourhood level, the objectives shift to integrating outcomes from the building level to enhance energy system operation, utilize predictive maintenance, and facilitate proof-of-concept for proposed solutions. This latter aspect is most relevant for our work. 488 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Bocullo et al. (2023) provide a concrete example of a neighbourhood level approach where a Digital Twin of a city block is developed and subsequently simulated to evaluate various renovation scenarios. They state that “the concept of a Digital Twin in deep renovation is a novel approach” (Bocullo et al., 2023, p. 3) and it serves a more holistic view on a renovation process. They evaluate scenarios comprising various renovation packages that utilize different technologies, and within those scenarios, they compare two alternatives with varying heat and electricity prices. However, Bocullo et al. (2023) do not elaborate on the context in which this design approach is applied. Similarly, HosseiniHaghighi et al. (2022) offer a comprehensive solution for evaluating energy transition scenarios using an extensive UDT approach, delivering detailed models of the housing stock. Through abstract model ing of the energy systems, they evaluate various energy system scenarios. However, they do not consider social perspectives in decision-making to select specific energy systems, so in that sense it remains a solely technical, although very impressive, exercise. Piaia and Frighi (2022) provide insight into application context with a social- technical approach, that includes monitoring health, comfort and wel being of residents. Piaia and Frighi (2022) suggest a six step approach based on (1) doing a baseline analysis case study, (2) modelling the baseline case, (3) simulating to test and monitor different design scenarios, (4) decision-making, (5) employing a multi- criteria decision-making (MCDA) approach to determine thresholds of interventions, and (6) employing a the Digital Twin as a guideline for the ongoing energy transition. In the sixth step, the Digital Twin could be employed to for example guide residents in learning about the effect of their behaviours, or to monitor progress of the transition (Calabuig-Moreno et al., 2022). They also note that the energy (or sustainable) transition cannot be seen separate from the digital transitions, referring to the concept of the ‘twin transition/transformation’ (Fouquet & Hippe, 2022; Graf-Drasch et al., 2023) or ‘dual transformation’ (Kürpick, Kühn, et al., 2023; Kürpick, Rasor, et al., 2023). 5 Towards Social Digital Twins As stated in section 3, an urban energy transition involves a multitude of stakeholders. Therefore, it is key to ensure a relevant representation and involvement of those stakeholders in their decision-making processes, and thus also in the Digital Twin designed to support these processes. Wan et al., (2019, p. 23) state that “system- S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 489 level optimization, though being the explicit purpose of some Digital Twins in the engineering sphere, may not be an ef ective approach to address “wicked” urban problems”. They further argue that Digital Twin developers who typical y have technical expertise are not equipped to address non-technical factors, therefore diminishing the meaningfulness of the subsequent model-based optimizations. We therefore propose a Social Digital Twin which can provide a municipality an overview of both social and technical variables that influence the urban energy transition. 5.1 Social Perspective Inclusion Strategies Based on our literature review, we have identified three strategies to include a social perspective in Digital Twins and will discuss these below. The first strategy addresses social characteristics at a macro-economic scale. For example, Savage et al. (2022) explore social inequality within the energy transition. Yossef Ravid and Aharon-Gutman (2023) introduce the concept of a Social Urban Digital Twin (SUDT) and highlight the lack of social considerations in smart city developments. Their implementation al ows for the analysis of detailed demographic characteristics, such as the accessibility of amenities for certain demographic groups. The second strategy engages residents and other social actors through a participatory approach. With respect to stakeholder involvement, Cureton and Hartley (2023) discuss the use of various user interfaces (VR, apps) within a City Information Model (or UDT). They conclude that virtualization and gamification of these models are “essential areas to build the socio-technical relationships”. In addition, Cureton and Hartley (2023) state that such a model “requires the engagement of stakeholders in the schematic design of these systems, including providing suitable training to attain the new opportunities described above”. The third strategy includes explicit model ing of the behaviour of these actors. Andriopoulos et al. (2023) propose a Consumer Digital Twin (CDT) that models a “human-oriented, simplified virtual replica that represents the entity of an electricity consumer”. For each consumer or user, preferences can be indicated through a user interface for various energy appliances. Subsequent results can be presented in a consumer-oriented Digital Twin dashboard. 490 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Given our case study, we would characterize the abovementioned first “macro- economic” strategy too superficial for a comprehensive understanding of decision- making processes. A deeper insight into decision-making and relevant behavioural characteristics of decision makers is essential. This, in principle, requires an additional layer on top of the Digital Twin information system that models the behaviour of these actors, as proposed in the third strategy. Of course, a blend between these strategies might be feasible as wel . 5.2 Conceptual Implementation Approach To enable a Digital Twin approach that supports decision-making, we follow the strategic approach offered by Nochta et al. (2021), who identify three key aspects that should be addressed. Firstly, it should be possible to translate high-level policy goals into practical policy chal enges for which potential solutions can be developed. In the case of Leidschendam-Voorburg, the municipality aims to transition neighbourhoods from gas-based to renewable energy systems. However, to support the day-to-day execution of the energy transition, it is crucial to provide information and guidance to residents, owners’ associations, and other energy renovation decision makers about specific renovation options suitable for their energy systems and housing situation. The model should include factors for municipality decision-making, such as their own prior decisions, stakeholder disposition and relevant local events, in addition to the more traditional “technical” state and renovation options for the neighbourhood. Furthermore, the model should provide concrete recommendations on where they can act, like subsidy support, information campaigns, timing of decisions, stakeholder engagement, and setting sustainable performance goals (KPIs), etc. Secondly, it is important to reflect the local governance structures appropriately. In our context, this entails accurate understanding the decision-making processes of the involved social actors. We suggest moving beyond traditional rational and individual decision-making and instead acknowledge the complex decision-making processes involved in e.g. owners’ associations by explicitly modelling a more complex group-based decision-making processes for those associations. S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 491 Thirdly, they argue that it is necessary to address both individual and organizational learning. A solely technology-focused approach often lacks human centricity, overlooking the diverse needs of individual and organizational (group) learning required to adopt and integrate CDTs into policy decision-making structures and processes. The municipality needs to learn how to apply these tools to encourage citizen participation in the process, understand how individuals interact, influence, and change their behaviour within the complex system, and facilitate this evolutionary process towards a collective sustainability goal at a macro level. Our intent is to implement this strategic approach into a simulation environment that utilizes a characterization of the built environment and its energy systems combined with an agent-based model that simulates people’s actions to understand their behaviour change in transition scenarios. We aim to characterize the agents in the model by abstraction of both quantitative and qualitative survey data among involved stakeholders into an actionable decision-making model. To this end, we build upon our previous approach (Haveman et al., 2020), which is based on the prevailing notion that “the agent-based approach is found to be uniquely suited for the complex adaptive sociotechnical systems that must be model ed” (Hoekstra et al., 2017). The basis of the decision-making model employed in our approach is the Theory of Planned Behaviour (Ajzen, 1991), which provides a widely understandable reference to understand decision-making and ultimately the social dynamics of an urban energy transition. 6 Discussion & Conclusion This work has given an outlook on potential strategies to include social perspectives in decision-making for urban energy transitions when employing Digital Twins. We have discussed three areas within the field of Digital Twins, being UDTs, EDTs and the usage of Digital Twins in energy transitions in general. This discussion provides an applicable background for extending urban energy transitions with social perspectives. We identified and discussed three strategies for incorporating social perspectives and elaborated the most relevant approach for our case study involves explicitly modelling decision-making through agent-based representations. 492 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION In this discussion, we would like to note that in general, when working with Digital Twins, data security and privacy are relevant themes to consider (Yossef Ravid & Aharon-Gutman, 2023). This wil of course be only more prevalent when working with social perspectives that include more “social data”. This work is part of an ongoing research project, and future work in the project aims to further elaborate, verify, and validate the approach in our current case study as wel as in potential other case studies. Acknowledgements This publication is part of the project “Optimizing the design and operation of Local Energy Systems for neighbourhoods with high self-sufficiency and high actor engagement” with project number 482.20.608 of the research programme The Merian Fund which is financed by the Dutch Research Council (NWO). We would like to thank our project partners for their contributions and col aboration in the overal project leading up to this publication. We also thank specifical y Ruqian Zhang and Pieter-Jan Hoes for their contributions in the conceptualization of this work References Agostinel i, S., Cumo, F., Nezhad, M. M., Orsini, G., & Piras, G. (2022). Renewable Energy System Control ed by Open-Source Tools and Digital Twin Model: Zero Energy Port Area in Italy. Energies, 15(5), Article 5. https://doi.org/10.3390/en15051817 Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T Alle Cijfers. (2024, February 27). Wijk De Heuvel / Amstelwijk (gemeente Leidschendam-Voorburg) in cijfers en grafieken. AlleCijfers.nl. https://allecijfers.nl/wijk/de-heuvel-amstelwijk-leidschendam-voorburg/ Alva, P., Biljecki, F., & Stouffs, R. (2022). USE CASES FOR DISTRICT-SCALE URBAN DIGITAL TWINS. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLVIII-4-W4-2022, 5–12. https://doi.org/10.5194/isprs- archives-XLVIII-4-W4-2022-5-2022 Andor, M. A., & Fels, K. M. (2018). Behavioral Economics and Energy Conservation – A Systematic Review of Non-price Interventions and Their Causal Effects. Ecological Economics, 148, 178–210. https://doi.org/10.1016/j.ecolecon.2018.01.018 Andriopoulos, N., Plakas, K., Mountzouris, C., Gialelis, J., Birbas, A., Karatzas, S., & Papalexopoulos, A. (2023). Local Energy Market-Consumer Digital Twin Coordination for S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 493 Optimal Energy Price Discovery under Thermal Comfort Constraints. Applied Sciences, 13(3), Article 3. https://doi.org/10.3390/app13031798 Association of Dutch Municipalities (VNG). (2023). Overzicht regelingen financiering energietransitie [Overview of energy transition financing schemes]. https://vng.nl/sites/default/files/2020-06/overzicht-regelingen-financiering- energietransitie.pdf Ataei, P., Gholamrezai, S., Movahedi, R., & Aliabadi, V. (2021). An analysis of farmers’ intention to use green pesticides: The application of the extended theory of planned behavior and health belief model. Journal of Rural Studies, 81, 374–384. https://doi.org/10.1016/j.jrurstud.2020.11.003 Bayer, D., & Pruckner, M. (2023). A digital twin of a local energy system based on real smart meter data. Energy Informatics, 6(1), 8. https://doi.org/10.1186/s42162-023-00263-6 Bel os, E. (2018). Sustainable energy development: How can the tension between energy security and energy transition be measured and managed in South Africa? Journal of Cleaner Production, 205, 738–753. https://doi.org/10.1016/j.jclepro.2018.08.196 Bocul o, V., Martišauskas, L., Gatautis, R., Vonžudaitė, O., Bakas, R., Milčius, D., Venčaitis, R., & Pupeikis, D. (2023). A Digital Twin Approach to City Block Renovation Using RES Technologies. Sustainability, 15(12), Article 12. https://doi.org/10.3390/su15129307 Bortolini, R., Rodrigues, R., Alavi, H., Vecchia, L. F. D., & Forcada, N. (2022). Digital Twins’ Applications for Building Energy Efficiency: A Review. Energies, 15(19), Article 19. https://doi.org/10.3390/en15197002 Buana, Y., Mursitama, T. N., Abdinagoro, S. B., & Pradipto, Y. D. (2023). Stakeholder engagement by power system experts of Indonesia electricity sector for sustainable energy transition. International Journal of Energy Sector Management, 17(3), 474–488. https://doi.org/10.1108/IJESM-05-2021-0021 Calabuig-Moreno, R., Temes-Cordovez, R., & Orozco-Messana, J. (2022). Neighbourhood Digital Modelling of Energy Consumption for Carbon Footprint Assessment. In J. R. Littlewood, R. J. Howlett, & L. C. Jain (Eds.), Sustainability in Energy and Buildings 2021 (pp. 541–551). Springer Nature. https://doi.org/10.1007/978-981-16-6269-0_45 Caprari, G., Castelli, G., Montuori, M., Camardelli, M., & Malvezzi, R. (2022). Digital Twin for Urban Planning in the Green Deal Era: A State of the Art and Future Perspectives. Sustainability, 14(10), Article 10. https://doi.org/10.3390/su14106263 Cureton, P., & Hartley, E. (2023). City Information Models (CIMs) as precursors for Urban Digital Twins (UDTs): A case study of Lancaster. Frontiers in Built Environment, 9. https://doi.org/10.3389/fbuil.2023.1048510 Deng, M., Peng, X., & Zhao, Y. (2023). Comparative analysis of market structures of P2P energy trading in a local energy system. https://doi.org/10.18690/um.fov.6.2023.34 Dijksterhuis, A. (2005). Why We Are Social Animals: The High Road to Imitation as Social Glue. In From neuroscience to social science (Vol. 2, pp. 207–220). https://direct.mit.edu/books/edited-volume/2583/chapter/69742/Why-We-Are-Social- Animals-The-High-Road-to Dimitrova, E., & Tomov, S. (2021). Digital Twins: An Advanced technology for Railways Maintenance Transformation. 2021 13th Electrical Engineering Faculty Conference (BulEF), 1–5. https://doi.org/10.1109/BulEF53491.2021.9690822 do Amaral, J. V. S., dos Santos, C. H., Montevechi, J. A. B., & de Queiroz, A. R. (2023). Energy Digital Twin applications: A review. Renewable and Sustainable Energy Reviews, 188, 113891. https://doi.org/10.1016/j.rser.2023.113891 Dorenbos, R., Dural, S., & Gitema, S. (2020). Kleine gemeenten aan de slag met de energietransitie [Small municipalities working on the energy transition]. Platform 31. https://open.overheid.nl/documenten/ronl-a7d7f1cf-5549-41b1-91df-2a12a764d383/pdf Dulaimi, A., Hamida, R., Naser, M., & Mawed, M. (2022). DIGITAL TWIN SOLUTION IMPLEMENTED ON ENERGY HUB TO FOSTER SUSTAINABLE SMART ENERGY 494 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION CITY, CASE STUDY OF SUSTAINABLE SMART ENERGY HUB. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, X-4-W3-2022, 41–48. https://doi.org/10.5194/isprs-annals-X-4-W3-2022-41-2022 Fathy, Y., Jaber, M., & Nadeem, Z. (2021). Digital Twin-Driven Decision Making and Planning for Energy Consumption. Journal of Sensor and Actuator Networks, 10(2), Article 2. https://doi.org/10.3390/jsan10020037 Fouquet, R., & Hippe, R. (2022). Twin transitions of decarbonisation and digitalisation: A historical perspective on energy and information in European economies. Energy Research & Social Science, 91, 102736. https://doi.org/10.1016/j.erss.2022.102736 Ghenai, C., Husein, L. A., Al Nahlawi, M., Hamid, A. K., & Bettayeb, M. (2022). Recent trends of digital twin technologies in the energy sector: A comprehensive review. Sustainable Energy Technologies and Assessments, 54, 102837. https://doi.org/10.1016/j.seta.2022.102837 Government of the Netherlands. (2019a). Klimaatakkoord [Climate Agreement]. Ministry of Economic Affairs and Climate Policy. www.rijksoverheid.nl/documenten/rapporten/2019/06/28/klimaatakkoord Government of the Netherlands. (2019b). Nationaal Programma Regionale Energiestrategieën [National Programme Regional Energy Strategies]. Ministry of the Interior and Kingdom Relations. https://www.regionale- energiestrategie.nl/documenten/handlerdownloadfiles.ashx?idnv=1313654 Graf-Drasch, V., Kauffeld, L., Kempf, L., Oberländer, A. M., & Teuchert, A. (2023). Driving twin transformation—The interplay of digital transformation and sustainability transformation. ECIS 2023 Research Papers. https://aisel.aisnet.org/ecis2023_rp/255 Haveman, S. P., Westerhof, M. W., Reyes Garcia, J. R. R., Kroon, R., & Bonnema, G. M. (2020). ADAM & EV: Developing an Adoption Dynamics Analysis Model for Electric Vehicles. 33rd Electric Vehicle Symposium & Exposition 2020 Hoekstra, A., Steinbuch, M., & Verbong, G. (2017). Creating agent-based energy transition management models that can uncover profitable pathways to climate-change mitigation Clinging to Equilibrium. Energy and Complexity, 2017, 1–25. Hogeschool Utrecht. (n.d.). Sustainable and Social Local Energy Systems. Retrieved 8 March 2024, from https://www.internationalhu.com/research/projects/sustainable-and-social-local-energy-systems HosseiniHaghighi, S., de Uribarri, P. M. Á., Padsala, R., & Eicker, U. (2022). Characterizing and structuring urban GIS data for housing stock energy modelling and retrofitting. Energy and Buildings, 256, 111706. https://doi.org/10.1016/j.enbuild.2021.111706 Huang, W., Zhang, Y., & Zeng, W. (2022). Development and application of digital twin technology for integrated regional energy systems in smart cities. Sustainable Computing: Informatics and Systems, 36, 100781. https://doi.org/10.1016/j.suscom.2022.100781 Jafari, M., Kavousi-Fard, A., Chen, T., & Karimi, M. (2023). A Review on Digital Twin Technology in Smart Grid, Transportation System and Smart City: Chal enges and Future. IEEE Access, 11, 17471–17484. https://doi.org/10.1109/ACCESS.2023.3241588 Kourtit, K., Nijkamp, P., & Scholten, H. (2023). Planning for Climate–Benign Cities – Design of a Mind Map for Smart Energy Transition. Economic Themes, 61(1), 41–61. https://doi.org/10.2478/ethemes-2023-0003 Kürpick, C., Kühn, A., Olszewski, L., & Dumitrescu, R. (2023). Framework for Dual Transformation: A Systematic Literature Review on the Interplays between Digitalization and Sustainability. 2023 IEEE Conference on Technologies for Sustainability (SusTech), 175– 182. https://doi.org/10.1109/SusTech57309.2023.10129630 Kürpick, C., Rasor, A., Scholtysik, M., Kühn, A., Koldewey, C., & Dumitrescu, R. (2023). An Integrative View of the Transformations towards Sustainability and Digitalization: The Case for a Dual Transformation. Procedia CIRP, 119, 614–619. https://doi.org/10.1016/j.procir.2023.02.155 S. P. Haveman et al.: Towards Social Digital Twins – An Integrated Sociotechnical Approach for the Urban Energy Transition 495 Lehtola, V. V., Koeva, M., Elberink, S. O., Raposo, P., Virtanen, J.-P., Vahdatikhaki, F., & Borsci, S. (2022). Digital twin of a city: Review of technology serving city needs. International Journal of Applied Earth Observation and Geoinformation, 114, 102915. https://doi.org/10.1016/j.jag.2022.102915 Lesnyak, E., Belkot, T., Hurka, J., Hörding, J. P., Kuhlmann, L., Paulau, P., Schnabel, M., Schönfeldt, P., & Middelberg, J. (2023). Applied Digital Twin Concepts Contributing to Heat Transition in Building, Campus, Neighborhood, and Urban Scale. Big Data and Cognitive Computing, 7(3), Article 3. https://doi.org/10.3390/bdcc7030145 Li, B., & Tan, W. (2023). A novel framework for integrating solar renewable source into smart cities through digital twin simulations. Solar Energy, 262, 111869. https://doi.org/10.1016/j.solener.2023.111869 Li, C., Lu, P., Zhu, W., Zhu, H., & Zhang, X. (2023). Intel igent Monitoring Platform and Application for Building Energy Using Information Based on Digital Twin. Energies, 16(19), Article 19. https://doi.org/10.3390/en16196839 Martinelli, V. (2023). Energy transition: Digital (t)win? TeMA - Journal of Land Use, Mobility and Environment, 16(3), 653–659. https://doi.org/10.6093/1970-9870/10321 Mourtzis, D., Angelopoulos, J., & Panopoulos, N. (2022). Development of a PSS for Smart Grid Energy Distribution Optimization based on Digital Twin. Procedia CIRP, 107, 1138–1143. https://doi.org/10.1016/j.procir.2022.05.121 Municipality Leidschendam-Voorburg. (2021). Transitievisie Warmte [Transition Vision Heating]. https://projectenboek.lv.nl/assets/docs/Bijlage_2_Transitievisie_Warmte.pdf Nijkamp, P., Kourtit, K., Scholten, H., & Willemsen, E. (2023). Citizen Participation and Knowledge Support in Urban Public Energy Transition—A Quadruple Helix Perspective. Land, 12(2), Article 2. https://doi.org/10.3390/land12020395 Nochta, T., Wan, L., Schooling, J. M., & Parlikad, A. K. (2021). A Socio-Technical Perspective on Urban Analytics: The Case of City-Scale Digital Twins. Journal of Urban Technology, 28(1– 2), 263–287. https://doi.org/10.1080/10630732.2020.1798177 Pagani, M. (2021). The Role of Individuals in the City-Scale Energy Transition: An Agent-Based Assessment [Doctoral Thesis, ETH Zurich]. https://doi.org/10.3929/ethz-b-000503285 Palm, A. (2017). Peer effects in residential solar photovoltaics adoption—A mixed methods study of Swedish users. Energy Research & Social Science, 26, 1–10. https://doi.org/10.1016/j.erss.2017.01.008 Peng, X. (2023). Stakeholder Engagement for a Local Energy Community in Energy Transition– Case of De Heuvel/Amstelwijk. Year one report of project No. 482.20.608 Funded by Dutch Research Council (NWO). Piaia, E., & Frighi, V. (2022). Energy transition of urban districts. A viewpoint for the development of a decision support platform. TECHNE - Journal of Technology for Architecture and Environment, 127–133. https://doi.org/10.36253/techne-12117 Rovers, T., Struck, C., Gieling, C., Mohammadi, S., Slagmolen, O., Dorée, A., Scholtenhuis, L. O., Vink, K., Poppe, H., Koopman, D., Maat, H. ter, & Boeve, B. (2022). Development and evaluation of digital twins for district-level heating energy demand simulation. CLIMA 2022 Conference. https://doi.org/10.34641/clima.2022.399 RTL Nieuws. (2024, February 20). Verduurzamen van VvE’s voor 2050 lijkt onmogelijk door kosten en rompslomp. RTL Nieuws. https://www.rtlnieuws.nl/economie/artikel/5436068/verduurzamen-van-vves-verloopt- stroef Saroj, A. J., Roy, S., Guin, A., & Hunter, M. (2021). Development of a Connected Corridor RealTime Data-Driven Traffic Digital Twin Simulation Model. Journal of Transportation Engineering, Part A: Systems, 147(12), 04021096. https://doi.org/10.1061/JTEPBS.0000599 Savage, T., Akroyd, J., Mosbach, S., Hillman, M., Sielker, F., & Kraft, M. (2022). Universal Digital Twin – the impact of heat pumps on social inequality. Advances in Applied Energy, 5, 100079. https://doi.org/10.1016/j.adapen.2021.100079 496 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Selvakkumaran, S., & Ahlgren, E. O. (2019). Determining the factors of household energy transitions: A multi-domain study. Technology in Society, 57, 54–75. https://doi.org/10.1016/j.techsoc.2018.12.003 Sepasgozar, S. M. E. (2021). Differentiating Digital Twin from Digital Shadow: Elucidating a Paradigm Shift to Expedite a Smart, Sustainable Built Environment. Buildings, 11(4), Article 4. https://doi.org/10.3390/buildings11040151 Strielkowski, W., Rausser, G., & Kuzmin, E. (2022). Digital Revolution in the Energy Sector: Effects of Using Digital Twin Technology. In V. Kumar, J. Leng, V. Akberdina, & E. Kuzmin (Eds.), Digital Transformation in Industry (pp. 43–55). Springer International Publishing. https://doi.org/10.1007/978-3-030-94617-3_4 Tanveer, A., Zeng, S., Irfan, M., & Peng, R. (2021). Do Perceived Risk, Perception of Self-Efficacy, and Openness to Technology Matter for Solar PV Adoption? An Application of the Extended Theory of Planned Behavior. Energies, 14(16), Article 16. https://doi.org/10.3390/en14165008 Tsado, Y., Jogunola, O., Olatunji, Femi. O., & Adebisi, B. (2022). A Digital Twin Integrated Cyber-physical Systems for Community Energy Trading. 2022 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), 134–140. https://doi.org/10.1109/SmartGridComm52983.2022.9961012 United Nations. (2015). Paris Agreement. https://unfccc.int/process-and-meetings/the-paris- agreement Wan, L., Yang, T., & Parlikad, A. (2019). City-Level Digital Twin Experiment for Exploring the Impacts of Digital Transformation on Journeys to Work in the Cambridge Sub-region. https://www.repository.cam.ac.uk/handle/1810/296272 Wang, B., Cai, Q., & Sun, Z. (2020). Determinants of Willingness to Participate in Urban Incentive-Based Energy Demand-Side Response: An Empirical Micro-Data Analysis. Sustainability, 12(19), 8052. https://doi.org/10.3390/su12198052 Weil, C., Bibri, S. E., Longchamp, R., Golay, F., & Alahi, A. (2023). Urban Digital Twin Challenges: A Systematic Review and Perspectives for Sustainable Smart Cities. Sustainable Cities and Society, 99, 104862. https://doi.org/10.1016/j.scs.2023.104862 Wolske, K. S., Gil ingham, K. T., & Schultz, P. W. (2020). Peer influence on household energy behaviours. Nature Energy, 5(3), 202–212. https://doi.org/10.1038/s41560-019-0541-9 Yossef Ravid, B., & Aharon-Gutman, M. (2023). The Social Digital Twin:The Social Turn in the Field of Smart Cities. Environment and Planning B: Urban Analytics and City Science, 50(6), 1455–1470. https://doi.org/10.1177/23998083221137079 Zinsmeister, D., & Perić, V. (2022). Implementation of a Digital Twin of the CoSES District Heating Prosumer Laboratory | Energy Proceedings. 26(Closing Carbon Cycles – A Transformation Process Involving Technology, Economy, and Society: Part I). https://doi.org/10.46855/energy-proceedings-10153 THE FACTORS THAT INFLUENCE THE ADOPTION OF TOOLS THAT IMPROVE THE AGILE WAY OF WORKING AND CONTINUOUS IMPROVEMENT AT THE DUTCH CENTRAL GOVERNMENT RENE WIERINGA, PAUL MORSCH, PASCAL RAVESTEIJN, WOUTER BRONSGEEST University of Applied Sciences Utrecht, Utrecht, Netherland rene.wieringa@student.hu.nl, paul.morsch@hu.nl, pascal.ravesteijn@hu.nl, wouter.bronsgeest@hu.nl Due to the changing technological possibilities of services, the demands that society places on the level of service provided by the Dutch Central Government (DCG) are changing rapidly. To accommodate this, the Dutch government is improving its processes in such a way that they become more agile and are continuously improved. However, the DCG struggles with the implementation of improvement tools that can support this. The research described in this paper aims to deliver key factors that influence the adoption of tools that improve the agile way of working and continuous improvement at the DCG. Therefore, a literature review has been conducted, from which 24 factors have been derived. Subsequently, 9 semi structured interviews have been conducted to emphasize the perspective of employees at the DCG. In total, 7 key factors have been derived from the interviews. The interviewees consisted of both employees from departments who already worked with tools to improve agile Keywords: working and continuous improvement as wel as employees from agile, departments who haven’t used such tools yet. An important development, insight based on this research is that the aims, way of working continuous improvement, and scope of the improvement tools must be clear for al the adoption, involved co-workers. government DOI https://doi.org/10.18690/um.fov.4.2024.29 ISBN 978-961-286-871-0 498 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction With an increasing amount of new technology available organizations need to be more agile when implementing this in their processes (El-Dardiry & Overvest, 2019). To enable agile working new methods and tools are developed to help organizations. There are several methods to facilitate agility, like Lean (SixSigma, 2022), SAFe (SAFe, 2023) and Scrum (Scrum.org, 2023). Continuous improvement is one of the basic principles of SAFe (Scaled Agile, 2022). Government organizations within the Netherlands also need to become more agile (CIO Rijk, 2019-2020). For the Dutch Central Government (DCG), the ICT advisory board has setup several guidelines to address governance and implementation of agile working methods (CIO Rijk, 2021). The DCG has addressed these developments and translated them into the deployment of methodologies that can further facilitate digitization and an agile way of working (Digitale Overheid, 2019). In the most recent version of the DCG strategic agenda 2021 (CIO Rijk, 2019), various strategic objectives are addressed for the coming years, where flexibility and developing in smal steps are mentioned as important principles. This ensures faster results, less risk of major mistakes and room for adjustments where needed. Within the Dutch central government, several departments and executive bodies have (partial y) replaced waterfall methods for an agile way of working (CIO Rijk, 2021). Additional to agile working methods, there are improvement tools that can support in becoming and staying agile. Examples of improvement tools are the Agile Maturity Model (Patel en Ramachandran, 2009) and the 4-D Framework (Qumer en Henderson-Sellers, 2008). Within the DCG the directorate of information management of the Dutch Tax Authority (DTA) has developed its own improvement tool, tailor-made for the organization. The tool is cal ed CiBia and is based on the principles and dimensions of continuous improvement and an agile way of working. The tool is intended to test a team, a team of teams and/or an entire IT-development chain for maturity on dimensions regarding agility and continuous improvement. While many methods and improvement tools are available, adoption is stil lacking. Implementation and use in organizations encounters many chal enges that are often human related instead of technical (Gandomani and Nafchi, 2016). Miller (2013) identified several aspects that can act as barrier when implementing agile methods R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 499 and tools. First, communication is important to create a different mindset and culture. Second, management is often focused on daily operational problems instead of the need to change the way of working. Third, next to gaining management support it is also important to get employees and customers onboard. Fourth, adoption of an agile way of working doesn’t occur overnight, it is imperative that experience is al owed to grow over time. Dutch central government organizations are also struggling with the adoption of improvement tools, for example, at the Department of Infrastructure (Auditdienst Rijk, 2016). Similarly, CiBia is currently not widely adopted within the DTA. It is not clear due to what factors the adoption of CiBia is lacking. Therefore, based on the above the following main research question is formulated: Which factors influence the adoption of methods and tools that improve the agile way of working and continuous improvement by central government organizations? The remainder of this paper is organized as follows; section 2 presents an overview of relevant literature fol owed by the research approach in section 3. Section 4 discusses the results and section 5 provides the conclusions, limitations, and recommendations for future research. 2 Literature Review A systematic literature review has been conducted, according to the approach of Bell, Bryman and Harley (2022) with the goal to generate insights into factors previously found in research. Using the university’s search engine HUGO as wel as ResearchGate and Google scholar the following primary keywords were used to find relevant articles: adoption, [critical success] factors, assessment, improvement tool, maturity, and model in combination with the secondary terms agile, lean, [continuous] improvement, SAFe, scrum. Based on the above 57 papers were retrieved. Each paper was scanned to determine the relevance: 500 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • 19 papers described the specifications of improvement tools, self- assessments and/or maturity models and gave no interpretation on adoption or factors. • 7 papers described practices of Lean and/or agile methodologies and did not provide interpretation on adoption or factors. • 31 contained studies on adoption factors, of which 5 more were excluded because they did not fit the research direction, as described in Section 3. Final y, 26 papers were selected for the literature review of this study. Several studies have examined factors of adoption for new methodologies. A study by Fryer et al. (2007) found 13 factors, retrieved from 24 papers that focused on continuously changing organizations. The sponsorship and commitment of leadership, an environment with possibilities for employees to learn and develop, the involvement of employees in the continuous improvement process are some of the col ected factors from this research. The literature review of Rafi et al. (2022) contains 9 success factors from 69 studies. Ef ective communication, customer feedback, learning and development of employees are the three most important factors. The purpose of the change, leadership vision, stakeholder management and the involvement of employees are the essential factors mentioned in a study by Mohamad et al. (2022) and in a case study at a healthcare institution Rosa et al. (2021) concluded that the staff adoption increased after knowledge transfer sessions. Support of c-level management was also an important factor. The involvement of employees and teams, learning and development, and involvement in decisions were also important factors mentioned. Overall, the literature review performed for this study provided 24 factors. Table 1 shows each factor, how often it was mentioned, and whether it was determined to be an obstacle (O), an incentive (I) or (neither) (X). Factors L1-L5 were found to be the most important factors in literature. The criterion for this is that these factors have been mentioned more than 5 times in the literature as either incentives or obstacles. The references are detailed in appendix A. R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 501 Table 1: Factors related to the adoption of Agile methods Factor Factor description Total O I X Source code 4, 5, 9, 10, 11, Management 13, 17, 18, 19, L1 commitment / 17 12 5 20, 21, 22, 23, leadership 24 10, 11, 12, 18, Training & learning L2 9 7 1 1 19, 20, 21, 22, (team) 23 Commitment / 5, 6, 10, 11, 19, L3 empowerment of 8 7 1 20, 23 employees 9, 10, 12, 14, L4 Organization culture 8 5 2 1 17, 23, 24, 25 Collaboration (in value 3, 4, 6, 15, 17, L5 7 5 1 1 chain) 20 L6 Team formation fit 6 3 3 3, 17, 18, 19, 21, 24 Employee involvement L7 6 3 3 5, 7, 11, 13, 16, in process 18 (Un)wil ingness to 3, 8, 10, 13, 20 L8 6 2 2 2 change L9 Costs & resources 5 3 2 1, 2, 5, 15, 20 Effective process 5, 6, 7, 12, 19 L10 5 3 2 communication Experience/skills with L11 4 1 3 4, 20, 23, 24 model/tool L12 Vision/goal 3 1 2 2, 8, 12 Transparency in 1, 3 L13 3 2 1 method Establishing mindset L14 3 2 1 3, 9, 22 (for new method) L15 Organization structure 2 2 17, 19 L16 Purpose of method 2 2 6, 16 Quality of data & L17 2 2 19, 21 reporting 502 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Factor Factor description Total O I X Source code L18 Method (mis)fit 1 1 10 Method & process L19 1 1 19 integration L20 PDCA 1 1 5 L21 Problem-solving 1 1 11 Eagerness for new L22 1 1 25 tech/methods L23 Employee attention 1 1 6 L24 Customer satisfaction 1 1 19 103 64 27 12 3 Research Methods For this research, a qualitative approach has been used. First, a literature study was conducted as described above according to the methods as described by Bell, Bryman and Harley (2022). This resulted in a first list of factors that are related, either as success factor or barrier, to the adoption of methods and tools to enable agile working. Second, semi-structured interviews were conducted according to the ‘nine questions-method’ of Kvale (1996) and the use of ‘the final-question that does not fit in any of these categories’ by Treviño et al. (2014) Finally both lists of factors (from the literature study and from the interviews) are combined and compared. The sample size was not defined in advance. The researcher adopted the approach of Guest et al. (2006) and started the interviews and data col ection until data saturation occurred. With this approach, the maximum is determined based on possible saturation of information rather than an impossible estimate of the correct number in advance. The researcher had 14 respondents available for interviews and saturation of information occurred with respondent 10. The organizations in the DCG contributing to this study were selected based on participation in a national network that focuses on the exchange of continuous improvement methodologies and project management (De Gast, 2023). The DTA itself participates in this network. From that network, respondents were selected on a voluntary basis, who were contacted by the researcher for participation in this R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 503 study. The criteria used for the interview selection is that the respondents have knowledge of project-based working, continuous improvement and/or continuous improvement processes. Experience with agile working was not specifical y included as a criterion, as not al organizations within the selection have experience with Agile working or have been through a transition to agile working. The same goes for the criteria of whether the organization has experience with improvement tools; not every organization and/or respondent has specific experience with improvement tools. Final y, we only selected participants (as shown in table 2) that worked for organizations that are part of the DCG. Table 2: Overview interviewees Level of #years #years experience Currently Participant experience Age in with using with Agile category current role improvement tools? tools DCG 1a 2 50-55 7 DCG 1b1 1 50-55 10 Novice Limited use DCG 2 20 50-55 10 Expert Actively using DCG 3 20 46-50 5 Expert Actively using DCG 4 2 40-45 1,5 Novice Not using DCG 5 2 40-45 1 Novice Not using DCG 6 12 56-60 4 Expert Actively using DTA 1 15 40-45 6 Expert Actively using DTA 2 15 50-55 2,5 Expert Actively using DTA 3 10 40-45 5 Expert Actively using Following the approach of Bel, Bryman and Harley (2022) the interview analysis process consists of 5 steps: • Step 1: Interviews: conducting semi-structured interviews with the respondents. The interview started with discussing the topic and the five most important factors found in literature. 1 During this interview there were two participants present thus the outcome is analyzed as one interview 504 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • Step 2: Transcribe the data: the recorded interviews are transcribed so they can be compared and analyzed. • Step 3: Coding: This step is fol owed to code the transcribed interview text according to the first and second cycle (Saldaña, 2015) 1. First coding step: descriptive coding. Quotations from the transcribed interview text are linked to a provisional code. The codes are linked to a factor. Figure 1 shows the schematic relationship between quotations, codes and factors. Figure 1: the schematic relationship between quotations, codes and factors Source: Own 2. Second coding step: refine, discover patterns. The quotations are analyzed and checked again. The coding is refined and standardized across the transcribed interview texts. • Step 4: Analyze: for the analysis of the interviews, in addition to the detailed coding in step 3, the method of (Ryan & Bernard, 2003) is used to detect and define themes from the interviews, for example by paying attention to repetitions, metaphors, transitions from one to the other topic. In addition to the analysis of the interviews, the following data required for comparing the factors is stored: 1. The factor code and description of the factor. 2. The code/coding; this concerns the code that the researcher uses to indicate the connection between the underlying quotations. 3. The quotations; this concerns the quotes from the interview to the test. • Step 5: Tests: the elaboration of the results is tested with al respondents. The feedback from the respondents wil be processed. After testing by the respondents and processing the feedback. R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 505 4 Results In this section the outcomes of the analysis of the interviews are described and subsequently this is compared to the factors that were derived from the literature study. As shown in table 2 there were 10 participants in the 9 interviews that were held: 3 persons worked for the DTA and 7 persons were employed by other organizations of the DCG. All DTA employees and the majority of the other participants had experience in using improvement tools. During the interviews the extent to which improvement tools are used by the participants was determined as wel as within which organizations they are active, and which improvement tools were used specifically. Table 3 lists the improvement tools mentioned by the participants, including the number of times these tools were mentioned during the interviews. Based on this it was determined that none of the organizations studied used an improvement tool unambiguously (based on explicit strategic or policy choices). Table 3: overview of methods & tools in use in organizations Target group Number of improvement tools Specification improvement in use tools Jira (4) PDCA (1) DCG 4 Obeya (1) SAFe Maturity Assessments (1) Jira (3) CIBIA (3) Kanban (1) DTA 5 Obeya (1) SAFe Maturity Assessments (1) JIRA is a software package from Atlassian (Atlas.ti, 2023). The software incorporates several tools for improvement, such as KANBAN boards. CIBIA is the improvement tool of the DTA and chal enges in adopting this tool was one of the reasons this study was conducted (Bronsgeest & Hofman, 2023). Obeya is an 506 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION improvement tool supporting Lean principles, where the term visual management room is also often used (Aasland & Blankenburg, 2012). Kanban is a methodology for structuring processes around software development delivery (Ahmad & Oivo, 2013). Withing the SAFe methods there are maturity models that can be used as tool, such as the Agile Maturity Assessment (ScaledAgile.com, 2023). PDCA stands for plan-do-check -act (Respondent RO1, 2023) and on this method a self-developed process and report is in use in the organization of one of the participants. From the semi-structured interviews, it is established that two different groups can be defined. First, a group of participants who work within an organization that has not yet undergone an agile transition and where there is virtually no or little experience with the use of improvement tools. Second, a group of participants who work at an organization that has already undergone an agile transition or is currently in transition and where there is ample experience with the use of improvement tools. Table 4: overview of factors at DCG and DTA Code Factor Description DCG DTA Total The usefulness and necessity of metrics of F1 12 7 19 improvement tools Adopt standards in improvement tooling and F2 7 12 19 practices The impact of culture and governance on the F3 10 7 17 use of improvement tools F4 Fit for purpose of the improvement tool 8 9 17 F5 Structure of the organization 8 0 8 Training and empowerment for use of F6 7 0 7 improvement tools The role of leadership in using improvement F7 3 3 6 tools Bottom-up approach to using improvement F8 3 3 6 tools F9 Connection to organizational goals 0 4 4 F10 The need for using an improvement tool 3 0 3 Total 61 45 106 R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 507 As is shown in table 4 there are 10 unique factors derived from the interviews. The table shows the number of times each factor was mentioned by participants working for respectively the DTA or other organizations within the DCG. Furthermore, the first seven factors were found the most important based on the criteria that a factor was mentioned at least 10 times across al interviews or were specifical y mentioned as either obstacle or incentive more than 5 times. The interview analysis identified what factors were cited as obstacle or incentive related to the adoption and use of an improvement tool for agile working and continuous improvement. Table 5 shows the number of quotations we found, where column X shows the number of times a factor was discussed but in a neutral manner. Table 5: Quotations mentioning factors as obstacle, incentive or neutral Code Factor Description O I X The usefulness and necessity of metrics of F1 22 9 4 improvement tools Adopt standards in improvement tooling and F2 4 6 2 practices The impact of culture and governance on the F3 17 3 0 use of improvement tools F4 Fit for purpose of the improvement tool 20 8 3 F5 Structure of the organization 0 13 0 Training and empowerment for use of F6 0 8 0 improvement tools The role of leadership in using improvement F7 12 5 3 tools Bottom-up approach to using improvement F8 2 5 0 tools F9 Connection to organizational goals 2 1 2 F10 The need for using an improvement tool 0 3 0 Based on the analysis it seems that factors F1, F3 and F4 are mostly seen as obstacles. Regarding the usefulness and necessity of metrics of improvement tools (F1) interviewees stated for example that “Improvement tools are used for scoring and not for improving” 2 and “We suf er from judgment in the scores of improvement tools” . However, this 2 All quotations are translated from Dutch 508 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION factor can also work as an incentive because “It helps if we can distinguish between the metrics of the line and the chain” . The impact of culture and governance on the use of improvement tools (F3) is often found to be an obstacle as the following statements make clear: “We are dealing with a culture of tempering and patronizing, which means nothing changes”, “We have a culture in which we continuously want to launch new improvement plans” and “There is a fear woven into the culture to adjust standards and thus maintain potential for improvement”. Finally looking at the fit for purpose of the improvement tool (F4) it was found that although it could help when it is “in line with the organization's objectives” more often it is found to be an hinderance as “There is no fol ow-up from the coaches once we have done a CiBia scan” and “Our method of implementing improvements is too separate from CiBia”. The most contributing factor regarding the adoption of tools seems to be (F5) Structure of the organization as “You want to create the same mindset together”, “Empowering employees is essential” and it should enable to “Share knowledge with the same focus” . Comparisons Table 6: Comparison of factors - literature versus interviews Factor Description Code Code interviews literature The usefulness and necessity of metrics of F1 improvement tools Adopt standards in improvement tooling and F2 practices The impact of culture and governance on the use of F3 L4 improvement tools Fit for purpose of the improvement tool F4 L17 Structure of the organization F5 L16 Training and empowerment for use of improvement F6 L2/L3 tools The role of leadership in using improvement tools F7 L1 R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 509 In table 6 we compare the top 7 factors derived from the interviews to those found in literature, the majority overlaps although there are two new factors (F1 and F2). Besides comparing the findings from the interview versus the literature we can also compare the two groups that are defined: (1) the participants whose organizations have not yet gone through an agile transition and where there is little or no experience with improvement tools and (2) those whose organization has gone through an agile transition or is in transition and where there is ample experience with the use of improvement tools. Table 7 shows the comparison between these groups. Table 7: Group comparison Group Factors # Quotations Training & empowerment (L2, L3, F6) 13 1 Structure of the organization (F5) 8 Fit for purpose of the improvement tool (F4) 25 The usefulness and necessity of metrics of improvement tools (F1) 29 Adopt standards in improvement tooling and 2 practices (F2) 16 The role of leadership in using improvement tools (L1, F7) 16 Culture & governance (L4, F3) 11 Based on the comparison we can see that the group that has not gone through an agile transition rates factors as the structure of the organization, receiving training and being empowered as being the most important when implementing and adopting tools to enable an agile way of working. For those that have already experienced such a transition, factors related to the specific tool to be used such as its fit for purpose, the adopted standards and usefulness of related metrics are becoming important. Even though this means that the group in which the organization finds itself influences which factors are deemed more relevant this does not mean that a factor relevant to group 1 is not mentioned by group 2 and vice versa. 510 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 5 Conclusions In this study, we investigated the factors influencing the adoption of methods and tools aimed at improving agile ways of working and continuous improvement within government organizations, with a specific focus on the Dutch Central Government. Our analysis encompassed a synthesis of literature and presents empirical data gathered through semi-structured interviews. The findings reveal that both organizational readiness and tool-specific considerations play an important role. It is evident that while many methods and tools are available, their adoption within government entities, including the DCG, faces substantial challenges, largely attributed to human factors rather than technical impediments. Our study identified several factors that influence adoption, with a notable emphasis on the structure of the organization, the fit-for-purpose nature of the improvement tool, and the impact of organizational culture and governance. Importantly, factors such as training, empowerment, and alignment with organizational objectives emerged as critical determinants of adoption success. Comparisons between organizations that have undergone agile transitions and those that have not underscore the differential importance placed on various factors, highlighting the dynamic nature of adoption dynamics within differing organizational contexts. Furthermore, our findings suggest that organizations should use a holistic approach encompassing both top-down structural changes and bottom-up empowerment initiatives as both are essential for fostering a conducive environment for the adoption of agile methodologies and improvement tools. Overall, this study contributes to the growing body of knowledge surrounding agile transformation within government organizations, offering insights into the multifaceted nature of adoption processes, and providing valuable guidance for practitioners and policymakers alike. However, it is essential to acknowledge the limitations of this study. First and foremost, only 10 persons were interviewed to cover various organizations of the DCG. While this provided a first insight into the research topic more data needs to be col ected to generalize the findings. Furthermore, not al participants to this study (and their organizations) have a clear understanding of what is understood by agile working and what it means to try to continuously improve working processes. Even though questions were asked to get a better understanding of the context within the organization of the interviewee, we R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 511 should be careful in comparing the outcomes as the perceptions of the participants might not fully fit with reality. Future research could expand upon these findings by conducting a quantitative study, with specific questions about adoption factors. Within this research, the factors were studied within the Dutch Central Government. However, future research outside the DCG can provide an additional view on the factors found in this study. References Aasland & Blankenburg. (2012). An analysis of the uses and properties of the Obeya. Engineering, Technology and Innovation (ICE), 2012, (p. 9). Fonte: 2023: https://leansixsigmagroep.nl/lean-agile-en-six-sigma/obeya/ Ahmad & Oivo. (2013). Kanban in software development: A systematic literature review. Software Engineering and Advanced Applications (SEAA), (p. 9). Oulo, Finland. Atlas.ti. (Mei de 2023). A smarter way to better research findings. Fonte: Atlas.ti - Scientific Software Development GmbH: https://atlasti.com/ Atlassian. (23 de 10 de 2023). JIRA Software. Fonte: Atlassian.com: https://www.atlassian.com/software/jira Auditdienst Rijk. (2016). Research on information security governance at the Ministry of Infrastructure and the Environment. The Hague, NL: CIO Infrastructure & Environment. Bel , Bryman & Harley. (2022). Business Research Methods, 6th edition. . Oxford, UK: Oxford University Press. Bell, Bryman, & Harley. (2022). Qualitative Data Analysis; More On Coding. Em B. a. Bell, Business Research Methods, 6th edition (p. 542). Oxford: Oxford Press. Bel , Bryman, & Harley. (2022). Reviewing The Literature. Em B. &. Bel , Business Research Methods (pp. 91-110). Oxford: Oxford University Press. Bronsgeest, W., & Hofman, J. (23 de 2 de 2023). Kick-off CiBia at the DTA. (Wieringa, Entrevistador) Carvalho. (2023). Continuos Improvement in organizations. Contonuous Improvement. Guimaeres, Portugal: River Publisher. CIO Rijk. (2019-2020). Strategic I-agenda Rijksdienst. The Hague, NL: Ministry of the Interior and Kingdom Relations. CIO Rijk. (2021). Handreiking Verantwoord Vernieuwen (Guide to Responsible Renewal). The Hague, NL: Ministry of Interior and Kingdom Relations. Digitale Overheid. (23 April 2019). Hoe Lean, Agile en Design Thinking elkaar versterken. DigitaleOverheid.nl: https://www.digitaleoverheid.nl/nieuws/hoe-lean-agile-en-design- thinking-elkaar-versterken/ El-Dardiry & Overvest. (2019). Position Paper Digitization. Den Haag: CPB. Fryer et al. (2007). Critical Success Factors of Continuous Improvement in the Public Sector: a review of literature and some key findings. Guest, Bunce, & Johnson. (2006). How many interviews are enough? An Experiment with Data Saturation and Variability. Em Field Methods 18 (pp. 59-82). Hofman, J. (15 de October de 2022). Interview Jos Hofman DTA. (Wieringa, Entrevistador) Linneberg & Korsgaard. (2019). Coding Qualitative Data: a synthesis guiding the novice. Emerald Insight, 259-270. 512 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Mohamad et al. (2022). Change Management Approaches for Successful Lean Adoption. Malaysia: Universiti Teknikal Malaysia Melaka. Patel en Ramachandran. (2009). Agile Maturity Model (AMM): A Software Process Improvement framework for Agile Software Development Practices. International Journal of Software Engineering, pp. 3-28. Qumer en Henderson-Sellers. (2008). An evaluation of the degree of agility in six agile methods and its applicability for method engineering. Information and Software Technology, pp. 280-295. Rafi et al. (2022). Decision-Making Taxonomy of DevOps Success Factors Using Preference Ranking Organization Method of Enrichment Evaluation. Mathematical Problems in Engineering, pp. Volume 2022, Article ID 2600160. Respondent RO1. (4 de July de 2023). Interview improvement tools. (R. Wieringa, Entrevistador) Rosa et al. (2021). Lean adoption in hospitals: the role of contextual factors and introduction strategy. BMC Health Services Research. Ryan & Bernard. (2003). Techniques to identify themes. Field Methods, 15, 85-109. SAFe. (December de 2023). SAFe. Fonte: Scaled Agile Framework: https://scaledagileframework.com/ Saldaña. (2015). The Coding Manual for Qualitative Researchers,. Thousand Oaks, CA: Sage. Scaled Agile. (2022). Business Agility. Fonte: ScaledAgileFramework.com: https://scaledagile.com/business-solutions/business-agility/ Scaled Agile. (11 de Oktober de 2022). SAFe Lean-Agile Principles. Fonte: ScaledAgileFramework.com: https://scaledagileframework.com/safe-lean-agile-principles/ Scaled Agile, Inc. (27 de Mei de 2023). Lean-Agile Leadership. Fonte: SAFe: https://scaledagileframework.com/lean-agile-leadership/ ScaledAgile.com. (23 de 10 de 2023). Measuring Agile Maturity with Assessments. Fonte: Scaledagile.com: https://scaledagile.com/blog/measuring-agile-maturity-with-assessments/ Scrum.org. (13 de December de 2023). What is Scrum? . Fonte: Scrum.org - The home of Scrum: https://www.scrum.org/resources/what-scrum-module SixSigma. (2022). An agile organization; the eight principles of an agile organization. Fonte: SixSigma.nl: https://www.sixsigma.nl/wendbare-organisatie-de-8-principes R. Wieringa et al.: The Factors That Influence the Adoption of Tools That Improve the Agile Way of Working and Continuous Improvement at the Dutch Central Government 513 Appendix A – References used in table 1 1. Gustavsson & Bergkvist, 2019 2. Salikhov et al, 2020 3. Putta et al, 2021 4. Maulana & Raharjo, 2021 5. Rosa et al, 2021 6. Dutra & Santos, 2020 7. Trzeciak & Banasik , 2022 8. Mohamad et al, 2022 9. Kulathunga & Ratiyala, 2018 10. McDermott & Anthony, 2022 11. Kaltenbrunner et al, 2017 12. Rafi et al, 2022 13. Benjamin & Chung, 2022 14. Rajapaske et al , 2022 15. Cater-Steel et al, 2006 16. Shankarani et al, 2015 17. Orantes-Jimenez et al, 2022 18. Unzueta et al, 2019 19. Fryer et al, 2007 20. De Jager et al, 2004 21. Jaca et al, 2012 22. Soliman, 2017 23. Mahanti, 2006 24. Papatheoucharous, 2014 25. Govender & Pretorius, 2014 514 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION SMART CITIES ON THE WATERFRONT: CITIZENS’ PERSPECTIVES ON SMART CITY IN THE CONTEXT OF A PORT CITY KRISTEL EDELMAN,1 RIIKKA FRANZEN,1 IRINA WAHLSTRÖM,2 ANASTASIA TSVETKOVA,2 YIRAN CHEN2 1 University of Turku, Centre for Collaborative Research, Turku, Finland kristel.edelman@utu.fi, riikka.franzen@utu.fi 2 Åbo Akademi University, Laboratory of Industrial Management, Turku, Finland irina.wahlstrom@abo.fi, anastasia.tsvetkova@abo.fi, yiran.chen@abo.fi This paper explores how smart city is defined by citizens in the context of a port city, and which aspects are prominent due to the significant role of a ferry port in such cities. Through a case study of a port city comprising a survey of its citizens, this research investigates citizens’ perceptions of a smart city, contextualised in the actual use of the area close to the port and expectations towards the development of this area. Utilizing thematic analysis of survey data, key themes identified include traffic, environmental sustainability, technology and K eywords: digitalization, and the importance of focusing on people, their smart city, wellbeing and quality of life. The findings emphasise the citizens’ citizens, wish for prioritising environmental sustainability and wellbeing digitalisation, in the development of the smart city in a context of a port city port city, and show that improvements in traffic around the port area and ferry urban planning are seen as most topical. port DOI https://doi.org/10.18690/um.fov.4.2024.30 ISBN 978-961-286-871-0 516 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction In the pursuit of sustainable, economic, and spatial co-existence with cities, an increasing number of ports have had to adjust, rearrange, and optimize their operations and infrastructure as a response to urbanization, expanding residential areas and other alternative land use needs by their seafronts. Previous studies suggest that through urban development and renewal projects, port areas become more physical y connected to the city, with the potential for the emergence of new urban spaces (Bruns-Berentelg et al, 2022; Carta & Rosnivalle, 2016; Hermelin & Jonsson, 2021; Nowacka-Rejzner, 2015). Therefore, the relationship between ports and cities is changing. Ports are not anymore viewed as economic entities only, but their role is becoming increasingly integrated into the residential functions of urban areas (Van den Berghe & Daamen, 2020). The port city case we are focusing on is experiencing the same kind of development pressures as many other old port cities: the lucrative seafront area would rather be seen in recreational or residential use, the traffic to and from the port is considered a problem for the city, the environmental concerns and demands are becoming stricter. In this paper we draw from the literature on smart cities and explore what kind of specific characteristics it entails when the context is a port city. The survey results discussed in a paper focusing on citizens are part of our ongoing study Smart Port City, which is inspired by recent research on the digital integration of cities and ports, emphasising the role of various actors in a smart city-port ecosystem (Beškovnik & Bajec, 2021). In addition to other actors we have concentrated on in our research, we consider citizens as one of the key actors in a smart integration of a city and a port1. Through a case study of a port city comprising a survey of its citizens, this research investigates citizens’ perceptions of a smart city, contextualised in the actual use of the area close to the port and expectations towards the development of this area. Our research question is as follows: how a smart city is perceived by citizens in the context of 1 The overall study includes interviews with representatives of the city, the port and the tourism sector followed by the survey analysed for the current paper. Our research continues with workshops with the representatives of key actors we have identified in a context of a smart port city, including citizens. K. Edelman et al.: Smart Cities on the Waterfront: Citizens’ Perspectives on Smart City in the Context of a Port City 517 a port city? The results of the study are structured based on focal themes as an outcome of the thematic analysis utilizing NVivo software. 2 Theoretical background 2.1 Smart City in a context of Port City Cities and ports have been “connected systems” with the path of “independent lives and developments” (Beškovnik & Bajec, 2021, p. 433). However, several old port cities have undergone a transformation process that has enabled the city and the port to continue their symbiotic co-existence. Similar changes are occurring in a port city, a case in our study, having a long history of being a port city with the recent developments in relation to smart city and smart port. In the current paper, we aim to bring the literature on smart city to the context of port city for building the framework for the analysis. The research on smart cities has general y adopted the understanding of smart city as “urban planning and administration project of embedding digital technologies into the urban fabric, and a reconfiguration of digital urban economies” (Burns et al., 2021 p. 463 on Kitchin et al., 2015; also Kitchin, 2014). Innovative uses of technology enable the optimisation of resources, support the enhancement of effective and fair governance, enable new forms of collaboration and contribute to sustainability and quality of life of city inhabitants (Gretzel et al., 2015; Ahvenniemi et al., 2017). In a context of physical infrastructure, the aim of the smart city is to blur the lines between physical and digital by contributing into the integration of technology (Gretzel et al., 2015). To contribute to the understanding of the digital side of the smart city Szpak et al (2024) make a distinction between the concepts of “smart city” and “digital city”. While smart city research emphasises the development of its sustainable environment, this aspect has not been the domain of interest of the digital city (Szpak et al., 2024, p. 5) mainly because the smart city literature has been evolving from the digital to the direction of sustainable cities (Zheng et al., 2020). From the perspective of information and communication technologies, cities “urbanize technologies” by testing new solutions and modify them accordingly (Szpak et al., p. 5 on Sassen, 2015). 518 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION By enabling the “networking of people, organisations, and infrastructures so that social, ecological, or economic added value is created and implemented in the city life” (Szpak et al., 2024, p. 5), the primary aim of the smart city is to deploy innovative technologies for the enhancement of its sustainable development (reducing greenhouse gas emissions as one of the examples, Ahvenniemi et al., 2017). For the conceptualisation of the smart city, the following dimensions might be brought out: (after Cohen, 2014 smart city wheel and Al Sharif & Pokharel, 2022; Moura & Silva, 2019; Qonita & Giyarsih, 2023): smart environment - smart buildings, resources management, sustainable urban planning; smart mobility - intelligent transport system and parking solutions, mobility-as-a-service; smart people - inclusion, education, creativity; smart living - healthy, safe and culturally vibrant; smart economy - entrepreneurship and innovation, productivity, local and global connection, and smart governance - contribution to decision-making, transparency of governance, public services and open data. With supporting sustainability, efficiency, and citizen participation in urban spaces (Townsend, 2013) as positive aspects, the concept of smart cities has been criticised due to its normative understanding of how the future cities should be, emphasising the importance of the flow of capital, human resources (Burns et al., 2021; Burns & Andrucki, 2021) and the neoliberal ideology (Grossi & Pianezzi, 2017). While one of the aims of smart city is to enhance citizens’ participation (Townsend, 2013), the research lacks such bottom-up approaches (Caragliu et al., 2019) as the studies on technological developments have gained too much prominence (Cardullo & Kitchin, 2019; Mosco, 2019). Ports cities are mainly characterised by their spatial relation with the water (Hein, 2016; Özgece et al., 2022). Located next to the sea, the water becomes a crucial element for the construction of the port city identity (Özgece et al., 2022), alongside its long history and the ongoing urban transformation process as in a context of a port city case presented in our paper. At the same time various port cities have a chal enge for dealing with traffic congestions and air pollution (Lehmacher et al., 2021). These chal enges are supported by smart solutions, such as different types of sensors and other means to digital y increase the situational awareness in port, as wel as to track and report the emissions from the port (see also Tsvetkova et al., 2021) to contribute into the smooth co-existence of the port and the city. The latter is crucial to consider as otherwise the outcome might be port and city separation in K. Edelman et al.: Smart Cities on the Waterfront: Citizens’ Perspectives on Smart City in the Context of a Port City 519 its “economic, spatial and cultural development” as it has happened in a case of Rotterdam (Hein & van der Laar, 2020, p. 265). On the one hand we emphasise the integration of physical and digital infrastructure from the side of the city and the port, that is supported by the new technologies. On the other hand, by incorporating the citizens’ perspectives on the port city, the approach broadens from infrastructure to the overal urban context where the urban developments of the case study area have a great potential for the recreational purposes as wel . Our research aligns with the recent approaches for digital infrastructure developments in a port city integration, which are understood from the framework of ecosystem, that enables to build a novel approach for digital integration of port and city when adjusting with increased urbanisation, mobility, and business developments (Beškovnik & Bajec, 2021). In this paper we consider citizens as one of the key actors in a port city ecosystem from the perspective of its smart development. 3 Methodology 3.1 Data col ection The study we present in the current paper is part of a single case study (Yin, 2003; 2009) of a port city located in Northern Europe. The broader aim of our research is to investigate how to balance the needs of the port and the city in its physical transformation and how digitalisation could support it. The expansion of the urban public space at the area under study, with the mix of residential and office spaces, will connect the city centre with the sea. The development process has started with the construction of the new passenger terminal and aims to bring more art, culture, and urban greening into the area, at the same time emphasising the history and preserving the current built environment with low scales of new constructions. The main underlying objective for developing the area is the Turku Climate Plan, as city of Turku was selected as one of the 100 cities in Europe to become climate-neutral by the end of 2029. In this paper we concentrate on citizens’ (local residents, recreational users, visitors, and ferry passengers) perceptions of a smart city based on a col ected data with online survey. Despite of the several attempts the city has made, the urban 520 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION development plans of this area have received little attention from the citizens. Therefore, an additional aim of comprising the survey was to raise awareness of the future developments of this specific area amongst the city inhabitants2. The survey was designed to understand the citizen’s perceptions of the smart city by studying the reasons for visiting this specific area and the respondents’ usage of digital applications and technologies in relation to their engagements with the evolving port city interface (for the survey questions please see the Appendix). Most of the questions were open-ended to al ow citizens to express their views and opinions in their own words and without time limitation to answer the questions. Initiated in October 2023, this survey remained accessible for two and a half months. To maximize inclusivity and reach, the survey was made available in (two) native languages of the targeted demographic and in English. The survey was promoted through the project’s dedicated web page, various communication platforms, and local media, culminating in 159 responses. Of the respondents 88% (139) were representing residents of the entire port city (in distinction from the port and city interface area studied), 8% (13) residents of peripheral municipalities and 4% (7) of other domiciles. Diverse digital tools were used for respondents to get aware about situation around port area. Even though 14% of respondents did not use digital tools, the rest of them mentioned using numerous digital tools. 50% of the respondents use urban public traffic applications, other mobility applications were also mentioned, such as bike, scooter, or ride hailing applications. Ship-travelling is another main reason for people to visit the port area, therefore, shipping companies’ website and applications are relevant information sources. Touristic attractions (museums), local businesses and events are also important information sources for respondents. 2 In a first phase of our study, while interviewing the representatives of the city in addition to the representatives of the port, we became aware of the challenge to reach the citizens when the question is about the development of this specific area we are concentrating on. This supported our choice of comprising the online survey, although we acknowledge the in-depth perspective the interviews would have enabled to provide. We continue this research in our workshops with citizens, that enable to go more deeper into the themes emerged from the analysis of the online survey. K. Edelman et al.: Smart Cities on the Waterfront: Citizens’ Perspectives on Smart City in the Context of a Port City 521 3.2 Data analysis In our analysis, we applied content analysis in its more qualitative form (Drisko & Maschi, 2016; Krippendorff, 2018; Schreier et al, 2019), meaning that we did not employ predeveloped categories, but they were data-based. Our analysis resembles thematic analysis (Lochmiller, 2021; Rapley, 2021), but also has characteristics of content analysis with calculating frequencies of themes mentioned in the data. The analysis was performed using NVivo software (Version 14) and initiated open coding of the survey answers, simultaneously developing data-based codes and subcodes. During and after the coding, the codes were organised by moving, merging, grouping, hence creating meaningful themes. The frequencies of mentioning a theme were calculated to enable identification of the most often mentioned themes. The themes structured for the analysis are supported by the quotes extracted from the answers to the open-ended questions of the survey. 4 Results 4.1 Reasons for visiting or using the area of the port city The 159 survey responses received on what currently and before the initiation of the urban transformation process attracts people or why people visit this specific area starting from the centre and reaching to the harbour and the sea, generated altogether 425 mentions fal ing under 13 main themes. These mentions and their frequencies are listed in the Table 1 below. As demonstrated by the mention distribution and frequency, the area’s cultural and service offering, historical value, geographic versatility, and verdancy provide several reasons for spending time or visiting the neighbourhood. The significance and exploitation of the area’s functionalities may be emphasized or differ to a certain extent between different respondent groups (residents, visitors, tourists, and ferry passengers) and by the distance of the place of domicile to the area in question. Among the neighbourhood residents, the opportunities for daily outdoor activities (walking, jogging, exercising), participation in events and the utilization of the area’s service offering (restaurants, cafés) in beautiful settings, are emphasized among the themes. 522 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION “I go there for walking the dog, strols, events, restaurants or just to soak in the atmosphere. When I want to refresh myself and see new people but stil be in my own peace, I go to the Castle environment. Yet, it feels like that area could have more potential. A bit like Sleeping Beauty’s slumber.” (R3 7) Table 1: Reasons for visiting and their frequencies Reason for visiting Examples Frequency physical activity, recreation, spend Outdoor activities, wel -being & time to find a tranquil hideaway, 85 leisure observe and meet other people travelling with ferries, leisurely visits, Tourism and travelling 50 hotel stays Attractions, historical settings, visits to historical medieval Castle 50 and nature values parks, verdancy, river, riverbank, Nature values 61 sea, uncrowdedness, tranquil ity Museums 35 Eating and drinking restaurants, cafés, pubs 33 Events and exhibitions 31 Port activities and environment observe ships, visit terminal facilities 20 Transport, mobility, and cycling and pedestrian routes, water 15 accessibility bus, public transportation educational institutions, errand Other 14 running, parking Residency within the living 13 neighbourhood workplace, meeting and conferences Work 12 attendance Hobbies photography, guided exercise 7 For city residents living further away and for people arriving from peripheral municipalities/other domiciles (as given in section 3.1), the outdoor and recreational activities are more accentuated to weekends, holidays, and/or summer season. For these groups of respondents, easy accessibility, good mobility, and transport options are highlighted in connection with the visits to the area. “I often cycle to the Castle area 3 R stands for Respondent K. Edelman et al.: Smart Cities on the Waterfront: Citizens’ Perspectives on Smart City in the Context of a Port City 523 and back to the city centre along the riverbank.” (R 33). Easy accessibility and transit to and from the ferry port, whether travel ing by car or public transportation, is of essence for those travelling by ferries, whether city residents, visitors, or tourists. “Smooth access to ships at the port, walking routes along the port and riverbank, and events” (R 22). For tourists and visitors, the most recurring themes encompassed the attractions, museums, and nature values. “When I have friends visiting from other places I take them to the castle” (R 139). Inquiring into the use of the focal area by the citizens al ows to better interpret the viewpoints on smart city presented in the following section. 4.2 Citizens’ view on smart city The citizen’s view on smart city is constructed by combining the answers to questions “What makes a city “smart” in your opinion?” and “What aspects would be important for you in the development/transformation process of the Linnanniemi?”. The most often mentioned themes are presented in Figure 1. Figure 1: Main themes of citizens’ view on the waterfront4 Source: Own Traffic was the most often mentioned theme among the answers (121, including green mobility). The respondents saw that the traffic of a smart city should be easy for people, safe, fluent, environmental y friendly, with functioning parking 4 The size of a bubble indicates how often a topic was mentioned; the overlaps among circles are not indicative of topic overlaps. 524 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION arrangements, easy connections with public transport (bus, train, tram, ferry) and new modes of transport. Utilizing technology and digital solutions were often mentioned related to traffic, for example when planning a trip, in public transport, in managing the traffic flow to and from the port or using AI in managing traffic lights. Concern for environmental sustainability was broadly present in the answers (103 mentions). Additional y, if we include the answers concerning green mobility, which are currently considered under the traffic theme, but also contribute to environmental sustainability, it will make this theme even more prominent. “A city is especial y climate smart when cars are not the first priority, when it comes to forms of mobility. In a smart city, it is extremely important that the main focus is on green and climate friendly alternatives such as col ective transport and of course walking and cycling. . Climate and environment should always be the first things to consider in al infrastructural planning. That makes a smart city.” (R 6) Various aspects of environmental sustainability were brought up, for example applying sustainable development and living (e.g. recycling), being climate friendly, adapting to climate change, and using renewable energy. Many respondents wished for better abilities to enjoy the nature and the waterfront, and more of green areas that are considered contributing to the attractiveness of the place and the wel -being of people. Utilizing technology and data was the third often mentioned theme (82). It was seen designed for making people’s life easier for example by offering easy access to information or services, in a simple, user-friendly way. Technology was seen to be useful in healthcare, traffic/mobility, communication, experiences, smart buildings, city planning and maintenance, but also in enhancing sustainability and diversity. “A city that employs lots of smart solutions that are important for residents and visitors, such as in public transport, tidiness and city maintenance” (R 130) The respondents wished for seamless, integrated, and secure digital services, and a long-lasting, sustainable digital infrastructure that is built not only for the sake of having technology but serving a purpose. Less than 40 % (60 out of 159) K. Edelman et al.: Smart Cities on the Waterfront: Citizens’ Perspectives on Smart City in the Context of a Port City 525 respondents linked smart with technology, digitalization, or data, and many addressed only built environment, sustainability or mobility in their answers. The People orientation theme highlights that smart should be used for the people, enhancing equality and inclusiveness, the wellbeing and quality of life of both residents and visitors. Listening to people’s views and engaging them was considered an important task for governance. On the other hand, people were asked to be involved, smart, liberal, and polite to others. From the perspective of urban planning, the smart city is seen as adapting to the citizen’s needs and is flexible according to the amount of its user’s. It contributes into the positive residing or visiting experience with the aim of being “more than expected” (R 49). The main expectation is people oriented urban planning, so that residents and visitors of the city feel that they are the priority. When developing the area that connects the city centre with the sea, the case study area in our research, it is expected that there would be a lot of public space, low constructions, urban greening, it would be inclusive, involving various activities and cultural happenings. There are several historical buildings in the area, the castle being the most important; the castle is a landmark that tel s the history of the city. The respondents saw the castle as “No. 1” (R 59) and “the crown of the city” (R 41), that all urban development needs to adapt to. 5 Discussion Our aim was to comprise an understanding of how citizens, including local residents, recreational users, visitors, and ferry passengers, perceive a smart city in the context of a port city that in our case study is located in the Northern Europe. We directed the citizens’ attention to the waterfront area near and around the port which is going to experience substantial urban development in the coming years, and where the port’s presence is most observable to the citizens. The sustainability aspect of the smart city (Ahvenniemi et al., 2017; Szpak et al., 2024) is clearly present in citizens perceptions of the meaning of the smart city. Especial y the environmental sustainability can be seen as a cross cutting theme in the data of our study, that could be supported by the enhancement of environmental y friendly traffic and smooth interconnectivity of its various modes, the aspects that become crucial in a context of this specific area perceived by citizens, especial y when travel ing with the ferry. 526 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Several attempts have been made for managing the traffic congestions (see also Lehmacher et al., 2022) from the side of the port and the city. However, the mobility- as-a-service or intelligent transport and parking solutions are currently the challenges of improving smart mobility (Cohen, 2014, also Al Sharif & Pokharel, 2022; Moura & Silva, 2019; Qonita & Giyarsih, 2023) in our case study area. Survey respondents have highlighted these aspects as crucial elements of the smart city within a context of a port city. Because of the location of our case city by the river and the sea, the river and the sea are an integral part of the identity of the city (Özgece et al., 2022) and the enhancement of its presence in everyday life of its inhabitants is perceived as part of the smart city in a context of a port city, as our study indicates. This, together with the increase of the general awareness of environmental issues, such as climate change and biodiversity loss, may explain the strong emphasis on environmental sustainability and nature in the data. Our case city is part of the EU Mission Climate Neutral and Smart Cities, which has influenced the strategies and plans for the development of the area close to the port. Therefore, the smart city is often linked to climate concerns in the survey responses, that could be supported by the people-oriented development of the infrastructure, seen as part of the sustainable urban planning under the smart environment (Cohen, 2014). As cities are facing the requirements of enhancing the wel being of their growing populations, this cannot be achieved without new technologies and digitalization (Szpak et al., 2024). In our study, this could be seen as an invisible side of the smart city as the citizens expect that developments of technology and digitalisation would just make their lives easier by offering easy access to information or services, in a simple, user- friendly way. The expectations of the survey respondents meet the city’s urban planning principles for this area, currently covered mainly with asphalt. The transformation in upcoming years wil be towards controlled rewilding with the mix of residential and office spaces and the expansion of public space with park and urban greening with the aim of bringing art and culture into the area. 6 Conclusions The aim of our study was to comprise an understanding of how city inhabitants and visitors perceive a smart city in the context of a port city. According to our findings, the most prevalent themes in the data were traffic, environmental sustainability, technology and digitalization, urban planning, people orientation and culture, art & history of this specific area under study. Our findings show that citizens emphasize K. Edelman et al.: Smart Cities on the Waterfront: Citizens’ Perspectives on Smart City in the Context of a Port City 527 the utilisation of smart solutions for people and their wel being, whether related to traffic, services, sustainability, or urban planning, as well as environmental sustainability. This echoes the insight that the human component should never be overlooked in development of smart cities (Cardullo & Kitchin, 2019; Kitchin 2014; Mosco 2019; Szpak et al, 2024). The key topics specific to a smart city within the context of a port city include seamless traffic and mobility from the city to the port, as well as the preservation of the natural, historic, and cultural environment of the waterfront area, including the sea and the port. The results of our study support the recent directions in the smart cities literature, with the emphasis on environmental sustainability (Ahvenniemi et al, 2017). These results presented in the current paper wil be developed further based on the results of the upcoming workshops with citizens. Further research could consider the impact of the different population characteristics on the perception of a smart port city, such as the gender, age, income levels, etc., which were not accounted for in this study. The effect of the primary use of the port area on the perception of the smart port city also requires further investigation. References Ahvenniemi, H., Huovila, A., Pinto-Seppä, I., Airaksinen, M. (2017). What are the differences between sustainable and smart cities?. Cities, 60, 234-245. Al Sharif, R., Pokharel, S. (2022). Smart city dimensions and associated risks: Review of literature. Sustainable Cities and Society, 77, 1-14. Beškovnik, B., Bajec, P. (2021). Strategies and approach for smart city–port ecosystems development supported by the internet of things. Transport, 36(5), 433-443. Bruns-Berentelg, J., Noring, L., Grydehøj, A. (2022). Developing urban growth and urban quality: Entrepreneurial governance and urban redevelopment projects in Copenhagen and Hamburg. Urban Studies, 59(1), 161-177. Burns, R., Andrucki, M. (2021). Smart cities: Who cares? Environment and Planning A: Economy and Space, 53(1), 12-30. Burns, R., Fast, V., Levenda, A., Miller, B. (2021). Smart cities: Between worlding and provincialising. Urban Studies, 58(3), 461-470. Caragliu, A., Del Bo, C. F. (2019). Smart innovative cities: The impact of Smart City policies on urban innovation. Technological Forecasting and Social Change, 142, 373-383. Cardullo, P., & Kitchin, R. (2019). Being a ‘citizen’ in the smart city: Up and down the scaffold of smart citizen participation in Dublin, Ireland. GeoJournal, 84(1), 1-13 Carta, M., Ronsivalle, D. (2016). The Fluid City Paradigm. Regeneration as an Urban Renewal Strategy. Springer International Publishing, Cham, Switzerland. Cohen, B. (2014). The Smartest Cities in The World 2015. https://www.fastcompany.com/3038818/the-smartest-cit-ies-in-the-world-2015- methodology Drisko, J. W., Maschi, T. (2016). Content analysis. Oxford University Press, New York, United States of America. 528 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Gretzel, U., Sigala, M., Xiang, Z., Koo, C. (2015). Smart tourism: foundations and developments. Electronic Markets, 25, 179-188. Grossi, G., Pianezzi, D. (2017). Smart cities: Utopia or neoliberal ideology? Cities, 69, 79-85. Hein, C. (2016). Port cities and urban waterfronts: How localized planning ignores water as a connector. Wiley Interdisciplinary Reviews: Water, 3(3), 419-438. Hein, C., van de Laar, P. T. (2020). The separation of ports from cities: The case of Rotterdam. In: Carpenter, A., Lozano, R. (Eds.). European port cities in transition: Moving towards more sustainable sea transport hubs. Springer International Publishing, Cham, Switzerland, 265- 286. Hermelin, B., Jonsson, R. (2021). Governance of waterfront regeneration projects: Experiences from two second‐tier cities in Sweden. International Journal of Urban and Regional Research, 45(2), 266-281. Kitchin, R. (2014). The real-time city? Big data and smart urbanism. GeoJournal, 79(1), 1-14. Krippendorff, K. (2018). Content analysis: An introduction to its methodology. 4th Ed. Sage Publications, Thousand Oaks, United States of America. Lehmacher, W., Lind, M., Poikonen, J., MeseguerS J., Cárcel Cervera, J. L. (2022) Reducing port city congestion through data analysis, simulation, and artificial intelligence to improve the wellbeing of citizens. Journal of Mega Infrastructure & Sustainable Development, 65-82, DOI: 10.1080/24724718.2022.2133524 Lochmiller, C. R. (2021) Conducting thematic analysis with qualitative data. The Qualitative Report, 26(6), 2029-2044. Mosco, V. (2019). The smart city in a digital world. Emerald Publishing Limited, Bingley, United Kingdom. Moura, F., Silva, J. D. A. (2019). Smart cities: Definitions, evolution of the concept and examples of initiatives. In: Leal Filho, W., Azul, A., Brandli, L., Özuyar, P., Wal , T. (Eds.). Industry, innovation and infrastructure. Springer Nature, Cham, Switzerland, 1-9. Nowacka-Rejzner, U. (2015). Post-harbour areas-new urban space. Civil and Environmental Engineering Reports, 19(4), 71-78. Özgece, N., Edgü, E., Ayıran, N. (2022). Assessing imageability of port cities through the visibility of public spaces: the cases of Famagusta and Limassol. Space and Culture, 25(4), 535-552. Qonita, M., Giyarsih, S. R. (2023). Smart city assessment using the Boyd Cohen smart city wheel in Salatiga, Indonesia. GeoJournal, 88(1), 479-492. Rapley, T. (2021). Some pragmatics of qualitative data analysis. In: Silverman, D. (Ed.). Qualitative research. 5th Ed. SAGE Publications, London, United Kingdom. Schreier M, Stamann C, Janssen M, Dahl T, Whittal A. (2019) Qualitative content analysis: Conceptualizations and chal enges in research practice - Introduction to the FQS special issue “qualitative content analysis I”. Forum: Qualitative Social Research, 20(3). https://ntnuopen.ntnu.no/ntnu-xmlui/handle/11250/2640898 Szpak, A., Modrzyńska, J., Dahl, M. (2024). Digitalization of Smart City—Sine Qua Non or an Option for Those Interested?. Urban Affairs Review, DOI: 10.1177/10780874241226674 Townsend, A. M. (2013). Smart cities: Big cata, civic hackers, and the quest for a new utopia. WW Norton, New York, United States of America. Tsvetkova A., Gustafsson, M., Wikström, K. (2021). Digitalising maritime transport: digital innovation as a catalyser of sustainable transformation. In: Montero, J., Finger, M. (Eds.). A modern guide to the digitalization of infrastructure. Edward Elgar Publishing, Cheltenham, United Kingdom, 123-148. Van den Berghe, K. B., Daamen, T. A. (2020). From planning the port/city to planning the port-city: Exploring the economic interface in European port cities. Springer International Publishing, Cham, Switzerland. Yin, R. K. (2003). Case Study Research: Design and Methods. 3rd Ed. SAGE Publications, Thousand Oaks, United States of America. K. Edelman et al.: Smart Cities on the Waterfront: Citizens’ Perspectives on Smart City in the Context of a Port City 529 Yin, R. K. (2009). How to do better case studies. In: Bickman, L., Rog, D. J. (Eds.). The SAGE handbook of applied social research methods. SAGE Publications, Los Angeles, United States of America, 254-282. Zheng, C., Yuan, J., Zhu, L., Zhang, Y., Shao, Q. (2020). From digital to sustainable: A scientometric review of smart city literature between 1990 and 2019. Journal of Cleaner Production, 258, 1-24. 530 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix The survey questions. 1. Please provide your residential postcode 2. What makes a city "smart" in your opinion? 3. What does a smart and sustainable city mean for you? 4. What attracts you or for what purposes do you use/visit the Linnanniemi area (the area extending from Forum Marinum through the Turku Castle Park to the Port of Turku)? 5. What aspects would be important for you in the development/transformation process of the Linnanniemi area? 6. What digital solutions (Apps, WEB pages etc.) have you used in connection with the harbour area and/or surrounding area of Turku Castle (e.g. mobility & transport, parking, tourism, events, cultural offerings, local businesses, carbon foot monitoring or other - what)? RELAXING OR EXCITING? MOBILE EMOTION RECORDING IN NATURAL SETTINGS ONDREJ MITAS,1 BEN WIELENGA,2 KEES EVERS,1 JELENA FARKIĆ,1 PETER WARD,3 JEROEN KLIJS,1 MARCEL BASTIAANSEN4 1 Breda University of Applied Sciences, Academy for Tourism, Breda, Netherland mitas.o@buas.nl, keesevers15@gmail.com, farkic.j@buas.nl, klijs.j@buas.nl 2 NHL Stenden University of Applied Sciences, European Tourism Futures Institute, Leeuwarden, Netherland ben.wielenga@nhlstenden.nl 3 Brigham Young University, Department of Experience Design and Management, Provo, Utah, United States of America peter_ward@byu.edu 4 Breda University of Applied Sciences, Academy for Leisure, Breda, Netherland bastiaansen4.m@buas.nl The health benefits of experiencing nature are wel -known. Several established theories, such as attention restoration, biophilia, and awe theories, suggest that lowered emotional arousal is a mechanism of the health effects of experiencing nature. This has not been tested in nature walking experiences in the field, and has not accounted for the recent trend of constructing built features such towers, bridges, and museums to bring visitors in closer touch with nature. Wearable skin conductance recording technology has recently opened this avenue for research. The present study shows that these built features were associated with lower emotional arousal than natural areas, or than purely functional built features. However, individuals reporting improvment in health over the visit experienced relatively lower arousal in natural areas, yet higher Keywords: health, arousal at built features such as bridges, towers, and museums nature, aimed to bring them closer to nature. These effects point to emotions, biophilia and attention restoration occurring in natural location tracking, environments, while built features focused on nature may be spatial triggering awe. analysis DOI https://doi.org/10.18690/um.fov.4.2024.31 ISBN 978-961-286-871-0 532 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Contact with nature is widely understood to enhance health. Natural environments restore attention and promote physical activity, resulting in positive emotions, reduced stress, and lower blood pressure in the short term, faster healing, and higher subjective wel -being in the long term (Kaplan & Kaplan, 1989; Kaplan, 1995; Ulrich, Dimberg, et al., 1991; Ulrich, Simons, et al., 1991). These wel -known outcomes are important reasons for governments to protect natural areas, and for park managers to invest in facilities for visitors such as interpretive programs, parking, trails, signage, benches, and lookout towers. In recent years, there has been a noticeable increase in architectural attention to the design of these built features. They are now frequently designed not only for low cost and basic function, but also for their own aesthetic beauty and fit with the natural landscape (Wielenga et al., 2022, Wielenga, 2021). Examples include intricate and sculptural lookout towers, such as on the Devinska Kobyla hill near Bratislava, and museums which appear to grow out of the landscape, such as The Whale Museum in Norway. One major hope behind increased investment in these facilities is to improve the visitor experience. More concretely, it is assumed that a wel - designed bench or lookout point provides closer and more intimate access to nature, and enhances the already-known positive effects of the natural environment. This assumption is untested, however, as emotional reactions to specific environmental stimuli are fleeting and difficult to capture in the moment (Bastiaansen et al., 2019). Developments in wearable tracking of emotional arousal have made such recording possible. When combined with GPS location tracking, wearable emotion recordings have opened up possibilities of linking emotional reactions to specific environmental stimuli (Mitas, Mitasova, et al., 2020; Strijbosch et al., 2021b) Situating the present study in two natural areas in the Netherlands, we mobilize wearable emotion and location tracking technologies to examine visitors’ emotional arousal at two protected natural areas, contrasting nature-enhancing built features with purely functional built features. We further link the differences herein to visitors’ self-reported health. O. Mitas et al.: Relaxing or Exciting? Mobile Emotion Recording in Natural Settings 533 2 Literature review 2.1 Health benefits of experiencing nature The salubrious effects of experiencing nature on physical health are wel - documented and have been explained by a number of potential mediators. Theories such as biophilia (Grinde & Patil, 2009; Wilson, 1986) and attention restoration theory (Basu et al., 2019; Ohly et al., 2016) suggest that built environments are too stimulating and information-rich, creating excess cognitive load and emotional arousal. The ‘soft fascination’ of natural environments, especially through the visual sense, decreases this excessive emotional arousal and thus allows restoration of cognitive and physiological capacities (Basu et al., 2019). In contrast, Keltner and colleagues suggest that awe, an arousing yet uniquely beneficial emotion, is triggered by nature stimuli. Awe leads to prosocial and other health-reinforcing behaviors (Anderson et al., 2018; Harker & Keltner, 2001; Keltner & Haidt, 2003; Piff et al., 2015). Thus, theoretical explanations of how nature experiences contribute to health disagree on the mediating role of emotional arousal. While biophilia and attention restoration theories assert that nature improves health by decreasing emotional arousal, research on awe suggests that nature improves health through (a specific) increase in emotional arousal. Resolving this ambiguity is crucial for the management of natural areas. Improving public health is often a stated goal of managing natural areas for recreational visitors. However, management interventions vary, and can be designed to either increase (e.g., sweeping views, perceived risk) or decrease (e.g., gradual curves and ascents, low-key, minimalist built features) emotional arousal. These interventions can also deliberately vary between built and natural location contexts within a single natural area. Furthermore, progress on the ambiguity around emotional mechanisms in natural areas has been difficult due to technical limitations of emotional arousal measurement. Field studies of nature experiences are typical y based on self-report, which compresses emotional arousal experienced over varying natural environmental stimuli into a single number. This number fails to capture the inherent ebb and flow of emotional arousal triggered by specific stimuli and at a precise time (Bastiaansen et al., 2019) and is subject to wel -known recal errors 534 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION (Zajchowski et al., 2016). Therefore, detailed physiological measurement of emotional arousal in real-world natural settings is necessary. 2.2 Wearable measurement of emotional arousal Several important technological innovations have recently made it possible to continuously and accurately monitor emotional arousal in real-world settings. First, phasic skin conductance has been established as a valid indicator of emotional arousal, even during physical movement (Li et al., 2022) as slow temperature-related changes are filtered out by focusing on the phasic component of skin conductance (Benedek & Kaernbach, 2010). Second, devices to record skin conductance have become smal , simple, affordable, and wearable. Third, software developments have enabled rigorous filtering of motion artifacts (Strijbosch et al., 2021a). These technological developments have unlocked new insights into the ebb and flow of emotional arousal over the course of even small variations in stimuli, such as museum exhibits (Kirchberg & Tröndle, 2015; Mitas, Cuenen, et al., 2020), twists and turns of a rol er coaster (Bastiaansen et al., 2022), and stops on a guided tour (Mitas, Mitasova, et al., 2020). Thus, it is now possible to address the issue of emotional arousal across natural and built features during a single visit to a nature area. Combining mobile skin conductance recording with self-report survey data also allows us to determine how variations in this emotional arousal correspond to health outcomes. 3 Methods We used a mixed-method approach which combined physiological recording of emotional arousal, GPS-based location tracking, and self-report questionnaires before and after visits to natural areas. For the purpose of this study, two natural areas in the Netherlands were selected – Fort de Roovere and Nationaal Park Sal andse Heuvelrug. Although they are different in size and characteristics, they are both valued for their biodiversity, recreational opportunities, and cultural significance. We intercepted visitors entering these sites in the spring and summer of 2023. Participants were invited to respond to an intake questionnaire, wear a wristband which recorded their skin conductance as a proxy of emotional arousal, O. Mitas et al.: Relaxing or Exciting? Mobile Emotion Recording in Natural Settings 535 and carry a smartphone which recorded their location via GPS. Upon completing their visit, participants filled out a brief post-experience questionnaire. 3.1 Study sites 3.1.1 Fort de Roovere Fort de Roovere is in the Netherlands between the cities of Steenbergen and Bergen op Zoom. Constructed in the 17th century, this fort witnessed numerous wars. Today, the fortification consists of four earthen bastions surrounded by a moat, with further outlying defensive works to the east and south. A small temporary building houses a lunchroom. We term this site a green fort as there are no stone or brick wal s; al wal s are ful y planted. The entire area of the fort, and the western side of the moat, are completely covered in grass. Most of the moat is flooded with water. The outlying defensive works are completely overgrown by forest with occasional canals. Figure 1: Fort de Roovere with Moses Bridge (center bottom) and Pompejus lookout tower (upper left) Source: Marc Bolsius, 2024 Besides the lunchroom, Fort de Roovere contains a smal , bare parking lot. The fort also includes two unique architectural features intended to enhance visitors’ nature experience. One is the Moses bridge, which crosses the moat from the fort into the forest on its eastern side. The bridge is sunken into the water, essential y placing visitors at about chest level with the surface of the water. This creates a compel ing 536 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION visual and spatial effect that enables visitors to view aquatic birds, amphibians, and insects from an unusual y close and unique perspective. The second such feature is the Pompejus lookout tower, which rises 25 meters above the northeast bastion, allowing a sweeping view of the surrounding agricultural landscape and forest (Figure 1). 3.1.2 Nationaal Park Sallandse Heuvelrug Nationaal Park Sallandse Heuvelrug is in the east-central Dutch province of Overijssel and contains two distinct landscape types, forest and heather fields, on 2217 hectares of rolling hills (Figure 2). Most visitors enter the park from its northern edge near the village of Nijverdal. Here a large parking lot is situated across from a visitor center which contains a museum, restaurant, and souvenir shop (Figure 3). Numerous hiking paths originate at this point. About a kilometer south of the visitor center is the peak of the Noestelerberg, the first of the park’s two hil tops. Here several benches form a popular lookout point. Figure 2: Sal andse Heuvelrug purple heather field (foreground) with forested hil top (background) Source: Staatsbosbeheer O. Mitas et al.: Relaxing or Exciting? Mobile Emotion Recording in Natural Settings 537 Figure 3: Nationaal Park Sal andse Heuvelrug visitor center Source: Staatsbosbeheer 3.2 Sample We used an intercept sampling approach, asking each visitor entering the site on foot from the main parking lot if they would be wil ing to participate in exchange for a free coffee or tea. To have a sufficient population of visitors to sample from, we limited data col ection to busy days (Wednesdays and weekends) with weather suitable for outdoor recreation (no or little precipitation). We collected data at Fort de Roovere in April of 2023, including Easter weekend, and at Sallandse Heuvelrug in May and June of 2023, including the Ascension and Pentecost weekends. Of 125 participants who provided data, either questionnaire, wearable, or GPS data were missing for 32 participants. From the remaining 93 participants, 21 exhibited wearable data that had too many motion artifacts in the physiological data to be useable, resulting in a final sample of 72 participants. 3.3 Measures To record location, we lent participants a smartphone with the popular workout application Strava that recorded GPS location once per second. These data were then grouped into three environmental stimulus types: nature-enhancing architecture, which were built features aimed at bringing visitors in contact with nature, such as lookout towers, benches at scenic viewpoints, and nature museums; non-nature architecture, including parking lots, roads, and restaurants, which served a functional 538 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION purpose and were not designed to bring visitors in contact with nature; and al other locations, which general y offered visitors unmediated contact with nature. To measure emotional arousal, we outfitted participants with an Empatica E4 wristband to measure skin conductance, a proxy for emotional arousal, at 4 times per second. The wristband records skin conductance from 2 wires which attach to pre-gel ed electrodes worn on the fingers (Figure 4). The skin conductance signal was cleaned from motion artifacts using the ArtifactZ function of the Breda Experience Lab Toolbox (Bastiaansen et al., 2022). Tonic changes in the signal due to temperature and wearing of the device were filtered out using deconvolution (Benedek & Kaernbach, 2010). Skin conductance signals were also Z-standardized to cancel out differences in skin responsiveness between participants, and log- transformed to reduce kurtosis. Figure 4: Empatica E4 Wristband Source: Own To assess self-reported health, we asked participants to fill out questionnaires at the beginning and end of their visit. The questionnaires contained the World Health Organization single item “How satisfied are you with your health?” on a 5-point scale ranging from “very dissatisfied” to “very satisfied” (Skevington et al., 2004). While such subjective measures are imperfect, they correlate at least moderately with some physical health measures and are able to discriminate between healthy and il O. Mitas et al.: Relaxing or Exciting? Mobile Emotion Recording in Natural Settings 539 subsamples (Bech, 2001). We subtracted participants’ pre-visit health from their post-visit health for an index of health change. Because most participants’ self- reported health was unchanged from before to after the visit, we col apsed this variable into two categories, coded 1 if participants’ health improved during the visit, and 0 if it did not. 3.4 Data analysis We modeled the data using mixed-effects linear models with random intercepts per participant. We built our models in two stages. First, we modeled phasic skin conductance as a function of environmental stimulus type, using a dummy-coded predictor for nature-enhancing and non-nature architecture, taking nature areas as the reference category. Subsequently, we included health change as a predictor, and al owed it to interact with each of the previously entered predictors representing environmental stimulus types. 4 Findings 4.1 Descriptive findings The 34 participant visitors to Fort de Roovere spent an average of 68 minutes visiting the fort (sd = 32 minutes). They reported fairly good health before (m = 3.79, sd = 0.73) and after (m = 3.88, sd = 0.77) their visit. Before- and after-visit health scores were identical for 31 of these visitors, and improved for 3. The 38 visitors to the Sallandse Heuvelrug stayed longer than visitors to the fort, spending an average of 108 minutres (sd = 93 minutes). Their self-reported health was likewise good before (m = 4.01, sd = 0.69) and after (m = 4.09, sd = 0.63) their visit. The change in health over the course of the visit was negative for 3 participants, positive for 6 participants, and remaining the same for 30 participants. 4.2 Spatial analysis of emotional arousal Maps reveal striking variations of emotional arousal at each site, with blue indicating lower emotional arousal, and red indicating higher emotional arousal. At Fort de Roovere, emotional arousal was low in the forested area around the fort, especial y to the east. On the other hand, the forested paths along canals just south of the fort 540 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION featured strong emotions. Basicaly, when in the forest, participants got more emotional the closer to the water they got. This could be because the elevation changes are interesting, but also a bit frightening, as they are steep. Strong emotions were also experienced at the entrance, the lunchroom, and the location on the fort wal where the Moses bridge comes into view. Remarkably, the bridge and the tower themselves seemed to have a rather calming influence on average (Figure 5). Figure 5: Map of emotional arousal across the entire Fort de Roovere site (left) and zoomed in to the most visited area (right) Note: Lower arousal Higher arousal Due to the much larger area covered, the map of the Sal andse Heuvelrug is more difficult to interpret qualitatively. It is clear, however, that most forested areas (upper half of the visited area) were lower in arousal rather than the heather fields (lower half). Furthermore, hiking paths with sweeping views of the landscape triggered relatively higher levels of arousal. Emotional response to the visitor center was mixed (Figure 6). O. Mitas et al.: Relaxing or Exciting? Mobile Emotion Recording in Natural Settings 541 Figure 6: Map of emotional arousal across the entire Sal andse Heuvelrug site (left) and zoomed in to the most visited area (right) Note: Lower arousal Higher arousal Statistical modeling of emotional arousal as a function of environmental stimulus type demonstrated that nature-enhancing architecture was associated with lower emotional arousal than natural areas (t = -5.022; p = <0.001), while non-nature architecture was associated with higher emotional arousal than natural areas (t = 32.183; p <0.001) (Table 1). Table 1: Multilevel model of phasic skin conductance as a function of environmental stimulus Variable Coefficient estimate t-value (Standard error) Constant 0.008432 (0.006466) 13.040*** Nature-enhancing architecture -0.001758 (0.000325) -5.022*** Non-nature architecture 0.01046 (0.000350) 32.183*** Note: Reference category: Nature areas. Significance codes for p values: *** < 0.001 < ** < 0.01 * < 0.05 542 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.3 Self-reported health change as moderator We entered the variable of self-reported health change as a moderator term in the model of emotional arousal as a function of environmental stimulus (Table 2). This al owed us to determine at which locations a different momentary health outcome (improved vs. not improved) was associated with greater, or lower, emotional arousal. Participants who reported improved health over the course of their visit experienced slightly higher emotional arousal in built areas and much lower emotional arousal in nature, compared to participants whose self-reported health did not change or decline over the visit (all p's < 0.001). Table 2: Multilevel model of phasic skin conductance as a function of environmental stimulus x health change Variable Coefficient estimate t-value (Standard error) Constant 0.08452 (0.006484) 13.036*** Nature-enhancing architecture -0.002309 (0.000357) -6.463*** Non-nature architecture 0.007973 (0.000346) 23.014*** Health change -0.001515 (0.000718) -2.109* Nature-enhancing architecture x Health change 0.009553 (0.001786) 20.578*** Non-nature architecture x Health change 0.02054 (0.000998) 5.350*** Note: Reference category: Nature areas. Health change = 0 if not improved over visit; =1 if improved over visit. Significance codes for p values: *** < 0.001 < ** < 0.01 * < 0.05 4 Discussion The present findings demonstrate that experiences of natural areas vary richly over space and time. Human-built features aimed at connecting visitors to nature, for example by creating opportunities to rest or gain a unique visual perspective on the landscape, had a calming effect on emotional arousal. However, participants who experienced higher emotional arousal at human-built features, but lower arousal in natural areas, were more likely to report increased self-reported health. O. Mitas et al.: Relaxing or Exciting? Mobile Emotion Recording in Natural Settings 543 Participants who experienced better health outcomes had attentuated emotional arousal in natural areas, a finding consistent with low-arousal theories of nature experiences, such as attention restoration theory (Ohly et al., 2016) and biophilia (Grinde & Patil, 2009). However, higher emotional arousal among healthier participants at built features aimed to increase contact with nature seems more in line with awe experiences, which are highly arousing (Anderson et al., 2018). One potential explanation is the way individuals interact with nature is fundamentally different on walking trails (comprising most of the nature settings studied) and built features, such as towers and bridges, that aim to enhance contact with nature. Walks offer slowly unfolding, calming visual stimuli characteristic of 'soft fascination,' a response to stimuli extensively studied in attention restoration experiments (Basu et al., 2019). Built features such as lookout towers, bridges, and museums instead offer overviews and interpretive material aimed to deepen appreciation and attention to nature as a whole, increasing chances of awe experiences. This potential explanation may be used to generate hypotheses for future research. To further disambiguate the meaning of these patterns would require more detailed self-report data, as well as more instances of built features. Furthermore, it is wel - known that social interactions are a key source of both emotions (Mitas et al., 2023) and health (Cacioppo et al., 2002) and not only occur in natural areas, but may also be facilitated by built environments therein. For example, a twisty, isolated trail will feature less social interaction than a restaurant in the visitor center. We did not measure social interaction in the current study. The possibility that it is interacting with the spatial effects we found warrants further research. Given that healthier participants experienced relatively higher emotional area at built features, and lower in natural areas (especially trails), suggests simply increased or attenuated emotional arousal does not make a nature experience healthier. Rather, both quiet trails which invite calm and contemplation (Kaplan, 1995) and an occasional lookout tower with an awe-inspiring overview of the entire natural area (Cajas, 2020) are probably best. In fact, previous research on skin conductance in cultural tourism contexts has confirmed that simply more emotional arousal is not necessarily better; rather it is the temporal profile bby which emotional arousal ebbs and flows that makes for an excel ent experience (Mitas, Mitasova, et al., 2020; Mitas et al.). We hereby extend importance of wel -designed emotional ebb and flow over an experience to the context of visiting natural areas. 544 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.1 Limitations It is important to note that these results capture the difference between 9 participants for whom health improved, and 63 participants for whom it did not. As the group of participants for whom health improved is very smal , owing to the generic single- item measure used, these findings should be seen as exploratory and interpreted with caution. Additional measures of outcomes (e.g., life satisfaction), potential psychological mechanisms (e.g., appreciation of beauty, cognitive restoration, social interaction), and qualitative data in participants' own words could enable further insights into the benefits of nature experiences. The findings may have been influenced by the intercept sampling approach, as frequent visitors to natural areas may have been over-represented. Natural sites are often managed and developed with the hope of attracting a broader scope of visitors, including those who do not wander into nature often. Future research could undertake a targeted approach to bring less-frequent visitors to natural areas to determine if they experience the differences between natural and built features differently. Final y, we recommend replication of the present study in other landscape morphologies. Changes in elevation are more dramatic in many natural areas, and produce emotional y intense awe and overview effects. In general, the research is limited by the choice of two sites. We therefore aim to continue col ecting data at a greater variety of sites, both to increase the number of participants, as wel as increasing the diversity of landscape features in each location context category. These would be worth capturing using the unique affordances of mobile emotion and location tracking. Acknowledgements We appreciate the input of Keri Schwab to the conceptualization of this research, Shelly van Winden, Anne-Wil Maris, Eric Goosen, Tessa van Ieperen, and Sibel van Lent to the data collection, and Wilco Boode and Hans Revers to the data processing and analysis. O. Mitas et al.: Relaxing or Exciting? Mobile Emotion Recording in Natural Settings 545 References Anderson, C. L., Monroy, M., & Keltner, D. (2018). Awe in nature heals: Evidence from military veterans, at-risk youth, and college students. Emotion, 18(8), 1195. Bastiaansen, M., Lub, X., Mitas, O., Jung, T. H., Passos Acenção, M., Han, D., Moilanen, T., Smit, B., & Strijbosch, W. (2019). Emotions as core building blocks of an experience. International Journal of Contemporary Hospitality Management, 31, 651-668. Bastiaansen, M., Oosterholt, M., Mitas, O., Han, D., & Lub, X. (2022). An emotional rol er coaster: Electrophysiological evidence of emotional engagement during a rol er-coaster ride with virtual reality add-on. Journal of hospitality & tourism research, 46(1), 29-54. Basu, A., Duval , J., & Kaplan, R. (2019). Attention restoration theory: Exploring the role of soft fascination and mental bandwidth. Environment and Behavior, 51(9-10), 1055-1081. Bech, V. N., Per. (2001). The WHO quality of life (WHOQOL) questionnaire: Danish validation study. Nordic journal of psychiatry, 55(4), 229-235. Benedek, M., & Kaernbach, C. (2010). Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology, 47(4), 647-658. Cacioppo, J. T., Hawkley, L. C., Crawford, L. E., Ernst, J. M., Burleson, M. H., Kowalewski, R. B., Malarkey, W. B., Van Cauter, E., & Berntson, G. G. (2002). Loneliness and health: Potential mechanisms. Psychosomatic Medicine, 64(3), 407-417. Cajas, B. (2020). Feeling ‘Awe’some! The impact of awe, produced by exposure to nature and interpersonal elicitors, on prosocial behaviour [MA, University of Twente]. Enschede, Netherlands. Grinde, B., & Patil, G. G. (2009). Biophilia: does visual contact with nature impact on health and wel -being? International Journal of Environmental Research and Public Health, 6(9), 2332-2343. Harker, L., & Keltner, D. (2001). Expressions of positive emotion in women's col ege yearbook pictures and their relationship to personality and life outcomes across adulthood. Journal of personality and social psychology, 80(1), 112. Kaplan, R., & Kaplan, S. (1989). The experience of nature: A psychological perspective. Cambridge university press. Kaplan, S. (1995). The restorative benefits of nature: Toward an integrative framework. Journal of environmental psychology, 15(3), 169-182. Keltner, D., & Haidt, J. (2003). Approaching awe, a moral, spiritual, and aesthetic emotion. Cognition & emotion, 17(2), 297-314. Kirchberg, V., & Tröndle, M. (2015). The museum experience: Mapping the experience of fine art. Curator: The Museum Journal, 58(2), 169-193. Li, S., Sung, B., Lin, Y., & Mitas, O. (2022). Electrodermal activity measure: A methodological review. Annals of tourism research, 96, 103460. Mitas, O., Cuenen, R., Bastiaansen, M., Chick, G., & van den Dungen, E. (2020). The War from both Sides: how Dutch and German Visitors Experience an Exhibit of Second World War Stories. International Journal of the Sociology of Leisure, 3(3), 277-303. https://doi.org/https://doi.org/10.1007/s41978-020-00062-3 Mitas, O., Mitasova, H., Mil ar, G., Boode, W., Neveu, V., Hover, M., van den Eijnden, F., & Bastiaansen, M. (2020). More is not better: The emotional dynamics of an excel ent experience. Journal of hospitality & tourism research, 1096348020957075. Mitas, O., Shahvali, M., Ward, P., Duerden, M., & Bastiaansen, M. (2023). Human connection: A crucial ingredient in vacation experience. In Handbook of Tourism and Quality-of-Life Research II: Enhancing the Lives of Tourists, Residents of Host Communities and Service Providers (pp. 225-236). Springer. Mitas, O., van Waalwijk, J., Bargeman, B., Calvi, L., van Esch, L., Hover, M., Boode, W., & Bastiaansen, M. The Tension is Rising: Storytelling as an Intervention. In Tourism Interventions (pp. 97-112). Routledge. 546 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Ohly, H., White, M. P., Wheeler, B. W., Bethel, A., Ukoumunne, O. C., Nikolaou, V., & Garside, R. (2016). Attention Restoration Theory: A systematic review of the attention restoration potential of exposure to natural environments. Journal of Toxicology and Environmental Health, Part B, 19(7), 305-343. Piff, P. K., Dietze, P., Feinberg, M., Stancato, D. M., & Keltner, D. (2015). Awe, the smal self, and prosocial behavior. Journal of personality and social psychology, 108(6), 883. Skevington, S. M., Lotfy, M., & O'Connell, K. A. (2004). The World Health Organization's WHOQOL-BREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Quality of life Research, 13, 299-310. Strijbosch, W., Mitas, O., van Blaricum, T., Vugts, O., Govers, C., Hover, M., Gelissen, J., & Bastiaansen, M. (2021a). Evaluating the Temporal Dynamics of a Structured Experience: Real-Time Skin Conductance and Experience Reconstruction Measures. Leisure Sciences, 1- 25. Strijbosch, W., Mitas, O., van Blaricum, T., Vugts, O., Govers, C., Hover, M., Gelissen, J., & Bastiaansen, M. (2021b). When the parts of the sum are greater than the whole: Assessing the peak-and-end-theory for a heterogeneous, multi-episodic tourism experience. Journal of Destination Marketing & Management, 20, 100607. Ulrich, R. S., Dimberg, U., & Driver, B. L. (1991). Psychophysiological indicators of leisure benefits. In B. L. Driver, L. R. Brown, & G. L. Peterson (Eds.), Benefits of Leisure. (pp. 73-89). Venture. Ulrich, R. S., Simons, R. F., Losito, B. D., Fiorito, E., Miles, M. A., & Zelson, M. (1991). Stress recovery during exposure to natural and urban environments. Journal of environmental psychology, 11(3), 201-230. Wielenga, B., Hartman, S., Heslinga, J.H. Architourism in nature areas: a ‘Bilbao effect’ in the bush? In Novel i, M., Cheer, J., Dolezal, C., Jones, A., Milano, C. Handbook of Niche Tourism, pp. 25-37.DOI: 10.4337/9781839100185.00011. Wielenga, B. (2021). New experiences in nature areas: architecture as a tool to stimulate transformative experiences among visitors in nature areas? Journal of Tourism Futures, 9(3), 304-310. DOI: 10.1108/JTF-09-2020-0155. Wilson, E. O. (1986). Biophilia. Harvard university press. Zajchowski, C. A., Schwab, K. A., & Dustin, D. L. (2016). The Experiencing Self and the Remembering Self: Implications for Leisure Science. Leisure Sciences, 1-8. IT’S A MATCH! – FINDING THE PERFECT PARTICIPANTS FOR HEALTH RESEARCH. STUDY ON THE WILLINGNESS TO SHARE HEALTH DATA WITH A RESEARCH REFERRAL PORTAL MELANIE MANDL, CHRISTINE MALIN, STEFAN THALMANN University of Graz, Business Analytics and Data Science-Center, Styria, Graz, Austria melli.mandl@hotmail.com, christine.malin@uni-graz.at, stefan.thalmann@uni-graz.at Finding suitable participants is a big challenge for health research and is considered a significant barrier. Research referral portals (RRPs) matching participants with requirements of researchers are intended to overcome this barrier. Here, the wil ingness to share health data is the key success factor for this data-driven matching process. However, the variables and incentives influencing the wil ingness to share have not been researched in this specific context so far. Therefore, this article presents a two- stage study exploring individuals’ willingness to share personal Keywords: health data with RRPs conducting two focus group interviews healthcare (n=13) and an online survey (n=1,223). The study investigated data, the willingness to share data among individuals for RRPs and research referral identified five influencing data categories. We also identified portals, factors such as attitudes towards technology, altruism, and matchmaking, science affecting wil ingness to share. Practical implications willingness to share include refining matching processes and enhancing data security, health while longitudinal studies are suggested for broader insights. data DOI https://doi.org/10.18690/um.fov.4.2024.32 ISBN 978-961-286-871-0 548 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Despite clinical research involving participants forming the backbone of our health system, finding suitable participants remains a major chal enge for health research (Capili, 2021; Chaudhari et al., 2020; Gul & Ali, 2010). Finding suitable participants is not trivial as the participants must fulfill a study-specific set of inclusion and exclusion criteria such as age or specific diseases. The more specific these requirements are, the more chal enging is it to find suitable participants (Capili, 2021; Evans & Ildstad, 2001). To start this recruitment process, a large amount of participants’ health data is necessary. As this is chal enging, the researchers may not be able to continue their project (Borg et al., 2024), slowing down health research. Research referral portals (RRPs) are one way to solve this major chal enge, as they recruit participants for researchers and facilitate connecting participants, researchers, and research institutions. The major success factor of RRPs is the availability of comprehensive health data about as many potential participants as possible. However, health data is considered sensitive and the wil ingness to share such data is limited (Woldaregay et al., 2020). Previous research (e.g., Seltzer et al. 2019) indicates that individuals are already willing to share their health data with researchers, but the contexts in which they would be most likely to share data for research use is unclear. To date, comprehensive research investigating the dynamics between the matching process, willingness to share health data, and incentives remains notably scarce. While several studies (e.g., Broekstra et al., 2020; Singer & Couper, 2008; Song et al., 2023) have delved into the aspects of incentives of why individuals take part in clinical trials, a comprehensive understanding of how the elements interplay in the context of an RRP is yet to be clarified. This article aims to investigate the willingness of individuals to share their health data with RRP. It answers the fol owing research question: Which factors influence the wil ingness to share personal health data for matching participants with health studies? 2 Background and Hypotheses Clinical trials involving humans are an essential basis for scientific progress in medicine (e.g., the development of drugs), and therefore, it is necessary to find volunteering participants who agree to share health data (Inan et al., 2020; Tishler & M. Mandl, C. Malin, S. Thalmann: It’s a Match! – Finding the Perfect Participants for Health Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 549 Bartholomae, 2002). There are several reasons why clinical trials might fail. The most common ones are issues with the study design, safety concerns, and insufficient funding, but one of the main hurdles is patient recruitment and retention (Fogel, 2018; Su et al., 2023). Chal enges accompanying recruitment and retention are among others participant interest, inclusion and exclusion criteria, geographical barriers, or patient burden (Fogel, 2018; Kelly & Halabi, 2018). Participation in clinical trials provides several advantages for participants such as intensive monitoring, access to new drugs, or active contribution to medical research. However, the data col ection in health care plays a crucial role and therefore cannot be overstated. Health data (e.g., electronic health records or patient/disease registries) affects every facet of the health care system, from health care providers to patients. To close the gap between participants and studies, RRPs intervene and use a matching process, where they compare two sets of data and match them against each other (Integrate.io, 2023). This process is designed to refer participants to studies based on specified characteristics. It starts with the researchers specifying the inclusion and exclusion criteria and passing them on to the RRP (1). Then, the future participants share their health-related data with the RRP (2). The focus of this matching now lies on the user profile. This data is matched with the study requirements and potential matches are identified (3). Eligible users are now notified and invited to participate in the clinical trials. This process enables the RRP to find suitable participants for the studies, while the participants can participate in studies that match their needs and interests (see Figure 1). Figure 1: Matching Process of Research Referral Portals (RRPs) Source: Own 550 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Based on this matching process, an RRP is therefore a specialized online platform used to connect researchers and subjects (Integrate.io, 2023). Such portals enable researchers to conduct studies and col ect health data from participants while allowing individuals to participate in these studies and potentially gain access to new treatments, medications, or monetary compensation. Participants’ motives and barriers for sharing personal health data for medical research influence their decision to share personal health data with an RRP (Broekstra et al., 2020; Permuth-Wey & Borenstein, 2009). The motives and barriers might explain differences between participation and non-participation, and therefore, it can be assumed that motives increase and barriers decrease the wil ingness to share personal health data. The literature distinguishes between two barriers to sharing personal data: (1) lack of individual benefits, and (2) non- contribution to public goods (Broekstra et al., 2020). The former describes the discrepancy between participants’ expectations and the benefits offered. Participants would still become ill irrespective of their research participation, denoting a lack of individual benefit from a medical examination. Moreover, the time and effort invested in each study may be barriers to participation in general (Broekstra et al., 2020). The latter barrier describes the fear that their health data might be sold or misused, often driven by skeptical news, or negative experiences during previous research visits (Broekstra et al., 2020). In the literature, three main motives for participating in health research are distinguished: altruism, survey-related, and egoistic reasons. Altruism describes a “behavior that benefits others at a personal cost to the behaving individual” (Kerr et al., 2004), e.g., donating a kidney (Lamanna, 1997). In the context of a health- related study, altruism means that the research furthers some purpose that is important to the participant, or the participant is fulfil ing a social obligation (Singer & Couper, 2008). Survey-related reasons are aspects a participant is interested in, or they find the interviewer/researcher appealing (Singer & Couper, 2008), whereas egoistic reasons mean that the participant likes to participate or just participates for the money (Singer & Couper, 2008). Even though monetary benefits are often used and mentioned as an effective incentive for research participation, there are stil other approaches (e.g., curiosity, scientific advancement, and desire for medical care/attention) to increase the wil ingness to share data and thus obtain sufficient participants for research (e.g., Permuth-Wey and Borenstein 2009). M. Mandl, C. Malin, S. Thalmann: It’s a Match! – Finding the Perfect Participants for Health Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 551 Since various factors, i.e., attitudes, might contribute to a variability in decisions regarding private data (Woldaregay et al., 2020). In this article, we investigate how different attitudes influence the wil ingness to share health (WILL). We assume that different attitudes influence people’s decisions regarding sharing their health data, since they seem to be essential for health research, facilitating informed decision- making, fostering participation, and ensuring the advancement of medical research for societal benefit (Colombo et al., 2019; McCormack et al., 2016). Those three are attitudes towards technology (ATTT), attitudes towards altruism (ATTA), and attitudes towards science (ATTS). ATTT: As technology drives growth and economic development, the impact of technology appears to have a heavy impact on society (R. Williams & Edge, 1996). The ATTT can be defined as an individual’s positive or negative evaluation of the introduction of new kinds of technology in an individual’s life (Elias et al., 2012). Kim and Choi (2019) concluded that individuals with a positive ATTT are more willing to share their socio-economic and health data with hospitals and researchers. Also, individuals with experience with information technology are more wil ing to share their health data (Naeem et al., 2022). Since this attitude seems to affect an individual’s mindset, it can be assumed that the ATTT might also affect the decision to share health data with RRPs. Thus, we formulated the following hypothesis: H1: The higher the attitude towards technology, the higher the wil ingness to share health data. ATTA: Altruism represents one of the main concepts when sharing data. In the context of a health-related study, altruism means that the research furthers some purpose that is important to the participant, or the participant is fulfil ing a social obligation (Singer & Couper, 2008). According to Manzur & Olavarrieta (2021), there exist several studies about the individual differences in altruistic behavior with different groups of people (adults, children, different countries, etc.). Studies suggest that although willingness to join multi-user data networks is low, altruism significantly predicts participation in such networks (Raj et al., 2020), reinforcing the notion that ATTA influence willingness of sharing data. Given this background, we assume that the altruistic attitude of a person influences the wil ingness to share health data positively: H2: The higher the attitude towards altruism, the higher the wil ingness to share health data. 552 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION ATTS is composed of six dimensions that affect behaviors in science: “[…] attitude toward scientists, scientific enquiry, science learning, science-related activities, science careers, and the adoption of ‘scientific attitudes’” (Mao et al., 2021). Prior research, such as Jamal et al (2014), has shown that high trust in researchers influences the willingness to share personal data positively, since participants feel engaged with researchers and valued. Therefore, a positive ATTS is an indicator of a high wil ingness to share data. Given this background, we assume the following hypothesis: H3: The higher the attitude towards science, the higher the wil ingness to share health data. In addition to ATTS, research on experience of clinical trial participation indicates that as soon as individuals have participated in health-related studies, they gain more trust in such study designs (Ohmann & Deimling, 2004). The concept and building of trust can be explained by the easy, accessible, and feasible sharing of health data (Naeem et al., 2022). Several studies indicate that the higher the level of trust, the more likely individuals will be willing to share data in the future (Naeem et al., 2022). Thus, we investigate the influence of earlier participation and formulated the following hypothesis: H4: Earlier participation in studies increases willingness to share data. Figure 2: Hypothesis Model Source: Own Not only the fundamental wil ingness to share data (WILL) is assumed to be influenced by attitudes. According to Naeem et al. (2022), the category of health information influences the wil ingness to share data. Hirst et al. (2023)‘s study showed differences between the wil ingness to share different types of data, while Woldaregay et al . (2020) concluded that health data sharing dependents on the type of data. Therefore, we assume that the data categories influence the wil ingness to M. Mandl, C. Malin, S. Thalmann: It’s a Match! – Finding the Perfect Participants for Health Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 553 share health data in a moderating way: H5: The data category influences the wil ingness to share. 3 Method and Procedure We conduct a two-stage study consisting of two focus groups and an online survey to investigate the wil ingness to share health data. Stage 1 – Workshops: Data categories and incentives for sharing health data with RRPs were identified in two separate workshops in March 2023 for participants who were registered at an RRP (n=7) and those who were not (n=6). Both workshops were designed equally and consisted of brainstorming, clustering, prioritization, and an in-depth discussion regarding the data categories and incentives. Both workshops were audio-recorded, transcribed, and analyzed using the thematic analysis by Braun and Clarke (2006). Stage 2 – Survey: Building on the data categories identified in the workshops, we conducted an online survey via LimeSurvey to test the hypotheses. We recruited our participants via the university distribution mail for student surveys, via social media platforms, and via NORSTAT. A total of 2,966 persons participated in the survey, whereby 1,743 participants had to be excluded due to incomplete or insufficient datasets. Thus, our final sample consisted of 1,223 participants. The age distribution of participants varied: 5.40% were aged 19 or younger, 18.32% were between 20 and 29 years old, 17.01% fell within the 30-39 age bracket, 15.37% were aged 40-49, 17.58% were between 50-59, 13.00% were in the 60-69 range, and 12.10% were 70 or older. Regarding gender distribution, 53.36% identified as female, 46.07% as male, and 0.41% as diverse; 3 participants did not specify. Educational backgrounds varied, with 33.14% completing middle or high school, 31.75% completing apprenticeships, and 22.50% holding university degrees. Employment status was diverse: 43.4% were employed full-time, 24.8% part-time, 12.4% marginally employed, 1.8% unemployed, 5.3% self-employed, 7.1% retired, and 5.3% unable to work. Notably, 3.19% did not understand the matching process. In terms of research experience, 41.09% had participated in research studies before, while 58.91% had not. 554 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Procedures and data colection: The online survey was structured the folowing way: The matching process was explained using visualizations and participants were asked whether it had been understood. If it was not understood, only the demographic information was requested. If the participants understood the process, the wil ingness to share data (WILL) was asked, and which clients were excluded. Next, we asked for each data group (e.g., health status) how easy it is for them to share a certain data group with an RRP and how they assess which motives influence their motivation and to what extent. Motives extracted from the literature and discussed in the workshops were questioned: monetary compensation, general interest, medical progress, and individual treatment. Finally, attitudes towards three topics were queried, i.e., technology, altruism, and science. The questionnaire was pre- tested by six people with different backgrounds (business analytics, languages, business administration). The survey was released on May 17, 2023, and ended on August 17, 2023. Measures: All items were answered on a Likert scale ranging from (1) “Ful y agree” to (5) “Do not agree at al ”. ATTT measured the average attitude of a person towards technology using the 10-item construct by Edison and Geissler (2003). ATTA measured an individual’s attitude towards altruism and sociality using the 9-item construct from Manzur and Olavarrieta (2021). ATTS measured participants’ attitude towards science using the 21-item construct by Akkuş (2019). Willingness to share data (WILL) measured the preferences of participants regarding sharing personal health data. 4 Results We identified five data categories relevant to a wel -performing matching process of RRPs: (1) health status, (2) attested diseases, (3) current medication, (4) mental health, and (5) lifestyle habits. Our findings show that the wil ingness to share data differs depending on the data category. Health status: Health status describes a person’s medical conditions, health care, medical history, as well as genetic information (e.g., World Health Organization, 2023). Overal , sharing data about the category health status with RRPs seems not to be a big deal for the individuals, “I would disclose the health data immediately, I don’t need anything in return. It would be cool to receive a report of the results directly […].” (I10). M. Mandl, C. Malin, S. Thalmann: It’s a Match! – Finding the Perfect Participants for Health Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 555 However, participants presuppose that it should not be of great effort to share this kind of data; it should be time-saving to share it with RRPs. Here, the more time they have to spend sharing that data, the higher should be the remuneration; participants want to receive some reward in exchange for this data as one participant explained, “The longer [the data input] lasts, the more intense it is, the more ef ort there is behind it, [I would want to receive] some form of compensation.” (I6) At ested diseases: A disease is a condition of the living body or of its parts that impairs normal functioning and is typical y shown by distinguishing symptoms (Amzat & Razum, 2014). The participants subdivided this data category into two subgroups, namely mild and severe diseases. One can also interpret severe diseases as sensitive; diseases that are e.g., embarrassing for the participants. In principle, participants are easily willing to share data on attested diseases, however, their wil ingness decreases the more severe the disease is. Given this background, individuals would be willing to share mild diseases without any problems or incentives. In contrast, in the case of severe diseases, individuals are more likely to be unwilling to share these data, or they demand to receive some sort of monetary reimbursement or individual treatment, “[.. ] dif erentiate between dif erent diseases, i.e., serious diseases such as leukemia, cancer, and other diseases. I would easily give that away to speed up treatment or to advance research. And yes, because such minor il nesses and so on, that is simply visual impairment or something like that, then perhaps probably monetary compensation, but also to advance research.” (I9) Current medication: Regarding this data category, participants are concerned about the risk of data breach, resulting in a low willingness to share this type, “But that often changes so quickly. And then the question arises of how up to date this list is.” (I6). Due to their concerns regarding data protection, participants general y expect to receive monetary reimbursement for sharing information within this data category. However, some participants noted that they would share this data freely to advance research. Mental health: This category is alone standing as Amzat and Razum (2014) clearly distinguish mental health from physical health. In general, participants are reluctant to share information about their mental health. Their reluctance is reflected in a low wil ingness to share this data, as the following statement indicates, “And when it comes to mental health, I find that difficult because there’s a stigma here and it’s often not easy, so I would find that difficult.” (I4). Having these concerns in mind, participants seemed to need 556 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION to get a feeling that this personal data is being treated and processed confidentialy. They want to advance research and create sensitivity by sharing their data. Lifestyle habits: This category covers personal interests and preferences, which are directly connected to a person’s health status, e.g., attitudes towards alcohol and smoking, drug use, sports activities, or nutrition (Mozaffarian et al., 2008). Individuals are willing to share information about their lifestyle habits without any problems, and any kind of remuneration or incentive. However, some of the participants could imagine receiving little monetary reimbursements, such as coupons or discounts, “Lifestyle habits – I have no problem at al .” (I3) Overal , it seems that the participants either want to advance research in return for their data sharing or they want to receive monetary benefits for doing so. Table 1 displays an overview of the identified data categories as wel as the respective incentives to increase individuals` wil ingness to share data belonging to one of the respective data categories. Descriptive results: Table 1 shows means and medians for al variables. In general, the participants showed a high willingness to share data. Of the final sample, 44.51% ful y agreed and 43.83% agreed to share data. Only a smal percentage disagreed or strongly disagreed, with 1.44% indicating they did not agree at all. Besides a high willingness to share data, participants showed a positive ATTT, ATTA, and ATTS. Here, mean values range from 2,988 for ATTA to 3,798 for ATTS. Concretely, individuals have a moderately positive ATTT, as indicated by a mean of 3.686. The median of 3.800 shows that half of the participants fal below this score, and the other half above it. Similar to ATTT, participants also hold a moderately positive outlook on ATTA (mean value=2.988). The median of 2.889 signifies that half of the respondents have attitudes towards altruism lower than this score, and the other half above it. In contrast, ATTS appears quite positive with a mean score of 3.798. The median of 3.810, indicates that half of the participants have an attitude towards science lower than this score, while the other half have attitudes above it. M. Mandl, C. Malin, S. Thalmann: It’s a Match! – Finding the Perfect Participants for Health Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 557 Table 1: Descriptive Results ATTT ATTA ATTS WILL Mean 3.686 2.988 3.798 4.274 Median 3.800 2.889 3.810 4.000 Correlation analysis: We conducted a correlation analysis using Pearson correlation tests to test our hypotheses that, a higher attitude towards technology (H1), a higher attitude towards altruism (H2), and a higher attitude towards science (H3) lead to an increased willingness to share health data. In addition, we also investigated the willingness to share different data categories (H5). These tests assume that the variables are normally distributed; this is significantly ensured with the col ected data. Additional y, we conducted a Mann-Whitney-U-test to investigate whether earlier participation in studies influences the willingness to share data (H4). Table 2 summarizes the correlations and the corresponding p-values of the hypothesis tests. The correlation value between ATTT and WILL is 0.216903, which shows a weak but still positive correlation. The p-value indicated in Table 2 is much lower than the assumed significance level of 0.05. The results suggest that a positive ATTT is associated with an increased WILL. Therefore, our results support H1. The correlation between ATTA and WILL is calculated as 0.17155. As expected, this value is positive but quite weak. Thus, as previously assumed, the altruistic behavior of individuals represents an important and influences the wil ingness to share private health data. Our results support H2. The correlation between ATTS and WILL is 0.200791, indicating a small positive correlation. As a result, the attitude towards science positively influences the wil ingness to share data. Thus, H3 is supported. The mean of people who have already participated (ALPA) in clinical trials is 4.35, and the mean of people with no experience is 4.22. This significant (p-value 0.02394) result indicates that individuals with experience are more willing to share their health data with an RRP compared to individuals with no experience. It can be assumed that WILL is significantly higher in experienced than in non-experienced participants. Thus, the results support H4. Our findings show that ATTT positively influences WILL but also the data categories’ willingness to share certain data. We observed only minor differences between the data categories. Here, only WILL about current medication is clearly smaller than the other correlations. Similar to ATTT, ATTA also positively influences WILL. However, it less influences different 558 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION data categories (see Table 2). Individuals’ ATTS also positively influences WILL. It is noticeable that the correlation between ATTS and the wil ingness to share lifestyle habits is quite high. The positive correlations of all variables (see Table 2) support H5. Table 2: Correlation Matrix towards wil ingness to share data (WILL) WILL WILL WILL WILL WILL WILL Health S. Current M. Attested D. Mental H. Lifestyle H. 0.216903 0.1748014 0.1670655 0.2120066 0.1802049 0.2122917 (0.000000 ATTT (0.0000000 (0.00000000 (0.00000000 (0.0000000 (0.00000000 00000002 007013) 3664) 000008452) 002114) 000007833) 251) 0.17155 0.1485261 0.1353442 0.1612933 0.1655567 0.1539476 ATTA (0.000000 (0.0000001 (0.00000147 (0.00000001 (0.0000000 (0.00000005 001417) 42) 9) 198) 05014) 092) 0.200791 0.2200518 0.2375699 0.1965512 0.1939512 0.3143476 (0.000000 (0.0000000 ATTS (0.00000000 (0.00000000 (0.0000000 (0.00000000 00000155 000000094 000000022) 0004467) 00008443) 000000022) 2) 51) 5 Discussion and Outlook We investigated the wil ingness of people to share health data needed for the matching process of RRPs. By doing so, we conducted a two-stage study that consisted of focus group workshops and an online survey to explore data categories and incentives for sharing health data with RRPs. This study makes two key contributions to both practice and theory: First, we contribute to the research on willingness to share data by identifying and validating those data categories relevant for sharing data with RRPs. We identified five data categories influencing the willingness to share: health status, attested diseases, current medication, mental health, and lifestyle habits based on the literature. Here, our findings indicate that participants’ wil ingness to share data could be influenced by different incentives such as monetary reimbursement or altruism. Our findings are consistent with previous studies such as Woldaregay et al. (2020) who highlight that participants’ wil ingness to share their diseases depends on different attitudes. Participants expressed a higher wil ingness to share mild diseases compared to severe ones, with M. Mandl, C. Malin, S. Thalmann: It’s a Match! – Finding the Perfect Participants for Health Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 559 a preference for rewards or individual treatment in the case of more serious diseases - concerns about data breaches and the changing status of current medication led to a lower wil ingness to share this category, often entailing monetary reimbursement. Mental health data raised reluctance, highlighting the need for confidentiality and a sense of trust. Lifestyle habits, on the other hand, were readily shareable without specific incentives. These results confirm that there are differences in the wil ingness to share data for different data categories shown in traditional settings (e.g., Hirst et al. 2023). Second, we contribute to the understanding of factors influencing data sharing by demonstrating the effects of various attitudes on the wil ingness to share data in the specific context of RRPs. Our study reveals that a positive attitude towards technology increases the wil ingness to share data. Thus, we show that this relationship holds also in the context of RRP. Kim and Choi (2019), which showed its impact on society and thus the individual mindsets about sharing private health data with research portals. Similarly, Broekstra et al. (2020), for example, highlighted that individuals consider participation in research to promote social progress and therefore see the sharing of health data as a donation. Moreover, our findings support Jamal et al.’s (2014) assumption that a positive attitude towards science is correlated with an increased wil ingness to share personal data for the context of RRPs. Like Naeem et al. (2022), our study revealed that participants with previous research experiences were found to be more wil ing to share their health data than those without. From a practical perspective, RRPs now can build upon these results and differentiate their data collection. Understanding the wil ingness to share health data across categories al ows for the development of targeted strategies, ensuring more effective participant recruitment for health studies and clinical trials. Such a data- driven approach, as this is with a customized data collection combined with the identified incentives, seems very promising (Huang et al., 2018). Current research (e.g. Woldaregay et al. 2020) highlighted the need for strict privacy measures in traditional study context. Participants also found such concerns about data protection, especial y regarding the data categories of current medication and mental health data, which is consistent with Woldaregay et al.’s (2020) results, emphasize the need for robust privacy measures in the context of RRPs. Thus, the focus on implementing and communicating strict data protection protocols to build trust among potential participants seems a key success factor for RRPs. 560 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Our study has one major limitation: All participants came from Austria. Nevertheless, the participants were of varying professional and cultural backgrounds and age ranges. It might be interesting to build on our results and research in other countries and world regions. Furthermore, our findings can serve as a starting point for future research on the wil ingness to share health data. By including perspectives from individuals who avoid data sharing, future studies could offer a fuller understanding of participation barriers. We encourage researchers to conduct longitudinal studies to explore how attitudes towards technology, altruism, and science evolve over time could provide a deeper understanding of the factors influencing individuals’ wil ingness to share health data. This could contribute to the development of more dynamic and adaptive participant recruitment strategies. Acknowledgments Funding from Probando and FFG is grateful y acknowledged. References Akkuş, A. (2019). “Developing a Scale to Measure Students’ Attitudes toward Science”, International Journal of Assessment Tools in Education, 6, 706–720. Amzat, J. and Razum, O. (2014). “Health, Disease, and Il ness as Conceptual Tools”, Medical Sociology in Africa, 21–37. Borg, D. J., Haritopoulou-Sinanidou, M., Gabrovska, P., Tseng, H.-W., Honeyman, D., Schweitzer, D., & Rae, K. M. (2024). Barriers and facilitators for recruiting and retaining male participants into longitudinal health research: A systematic review. BMC Medical Research Methodology, 24(1), 46. Braun, V. and Clarke, V. (2006). “Using thematic analysis in psychology”, Qualitative Research in Psychology, 3, 77–101. Broekstra, R. et al. (2020). “Motives of contributing personal data for health research: (non- )participation in a Dutch biobank”, BMC Medical Ethics, 21, 62. Capili, B. (2021). Selection of the Study Participants. The American Journal of Nursing, 121(1), 64. Chaudhari, N., Ravi, R., Gogtay, N. J., & Thatte, U. M. (2020). Recruitment and retention of the participants in clinical trials: Challenges and solutions. Perspectives in Clinical Research, 11(2), 64– 69. Colombo, C., Roberto, A., Krleza-Jeric, K., Parmelli, E., & Banzi, R. (2019). Sharing individual participant data from clinical studies: A cross-sectional online survey among Italian patient and citizen groups. BMJ Open, 9(2), e024863. Edison, S.W. and Geissler, G.L. (2003). “Measuring attitudes towards general technology: Antecedents, hypotheses and scale development”, Journal of Targeting, Measurement and Analysis for Marketing, 12(2), 137–156. Elias, S.M., Smith, W.L. and Barney, C.E. (2012). “Age as a moderator of attitude towards technology in the workplace: work motivation and overal job satisfaction”, Behaviour & Information Technology, 31(5), 453–467. Evans, C.H. and Ildstad, S.T. (eds) (2001). Smal Clinical Trials: Issues and Chal enges. Washington (DC): National Academies Press (US). M. Mandl, C. Malin, S. Thalmann: It’s a Match! – Finding the Perfect Participants for Health Research. Study on the Wil ingness to Share Health Data with a Research Referral Portal 561 Fogel, D.B. (2018). “Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: A review”, Contemporary Clinical Trials Communications, 11, 156– 164. Gul, R.B. and Ali, P.A. (2010). “Clinical trials: the challenge of recruitment and retention of participants”, Journal of Clinical Nursing, 19(1–2), 227–233. Hirst, Y. et al. (2023). “Understanding Public Attitudes and Willingness to Share Commercial Data for Health Research: Survey Study in the United Kingdom”, JMIR Public Health and Surveillance, 9(1), e40814. Huang, G.D. et al. (2018). “Clinical trials recruitment planning: A proposed framework from the Clinical Trials Transformation Initiative”, Contemporary Clinical Trials, 66, 74–79. Inan, O. T., Tenaerts, P., Prindivil e, S. A., Reynolds, H. R., Dizon, D. S., Cooper-Arnold, K., Turakhia, M., Pletcher, M. J., Preston, K. L., Krumholz, H. M., Marlin, B. M., Mandl, K. D., Klasnja, P., Spring, B., Iturriaga, E., Campo, R., Desvigne-Nickens, P., Rosenberg, Y., Steinhubl, S. R., & Califf, R. M. (2020). Digitizing clinical trials. Npj Digital Medicine, 3(1), 1– 7. Integrate.io (2023). What is Data Matching? URL: https://www.integrate.io/glossary/what-is-data-matching/ (visited on November 9, 2023). Jamal, L. et al. (2014). “Research participants’ attitudes towards the confidentiality of genomic sequence information”, European Journal of Human Genetics, 22(8), 964–968. Kelly, W. K., & Halabi, S. (2018). Oncology Clinical Trials: Successful Design, Conduct, and Analysis. Springer Publishing Company. Kerr, B., Godfrey-Smith, P., & Feldman, M. W. (2004). What is altruism? Trends in Ecology & Evolution, 19(3), 135–140. Kim, T.K. and Choi, M. (2019). “Older adults’ wil ingness to share their personal and health information when adopting healthcare technology and services”, International Journal of Medical Informatics, 126, 86–94. Lamanna, M.A. (1997). “Giving and getting: altruism and exchange in transplantation”, The Journal of Medical Humanities, 18(3), 169–192. Manzur, E. and Olavarrieta, S. (2021). “The 9-SRA Scale: A Simplified 9-Items Version of the SRA Scale to Assess Altruism”, Sustainability, 13(13), 6999. Mao, P. et al. (2021). “The Relationship Between Attitude Toward Science and Academic Achievement in Science: A Three-Level Meta-Analysis”, Frontiers in Psychology, 12. McCormack, P., Kole, A., Gainotti, S., Mascalzoni, D., Molster, C., Lochmül er, H., & Woods, S. (2016). ‘You should at least ask’. The expectations, hopes and fears of rare disease patients on large-scale data and biomaterial sharing for genomics research. European Journal of Human Genetics, 24(10), Article 10. Mozaffarian, D., Wilson, P.W.F. and Kannel, W.B. (2008). “Beyond established and novel risk factors: lifestyle risk factors for cardiovascular disease”, Circulation, 117(23), 3031–3038. Naeem, I. et al. (2022). “Factors Associated With Willingness to Share Health Information: Rapid Review”, JMIR Human Factors, 9(1), e20702. Ohmann, C. and Deimling, A. (2004). “Attitude towards clinical trials: results of a survey of persons interested in research”, Inflammation Research: Official Journal of the European Histamine Research Society . . [et Al.], 53 Suppl 2, S142-147. Permuth-Wey, J. and Borenstein, A.R. (2009). “Financial Remuneration for Clinical and Behavioral Research Participation: Ethical and Practical Considerations”, Annals of Epidemiology, 19(4), 280–285. Raj, M., Vries, R. D., Nong, P., Kardia, S. L. R., & Platt, J. E. (2020). Do people have an ethical obligation to share their health information? Comparing narratives of altruism and health information sharing in a national y representative sample. PLOS ONE, 15(12), e0244767. Seltzer, E. et al. (2019). “Patients’ wil ingness to share digital health and non-health data for research: a cross-sectional study”, BMC Medical Informatics and Decision Making, 19(1), p. 157. 562 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Singer, E., and Couper, M.P. (2008). “Do incentives exert undue influence on survey participation? Experimental evidence”, Journal of empirical research on human research ethics: JERHRE, 3(3), 49–56. Song, X., Mehrotra, M., & Rajapakshe, T. (2023). An Analysis of Incentive Schemes for Participant Retention in Clinical Studies. Manufacturing & Service Operations Management, 25(3), 1033–1050. Su, Q., Cheng, G., & Huang, J. (2023). A review of research on eligibility criteria for clinical trials. Clinical and Experimental Medicine, 23(6), 1867–1879. Tishler, C.L. and Bartholomae, S. (2002). “The recruitment of normal healthy volunteers: a review of the literature on the use of financial incentives”, Journal of Clinical Pharmacology, 42(4), pp. 365–375. Williams, R. and Edge, D. (1996). “The Social Shaping of Technology”, Research Policy, 25, 865– 899. Woldaregay, A.Z. et al. (2020). “User Expectations and Willingness to Share Self-Collected Health Data”, Stud Health Technol Inform, (270), 894–898. World Health Organization (2023). Constitution of the World Health Organization. URL: https://www.who.int/about/accountability/governance/constitution (visited on November 17, 2023). EVALUATING COVID-19 MOVEMENT RESTRICTIONS: A FINNISH CASE STUDY VALTTERI WANNE,1 XIAOLU WANG,2 ANNA SELL2 1 Azets, Åbo Akademi University, Faculty of Social Sciences, Business and Economics, and Law, Turku, Finland valtteri.wanne@gmail.com 2 Åbo Akademi University, Faculty of Social Sciences, Business and Economics, and Law, Turku, Finland xiaolu.wang@abo.fi, anna.sel @abo.fi A case study has been conducted to substantiate the effects of the lockdown imposed on the Uusimaa area in Finland. In particular, the case study aims to find out if the Uusimaa lockdown has mitigated the spread of the coronavirus from Uusimaa to other parts of Finland. The study shows that after the lockdown was imposed, it took approximately two weeks Keywords: before the daily count of new COVID-19 cases outside the Covid-19, Uusimaa area reached its peak and started to decrease. The movement phenomenon aligns with the findings of relevant studies that a restrictions, non- peak in the curve of diagnosed cases of infection occurs 14 to 18 pharmaceutical days from lockdown. It also emerges that the lifting of the interventions, lockdown on Uusimaa caused immediate outflowing mobility visual analytics, but did not result in an increased number of new cases in other machine parts of Finland. learning DOI https://doi.org/10.18690/um.fov.4.2024.33 ISBN 978-961-286-871-0 564 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction In early 2020, the world faced the COVID-19 pandemic caused by the SARS-CoV- 2 virus. This unprecedented health crisis prompted nations to impose travel restrictions and other public health measures, including mandatory mask-wearing, the closure of non-essential businesses, and restrictions on public gatherings. Alongside these measures, many countries implemented national or regional restrictions on movement. These restrictions raised concerns about their impact on personal freedoms and the necessity for substantial research to assess their effectiveness in reducing the transmission of COVID-19. This research is crucial given that movement-based disease control represents a relatively new approach in modern public health strategies. In Finland, during the early stage of the pandemic in 2020, a movement restriction in the form of a regional lockdown was introduced. Finland’s most populated region Uusimaa was put under a three-week lockdown from March 28 to April 15. Such a radical measure raises the question of its efficacy, what was the effect of this lockdown in mitigating the spread of the virus? Our research examines the extent to which the Uusimaa region lockdown was effective in attenuating the propagation of the coronavirus from the Uusimaa region to other regions within Finland. This study seeks to offer critical understanding that may inform responses to future infectious disease outbreaks, as no similar research on this specific subject has been conducted in Finland, making this study a pioneering contribution to the field. We test the hypothesis that mobility is linked to the number of new COVID-19 cases. The main research question is: ● Is there a relationship between change in mobility and the number of new cases? 2 Literature review Direct evidence of the effectiveness of movement restrictions in mitigating the spread of COVID-19 in Finland is limited. Consequently, we commence with an analysis of outcomes from nations that have enforced comparable non- pharmaceutical interventions (NPIs). The scope of this literature review is confined V. Wanne, X. Wang, A. Sel : Evaluating Covid-19 Movement Restrictions: A Finnish Case Study 565 to examining the impact of NPIs instituted in the initial phase of the pandemic, aligning with the temporal context of our case study in Finland. 2.1 Global Perspective on NPI Effectiveness We first examine a global perspective on the implementation and outcomes of various governmental interventions. Haug et al. (2020) applied four different regression techniques (namely case-control analysis, step function Lasso regression, random forest regression, and transformers modeling) to quantify the impact of 46 NPIs implemented in the first infection wave in 79 territories (Desvars-Larrive et al., 2020). To be more specific, they investigated their effects on the reduction in the effective reproduction number, Rt, which is an essential epidemiological quantity that represents the average number of infections generated at time t by each infected case throughout their infection. The results suggested that no individual NPI had been successful at reducing Rt to values below 1. Nevertheless, by combining the results yielded by the above methods, a set of six NPIs which al four methods show significant results for was identified. Small gathering cancellations (estimated ∆Rt ranging from -0.22 to -0.35), the closure of educational institutions (estimated ∆Rt ranging from -0.15 to -0.21), and border restrictions (estimated ∆Rt ranging from - 0.057 to -0.23) were the three most effective NPIs, while cordon sanitaire was found to have a significantly positive effect on reducing ∆Rt (estimated ∆Rt being around -0.09) by only random forest regression and transformers modeling. Hsiang et al. (2020) investigated the direct health benefits of NPIs, which were deployed by the governments of China, South Korea, Italy, Iran, France, and the United States. The dataset commences with the first travel ban imposed in Wuhan, China, on January 23, 2020, and encompasses al subsequent restrictive anti- contagion policies adopted in the aforementioned countries up to April 6, 2020, together with the corresponding daily infection rates. They applied panel regression models to estimate how the daily growth rate of infections changed over time within a location when different combinations of NPIs were deployed. They estimated that, in the absence of NPIs, early infection rates of COVID-19 would have grown 43% per day on average across the subject countries, which corresponds to a staggering doubling time of two days. However, once al anti-contagion policies were implemented, their estimated combined effect would reduce the daily growth rate of 566 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION infections by a substantial and statistically significant amount. Among all the NPIs, travel bans, and social distancing were found to be the most effective, which were expected to lower the daily growth rate by 28% and 22%, respectively. Flaxman et al. (2020) analyzed the impact of five types of governmental interventions across 11 European countries for the period from February to May 2020, and they were also interested in assessing their effectiveness in reducing the reproduction number Rt. Unlike the work mentioned above, a Bayesian inference framework, which involves back-calculating from observed deaths to infer the total population infected and the subsequent impact of interventions, was utilized in this study. It concluded that interventions were effective in reducing the reproduction number Rt below 1 in al countries considered, hence containing the epidemic. Among the five governmental interventions included in this study, only the effectiveness of lockdown was identifiable, and it was estimated to have led to a remarkable reduction of 81% in Rt if the model was fitted to the pooled data from all 11 European countries. In conducting our literature review, we also endeavored to incorporate research focused on the Nordic region due to their shared geographical, political, and social characteristics. Our emphasis was primarily on Finland, Norway, and Denmark, which was motivated by these countries' implementation of comparatively stringent restrictions on population mobility. Banholzer et al. (2021) proposed a model that linked two unobserved quantities (i.e., the daily number of contagious subjects and the daily number of new infections) to an observed quantity (i.e., the number of reported new cases). They assessed the impact of seven NPIs on curbing the number of new infections, deduced from the number of reported new cases through a semi- mechanistic Bayesian hierarchical model. This research spanned 20 nations, including Finland, Norway, and Denmark. The findings revealed that prohibitions on large gatherings were the most effective, associated with a 37% reduction in new infections. This was fol owed by venue closures at 18% and school closures at 17%. Despite these findings, the study did not individually exhibit the effects of these interventions in any of these countries. As we progress, it is crucial to delve into the impact of NPIs on an individual country level. We first turn our attention to China, the epicenter where the outbreak began, to understand the genesis of the containment strategies that would later sweep across V. Wanne, X. Wang, A. Sel : Evaluating Covid-19 Movement Restrictions: A Finnish Case Study 567 the globe. Additionally, a review of Italy's experience is imperative as it represents the first European nation to confront the pandemic head-on with extensive lockdown measures. Incorporating studies specifically from the Nordic countries would have added valuable insights; however, our search did not uncover any research exclusively focused on these nations. Utilizing real-time human mobility data from Wuhan and epidemiological data from other provinces, including travel histories, Kraemer et al. (2020) aimed to elucidate whether epidemics outside of Wuhan could be predicted by the volume of human movement out of Wuhan and to evaluate the efficacy of the cordon sanitaire. For these purposes, they built three generalized linear models (namely Poisson regression, negative Binomial, and log-linear regression) of daily case counts. The models suggested that the volume of human movement out of Wuhan alone was wel predictive of the magnitude of the early epidemic outside of Wuhan. However, the correlation decreased after February 1, 2020, corresponding to one mean plus one standard deviation of the incubation period after the interventions were implemented. Quilty et al. (2020) analyzed the impact of Wuhan's travel restrictions on slowing the virus's spread. They revealed that the cordon sanitaire, which was put into place on January 23, 2020, significantly reduced Wuhan's outflow by 92.7%. Despite this, local transmissions in major cities like Beijing, Chongqing, Hangzhou, and Shenzhen likely began well before the restrictions, diminishing the measure's overall effectiveness in altering the spread of the infection. They also modeled a scenario in which no cordon sanitaire was implemented. The simulation suggested that the cordon sanitaire's influence was negligible in these larger cities due to pre-existing high infection rates. However, Quilty et al. argued that the cordon sanitaire demonstrated a more pronounced effect in smal er cities. They concluded by suggesting other stringent NPIs had a greater effect on the development of COVID- 19 than travel restrictions (Quilty et al., 2020). Santamaria et al. (2020) conducted a study about movement restrictions and the effect of COVID-19 in Europe. In the study, they developed a mobility indicator to evaluate the effects of lockdowns on movement across European countries, focusing on Italy's response to COVID-19. Utilizing anonymized mobile data, the researchers crafted an origin-destination matrix to track mobility trends. Their 568 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION analysis revealed significant mobility reductions in countries with stringent lockdowns, such as Spain, Italy, and France, compared to those with milder restrictions. The study also examined the relationship between mobility and the effective reproduction number Rt in Italy, finding a strong correlation in the early lockdown phase. This association weakened over time, suggesting that increased public awareness and compliance to precautionary measures mitigated the potential rise in Rt despite the increase in mobility. This indicated that while mobility restrictions effectively reduced virus transmission initially, long-term outcomes also depended on public behavior and awareness. 2.2 Finland's NPI Approach Regarding the impact of movement restrictions in Finland, research is sparse. Nevertheless, adjacent studies provided insight into this area. For instance, Hakola- Uusitalo et al. (2020) utilized Google's mobility trend data to examine the effects of early interventions against COVID-19, comparing Finland's mobility with other Nordic countries. The study found that mobility, especial y to retail and recreation venues, workplaces, and public transport, significantly decreased fol owing the first recommendations on March 12, 2020. However, mobility in parks increased, suggesting a shift towards outdoor activities due to indoor restrictions. Interestingly, the study also indicated that the Uusimaa lockdown did not have a significant impact on mobility. The second study, by Willberg et al. (2021), focused on the movement between urban and rural areas, highlighting the Finnish tendency to retreat to secondary homes during the pandemic's first wave. Despite government advice against such movements, data showed a significant population increase in rural municipalities with many holiday cottages, also indicating a substantial urban-to- rural mobility shift. 3 Methodology 3.1 Mobility and COVID-19 data To effectively examine the impact of movement restrictions on the spread of COVID-19, it is essential to analyze mobility data in conjunction with data on the virus's spread. Numerous methods exist for measuring mobility, with mobile network analysis being the most common. Yet, due to the unavailability of such data, V. Wanne, X. Wang, A. Sel : Evaluating Covid-19 Movement Restrictions: A Finnish Case Study 569 we opted to use traffic data as a practical alternative. The Finnish Transport Infrastructure Agency (FTIA) gathers road traffic data through more than 500 traffic monitoring systems (TMS) positioned across Finland's road network. These TMS units, consisting of electrical y conducting loops embedded in the pavement, record each vehicle that passes over them. This methodology aligns perfectly with the objectives of this case study, especial y since numerous TMS units are situated near the Uusimaa border, providing precise traffic flow measurements. For this case study, TMS data from six out of the seven major roads crossing the Uusimaa border were utilized. The specific TMS locations used in this study are depicted in Figure 1. For our analysis, we combined the traffic volumes from each station on a daily level. Figure 1: The border of Uusimaa and the TMS locations utilized in this study. Source: https://liikennetilanne.fintraffic.fi/kartta/ The monitoring of COVID-19's spread can be approached through various methods, such as observing excess mortality rates. However, the most widely used method, which is adopted in this case study, involves tracking daily new confirmed cases of COVID-19. The spread of COVID-19 was quantified using data provided by the Finnish Institute for Health and Welfare. Analysis of the daily case data was conducted across different regional groups. However, given that the primary goal of the Uusimaa lockdown was to prevent the virus's spread from Uusimaa to other areas, the most essential focus was on all regions outside of Uusimaa. To reveal underlying trends more accurately, both the traffic and COVID-19 data were smoothed using a five-day moving average. 570 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3.2 Visual and Statistical Analysis In our analysis, we first applied visual analytics, enhancing our ability to discern patterns and correlations within datasets. Visual analytics facilitates an exploration of the combined mobility and COVID-19 datasets, offering a dynamic approach to identify and visualize temporal trends and anomalies (Cui, 2020). This method supports an intuitive understanding of how mobility changes may relate to fluctuations in COVID-19 case numbers. To complement the insights gained from the visual analytics, we employed the Poisson regression model to analyze the impact of the lockdown and human mobility out of Uusimaa on the number of cases reported outside Uusimaa. Poisson regression was selected because it is a widely used general linear model for predicting non-negative integer values, or counts. We fitted a Poisson regression model to the abovementioned data on mobility and COVID-19 incidence, utilizing the following link function: log[E(Yt)] = β0 + β1Yt-7 + β2ωt + β3Xt-6 (1) where • Yt represents the daily number of new cases in other parts of Finland outside of the Uusimaa region on day t • Yt-7 represents the daily number of new cases in other parts of Finland outside of the Uusimaa region seven days prior to day t • ωt is a binary variable, which is 1 if the lockdown was in place on day t and 0 otherwise • Xt-6 represents the daily number (in thousands) of human mobility out of the Uusimaa region six days prior to day t. Our statistical analysis exploited the knowledge on the doubling time of the COVID- 19 epidemic, and the biological lag between virus transmission and the appearance of the first symptoms (i.e., the incubation period). Evidence suggests that, in its early stages, the epidemic doubled in size about every seven days (Li et al., 2020). The exact incubation period for COVID-19 remains unclear as original chain-of- infection data may not be ful y accessible. Nevertheless, multiple studies (Backer et V. Wanne, X. Wang, A. Sel : Evaluating Covid-19 Movement Restrictions: A Finnish Case Study 571 al., 2020; Men et al., 2023) exploiting confirmed cases in China in the early outbreak phase report a mean incubation period of six days. 4 Results 4.1 Visual Analytics of Mobility Data and COVID-19 Cases The study aimed to evaluate the impact of restrictions on COVID-19's spread beyond Uusimaa. Therefore, we focused exclusively on outflowing traffic from Uusimaa and COVID-19 cases outside of Uusimaa during the period of March 1, 2020, to May 31, 2020. Figure 2 shows the results for the traffic data, where the blue bar indicates the daily traffic volumes, and the black line indicates the five-day moving average. Also, key moments are highlighted: the first black dot marks the announcement of initial government recommendations on March 12, and the second dot represents March 17, when further COVID-related restrictions were imposed. The horizontal red line outlines the duration of the Uusimaa lockdown. Figure 2: Outflowing traffic from the Uusimaa region The analysis reveals that outbound mobility began to decline following the initial recommendations on March 12. Subsequent restrictions introduced on March 17 had no additional effect on the already decreasing outflow of traffic. A notable decrease in mobility occurred with the onset of the Uusimaa lockdown on March 28. Mobility levels remained substantially low until the lifting of the lockdown on April 15. 572 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION In Figure 3, the compilation of daily COVID-19 cases from hospital districts outside of Uusimaa is displayed as blue bars, with a five-day moving average depicted by the black line. The figure shows a very sharp increase in cases between March 17 and April 3. A few days after the peak, the number of daily cases decreased at a nearly similar rate as it increased. Figure 3: New confirmed cases of COVID-19 outside of Uusimaa 4.2 Analysis of Changes in Mobility and COVID-19 Cases Figure 4 il ustrates the relationship between mobility trends and confirmed cases of COVID-19, with the start and end dates of the Uusimaa lockdown outlined by two vertical black lines. The figure suggests a relationship between mobility and case numbers. It seems that a reduction in mobility to a certain threshold causes a subsequent decrease in COVID-19 cases, with a time lag before this impact becomes apparent. 4.3 Estimated Effects of Lockdown and Mobility The summary for the Poisson regression model fitted to estimate the daily number of new cases outside Uusimaa is shown in Table 1. These results indicate that al selected independent variables in (1) have a statistically significant relationship with the number of cases reported outside Uusimaa. Interestingly, the coefficient associated with lockdown_Yes_No[T.Yes] (ωt in (1)) is positive, which indicates that enforcing the Uusimaa lockdown will result in a 60.43% increase in the number of V. Wanne, X. Wang, A. Sel : Evaluating Covid-19 Movement Restrictions: A Finnish Case Study 573 new cases in other regions on average. The outcome, while initially seeming paradoxical, is in alignment with the empirical evidence delineated in Figure 4. This figure illustrates that the number of daily COVID-19 cases continued to climb following the initiation of the lockdown, and it wasn't until approximately two weeks later that a decline was observed. Figure 4: Outflowing mobility from Uusimaa and new COVID-19 cases. Table 3: Results for the Poisson regression model 5 Discussion To answer the research question “Is there a relationship between change in mobility and the number of new cases?” a case study was conducted. The findings indicate that there was a rising trend in daily COVID-19 cases leading up to the Uusimaa lockdown. Fol owing the implementation of the lockdown, there was a significant drop in traffic volume. Approximately two weeks elapsed before the peak in 574 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION COVID-19 cases was reached and soon after the peak, the trend started to decline. After the lockdown was lifted on April 15, traffic volumes quickly rebounded, nearly to pre-lockdown levels. Yet, the case study revealed no corresponding increase in COVID-19 cases. These results mirror previous research by Santamaria et al. (2020). Thus, it appears that the restrictions were able to reduce the spread of COVID-19 in the short term, despite the increased mobility after the lifting of movement restrictions. Santamaria et al. (2020) argued that this was due to increased compliance with preventive measures. Similarly, Haug et al. (2020) suggested in their study that no individual NPI managed to reduce Rt below 1. While the Uusimaa case study did not specifical y explore this factor, it's plausible that heightened public awareness and compliance played a role. Furthermore, it should be acknowledged that other restrictive measures were implemented before and during the Uusimaa lockdown and were not lifted at the end of the lockdown, which likely have had an impact. Even though the results of the Uusimaa case study aligned wel with the European and Scandinavian research, differences can be found in the study on China by Quilty et al. (2020). In China, the effect was minimal and only temporary (Quilty et al., 2020), while in Europe, the lifting of the lockdown did not result in an increase in cases. Potential explanations for this discrepancy are differences in the size of the population, culture, and demographics, to name a few. The study also examined traffic volumes by vehicle type, as TMSs' are capable of differentiating cars, buses, trucks, etc., which yielded intriguing insights, particularly regarding bus traffic. The volume started to decline around the time the first recommendations were announced and continued to decrease at a linear pace until the end of the lockdown. However, bus traffic showed minimal evidence of recovery after the lockdown ended, suggesting a continued reluctance among the public to use crowded modes of transportation. While this case study spans from March 1, 2020, to May 31, 2020, making long-term trends indeterminate, the enduring effects on bus traffic patterns present a compel ing subject for future research. It is essential to acknowledge several limitations associated with the Uusimaa case study. The sample size is relatively small when compared to similar studies. For instance, the incidence of COVID-19 cases in Finland was significantly lower than that reported for other countries discussed in the literature review. In addition, as V. Wanne, X. Wang, A. Sel : Evaluating Covid-19 Movement Restrictions: A Finnish Case Study 575 was mentioned earlier, to achieve a more accurate measurement of mobility, mobile phone positioning data could have been used, rather than relying solely on movement occurring on public roads, since it is impossible to ensure how many individuals are traveling in each vehicle. Although the Uusimaa case study has certain limitations, the results align closely with other relevant research, providing robust evidence in support of answering the research question. As noted earlier in this chapter, the exact cause of the differing effects of movement restrictions in European countries as opposed to China remains unidentified. Hence, further investigation is required to understand the underlying factors, be they geographical, demographic, or otherwise. 6 Conclusion The findings suggest that the imposed movement restrictions in Finland, as wel as in other European countries, effectively reduced human contact and successful y mitigated the spread of the virus. Moreover, the restrictions raised people's awareness of the ongoing situation, which led them to adopt other precautionary measures. These results suggest that mobility is a key element in reducing the spread of the virus in the initial phase of an outbreak, but its effect diminishes as other restriction measures are in place. References Backer, J. A., Klinkenberg, D., & Wal inga, J. (2020). Incubation period of 2019 novel coronavirus (2019-nCoV) infections among travellers from Wuhan, China, 20–28 January 2020. Eurosurveillance, 25(5), 2000062. https://doi.org/10.2807/1560-7917.ES.2020.25.5.2000062 Banholzer, N., Weenen, E. van, Lison, A., Cenedese, A., Seeliger, A., Kratzwald, B., Tschernutter, D., Sal es, J. P., Bottrighi, P., Lehtinen, S., Feuerriegel, S., & Vach, W. (2021). Estimating the effects of non-pharmaceutical interventions on the number of new infections with COVID-19 during the first epidemic wave. PLOS ONE, 16(6), e0252827. https://doi.org/10.1371/journal.pone.0252827 Cui, W. (2019). Visual Analytics: A Comprehensive Overview. IEEE Access, 7, 81555–81573. https://doi.org/10.1109/ACCESS.2019.2923736 Desvars-Larrive, A., Dervic, E., Haug, N., Niederkrotenthaler, T., Chen, J., Di Natale, A., Lasser, J., Gliga, D. S., Roux, A., Sorger, J., Chakraborty, A., Ten, A., Dervic, A., Pacheco, A., Jurczak, A., Cserjan, D., Lederhilger, D., Bulska, D., Berishaj, D., … Thurner, S. (2020). A structured open dataset of government interventions in response to COVID-19. Scientific Data, 7(1), Article 1. https://doi.org/10.1038/s41597-020-00609-9 576 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Duchene, S., Featherstone, L., Blasio, B. F. de, Holmes, E. C., Bohlin, J., & Pettersson, J. H.-O. (2021). The impact of public health interventions in the Nordic countries during the first year of SARS-CoV-2 transmission and evolution. Eurosurveillance, 26(44), 2001996. https://doi.org/10.2807/1560-7917.ES.2021.26.44.2001996 Flaxman, S., Mishra, S., Gandy, A., Unwin, H. J. T., Mel an, T. A., Coupland, H., Whittaker, C., Zhu, H., Berah, T., Eaton, J. W., Monod, M., Ghani, A. C., Donnel y, C. A., Riley, S., Vol mer, M. A. C., Ferguson, N. M., Okel , L. C., & Bhatt, S. (2020). Estimating the effects of non- pharmaceutical interventions on COVID-19 in Europe. Nature, 584(7820), Article 7820. https://doi.org/10.1038/s41586-020-2405-7 Hakola-Uusitalo, T., Heinonen, M., & Sieppi, A. (2020). Koronan ja rajoitustoimien vaikutukset liikkumiseen. KKV. https://www.kkv.fi/uploads/sites/2/2021/12/koronan-ja-rajoitustoimien-vaikutukset-liikkumiseen-2020.pdf Haug, N., Geyrhofer, L., Londei, A., Dervic, E., Desvars-Larrive, A., Loreto, V., Pinior, B., Thurner, S., & Klimek, P. (2020). Ranking the effectiveness of worldwide COVID-19 government interventions. Nature Human Behaviour, 4(12), Article 12. https://doi.org/10.1038/s41562- 020-01009-0 Hsiang, S., Al en, D., Annan-Phan, S., Bel , K., Bol iger, I., Chong, T., Druckenmil er, H., Huang, L. Y., Hultgren, A., Krasovich, E., Lau, P., Lee, J., Rolf, E., Tseng, J., & Wu, T. (2020). The effect of large-scale anti-contagion policies on the COVID-19 pandemic. Nature, 584(7820), Article 7820. https://doi.org/10.1038/s41586-020-2404-8 Kraemer, M. U. G., Yang, C.-H., Gutierrez, B., Wu, C.-H., Klein, B., Pigott, D. M., OPEN COVID-19 DATA WORKING GROUP, du Plessis, L., Faria, N. R., Li, R., Hanage, W. P., Brownstein, J. S., Layan, M., Vespignani, A., Tian, H., Dye, C., Pybus, O. G., & Scarpino, S. V. (2020). The effect of human mobility and control measures on the COVID-19 epidemic in China. Science, 368(6490), 493–497. https://doi.org/10.1126/science.abb4218 Li, Q., Guan, X., Wu, P., Wang, X., Zhou, L., Tong, Y., Ren, R., Leung, K. S. M., Lau, E. H. Y., Wong, J. Y., Xing, X., Xiang, N., Wu, Y., Li, C., Chen, Q., Li, D., Liu, T., Zhao, J., Liu, M., … Feng, Z. (2020). Early Transmission Dynamics in Wuhan, China, of Novel Coronavirus– Infected Pneumonia. New England Journal of Medicine, 382(13), 1199–1207. https://doi.org/10.1056/NEJMoa2001316 Men, K., Li, Y., Wang, X., Zhang, G., Hu, J., Gao, Y., Han, A., Liu, W., & Han, H. (2023). Estimate the incubation period of coronavirus 2019 (COVID-19). Computers in Biology and Medicine, 158, 106794. https://doi.org/10.1016/j.compbiomed.2023.106794 Quilty, B. J., Diamond, C., Liu, Y., Gibbs, H., Russel , T. W., Jarvis, C. I., Prem, K., Pearson, C. A. B., Clifford, S., Flasche, S., Emery, J. C., Auzenbergs, M., Davies, N., Nightingale, E. S., van Zandvoort, K., Jombart, T., Deol, A. K., Edmunds, W. J., Hel ewel , J., … CMMID COVID-19 working group. (2020). The effect of travel restrictions on the geographical spread of COVID-19 between large cities in China: A modelling study. BMC Medicine, 18(1), 259. https://doi.org/10.1186/s12916-020-01712-9 Santamaria, C., Sermi, F., Spyratos, S., Iacus, S. M., Annunziato, A., Tarchi, D., & Vespe, M. (2020). Measuring the impact of COVID-19 confinement measures on human mobility using mobile positioning data. A European regional analysis. Safety Science, 132, 104925. https://doi.org/10.1016/j.ssci.2020.104925 Wil berg, E., Järv, O., Väisänen, T., & Toivonen, T. (2021). Escaping from Cities during the COVID-19 Crisis: Using Mobile Phone Data to Trace Mobility in Finland. ISPRS International Journal of Geo-Information, 10(2), Article 2. https://doi.org/10.3390/ijgi10020103 TOWARDS TECHNO-PSYCHOLOGICAL IMMERSION: A NARRATIVE LITERATURE REVIEW OF IMMERSION AND ITS RELATED CONCEPTS TIINA PAANANEN,1 TIINA KEMPPAINEN,2 MARKUS MAKKONEN,3,1 MATILDA HOLKKOLA,1 OLLI TYRVÄINEN,1 LAURI FRANK1 1 University of Jyvaskyla, Faculty of Information Technology, Jyvaskyla, Finland ti na.e.paananen@jyu.fi, markus.v.makkonen@jyu.fi, matilda.i.holkkola@jyu.fi, olli.p.tyrvainen@jyu.fi, lauri.frank@jyu.fi 2 University of Jyvaskyla, School of Business and Economics, Jyvaskyla, Finland tikemppa@jyu.fi 3 Tampere University, Faculty of Management and Business, Tampere, Finland markus.makkonen@tuni.fi Immersion is re-trending interdisciplinary topic in academic research due to new digital innovations, such as augmented reality smart glasses. Immersion is, however, still criticized as being a vague concept which should be clarified. For instance, immersion can be analyzed from both psychological and technological perspectives, yet many studies solely focus on one aspect while neglecting the other. Moreover, technological immersion and immersive technologies can be incorrectly used as synonyms. Thus, in this study, we conducted a narrative Keywords: literature review of selected articles on immersion. As a result, immersion, technological we present our summary, which includes four sense stimulators immersion, (visual, auditory, haptic, and olfactory) of technological psychological immersion and three dimensions (spatial, narrative, and strategic) immersion, immersive of psychological immersion. Also, we suggest that immersive technologies, technologies should be separated from technological immersion. techno- Lastly, we propose a new perspective to immersion: techno- psychological immersion, psychological immersion, which combines technological and narrative literature psychological viewpoints of immersion. review DOI https://doi.org/10.18690/um.fov.4.2024.34 ISBN 978-961-286-871-0 578 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction New technology trends are estimated to emerge during the next few years (Pucihar, 2020), enabled by digital transformation (Zimmermann, 2016). One concept experiencing a resurgence in popularity is immersion. Previously, the concept of immersion has been popular particularly in digital games and gaming (Ermi & Mäyrä, 2005). Immersion means deep concentration and attention into something (e.g., a game), where a user relies on their instincts (Brown & Cairns, 2004) and gets closer to being immersed into a virtual world as it would present the real world – blurring boundaries between these two environments (Lee et al., 2013). In immersion, a user becomes part of the experience, physically or virtually (Ermi & Mäyrä, 2005), and immersion can be viewed as stimulating the user’s senses via technology – technological immersion or as a subjective experience – psychological immersion (Nilsson et al., 2016). The concept of immersion is used across diverse domains, from retail to digital services. Hudson et al. (2019) studied immersive shopping experiences in metaverse environments. Acikgoz and Tasci (2022) found brand immersion interesting in brand community contexts. Also, everyday digital services (e.g., Netflix) are immersing their users (Kemppainen & Paananen, 2024a), with some studies suggesting that this engagement can even resemble a form of relationship (Paananen et al., 2022). Thus, it should be investigated how technology can lead a user to immersion (Soliman et al., 2017) or what are the causes and attributes of psychological immersion (Agrawal et al., 2020). This information could be used to enhance desired customer behavior during the customer journey. The concept of immersion has been criticized to be widely used but unclear (Brown & Cairns, 2004), and it is still seen as a vague (Agrawal et al., 2020) and diverging (Sun & Botev, 2023) concept. The notion of immersion lacks a standard definition, with interpretations varying across different studies. Immersion can even be left undefined in research as in Tonteri et al.’s (2023) study. Hence, in this work, we aim to clarify the concept of immersion. Our research question is: how has immersion been conceptualized in research literature? We approach this question with a narrative literature review by reviewing both sides of the immersion concept. Previous recent literature reviews of immersion have focused on either technological immersion (e.g., Ambika et al., 2023; Queiroz et al., 2018; Suh & Prophet, 2018) or T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 579 psychological immersion (e.g., Agrawal et al., 2020; Nilsson et al., 2016), but not both sides simultaneously – and equally. The structure of this article is as follows: First, we introduce our methodological approach, fol owed by the literature review. Next, we present a summary of the main findings of the literature review and discuss these findings in more detail. Finally, the limitations and potential paths for future research conclude this work. 2 Methodological Approach A narrative literature review provides a summarized overview of selected articles related to the chosen phenomenon. This article seeks to understand immersion and investigates it from a technological and psychological perspective. Related concepts flow, presence, and transportation are included and compared to immersion because these are commonly reflected concepts within immersion (see e.g., Agrawal et al., 2020). This article follows the narrative literature review method by Cook et al. (1997). This method has been previously employed in immersion research in the study by Nilsson et al. (2016), whereas most previous literature reviews of immersion have been conducted either systematically (e.g., Ambika et al., 2023; Cummings & Bailenson, 2016; Queiroz et al., 2018; Suh & Prophet, 2018) or without a clearly described method, such as Agrawal et al.’s (2020) study. The reason for selecting the narrative method is that it provides a more extensive scope to phenomena than the systematic method, which focuses on certain specific questions (Cook et al., 1997). The goal of the narrative method is to summarize prior knowledge. However, it can lead to developing new theoretical perspectives (Paré et al., 2015), such as, by combining previous research into a map form, in other words, ‘a greater whole’ (Dijkers, 2009). The narrative method does not usual y provide a repeatable systematic review (Dijkers, 2009; Paré et al., 2015) but, instead, as a selective approach, it should provide a careful y considered selection of articles on the phenomena (Cook et al., 1997). We wanted to include immersion-related peer-reviewed studies written in English representing either the key studies on the topic or new studies introducing fresh perspectives. The content was sourced through exploration from Google Scholar and databases (e.g., AIS Electronic Library, MIS Quarterly Journal Achieve, Science Direct) and an examination of key studies pertaining to the topic. Key 580 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION studies were identified by the number of citations and their frequency in studies when familiarizing themselves with the topic. Furthermore, snowbal ing tact was utilized to identify newer sources or prominent works that were frequently referenced. We fol owed Webster’s and Watson’s (2002) advise to review outside the main field and, thus, included studies also from other fields than information systems (IS). Also, the narrative method gives the possibility to bring new perspectives to immersion research in IS. For example, the immersion-related studies in MIS Quarterly have focused on quantitative data (e.g., Agarwal & Karahanna, 2000; Lee et al., 2012; Nah et al., 2011; Saunders et al., 2011), and immersion is not the main research subject of these studies. This research offers a broader perspective, resulting in a deeper understanding of the phenomenon. Such insights can be valuable in immersion-related concepts like the metaverse, as noted in immersive VR study by Dincelli and Yayla (2022). 3 Immersion and Related Concepts 3.1 Immersion Merriam-Webster dictionary describes the verb immerse as ‘ to plunge into something that surrounds or covers’ or ‘ engross, absorb’. In turn, Murray (1997) describes immersion in her widely cited book on future cyberspace narratives as fol ows: ‘Immersion is a metaphorical term derived from the physical experience of being submerged in water. We seek the same feeling from a psychologically immersive experience that we do from a plunge in the ocean or swimming pool: the sensation of being surrounded by a completely other reality, as dif erent as water is from air, that takes over al of our attention, our whole perceptual apparatus. (Murray, 1997, p. 98)’ Moreover, Slater and Wilbur (1997) present a technology-related definition to immersion: ‘Immersion is a description of a technology and describes the extent to which the computer displays are capable of delivering an inclusive, extensive, surrounding, and vivid illusion of reality to the senses of a human participant. (Slater & Wilbur, 1997, p. 3)’ T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 581 As these two citations suggest, immersion can be separated into two major perspectives: technological immersion (representing the technology’s or system’s objective property) and psychological immersion (representing the individual’s psychological state) (Agrawal et al., 2020). 3.2 Technological Immersion Previous research indicates that studies emphasizing technological immersion primarily examine immersion from a technological standpoint. Sun and Botev (2023) describe technological immersion as a device’s capacity to match, represent, and mediate the environment as in the physical world. Ambika et al. (2023) explain the goal is to broaden the user’s reality and enable unprecedented experiences. In literature, technological immersion seems to be understood via different technologies and concentrates on two perspectives: ‘technology immersion’ and ‘immersive technologies’. Lee et al. (2013) describe immersive technology as a technology that blurs the boundaries between the physical world and the simulated or digital world, leading to a user’s sense of immersion, such as, when using an interactive real- time theatre. Daassi and Debbabi (2021) describe immersive technologies as multi- sensory digital environments extending or replacing the natural surroundings of a user. Thus, we separate technological immersion and immersive technologies into two different concepts. First, technological immersion concerns a technology- enabled immersion via sensory experiences. Second, immersive technologies represent the used technologies, such as augmented reality or virtual reality technologies. Immersive technologies are explained in more detail in the next section (3.3.2). There are different perspectives on the concept of technological immersion, and Sun and Botev (2023) have criticized the immersion concept as diverging in academic research. For example, Agrawal et al. (2020) distinguish system and/or technology from the definition of immersion and propose using Slater’s (2003) term ‘system immersion’ instead, which means technology or system as a facilitator of the experience. Slater (2009) also highlights that the system’s physical properties determine the level of immersion. In our work, the emphasis on technological immersion lies in the sensory experience enabled by technology. 582 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The core of technology immersion is the sensory experience where user’s senses are stimulated. In a prior research, Slater (2003) have identified different types of technological immersion, which are visual, auditory, haptic, and olfactory fidelity. Similarly, in the gaming context, immersion is understood to rely on the player’s own senses alongside involvement (Brown & Cairns, 2004). Following this, immersive virtual reality (IVR) has been defined as consisting of visual, auditory, and haptic displays and a tracking system (Slater, 2009). Agrawal et al. (2020) found one reason leading to immersion to be multisensory simulation, even though their paper had more of a psychological immersion perspective. Following the multisensory perspective, in their study, Xu et al. (2018) described that sound is essential with immersive videos and, thus, they combined sound via earphones in their 360° videos. Various sensory experiences seem significant, especial y emphasizing visual and auditory sense stimulators. In this study, we included four types of technological immersion stimulating user’s senses towards immersion: I) visual, II) auditory, III) haptic, and IV) olfaction. In visual immersion, the amount and type of display screens (Queiroz et al., 2018) and image quality (Cummings & Bailenson, 2016) are essential factors when stimulating the sense of sight. In auditory immersion, Cummings and Bailenson (2016) mentioned that sound quality is essential, and audio can be flat audio or 3D audio (Queiroz et al., 2018) in stimulating the sense of hearing. Haptic immersion concerns haptic features (Cummings & Bailenson, 2016) where immersion is linked to haptic sense via hand, vehicle, or body (Queiroz et al., 2018) and the purpose is to stimulate the sense of touch. Lastly, olfactory fidelity or features (Cummings & Bailenson, 2016; Queiroz et al., 2018) concern olfaction – stimulating the sense of smell – and it can be used in immersive experiences through olfactory displays when the chosen odor (e.g., perfume smell) is released from a device connected to the computer during the use of immersive technology (Herrera & McMahan, 2014). To conclude, a summary of technological immersion is provided in Appendix A. 3.3 Immersive Technologies In turn, immersive technologies concern immersion-enabling technologies, of which several different technologies have emerged in previous research. According to Lee et al. (2013) immersive technologies refer specifically to the technology enabling immersion. Similarly, Slater (2009) characterize technology that provides users with T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 583 high-quality sensory information as immersive technology. There are different types of immersive technologies (Daassi & Debbabi, 2021), for example, virtual reality (VR) (Tonteri et al., 2023; Winkler et al., 2020), augmented reality (AR), three-dimensional (3D) views, mixed reality (MR: a combination of AR and VR) (Ambika et al., 2023), extended reality (XR: a combination of AR, VR and MR) (Adams, 2022) , and 360° videos (where users can rotate) (Xu et al., 2018). Several technologies have been examined, but the primary emphasis appears to be on virtual reality and augmented reality. Immersive technologies have been investigated in the virtuality-reality continuum (Suh & Prophet, 2018) and metaverse environments (Hudson et al., 2019). Compared to the physical world’s experience, in immersive technology-mediated experience (e.g., virtual reality), the user must learn to use the technology before using and focusing (Tonteri et al., 2023). Additionally, it is argued that the above listed immersive technologies can be considered as non-immersive if usage does not require special equipment (e.g., a head-mounted display). Thus, for example, web- based environments or Minecraft would be considered as being non-immersive because they are used via a computer, keyboard, and mouse (Suh & Prophet, 2018). The role of immersive technologies is to transport the user to immersion through the users’ senses. A summary of immersive technologies and the related studies is presented in Appendix B. 3.4 Psychological Immersion Agrawal et al. (2020) described psychological immersion as an individual’s deep mental state that enables the cognitive process even to dissociate a person from the physical world’s awareness. Psychological immersion can also be a subjective experience (Nilsson et al., 2016). In their article, Queiroz et al (2018) use psychological and subjective immersion. On the other hand, Agrawal et al. (2020), raise the concept of subjectivity to the definitions of immersion and use the term psychological immersion. In this study, we employ the term psychological immersion, which is frequently utilized. Psychological immersion can be further understood as follows. Agrawal et al. (2020) list three reasons leading to immersion: 1) the subjective sense of being surrounded or experiencing the multisensory simulation, 2) absorption in the narrative or the 584 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION depiction of the narrative, and, 3) absorption when facing strategic and/or tactical challenges. In this work, multisensory simulation is included in technological immersion. On the other hand, Queiroz et al. (2018) presented four different types of psychological immersion: spatial, strategic, narrative, and tactical. Furthermore, Ermi and Mäyrä (2005) describe that gamers can absorb themselves into a game and become immersed either aesthetical y (passively) or escapistical y (active participation). Finally, Kim (2013) propose ‘context immersion’ emphasizing the psychological state of immersion where the user’s immersion is embodied through mobile interaction and user experience connecting the user to real life. In this study, we understand psychological immersion as deep mental involvement in the user’s cognitive process leading to immersion, following Agrawal et al.’s (2020) definition (cited in Appendix C). This work uses three dimensions of psychological immersion found in previous research: spatial, narrative, and strategic. These dimensions describe the different deep mental involvement styles in the user’s cognitive process. First, the spatial dimension is the feeling of being surrounded (Agrawal et al., 2020). Kukkakorpi and Pantti (2021) explained space as transitioning from physical to digital. For example, how a story absorbs the user into another environment. Second, the narrative dimension concerns narrative aspects and the user’s imagination, such as how the user relies on the story through imagination (Ermi & Mäyrä, 2005). Third, the strategic dimension concerns strategic, tactical, and chal enge-based aspects. For example, the user is immersed when meeting tactical or strategical challenges (Agrawal et al., 2020) or when facing the game’s appropriately balanced challenge level (Ermi & Mäyrä, 2005; Nacke & Lindley, 2008). Similarly, Frank et al. (2015) noted in their survey that the game’s playfulness increases immersion in the game and effects the user’s hedonic motivation. Psychological immersion concepts are summarized in Appendix C. 3.5 Concepts Related to Immersion In previous research similar concepts are related to immersion, such as flow, presence, and transportation (Agrawal et al., 2020). Flow and immersion are two different concepts but close to each other. Csikszentmihalyi (1990) describes flow as ‘ the state in which individuals are so involved in an activity that nothing else seems to mat er’. In game studies, attention is central to flow, but in immersion, gamers sense experiences and emotions, and sensory simulation separates the concept of flow T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 585 from immersion (Brown & Cairns, 2004). On the contrary, Jennett et al. (2008) describe flow as an ‘ extreme end of immersion’ because, in their opinion, immersion is not always that strong. Agrawal et al. (2020) describe that flow and immersion might overlap, but these concepts are independent ideas. Another concept related to immersion is presence (Nilsson et al., 2016), which means, according to Slater (2009), ‘ being there’ in a virtual place despite being somewhere else in the physical world and being aware of that. Immersion and presence are firmly related but logical y separable concepts where presence represents the response to (a certain level of) immersion (Slater, 2003). On the contrary, Queiroz et al. (2018) include presence in immersion, whereas Ambika et al. (2023) define presence as addition to immersion. Additionally, Steuer (1992) distinguish telepresence from presence because telepresence includes ‘ the mediated perception of an environment’ (e.g., virtual reality), whereas presence is referring environments’ natural perceptions. However, previous research note teleoperations or teleconferencing applications to be the primary use of telepresence (Agrawal et al., 2020), not Steuer’s (1992) mentioned VR. Moreover, in the study by Mütterlein (2018), telepresence was supported to have a direct positive influence on immersion. Lastly, transportation can be absorbed into the narrative or detached from the environment (Green & Brock, 2000; Van Laer et al., 2014). Agrawal et al. (2020) define narrative immersion as similar to transportation. 4 Summary of the Literature Review Immersion can be categorized into technological and psychological immersion, where users can become immersed in one or both ways. A summarization of the above reported narrative literature review is visualized in a Figure 1. Figure 1 shows that in technological immersion can include one or more sense stimulants (visual, auditory, haptic, and odor), for example, how a user experiences a digital environment through sense stimulation. The role of immersive technology is to enable a sense of immersion in the possibilities brought by technology. We understand immersive technology as a technology, such as VR or AR. In addition to previous research, we added new innovations as one of the technologies. For example, the metaverse can bring new immersive technology possibilities. Technological immersion of a user can be simplified as ‘how much of the sense is 586 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION digitally/virtually covered’ enabled by immersive technology. For example, how much augmented reality smart glasses as an immersive technology are covering their user’s eyesight (i.e., visual sense), causing the user to experience technological immersion. Lastly, psychological immersion concerns the cognitive elements of immersion. Thus, sensory experience in immersion provided by technology is not mandatory but possible. In our summarization, we include three reasons leading to psychological immersion: spatial, narrative, and strategic. Thus, the user can be immersed, and the user can change from one space to another (spatial) through imagination and story (narrative) or with appropriately balanced chal enges (strategic). Figure 1: Summary of Immersion Source: Own All in all, immersion is not always either technological or psychological, but it can be a combination of both based on our summarization visualized in Figure 1. We call this techno-psychological immersion, referring to a situation when aspects from both sides of immersion are realized in the user’s immersion experience. Moreover, we T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 587 propose the immersion happening only in the physical world (e.g., reading a book) to present non-digital immersion. This immersion can evolve into digital immersion if technology is included, for example, by listening to music via earphones when reading. In contrast, listening to a podcast via earphones can present techno- psychological immersion if a user immerses oneself into the podcast’s story (narratively) via the auditory sense. Thus, techno-psychological immersion is an excel ent example of the merging of the digital and physical – phygital – worlds because even if we play videogame with al our senses on the computer, our sense of touch stil knows the mouse and keyboard of the physical world – unless we use a digital keyboard offered by smart glasses. 5 Discussion This study contributes to an understanding of the immersion concept. The prior literature on the immersion concept reveals two sides of immersion: technological and psychological. Also, immersive technologies are more commonly represented in the research but are sometimes used as a synonym with technological immersion such as in Lee et al.’s (2013) study. Some of the literature investigates immersive technologies through a technological approach rather than as a sensory experience or actual immersion, such as Suh and Prophet (2018) as well as Fan et al. (2022). Thus, our clarification of immersive technologies followed previous research ideas, but we clarified the concepts by separating the concepts of immersive technologies (e.g., VR and AR) and technological immersion (i.e., multisensory experience that leading to immersion). We also created a framework (Figure 1) in which we present immersion as an entity consisting of psychological and technological immersion, which can lead to techno-psychological immersion where both sides of immersion are realized. Different technologies are integrating into various aspects of people’s lives, from school to entertainment, and thus, customers might expect more immersive experiences, service providers, and products. Our techno-psychological immersion framework can be utilized as inspiration or the discussion stimulator for product design to determine ways to enhance user immersion, such as what aspects should be considered related to enabling or enhancing immersion. For example, designing IT devices covering the user's face (e.g., Dyson’s air-purifying headphones) could be further developed to enhance al sense experiences mentioned in our framework, 588 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION significantly improving the simulation of the sense of smell. Otherwise, there is potential for augmenting the factors that contribute to psychological immersion within VR headsets and other increasingly prevalent immersive technologies. For example, by bringing more robust storytel ing to the space between the user and the service with interactive communication or the depth of the narrative. A more detailed product design model should be developed in future research, and our framework will provide the first ideas. Due to possibilities provided by technological innovations, people are also integrated to use everyday digital services (e.g., Spotify), essential in people’s daily routines, as noted in the study by Kemppainen and Paananen (2024a). Service designers should consider our framework’s elements leading to techno-psychological immersion. This can mean, for example, helping a user transfer spatially (e.g., while waiting transfer for a more exciting digital environment) and providing appropriate sensory stimuli (e.g., improving concentration at work with suitable digital content – music or podcast), Thus, this can enhance positive digital wel -being as in study by Kemppainen & Paananen (2024b). This approach can also enhance spatial psychological immersion, as seen in platforms like Instagram reels, where users can feel detached from the outside world while scrol ing. Lastly, immersion is essential for companies pursuing metaverse opportunities. Immersion in metaverse is trending research topic right now in various fields (e.g., Dincelli & Yayla, 2022; Tang et al., 2022; Song et al., 2020). As commerce has evolved from electronic commerce to multichannel retailing, and onwards to omnichannel retailing (Brynjolfsson et al., 2013; Mali et al., 2022), the next step in this evolution seems to be the metaverse (Mystakidis, 2022), which emphasizes immersion (Hudson et al., 2019). As an example, Roblox is a metaverse VR game (Rospigliosi, 2022) where, in addition to playing and interacting with other players, the user can purchase branded clothing items in the game, which can be digital replicas of the physical world’s products. Thus, a company can use the same designs to sel products in new ways in the metaverse, which highlights the importance of understanding the user’s techno-psychological immersion. T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 589 6 Limitations and Future Research Suggestions Due to the narrative literature review method, this study only gives a greater scope of understanding of the phenomena. Thus, it does not provide the reproducibility of retrieving sources or research processes. However, the study presents a summarization of careful y selected articles. Narrative review maps previous research into a new form (Dijkers, 2009), which can lead to new theoretical perspectives (Paré et al., 2015) and, thus, our framework can be utilized as a discussion stimulator to inspire future research related to immersion. Hence, empirical and experimental research of techno-psychological immersion could be conducted using different methods. Also, immersion is trending in retail and, thus, further research should be conducted on how customers experience techno- psychological immersion in different contexts (e.g., omnichannel, brick-and-mortar stores, metaverse, or showrooms). Also, retail immersion barriers would be an exciting topic, as, for example, QR code usage barriers have been noticed during brick-and-mortar shopping, and people are not utilizing digital opportunities (see Paananen et al., 2023). Lastly, experimental studies of customer’s techno- psychological immersion with different immersion technologies (e.g., VR, AR) would be interesting in different retail contexts. References Acikgoz, F., & Tasci, A. D. A. (2022). Brand cocreation and immersion: The link between sense of brand community and attitude toward a brand. Journal of Hospitality and Tourism Insights, 5(2), 465-500. Adams, D. (2022). Virtual Retail in the Metaverse: Customer Behavior Analytics, Extended Reality Technologies, and Immersive Visualization Systems. Linguistic and Philosophical Investigations, 21, 73-88. Agarwal, R., & Karahanna, E. (2000). Time Flies When You’re Having Fun: Cognitive Absorption and Beliefs about Information Technology Usage. MIS Quarterly, 24(4), 665. Agrawal, S., Simon, A., Bech, S., Bærentsen, K., & Forchhammer, S. (2020). Defining Immersion: Literature Review and Implications for Research on Audiovisual Experiences. Journal of the Audio Engineering Society, 68(6), 404-417. Ambika, A., Shin, H., & Jain, V. (2023). Immersive technologies and consumer behavior: A systematic review of two decades of research. Australian Journal of Management. 03128962231181429. Brown, E., & Cairns, P. (2004). A grounded investigation of game immersion. CHI ’04 Extended Abstracts on Human Factors in Computing Systems, 1297-1300. Brynjolfsson, E., Hu, Y. J., & Rahman, M. S. (2013). As technology blurs the distinctions between physical and online retailing, retailers and their supply-chain partners wil need to rethink their competitive strategies. Digital transformation, 9. 590 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Cook, D. J., Mulrow, C., & Haynes, R. (1997). Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions. Annals of Internal Medicine, 126(5), 376-380. Csikszentmihalyi, M., 1990. Flow: The Psychology of Optimal Experience. Harper and Row, New York. Cummings, J. J., & Bailenson, J. N. (2016). How Immersive Is Enough? A Meta-Analysis of the Effect of Immersive Technology on User Presence. Media Psychology, 19(2), 272-309. Daassi, M., & Debbabi, S. (2021). Intention to reuse AR-based apps: The combined role of the sense of immersion, product presence and perceived realism. Information & Management, 58(4), 103453. Dijkers, M. P. J. M. (2009). The Value of “Traditional” Reviews in the Era of Systematic Reviewing. American Journal of Physical Medicine & Rehabilitation, 88(5), 423-430. Dincelli, E., & Yayla, A. (2022). Immersive virtual reality in the age of the Metaverse: A hybrid-narrative review based on the technology affordance perspective. The Journal of Strategic Information Systems, 31(2), 101717. Ermi, L., & Mäyrä, F. (2005). Fundamental Components of the Gameplay Experience: Analysing Immersion. In DeCastel S., Jenson J. (Eds.), Proceedings of DiGRA 2005 conference: Changing Views—Worlds in Play. 15-27. Vancouver, British Columbia, Canada: Simon Fraser University. Fan, X., Jiang, X., & Deng, N. (2022). Immersive technology: A meta-analysis of augmented/virtual reality applications and their impact on tourism experience. Tourism Management, 91, 104534. Frank, L., Salo, M. & Toivakka, A. (2015). Why Buy Virtual Helmets and Weapons? Introducing a Typology of Gamers. In the Proceedings of the 25th Bled eConference, 502-519. Green, M. C., & Brock, T. C. (2000). The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psychology, 79(5), 701-721. Haywood, N., & Cairns, P. (2006). Engagement with an Interactive Museum Exhibit. In T. McEwan, J. Gul iksen, & D. Benyon (Eds.), People and Computers XIX — The Bigger Picture. Proceedings of HCI2005, 113-129. Herrera, N. S., & McMahan, R. P. (2014). Development of a Simple and Low-Cost Olfactory Display for Immersive Media Experiences. Proceedings of the 2nd ACM International Workshop on Immersive Media Experiences, 1-6. Hudson, S., Matson-Barkat, S., Pal amin, N., & Jegou, G. (2019). With or without you? Interaction and immersion in a virtual reality experience. Journal of Business Research, 100, 459-468. Jennett, C., Cox, A. L., Cairns, P., Dhoparee, S., Epps, A., Tijs, T., & Walton, A. (2008). Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies, 66(9), 641-661. Kemppainen, T., & Paananen, T. E. (2024a). Dualities of digital services: Everyday digital services as positive and negative contributors to customer well-being. Journal of Service Theory and Practice, A-Head-of-Print. Kemppainen, T., & Paananen, T. (2024b). How Can Favorite Digital Services Enhance Users’ Digital Wel -Being? A Qualitative Study. In M. Papadaki, M. Themistocleous, K. Al Marri, & M. Al Zarouni (Eds.), Information Systems: 20th European, Mediterranean, and Middle Eastern Conference, EMCIS 2023 (pp. 65-76). Springer. Lecture Notes in Business Information Processing. Kim, M. J. (2013). A framework for context immersion in mobile augmented reality. Automation in Construction, 33, 79-85. Kukkakorpi, M., & Pantti, M. (2021). A Sense of Place: VR Journalism and Emotional Engagement. Journalism Practice, 15(6), 785-802. Lee, Y., Chen, A. N., & Ilie, V. (2012). Can Online Wait Be Managed? The Effect of Fil er Interfaces and Presentation Modes on Perceived Waiting Time Online. MIS Quarterly, 36(2), 365-394. Lee, Y.-C. N., Shan, L.-T., & Chen, C.-H. (2013). System development of immersive technology theatre in museum. In Proceedings of International Conference on Virtual, Augmented and Mixed reality, 400-408. T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 591 Mali, E., Paananen, T., Frank, L., & Makkonen, M. (2022). A Customer Perspective on Omnichannel Customer Journey and Channel Usage : A Qualitative Study. In P. Bednar, A. S. Islind, H. Vallo-Hult, A. Nolte, M. Rajanen, F. Zaghloul, A. Ravarini, & A. M. Braccini (Eds.), Proceedings of the 8th International Workshop on Socio-Technical Perspective in Information Systems Development (STPIS 2022), 299-310. RWTH Aachen. CEUR Workshop Proceedings, 3239. Murray, J. B. (1997). Hamlet on the holodeck: The future of narrative in cyberspace. New York, NY, USA: Free Press. Mütterlein, J. (2018). The Three Pillars of Virtual Reality? Investigating the Roles of Immersion, Presence, and Interactivity. . In T. X. Bui (Ed.), the Proceedings of 51st Annual Hawaii International Conference on System Sciences. 1407-1415. Mystakidis, S. (2022). Metaverse. Encyclopedia, 2(1), 486-497. Nacke, L., & Lindley, C. (2008). Boredom, Immersion, Flow—A pilot study investigating player experience. In Proceedings of the 2008 IADIS International Conference Gaming 2008: Design for Engaging Experience and Social Interaction, 103-107. Nah, Eschenbrenner, & DeWester. (2011). Enhancing Brand Equity Through Flow and Telepresence: A Comparison of 2D and 3D Virtual Worlds. MIS Quarterly, 35(3), 731-747. Nilsson, N. C., Nordahl, R., & Serafin, S. (2016). Immersion Revisited: A review of existing definitions of immersion and their relation to different theories of presence. Human Technology, 12(2), 108-134. Paananen, T., Frank, L., & Kemppainen, T. (2022). Customer-Brand Relationships in the Context of Digital Brands. In A. Pucihar, M. Kljajić Borštnar, R. Bons, A. Sheombar, G. Ongena, & D. Vidmar (Eds.), 35th Bled eConference : Digital Restructuring and Human (Re)action, 495- 510. University of Maribor. Paananen, T., Holkkola, M., Makkonen, M., Frank, L. & Kemppainen, T. (2023). Customers’ QR Code Usage Barriers in a Brick-and-Mortar Store : A Qualitative Study. In A. Pucihar, M. Kljajić Borštnar, R. Bons, G. Ongena, M. Heikkilä, & D. Vidmar (Eds.), 36th Bled eConference : Digital Economy and Society : The Balancing Act for Digital Innovation in Times of Instability, 171-188. University of Maribor. Paré, G., Trudel, M.-C., Jaana, M., & Kitsiou, S. (2015). Synthesizing information systems knowledge: A typology of literature reviews. Information & Management, 52(2), 183-199. Pucihar, A. (2020). The digital transformation journey: Content analysis of Electronic Markets articles and Bled eConference proceedings from 2012 to 2019. Electronic Markets, 30(1), 29-37. Queiroz, A. C. M., Nascimento, A. M., Alejandro, T. B., Tori, R. de Melo, V., de Souza Meirel es, F., & da Silva Leme, M. I. (2018). Virtual Reality in Marketing: Technological and Psychological immersion. In Proceedings of the 24th Americas Conference on Information Systems. Rospigliosi, P. A. (2022). Metaverse or Simulacra? Roblox, Minecraft, Meta and the turn to virtual reality for education, socialisation and work. Interactive Learning Environments, 30(1), 1-3. Saunders, Rutkowski, Genuchten Van, Vogel, & Orrego. (2011). Virtual Space and Place: Theory And Test. MIS Quarterly, 35(4), 1079-1098. Slater, M. (2003). A Note on Presence Terminology. Presence Connect, 3(3), 1-5. Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549-3557. Slater, M., & Wilbur, S. (1997). A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments, 6(6), 603-616. Soliman, M., Peetz, J., & Davydenko, M. (2017). The Impact of Immersive Technology on Nature Relatedness and Pro-Environmental Behavior. Journal of Media Psychology, 29(1), 8-17. Song, H. K., Baek, E., & Choo, H. J. (2020). Try-on experience with augmented reality comforts your decision: Focusing on the roles of immersion and psychological ownership. Information Technology & People, 33(4), 1214-1234. 592 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Steuer, J. (1992). Defining Virtual Reality: Dimensions Determining Telepresence. Journal of Communication, 42(2), 73-93. Suh, A., & Prophet, J. (2018). The state of immersive technology research: A literature analysis. Computers in Human Behavior, 86, 77-90. Sun, N., & Botev, J. (2023). Technological Immersion and Delegation to Virtual Agents. Multimodal Technologies and Interaction, 7(11), 106. Tang, Y. M., Chau, K. Y., Kwok, A. P. K., Zhu, T., & Ma, X. (2022). A systematic review of immersive technology applications for medical practice and education-trends, application areas, recipients, teaching contents, evaluation methods, and performance. Educational Research Review, 35, 100429. Tonteri, T., Holopainen, J., Lumivalo, J., Tuunanen, T., Parvinen, P., & Laukkanen, T. (2023). Immersive Virtual Reality in Experiential Learning—A Value Co-creation and Co-destruction Approach. In T. X. Bui (Ed.), the Proceedings of 56th Annual Hawai International Conference on System Sciences. 1313-1322. Van Laer, T., De Ruyter, K., Visconti, L. M., & Wetzels, M. (2014). The Extended Transportation-Imagery Model: A Meta-Analysis of the Antecedents and Consequences of Consumers’ Narrative Transportation. Journal of Consumer Research, 40(5), 797-817. Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS Quarterly, 26(2), xiii–xxiii. Winkler, N., Röthke, K., Siegfried, N. & Benlian, A. (2020). Lose Yourself in VR: Exploring the Effects of Virtual Reality on Individuals’ Immersion. In the Proceedings of 54th Annual Hawai International Conference on System Sciences. 1510-1519. Xu, Y., Dong, Y., Wu, J., Sun, Z., Shi, Z., Yu, J., & Gao, S. (2018). Gaze Prediction in Dynamic 360° Immersive Videos. In the Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5333–5342. Zimmermann, H.-D. (2016). Digital Transformation—The Emerging Digital Economy. In Proceedings of International Conference: Liberec Informatics Forum: ICT in the Role of Services: State of the art and perspectives. 138–146. T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 593 Appendix A: Summary of Technological Immersion and Senses Concept Explanation ‘The capacity of a media device to mediate and represent an environment in a way that matches human perception of the physical world’ (Sun & Botev, 2023) Technological immersion ‘Let’s reserve the term ‘immersion’ to stand simply for what the technology delivers from an objective point of view. The more System that a system delivers displays (in all sensory modalities) and immersion tracking that preserves fidelity in relation to their equivalent real- world sensory modalities, the more that it is ‘immersive’.’ (Slater, 2003) - Sensory immersion: ‘Large screens close to player’s face and powerful sounds easily overpower the sensory information coming Sensory from the real world, and the player becomes entirely focused on immersion the game world and its stimuli.’ (Ermi & Mäyrä, 2005) - Experience of multisensory simulation (Agrawal et al., 2020) - System-focused immersion (i.e., sensory immersion, based on media features) (Daassi & Debbabi, 2021) - Visual immersion (single screen, multiple screens, videowall, cavern automatic virtual environment) (Queiroz et al., 2018) - Stereoscopic vision, image quality, field of view, update rate Visual overal high versus low, (Cummings & Bailenson, 2016) - ‘Audiovisual implementation has something to do with immersive experiences, but it is by no means the only or even the most significant factor.’ (Ermi & Mäyrä, 2005) - Auditory immersion (flat audio, 3D audio) (Queiroz et al., 2018) Auditory - Sound quality (Cummings & Bailenson, 2016) - Haptic immersion (hand, vehicle, body) (Queiroz et al., 2018) Haptic - Haptic features (Cummings & Bailenson, 2016) - Olfactory fidelity (odor) (Queiroz et al., 2018), - Olfactory features (Cummings & Bailenson, 2016) - Olfaction is sense of smell, which is described as an important Olfaction perceptual function and olfactory display enable using odorants (e.g., the smel of orange) during immersive technology usage (Herrera & McMahan, 2014) 594 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix B: Summary of Immersive Technologies Concept Explanation ‘Immersive technology refers to technology that blurs the line Immersive between the physical world and digital or simulated world, thereby technologies creating a sense of immersion.’ (Lee et al., 2013) Equipment Immersion requires equipment (e.g., head-mounted display) from an perspective user (Suh & Prophet, 2018) - Augmented reality (AR) (Ambika et al., 2023; Daassi & Debbabi, 2021; Fan et al., 2022), mobile AR (Kim, 2013) - Virtual reality (VR) (Ambika et al., 2023; Kukkakorpi & Pantti, Enabling 2021; Queiroz et al., 2018; Tonteri et al., 2023; Winkler et al., 2020) immersive - Mixed reality (MR) (Ambika et al., 2023) technologies - Extended reality (XR) (Adams, 2022) - Three dimensional views (3D) (Ambika et al., 2023) - 360° videos (Xu et al., 2018) T. Paananen et al.: Towards Techno-Psychological Immersion: A Narrative Literature Review of Immersion and Its Related Concepts 595 Appendix C: Summary of Psychological Immersion and Dimensions Concept Explanation ‘Immersion is a phenomenon experienced by an individual when they are in a state of deep mental involvement in which their Psychological cognitive processes (with or without sensory stimulation) cause a immersion shift in their attentional state such that one may experience disassociation from the awareness of the physical world.’ (Agrawal et al., 2020) - Spatial dimension (Queiroz et al., 2018) - ‘Place refers to factual and inter-textual qualities, whereas space Spatial alludes to the digital environment in which the user is immersed.’ (Kukkakorpi & Pantti, 2021) - Subjective sense of being surrounded (Agrawal et al., 2020) - Absorption in the narrative or the depiction of the narrative (Agrawal et al., 2020) - Imaginative immersion: ‘The game offers the player a chance to use her imagination, empathize with the characters, or just enjoy Narrative the fantasy of the game.’ (Ermi & Mäyrä, 2005) - Children create narratives, which al ow for the use of fantasy while stil following the linear structure (in the museum context) (Haywood & Cairns, 2006) - Absorption when facing strategic and/or tactical chal enges (Agrawal et al., 2020) - Chal enge-based immersion: ‘This is the feeling of immersion Strategic that is at its most powerful when one is able to achieve a satisfying balance of chal enges and abilities.’ (Ermi & Mäyrä, 2005) - Immersion is more closely achieved with appropriately chal enging game tasks (Nacke & Lindley, 2008) 596 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION THE DRIVERS OF SHOWROOMING BEHAVIOR: A META-ANALYSIS MATILDA HOLKKOLA,1 OLLI TYRVÄINEN,1 MARKUS MAKKONEN,2, 1 HEIKKI KARJALUOTO,1 TIINA KEMPPAINEN,3 TIINA PAANANEN,1 LAURI FRANK1 1 University of Jyvaskyla, Faculty of Information Technology, Jyvaskyla, Finland matilda.i.holkkola@jyu.fi, olli.p.tyrvainen@jyu.fi, markus.v.makkonen@jyu.fi, heikki.karjaluoto@jyu.fi, tiina.e.paananen@jyu.fi, lauri.frank@jyu.fi 2 Tampere University, Faculty of Management and Business, Tampere, Finland markus.makkonen@tuni.fi 3 University of Jyvaskyla, School of Business and Economics, Jyvaskyla, Finland ti na.j.kemppainen@jyu.fi Showrooming behavior refers to consumer behavior where consumers first physically evaluate products in offline channels and then compare the potential purchases in online channels. Although the drivers of showrooming behavior have gained interest from many quantitative researchers and resulted in multiple conflicting results, there is no established framework for these drivers. Therefore, we made a meta-analysis of the drivers of showrooming behavior. To analyze prior results, we conducted a systematic literature review resulting in 24 independent study samples that fit our criteria. Of these samples, 18 drivers were meta-analytically analyzed, resulting in 13 drivers being found to have a statistical y significant association and five Keywords: drivers being found to have no statistically significant association showrooming behavior, with showrooming behavior. As a theoretical contribution, we meta-analysis, provide an established framework and solve prior conflicting omnichannel, findings. As a managerial contribution, we provide advice to consumer behavior, decrease customers’ competitive showrooming behavior cross-channel according to the identified main drivers. behavior DOI https://doi.org/10.18690/um.fov.4.2024.35 ISBN 978-961-286-871-0 598 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction In the retail context, new means and technologies to diversify consumers’ options in their decision-making process have multiplied. Thanks to advancements in information and communication technologies (ICTs), today’s smart consumers can weigh their options based on online information, also simultaneously when shopping in offline stores (Verhoef et al., 2015; Holkkola et al., 2023a). These possibilities to seamlessly utilize both offline and online channels of the same retailer are referred to as omnichannel retailing, which is considered the next step of multichannel retailing (Lin et al., 2023; Makkonen et al., 2023; Rigby, 2011). However, also comparing multiple retailers’ products is easy for smart consumers in the digital age. The phenomenon of consumers physical y evaluating products in offline channels and comparing or buying the product in online channels is referred to as showrooming behavior (Fiestas & Tuzoiv, 2021; Grewal et al., 2016). The verb “to showroom” originates from physical showrooms, where instead of buying the product directly, consumers can gain knowledge and consultancy of the displayed products and leave an order or buy it in other channels (Rapp et al., 2015; Fan et al., 2021). Thus, today’s showroomers can be perceived as using offline stores as showrooms for products purchased online (Mehra et al., 2018; Brynjolfsson et al., 2013). According to statistics, showrooming behavior is very popular – it is estimated that 84% of consumers are doing it (Retail Touch Points, 2018). Although showrooming can happen in the same retailer’s channels and, thus, be so-called loyal showrooming (Schneider & Zielke, 2020), showroomers have shown a tendency to ultimately buy the product via competing retailer’s online channels (Spaid et al., 2019). This kind of competitive showrooming makes it a particularly challenging dilemma for brick-and-mortar (B&M) store retailers (Rapp et al., 2015). Indeed, showroomers are often attracted by the possibility of physical y touching and feeling the product and still utilizing lower prices offered by online retailers, but the reasons and motives behind this cross-channel behavior are suggested to be more diverse than that (Gensler et al., 2017; Frasquet & Miquel-Romero, 2021). Therefore, identifying the drivers of showrooming behavior becomes important (Arora et al., 2022). However, there is a research gap in systematical y and statistical y combining the existing quantitative results of the drivers of showrooming behavior. Also, our literature review shows that up to six drivers have resulted in conflicting findings: M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 599 gender, age, brand loyalty, online trust, offline service, and exploratory shopping, which need further research. In the past decades of Information Systems (IS) research, meta-analysis has been proven as an efficient way to synthesize prior results and tackle contradictory findings and, thus, provide more reliable knowledge (Jeyaraj & Dwivedi, 2020). Meta-analysis consists of a Systematic Literature Review (SLR) and a statistical analysis where the data consists of samples from existing studies. Synthesizing the data from prior showrooming studies is vital for retail practitioners who want to retain existing or find new customers in the digital age (Mehra et al., 2018). Arora et al. (2017, 2022) also called for more research on the factors behind showrooming behavior. In addition, Holkkola et al. (2022a) call for research on showrooming drivers that have resulted in contradictory study results, such as gender. The goal of this paper is to fill this gap in the literature. Thus, we statistically synthesize the existing quantitative results concerning the drivers of showrooming behavior by identifying (1) what the main drivers of showrooming behavior are and (2) whether the drivers that seem contradictory in prior literature actual y drive showrooming behavior. Despite the researchers’ growing interest and multiple quantitative studies on showrooming behavior, no meta-analytical framework for the drivers of showrooming behavior has been proposed. Sahu et al. (2021) have made a descriptive SLR on showrooming and webrooming. Webrooming refers to behavior where the information search and actual purchase happen in the opposite channels compared to showrooming (Konuş et al., 2008). The findings of Sahu et al. (2021) bring together various drivers of showrooming behavior but do not provide a statistical synthesis of drivers’ average associations, statistical significance, and the correctness of conflicting prior results. Nor do they consider publication bias, which arises when statistically significant rather than not significant findings are more typical y submitted to and accepted by peer-reviewed publications (Jeyaraj & Dwivedi, 2020). Therefore, in this paper, we statistically synthesize the existing quantitative results concerning the drivers of showrooming behavior. To find al the drivers studied, we carried out an SLR on existing showrooming literature. Then, we integrated the existing constructs and executed a meta-analysis to find out the mean associations of the existing samples. In the next section, prior findings on showrooming behavior are presented. In the third section, the meta-analysis method is presented. The fourth section presents the findings of this study and, final y, the fifth section provides a discussion and conclusion. 600 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 2 Showrooming behaviour The causes and consequences of showrooming behavior have gained interest from researchers. The consequences of showrooming behavior have included, for example, an increase in consumers’ innovative purchase tendencies (Sahu et al., 2021), a negative impact on offline store staff’s performance (Rapp et al., 2015; Park & Hur, 2023), and a positive effect on revisit intention (Holkkola et al., 2023b). Thus, although the showrooming phenomenon could be perceived as a chal enge for offline retailers, the findings in prior literature seem multifaceted. Also, the drivers of showrooming have been studied with a great variety of variables. Sahu et al.’s (2021) SLR found 42 drivers of showrooming and webrooming behavior from prior studies. They classified these drivers into three categories: customer-led, company- led, and situational drivers. According to Sahu et al. (2021), customer-led showrooming drivers include, for instance, consumers’ capabilities and normative beliefs. Also, consumers’ socio- demographic characteristics behind showrooming behavior have been studied (Holkkola et al., 2022a). Some studies report that younger age increases showrooming behavior (Kolehmainen, 2018; Holkkola et al., 2022a) whereas other studies propose that age has no effect on the matter (Dahana et al., 2018; Li et al., 2018; Fang et al., 2021). This raises the question of which result is correct. Also in terms of gender, contradictory results have been found. For instance, Dahana et al. (2018) found that gender has no effect on showrooming behavior while Holkkola et al. (2022a) found women to showroom more than men. Regarding consumers’ income, higher income has been associated with more active showrooming behavior (Fang et al., 2021; Holkkola et al., 2022a). However, Jo et al. (2020) found no association between income and multichannel shopping behavior. In prior literature, consumers’ online trust and the lack of perceived online risks have also resulted in conflicting findings. Arora and Sahney (2018) found that consumers’ online trust increases their showrooming behavior. However, Quach et al. (2022) found that privacy risk has no effect on showrooming behavior, although, based on Arora and Sahney’s (2018) findings, the perceived privacy risk could be hypothesized to decrease showrooming behavior and the perceived lack of privacy risk to increase showrooming behavior. Similarly, Kolehmainen (2018) found no association between security risk and showrooming behavior. M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 601 The company-led showrooming drivers, in turn, consist of the things that are under a retailer’s control, such as price, customer service, and channel integration (Sahu et al., 2021). In prior quantitative studies, many of these company-led showrooming drivers have resulted in effects with the same direction: either positive or negative. For instance, Li et al. (2018), Fang et al. (2021), and Goraya et al. (2022) all found a positive effect of channel integration on showrooming behavior, although the strength of these effects varied. In line with this positive effect, utilizing a retailer’s online channels is suggested to enhance consumers’ perceptions of the same retailer’s channel integration and available services (Fang et al., 2021). However, some associations between showrooming behavior and company-led drivers have even resulted in opposite results. For instance, the effects of customer service in an offline store on showrooming behavior have been found both positive (Arora & Sahney, 2018; Shankar et al., 2021) and negative (Burns et al., 2018), whereas other studies (Kang, 2018) have found no association between them, thus underlining the need for this meta-analytical review. Regarding situational showrooming drivers, brand loyalty and exploratory shopping have resulted in contradictory results. Brand loyalty has been associated both positively (Quach et al., 2022) and negatively (Borges, 2018) with showrooming behavior. In addition, Burns et al. (2018) found no association between these (Burns et al., 2018). In exploratory shopping, consumers are involved and immersed in products (Christodoulides & Michaelidou, 2010; Quach et al., 2022) and may experience flow, which consists of immersion, enthusiasm, and losing track of time (Rose et al., 2012). Exploratory shopping has resulted in positive (Quach et al., 2022) and statistically not significant (Herrero-Crespo et al., 2022) associations with showrooming behavior. Banerjee and Longstreet (2016) conceptualized showroomers as having high consciousness in both physical and virtual dimensions, which is related to the immersion aspect of exploratory shopping. Also, shopping enjoyment, which is a component of customers’ flow, is more typical for multi- channel shoppers than for single-channel or low-commitment shoppers (Konuş et al., 2008). However, shopping enjoyment did not affect customers’ showrooming intention (Kolehmainen, 2018). Thus, exploratory shopping and its related components have resulted in both positive and statistical y not significant effects on showrooming and multichannel behaviors in general. Based on the above, multiple conflicting drivers need further analysis. 602 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3 Methodology 3.1 Data Collection and Coding The literature search for the meta-analysis was performed using various search terms, such as “showrooming”, “research shopping”, “omnichannel retailing”, “multichannel retailing”, and “cross-channel retailing” in several databases (ABI/INFORM, Scopus, ProQuest Central, Emerald, EBSCO Business Source Premier, ProQuest Dissertations and Theses, and Google Scholar). In addition, several proceedings of IS conferences (AMCIS, Bled eConference, ECIS, HICCS, ICIS, MCIS, PACIS, WHICEB, and Wirtschaftsinformatik) were searched or manually screened. In our inclusion criteria, studies had to 1) address showrooming behavior; 2) provide quantitative empirical results based on independent samples; 3) provide the required information for effect size integration; and 4) be written in English. The search resulted in 24 independent samples with a total of 12,129 respondents. These samples were from studies that were published between 2017 and 2024 (see Appendix 1). The resulting data was coded according to the guidelines of Rust and Cooil (1994). More specifical y, information representing effect sizes, sample sizes, and reliability of measurements was extracted. Correlation coefficients were selected to represent effect sizes. If the studies did not report correlation coefficients, we converted other statistics to correlations using the procedures by Lipsey and Wilson (2001) as well as Peterson and Brown (2005). Also, if studies reported multiple correlations for the same relationship, average correlations were calculated. 3.2 Effect-Size Integration and Construct Integration Effect size integration followed the random-effect approach by Hunter and Schmidt (2004). First, we corrected effect sizes in terms of reliability: effect sizes were divided by the square root of the product of reliabilities of independent and dependent variables. If this information was missing, the average correlation of the construct was used. Next, effect sizes were corrected in terms of sample sizes. Average correlations were calculated using the random-effect approach (Hunter & Schmidt, 2004). Regarding constructs, we found 86 constructs that were studied as drivers of showrooming behavior. Some of them had only been used in a single study and some in several studies. Some constructs measured the same thing as other M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 603 constructs in other studies, such as the constructs of online risk and privacy risk. When analyzing the data, we integrated these overlapping constructs which are presented in Table 1. Table 1: Results of construct integration Construct Definition Aliases Showrooming Consumers’ judgments of their capabilities Perceived behavioral self-efficacy and resources to showroom (Makkonen et al., 2022) control Consumer Consumers’ perceived innovativeness and Smart shopper innovative- power to seek information in the channels feelings, consumer ness of their choice (Huh et al., 2022) empowerment Trust in online vendors (Tan & Security risk Online trust Sutherland, 2004) and data protection (reversed), privacy (Mahrous & Hassan, 2017) risk (reversed) Attitude Customers’ attitudes toward and positive toward evaluations of showrooming (Arora et al., – showrooming 2020) The extent to which consumers’ Social showrooming behavior is influenced by Socialization, influence other people and social norms (Rejón- subjective norm Guardia & Luna-Nevarez, 2017) Offline search The extent how much offline evaluation In-store search value helps consumers (Rajkumar et al., 2021; value, perceived Kim, 2004). search ben-efits, feel of product Desire for customer The desire for offline assistance (Kim & service, sales staff as-sistance, desire Offline service Stoel, 2005) and social encounters (Haytko & Baker, 2004) as wel as satisfaction with for so-cial the store staff (Reynolds & Beatty, 1999) interaction, atten- tiveness convenience The extent to which consumer perceives Cross-channel inte- Channel al information systems and their gration, information integration management successful y integrated across integration, channels (Shi et al., 2020) perceived integration Ease of use of The degree to which customers believe online that switching to online purchasing would Effort expectancy purchase be effortless (Davis, 1989; Arora & Sahney, 2018) 604 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Construct Definition Aliases Monetary The expected monetary saving benefits of Deals and discounts, savings showrooming (Atkins & Kim, 2012) cost savings, price comparison The access to assortments with a wide Assortment seeking, Better range of products, brands, prices, and perceived assortment qualities (Eastlick & Feinberg, 1999; Kahn assortment, better & Wansink, 2004; Emrich et al., 2015) product assortment Perceived The expected usefulness and functionality usefulness of of showrooming to achieve desired Performance showrooming outcomes (Davis, 1989; Venkatesh et al., expectancy 2003; Chimborazo-Azogue et al., 2021) Customers’ attitudinal and behavioral Brand loyalty loyalty to a brand (Baldinger & Rubinson, – 1996) Product The level of importance and relevance of Purchase involvement the purchase to a consumer (Zaichkowsky, involvement 1986) Exploratory Shopping by being involved Exploratory shopping (Christodoulides & Michaelidou, 2010) informa-tion and immersed (Quach et al., 2022) in seeking, explo-ratory products acquisition, flow After having integrated paral el constructs, we excluded the remaining constructs that had been used in less than three studies (Tyrväinen et al., 2023). After this, 18 constructs remained in the final model. We wanted to include every construct that had been studied in a sufficient number of samples, because, as Dahana et al. (2018) reasoned, “any factor associated with these [offline and online] behaviors is expected to eventual y influence the extent to which consumers engage in showrooming”. 4 Results The results of effect-size integration for each integrated construct in terms of the number of analysed samples (k), the total N of these samples, the reliability-adjusted, sample size weighted average correlation (RC), the lower (CIlow) and upper limits (CIhigh) of its 95% confidence intervals, the Q-statistic, I2 statistic, and fail-safe N (FSN) to address the file-drawer problem are shown in Table 2. M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 605 Table 2: Results of effect-size integration k N RC CIlow CIhigh Q I2 FS N Customer-led drivers Age 7 3721 -0.031 -0.254 0.200 279.388* ** 97.852 – Gender 6 3225 -0.002 -0.093 0.089 29.818*** 83.230 – Income 5 2725 0.109** * 0.072 0.146 3.011 0.000 34 Showrooming self- efficacy 8 3693 0.385** * 0.276 0.485 91.622*** 92.360 113 0 Consumer innovativeness 4 1287 0.291** * 0.165 0.408 17.674** 83.026 117 Online trust 3 1365 0.201 -0.183 0.531 92.099*** 97.828 – Attitude toward showrooming 5 1862 0.557** * 0.456 0.637 28.288*** 85.860 883 Social influence 4 1676 0.375** 0.157 0.559 61.107*** 95.091 193 Company-led drivers Offline search value 4 1230 0.419** * 0.225 0.581 44.814*** 93.305 242 Offline service 4 1513 0.243 -0.071 0.513 125.893* ** 97.617 – Channel integration 3 2148 0.327** * 0.221 0.426 8.147* 75.450 107 Ease of use of online purchase 4 2097 0.357** * 0.163 0.524 65.312*** 95.410 325 Monetary savings 9 3544 0.361** * 0.175 0.523 293.238* ** 97.270 978 Better assortment 3 1275 0.221** 0.063 0.386 17.639*** 88.660 42 Situational drivers Perceived usefulness of showrooming 3 1794 0.537** * 0.395 0.654 28.397*** 92.960 459 Brand loyalty 4 1906 0.078 -0.113 0.264 49.471*** 93.940 20 Product involvement 6 2464 0.320** * 0.183 0.445 65.410*** 92.360 424 Exploratory shopping 3 2480 0.207* 0.022 0.379 45.662*** 95.620 81 606 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Based on our analysis, 18 constructs have been commonly studied related to showrooming, and, of them, 13 constructs were found to drive showrooming behavior. Regarding customer-led drivers, we found that showrooming behavior positively correlated with income (RC = 0.109, p < 0.001), social influence (RC = 0.375, p < 0.01), showrooming self-efficacy (RC = 0.385, p < 0.001), consumer innovativeness (RC = 0.291, p < 0.001), and attitude toward showrooming (RC = 0.557, p < 0.001). In contrast, the correlations with age, gender, and online trust were not statistical y significant. Regarding company-led drivers, we found that channel integration (RC = 0.327, p < 0.001), monetary savings (RC = 0.361, p < 0.001), better assortment (RC = 0.221, p < 0.01), the ease of use of online purchasing (RC = 0.357, p < 0.001), and offline search value (RC = 0.419, p < 0.001) al positively correlated with showrooming behavior. Interestingly, the correlation with offline service was not statistically significant. Regarding situational drivers, product involvement (RC = 0.320, p < 0.001), exploratory shopping (RC = 0.207, p < 0.05), and the perceived usefulness of showrooming (RC = 0.537, p < 0.001) positively correlated with showrooming behavior, whereas the correlation with brand loyalty was not statistically significant. The statistically significant Q-statistics for the homogeneity test and I2 statistics indicate heterogeneity across effect sizes for most of the relationships. Thus, further studies should test the moderating effects of these relationships. 5 Discussion and Conclusion Although consumers’ showrooming behavior has gained interest from IS and marketing researchers, there is no common consensus about the phenomenon and its main drivers. Therefore, in this study, we aimed to produce a comprehensive framework for the drivers of showrooming behavior. We conducted a meta-analysis, which is a useful way of drawing more consistent conclusions from prior and possibly contradictory results (Grewal et al., 2018). This meta-analysis includes results from 24 independent study samples from studies published between 2017 and 2024. In total, these study samples included 12,129 respondents. To our best knowledge, this is the first meta-analysis concerning showrooming behavior. Therefore, this meta-analysis fills this research gap and answers Arora et al.’s (2017, 2022) and Holkkola et al.’s (2022a) calls for further research on drivers of showrooming behavior. By doing so, this study provides useful generalizations by identifying (1) what the main drivers of showrooming behavior are and (2) whether M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 607 the drivers that seem contradictory in prior literature actual y drive showrooming behavior. Based on our findings, we also make two additional observations concerning possible moderators and the applicability of the Technology Acceptance Model (TAM) (Davis, 1986). The findings of this study are summarized in Figure 1 and discussed below. Figure 1: Established framework of the drivers of showrooming behavior Source: Own The theoretical implications of this meta-analysis are twofold. Firstly, we provide an established framework of the drivers of showrooming behavior presented above. The drivers are categorized as customer-led, company-led, and situational drivers according to Sahu et al.’s (2021) proposal. Within each category, the order of the drivers is determined according to the strength and statistical significance of their association with showrooming behavior. The strongest drivers are in line with prior quantitative showrooming studies that are presented in Appendix 1. Among the strongest drivers are attitude toward showrooming, the perceived usefulness of showrooming, and the ease of use of online purchase, which are also in line with the TAM model (Davis, 1986). Secondly, this meta-analysis resolves how the conflicting drivers from prior studies relate to showrooming behavior. These conflicting drivers are gender, age, brand loyalty, online trust, offline service, and exploratory shopping. Additionally, income’s positive effect on showrooming behavior (Fang et al., 2021) but statistically not significant effect on multichannel shopping (Jo et al., 2020) have 608 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION raised questions. The statistically not significant drivers found in this meta-analysis are presented in the grey box in Figure 1. Although Holkkola et al. (2022a) suggest that women are more probable showroomers, gender is not associated with consumers’ showrooming behaviors, in line with Dahana et al. (2018). Also, although younger age has been suggested to increase one’s showrooming behavior (Kolehmainen, 2018; Holkkola et al., 2022a), we find that age has no effect either. This is again in line with Dahana et al. (2018). Further, we find that customers’ brand loyalty is not associated with their showrooming behavior. This is in line with Burns et al. (2018) and refutes the opposing effects proposed by Quach et al. (2022) and Borges (2018). Regarding consumers’ online trust, its association with showrooming behavior is statistically not significant although Arora and Sahney (2018) suggested that online trust increases showrooming. Our finding is in line with Kolehmainen (2018) and Quach et al. (2022). Regarding offline service, the positive but statistically not significant association is in line with Kang (2018). Thus, our meta-analysis refutes Burns et al.’s (2018) suggestion that a negatively perceived offline service increases showrooming behavior. Our finding that offline service does not associate with showrooming probes one to think why the desire for customer service as wel as its availability and quality is not connected to showrooming behavior. Unlike other conflicting drivers, exploratory shopping was found to drive showrooming behavior. This is in line with Quach et al. (2022) and supports Konuş et al.’s (2008) findings regarding multichannel shopping. The managerial insights provided by this study help offline retailers develop strategies to prevent competitive showrooming. For them, company-related drivers are not easily managed because it is difficult to compete against online retailers in terms of monetary savings and wide assortment. Also, as offline service and brand loyalty do not decrease showrooming behavior, it seems that new means are needed to retain the potential showroomers loyal. By recognizing the customer segments and situations prone to showrooming, retailers can better target their measures. Also, high product involvement increases showrooming behavior, and we believe this is because consumers want to have more information about the product and different options when shopping for high-involvement products. Thus, sufficient product information provided by the store is recommended for preventing competitive M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 609 showrooming. This could also diminish one of the strongest drivers of this framework: the perceived usefulness of showrooming behavior. This study has certain limitations. Despite the conducted SLR, it is possible that some samples, especial y those of unpublished works and dissertations, have inadvertently been left outside this meta-analysis. In addition, the drivers have been analyzed separately, and thus some drivers might not necessarily have been found to have a statistical y significant effect on showrooming behavior if analyzed together in the same model. Also, regarding the socio-demographic drivers, it is worth noting that the effects must be interpreted with caution as typical y most studies were not representative samples of any target population. Future research should investigate the potential moderators for the identified drivers. For instance, the product type’s moderating effect could be investigated. Empirical future research could study novel showrooming drivers and platforms. For instance, consumers’ sustainability attitudes could be studied as a new driver for showrooming behavior. According to our SLR, sustainability attitudes have not been studied as drivers of showrooming behavior, although responsible consumers are suggested to search for sustainability information online (Holkkola et al., 2022b; Wilska et al., 2023). Also, future research should study which types of information and platforms would retain the potential showroomers in the same retailers’ offline or online channels. For instance, exploratory shopping via different in-store technologies (Paananen et al., 2023), immersive technologies, online showrooms, and metaverse environments could be studied. References Arora, S., Parida, R. R., Sahney, S. (2020). Understanding consumers’ showrooming behaviour: a stimulus–organism–response (SOR) perspective. International Journal of Retail & Distribution Management, 48(11), 1157¬–1176. Arora, S., Sahney, S. (2018). Antecedents to consumers’ showrooming behaviour: An integrated TAM-TPB framework. Journal of Consumer Marketing, 35(4), 438–450. Arora, S., Sahney, S., Parida, R.R. (2022). Drivers of showrooming behaviour: insights from integrated perspectives. International Journal of Retail & Distribution Management, 50(3), 398–413. Arora, S., Singha, K., Sahney, S. (2017). Understanding consumer’s showrooming behaviour: Extending the theory of planned behaviour. Asia Pacific Journal of Marketing and Logistics, 29(2), 409–431. Banerjee, S., Longstreet, P. (2016). Mind in eBay, body in Macy’s: Dual consciousness of virtuo-physical consumers and implications for marketers. Journal of Research in Interactive Marketing, 10, 288–304. 610 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Borges, R. M. F. (2018). Showrooming: a look into the Portuguese electronics market. Universidade Católica Portuguesa. Portugal. Brynjolfsson, E., Hu, Y., Rahman, M. (2013). Competing in the age of omnichannel retailing. MIT Sloan Management Review, 54, 23–29. Burns, D. J., Gupta, P. B., Bihn, H. C., Hutchins, J. (2018). Showrooming: An exploratory empirical investigation of students’ attitudes and behavior. Information Systems Management, 35(4), 294–307. Chimborazo-Azogue, L., Frasquet, M., Mol a-Descals, A., Miquel-Romero, M. (2021). Understanding mobile showrooming based on a technology acceptance and use model. Sustainability (Basel, Switzerland), 13(13), 7288. Chimborazo-Azogue, L. E., Mol á-Descals, A., Miquel-Romero, M., Frasquet, M. (2022). Mobile dependency and uncertainty reduction: Influence on showrooming behaviours and user- generated content creation. International journal of retail & distribution management, 50(8/9), 996–1014. Chokkannan, P., Bhavani Shankar, S., Pattusamy, M. (2023). The fashion retailer’s opportunity: Effect of deal-seeking behavior on mobile shopping intention of showroomers. Asia Pacific Journal of Marketing and Logistics, 35(5), 1187–1205. Christodoulides, G., Michaelidou, N. (2010). Shopping motives as antecedents of e-satisfaction and e-loyalty. Journal of Marketing Management, 27(1–2), 181–197. CSA. (2019). Survey: Webrooming, showrooming popular with shoppers https://chainstoreage.com/survey-webrooming-showrooming-popular-shoppers. Retrieved 31.01.2024. Dahana, W., Heejae, S., Katsumata, S. (2018). Influence of individual characteristics on whether and how much consumers engage in showrooming behavior. Electronic Commerce Research, 18. Davis, F.D. (1986). A technology acceptance model for empirical y testing new end-user information systems: theory and results. MIT Sloan School of Management, Cambridge, MA. Fan, X., Wang, J., Zhang, T. (2021). For showing only, or for selling? The optimal physical store mode selection decision for e-tailers under competition. International Transactions in Operational Research, 28(2), 764–783. Fang, J., Liu, H., Li, Y. (2021). Balance cues of online-offline channel integration: Considering the moderating role of customer's showrooming motivation. Information & Management, 58(8), 103535. Fiestas, J.C., Tuzovic, S. (2021). Mobile-assisted showroomers: understanding their purchase journey and personalities. Journal of Retailing and Consumer Services, 58, 102280. Flavián, C., Raquel, G., Carlos, O. (2019). Feeling confident and smart with webrooming: understanding the consumer's path to satisfaction. Journal of Interactive Marketing 47, 1–15. Frasquet, M., Miquel-Romero, M. J. (2021). Competitive (versus loyal) showrooming: An application of the push-pul -mooring framework. Journal of Retailing and Consumer Services, 62, 102639. Gensler, S., Neslin, S.A., Verhoef, P.C. (2017). The showrooming phenomenon: It’s more than just about price. Journal of Interactive Marketing, 38, Supplement C, 29–43. Goraya, M. A. S., Zhu, J., Akram, M. S., Shareef, M. A., Malik, A., Bhatti, Z. A. (2022). The impact of channel integration on consumers’ channel preferences: do showrooming and webrooming behaviors matter? Journal of Retailing and Consumer Services, 65, 102130. Grewal, D., Puccinelli, N., Monroe, K. B. (2018). Meta-analysis: Integrating accumulated knowledge. Journal of the Academy of Marketing Science, 46, 9–30. Grewal, D., Roggeveen, A.L., Nordfält, J. (2016). Roles of retailer tactics and customer-specific factors in shopper marketing: substantive, methodological, and conceptual issues. Journal of Business Research, 69(3), 1009–1013. Herrero-Crespo, Á., Viejo-Fernández, N., Col ado-Agudo, J., Sanzo Pérez, M. J. (2022). Webrooming or showrooming, that is the question: explaining omnichannel behavioural intention through the technology acceptance model and exploratory behaviour. Journal of Fashion Marketing and Management: An International Journal, 26(3), 401–419. M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 611 Holkkola, M., Frank, L., Kemppainen, T., Paananen, T., Luhtanen, V. (2022b). The role of sustainability in online customer experiences: A qualitative study on female fashion shoppers. In: Proceedings of the 14th Mediterranean Conference on Information Systems. Association for Information Systems. Holkkola, M., Nyrhinen, J., Makkonen, M., Frank, L., Karjaluoto, H., Wilska, T.-A. (2022a). Who are the showroomers? Socio-demographic factors behind the showrooming behavior on mobile devices. In: 35th Bled eConference: Digital Restructuring and Human (Re)action. Ed. by A. Pucihar, M. Kljajić Borštnar, R. Bons, A. Sheombar, G. Ongena, and D. Vidmar. University of Maribor, 113–128. Holkkola, M., Nyrhinen, J., Makkonen, M., Frank, L., Karjaluoto, H., Wilska, T. A. (2023b). Showrooming behavior, omnichannel self-efficacy, and perceived channel integration as antecedents of revisit intention. In: Proceedings of the 15th Mediterranean Conference on Information Systems. Association for Information Systems. Holkkola, M., Paananen, T., Frank, L., Kemppainen, T., Makkonen, M. (2023a). How do QR codes enhance customer experience? Omnichannel customer experiences in a brick-and-mortar fashion store. In: 36th Bled eConference: Digital Economy and Society: The Balancing Act for Digital Innovation in Times of Instability. Ed. by A. Pucihar, M. Kljajić Borštnar, R. Bons, G. Ongena, M. Heikkilä, & D. Vidmar. University of Maribor, 319–334. Huh, J., Kim, H. (2022). Showrooming versus webrooming: Examining motivational differences in omnichannel exploratory behaviors. The International review of retail, distribution and consumer research, 32(5), 532–548. Hunter, J. E., Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and Bias in research findings. Thousand Oaks: Sage Publications. Jeyaraj, A., Dwivedi, Y. K. (2020). Meta-analysis in information systems research: Review and recommendations. International Journal of Information Management, 55, 102226. Jo, W., Kim, J., Choi, J. (2020). Who are the multichannel shoppers and how can retailers use them? Evidence from the French apparel industry. Asia Pacific Journal of Marketing and Logistics, 33(1), 250–274. Kang, J. Y. M. (2018). Showrooming, webrooming, and user-generated content creation in the omnichannel era. Journal of Internet Commerce, 17(2), 145–169. Kolehmainen, T. (2018). The drivers of showrooming: The role of channel benefits, consumer attributes and products. Aalto University (Master's thesis). Konuş, U., Verhoef, P.C., Neslin, S.A. (2008). Multichannel shopper segments and their covariates. Journal of Retailing, 84(4), 398–413. Li, Y., Liu, H., Lim, E. T., Goh, J. M., Yang, F., Lee, M. K. (2018). Customer’s reaction to cross-channel integration in omnichannel retailing: the mediating roles of retailer uncertainty, identity attractiveness, and switching costs. Decision Support Systems, 109, 50–60. Lin, S.-W., Huang, E. Y., Cheng, K.-T. (2023). A binding tie: why do customers stick to omnichannel retailers? Information Technology & People, 36(3), 1126–1159. Lipsey, M. W., Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks: Sage. Liu, T., Liu, M. (2024). Does cross-channel consistency always create brand loyalty in omni-channel retailing? International journal of retail & distribution management, 52(1), 125–145. Makkonen, M., Frank, L., Paananen, T., Holkkola, M., Kemppainen, T. (2023). The cross-channel effects of in-store customer experience in the case of omnichannel fashion retailing in Finland. In: 36th Bled eConference: Digital Economy and Society: The Balancing Act for Digital Innovation in Times of Instability. Ed. by A. Pucihar, M. Kljajić Borštnar, R. Bons, G. Ongena, M. Heikkilä, D. Vidmar. University of Maribor, 609–625. Makkonen, M., Nyrhinen, J., Frank, L., Karjaluoto, H. (2022). The effects of general and mobile online shopping skilfulness and multichannel self-efficacy on consumer showrooming behaviour. In: 35th Bled eConference: Digital Restructuring and Human (Re)action. Ed. by A. Pucihar, M. Kljajić Borštnar, R. Bons, A. Sheombar, G. Ongena, and D. Vidmar. University of Maribor, 113–128. 612 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Mehra, A., Kumar, S., Raju, J. (2018). Showrooming and the competition between store and online retailers. Management Science, 64(7), 3076–3090. Park, H., Hur, W. M. (2023). Customer showrooming behavior, customer orientation, and emotional labor: Sales control as a moderator. Journal of Retailing and Consumer Services, 72, 103268. Peterson, R., Brown, S. (2005). On the use of beta coefficients in meta-analysis. Journal of Applied Psychology, 90(1), 175–181. Quach, S., Barari, M., Moudrý, D. V., Quach, K. (2022). Service integration in omnichannel retailing and its impact on customer experience. Journal of Retailing and Consumer Services, 65, 102267. Rajkumar, N., Vishwakarma, P., Gangwani, K. K. (2021). Investigating consumers’ path to showrooming: A perceived value-based perspective. International journal of retail & distribution management, 49(2), 299–316. Rapp, A., Baker, T. L., Bachrach, D. G., Ogilvie, J., Beitelspacher, L. S. (2015). Perceived customer showrooming behavior and the effect on retail salesperson self-efficacy and performance. Journal of Retailing, 91(2), 358–369. Retail Touch Points (2018). Shoppers prefer self-service kiosks, mobile-armed associates in-store. https://retailtouchpoints.com/topics/store-operations/study-reveals-why-96-of-shoppers- leave-stores-empty-handed Rose, S., Clark, M., Samouel, P., Hair, N. (2012). Online customer experience in e-retailing: an empirical model of antecedents and outcomes. Journal of Retailing, 88(2), 308–322. Rigby, D. (2011). The Future of Shopping. Harvard Business Review, 89(12), 64–75. Rust, R. T., Cooil, B. (1994). Reliability measures for qualitative data: Theory and implications. Journal of Marketing Research, 31(1), 1–14. Sahu, K. C., Naved Khan, M., Gupta, K. D. (2021). Determinants of webrooming and showrooming behavior: A systematic literature review. Journal of Internet Commerce, 20(2), 137–166. Schneider, P. J., Zielke, S. (2020). Searching Offline and Buying Online: An Analysis of Showrooming Forms and Segments. Journal of Retailing and Consumer Services, 52. Shankar, A., Gupta, M., Tiwari, A. K., Behl, A. (2021). How does convenience impact showrooming intention? Omnichannel retail strategies to manage global retail apocalypse. Journal of Strategic Marketing, 1–22. Sharma, N., Sharma, A., Dutta, N. Priya, P. (2023). Showrooming: a retrospective and prospective review using the SPAR-4-SLR methodological framework. International Journal of Retail & Distribution Management, 51(11), 1588–1613. Spaid, B. I., O’Neil , B. S., Ow, T. T. (2019). The upside of showrooming: How online information creates positive spil -over for the brick-and-mortar retailer. Journal of Organizational Computing and Electronic Commerce, 29(4), 294–315. Tyrväinen, O., Karjaluoto, H. Ukpabi, D. (2023). Understanding the role of social media content in brand loyalty: A meta-analysis of user-generated content versus firm-generated content. Journal of Interactive Marketing, 58(4). Verhoef, P. C., Kannan, P. K., Inman, J. J. (2015). From multi-channel retailing to omni-channel retailing. Journal of Retailing, 91(2), 174–181. Wilska, T.-A., Holkkola, M., Tuominen, J. (2023). The role of social media in the creation of young people’s consumer identities. Sage Open, 13(2) M. Holkkola et al.: The Drivers of Showrooming Behavior: A Meta-Analysis 613 Appendix 1: Selected samples and constructs for the meta-analysis Paper Selected constructs 1 Holkkola et al. (2023b) self-efficacy, age, gender, income sales staff assistance (offline service), feel of the product (off-line search value), socialization (social influence), subjective norm (social influence), online trust, perceived behavioral control (showrooming self- 2 Arora & Sahney efficacy), deals and discounts (mo-netary savings), cost (2018) savings (monetary savings), better pro-duct assortment (better assortment), ease of use of online pu-rchase, perceived usefulness of showrooming, attitude toward showrooming, perceived integration (channel integration) 3 Fang et al. (2021) information integration (channel integration), age, gender 4 Li et al. (2018) cross-channel integration (channel integration), age, gender, income 5 Liu & Liu (2024) brand loyalty 6 Shankar et al. (2021) attentiveness convenience (offline service), product involvement 7 Dahana et al. (2018) product involvement, age, gender smart shopper feelings (consumer innovativeness), 8 Rajkumar et al. (2021) enhanced product evaluation (offline search value), monetary savings 9 Chimborazo-Azogue et al. (2022) attitude toward showrooming 10 Quach et al. (2022) flow (exploratory shopping), reversed privacy risk (online trust), brand loyalty 11 Huh & Kim (2022) consumer innovativeness desire for social interaction (offline service), price 12 Kang (2018) comparison (monetary savings), assortment seeking (better assortment) 13 Borges (2018) brand loyalty, product involvement, age, gender, income 14 Burns et al. (2018) desire for customer service (offline service), brand loyalty reversed security risk (online trust), perceived 15 Kolehmainen (2018) behavioral control (showrooming self-efficacy), attitude toward showrooming 16 Chokkannan et al. (2023) product involvement, age 614 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Paper Selected constructs consumer empowerment (consumer innovativeness), 17 Goraya et al. (2022), sample 1 perceived assortment (better assortment), channel integration consumer empowerment (consumer innovativeness), 18 Goraya et al. (2022), sample 2 perceived assortment (better assortment), channel integration in-store search value (offline search value), 19 Arora et al. (2020) showrooming self-efficacy, attitude toward showrooming, product involvement perceived search benefits (offline search value), 20 Arora et al. (2017) subjective norm (social influence), showrooming self- efficacy, perceived behavioral control (showrooming self-efficacy), attitude toward showrooming subjective norm (social influence), ease of use of online 21 Chimborazo-Azogue et al. (2021) purchase, perceived usefulness of showrooming, product involvement exploratory information search (exploratory shopping), 22 Herrero-Crespo et exploratory acquisition (exploratory shopping), ease of al. (2022) use of online purchase, perceived usefulness of showrooming 23 Holkkola et al. (2022a) age, gender, income 24 Makkonen et al. (2022) self-efficacy EXPLORING SUSTAINABLE VALUE PROPOSITIONS IN LOGISTICS SERVICES – HOW DIGITALIZATION AND DATA SUPPORT GREENER SERVICES? FUAD KHAN, TAINA ERIKSSON, MARIKKA HEIKKILÄ, TITIANA ERTIÖ University of Turku, Turku School of Economics, Turku, Finland fuad.khan@utu.fi, taina.eriksson@utu.fi, marikka.heikkila@utu.fi, titiana.ertio@utu.fi Employing a multiple-case study method, this study examines the value propositions in sustainable logistics services. We explored publicly accessible data on the financial, functional, and emotional aspects of the value propositions offered by four companies. We first discovered that digitalization plays a crucial role in value proposition creation supporting low emission logistics ecosystems. Specifically, digital solutions promise cost reduction opportunities across the entire value chain as financial value proposition; route optimization and regulatory compliance as functional value propositions; and transparency, trust, and credibility as emotional value propositions. Second, the reduction of fossil fuels as a functional value proposition is identified as a target for all companies. Third, functional value propositions prelude emotional value propositions, i.e. through sustainability certificates, service providers establish transparency, ensure value chain compliance, and ultimately Keywords: sustainability, build trust among customers. By highlighting the differences and logistics, similarities in value propositions, this study explores sustainable value business model innovation and offers critical perspectives for proposition, digitalization, organizations aiming to shift towards sustainable logistic service green provision. service DOI https://doi.org/10.18690/um.fov.4.2024.36 ISBN 978-961-286-871-0 616 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Holistically, the EU aims to decrease its greenhouse gas emission through initiatives such as the European Green Deal and Fit for 55 legislative packages. Simultaneously, the twining of the green with digital transition (i.e., the transitions mutual reinforcement) provides ample opportunities to achieve EU’s climate neutrality goals by 2050. The logistics industry, recognized as a vital component of the global economy, generates a third of global carbon dioxide (CO2) emissions and has come under increasing pressure to minimize its environmental impact (Towards a Net- zero Logistics Sector, 2023). Notably, the European Commission has implemented a series of policies aimed at reducing the industry's carbon emissions (Completion of Key ‘Fit for 55’ Legislation, 2023). In the face of such pressure towards sustainability, logistics companies have started to rethink and innovate their business models, to both align with environmental goals but also carve out a competitive edge in the changing market/ industry. The exploration of innovative, sustainable business models is not merely an environmental necessity but a strategic choice to stay relevant and appeal to a consumer base which increasingly values sustainability (Bask et al., 2018; Martins et al., 2019). Sustainable business models cover both environmental and social dimensions of sustainability (Mignon & Bankel, (2022); Brenner, 2018). The present study focuses deeply on the environmental sustainability of business models, specifically on carbon dioxide emission reductions. Hence, the concept of sustainable business model is used in this study to refer to environmentally sustainable business model. Central to sustainable business models is the customer value proposition (De Giacomo & Bleischwitz, 2020). The value proposition reflects the company's commitment to delivering value that goes beyond the traditional economic metrics, encompassing environmental stewardship. However, defining what exactly constitutes a sustainable value proposition remains a challenge. Digital technologies hold immense potential to achieve climate neutrality as exemplified in the European Commission’s Twin Transition efforts. However, the extent to which digital technologies are integrated into sustainable value propositions is still under- explored. Hence, our central research question is: How does digitalisation support value propositions in environmentally sustainable business models in logistics services? Employing a comparative case study methodology, this paper examines sustainable value propositions of four logistics companies. Our F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 617 investigation first delves into the value companies propose, while later illustrating the role of digitalization in providing environmentally sustainable logistics services. The market for sustainable logistics services is still in its infancy, with cases analysed in this paper emerging among the first services offered. The drivers for these services stem from the interplay between market opportunities, increased regulatory demands to lower carbon dioxide emissions and company’s desire to contribute to environmental sustainability. Emission data provision contributes to transparency in operations. This study is organized as follows: Section 2 reviews relevant literature on sustainability in the logistics industry, digital technologies supporting the sustainability transition, and sustainable BMs including their value propositions. Section 3 describes the methodology behind our multiple-case study. Section 4 presents the findings of the cross-case analysis. Section 5 discusses the implications of our findings and concludes the paper, summarizing key insights and proposing directions for future research. 2 Literature review 2.1 Sustainability in the logistics industry The push towards sustainability is increasingly capturing the interest of international transport and logistics providers, driven by market demands. They view sustainability to gain a competitive edge (Bask et al., 2018; Martins et al., 2019). Incorporating sustainability into logistics requires re-evaluation of conventional operations to reduce environmental impact and improve efficiency. However, researchers found that due to inconsistent measurement method of sustainability indicators across the supply chain, logistics providers fail to share cost and benefits with partners and differentiate themselves from competitors (Bask et al., 2018; Martins et al., 2019). Like fuel cost, amount of emissions is an indicator of sustainability. Replacing fossil fuels with less emitting alternatives is the major mean to make logistics more environmentally sustainable. While Electrifying freight deliveries has significant prospect to diminish the CO2 emissions of the sector (Gillström, 2023), especially in land logistics, LNG, biofuels, methanol and ammonia are potential renewable 618 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION energy sources in sea logistics (Solakivi et al., 2022). Also speed and route optimization has received much attention in past research as it can both save fuel cost and achieve emission reduction. One strategy is to design new vessels with a lower maximum speed. A notable example is Maersk's recent initiative, where they decreased the top speed of their new vessels from the range of 20–26 knots down to 23 knots. This adjustment led to 20% reductions in both fuel consumption and carbon emission (Macguire, 2013). However, there are drawbacks to consider with new constructions, including the emissions generated during the shipbuilding process and the high initial investment required. The second alternative is to reduce the speed of the vessel; higher fuel prices or lower freight rates encourage slower sailing. Slow sailing impacts the cargo owners most, leading to higher freight costs and delayed deliveries. This may push them to choose faster and potential y more polluting transport modes like rail or air. Interestingly, higher-value cargo tends to sail faster and hence emits more. Port congestion also affects this dynamic; optimizing arrival times can reduce emissions, highlighted by initiatives like "virtual arrival" and financial incentives (Psaraftis and Kontovas, 2015; Bektaş et al., 2019). Adopting green practices in logistics involves navigating through various options, each presenting its unique set of benefits and compromises. For instance, opting for reusable materials can enhance resource efficiency and reduce expenses, but it may inadvertently increase carbon emissions during the return logistics phase. In this context, adopting renewable energy stands out as an effective approach to reduce emissions while also offering economic advantages (Bhattacharya et al., 2016; Khan & Dong, 2017; Trivellas et al., 2020). Selecting various modes of transport, the average frequency of mode changes, haul duration, incidence of empty runs, energy efficiency per transport mode, and emissions per unit of energy consumption are al critical elements for developing sustainable transport routes. (McKinnon, 2016; Trivellas et al., 2020). 2.2 Digital technologies as means to achieve logistics sustainability Digitalization, leveraging data, and improved computational capabilities, plays another important role in transforming industries by creating innovative services and business models, enhancing operational efficiency, and enabling cost savings for further digital investments (Tagliapietra et al. 2020; Agarwala et al., 2021; Broccardo F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 619 et al., 2023). It encompasses both information and operational technology tools to streamline processes and reduce transactional costs (Felser et al., 2019; Mas et al., 2020; Broccardo et al., 2023). Furthermore, empirical research in European cargo transportation context shows that information and communication technology can be utilised for sustainability promotion (De Andres Gonzalez et al., 2021). In logistics, technologies such as robotics, AI, and big data offer significant decarbonization potential (Brinken et al., 2023). Various digital solutions, from sensors to blockchain, aim to optimize operations and reduce emissions (Durkin, 2021; Plaza-Hernández, et al. 2021; Lind et al., 2020; Boison & Antwi-Boampong, 2020; Di Silvestre et al., 2018; N. Agarwala & Guduru, 2021; Agarwala et al., 2021). Sustainable business models highlight the role of digital platforms in facilitating resource sharing, co-creation, and operational efficiencies, suggesting that digital tools are instrumental but not sufficient without integrating green technologies for true decarbonization (De Andres Gonzalez et al. 2021; Hiteva & Foxon, 2021; Broccardo et al., 2023). 2.3 Value proposition in sustainable business model Business models outline how organizations generate, deliver, and capture value, as described by Teece (2010). The business model canvas presented by Osterwalder & Pigneur (2010) includes various elements like the value proposition, customer engagement and segmentation, distribution channels, essential activities and resources, partnerships, and the revenue strategy. Companies can adjust these elements to seize new opportunities for growth or adapt to shifts in the market (Saebi et al., 2017; Eriksson et al., 2022). Sustainable business models can integrate economic, environmental, and social dimensions. As highlighted by researchers such as Lüdeke-Freund (2019), Schaltegger et al. (2012) and Schneide and Spieth (2013), sustainable business model aims to create positive impact and/or reduce negative impact on environment and society, while providing value generation and capture opportunity for all parties involved (Bocken et al., 2014, p. 44; Chevrollier et al., 2023). Sustainable forerunner firms may employ long term partnerships with their stakeholders to share their resources and knowledge. Likewise, rethinking the key partners wil include partners who support the creation of sustainable ecosystems (Broccardo et al., 2023). 620 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION There are substantial evidences from recent research that especialy the value proposition is the most central component of both traditional and Sustainable Business Model (De Giacomo & Bleischwitz, 2020; Eyring et al., 2011; Geissdoerfer et al., 2018; Schaltegger, Hansen, & Lüdeke-Freund, 2016; Bähr & Fliaster, 2022; Baldassarre et al., 2017; Boons & Lu; deke-Freund, 2013; Kristensen and Remmen, 2019; Laukkanen & Tura, 2022). Using value propositions firms can solve environmental problems through their products and services (Awan, 2020; Baldassarre et al., 2017; Chevrollier et al., 2023). New entrants or the biggest players in the market can also disrupt the market's established value propositions and propose a new type of values using non-conventional ways, resources and by revamping processes (Brenner, 2018; Broccardo et al., 2023). Hence, service providers must understand clearly what customers perceive as valuable, because the mismanagement of the expected and realised value will have implications and impacts on stakeholders (Hlady-Rispal and Servantie, 2018; Chevrollier et al., 2023). Customers likely value something that solves their problems or meet their needs, which would help service providers to build a sustainable value proposition (Foss & Saebi, 2016; Schneider & Spieth, 2013; Chevrollier et al., 2023) through which customers gain certain benefits (Patala et al., 2016; Osterwelder & Pigneur, 2010; Laukkanen & Tura, 2022). 2.4 Conceptual Framework To provide a value adding solution, firms must understand the utility customers gain from the value. Based on the utility gain, value proposition dimensions are categorised into financial (Borg et al., 2020), functional, emotional, and social (Sheth et al., 1991; Gilly et al., 1992; Sweeney & Soutar, 2001), epistemic i.e., utility of gathering new knowledge through products and service and conditional i.e., utility obtained in a particular situation (Sánchez-Fernández & Iniesta-Bonillo, 2007; Zeithaml et al., 2020; Laukkanen & Tura, 2022). We adapt the initial conceptual frameworks developed by Sheth and colleagues (1991), Gilly et al. (1992) for the logistics transportation industry. In this context, the epistemic and conditional dimensions have been excluded as they are linked to the customers, whereas here we focus on service providers’ perspectives only and their value proposition for achieving sustainability in logistics. Our working hypothesis is that businesses which provide products and services will prioritize financial, F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 621 functional, and emotional value propositions over epistemic and conditional value propositions. Epistemic value (i.e., value accrued through new knowledge) can be beneficial specifical y to end consumers rather than service providers. Service providers are continuously pressured by environmental regulations and thus already familiar with sustainability added value than consumers. Similarly, understanding the conditional value of green logistic services would be challenging as such services are new to the market and would require in-depth understanding of the customer context in which logistics services are purchased. In summary, our framework for sustainable value propositions is broken down into three dimensions: 1. Financial (Borg et al., 2020): The financial aspect focuses on cost reduction and avoidance, specifically in the process of fuel consumption, regulatory charges, and material costs. This implies that sustainability efforts can lead to significant savings and more efficient resource utilization from a financial perspective. In addition, by taking sustainability initiatives and abiding by the regulations, firms can avoid regulatory penalties. 2. Functional (Sheth et al., 1991; Gilly et al., 1992; Sweeney & Soutar, 2001): Functionality is enhanced through improved performance efficiency, which includes speed, waste reduction, fuel consumption reduction (by decreasing fossil fuel dependency), and optimized routing to reduce idle time at ports. Additional y, the sharing of real-time data and information is highlighted, suggesting that timely and accurate information flow can greatly enhance operational efficiency. Moreover, replacing fossil fuel with renewables also indicates sustainable performance improvement. 3. Emotional (Sheth et al., 1991; Gilly et al., 1992; Sweeney & Soutar, 2001): On an emotional level, the goal is to elicit a positive reaction. This suggests that sustainability initiatives should also resonate with customers and other stakeholders on a level beyond mere financial or functional benefits, possibly enhancing the company's reputation. 4. Digital technologies: serve to achieve logistics sustainability, suggesting that leveraging technology is central to making these dimensions work together effectively in pursuit of sustainable operations. 622 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION A key objective of this framework is the reduction of carbon emissions, aligning business practices with environmental sustainability goals. Figure 1: Framework for sustainable value proposition dimensions Source: adapted from Sheth et al. (1991), Gilly et al. (1992), Sweeney & Soutar (2001) 3 Methodology This study’s aim is to examine sustainable value propositions of logistics services. With tightening emissions regulations and standards, the cases studied here exemplify companies that meet these chal enges by providing innovative, environmental y sustainable services to their customers. The primary focus in each case was the value proposition; our secondary focus was to il ustrate how digitalisation supports these value propositions. Case selection criteria: The cases were selected through convenience sampling, because there is a limited number of sustainable logistics services on the market with sufficient public data available to study. The selection aimed to contrast and compare services offered by both large and small businesses, covering global and regional scopes, as well as single and multi-modal transportation options. Consequently, this methodology resulted in the inclusion of four cases in the final analysis: two cases represent large, global, and multi-modal logistics services, two cases contrast as small, local, and single-modal transportation services. Data: The data analysed in this study is secondary data that was col ected from public sources, such as companies’ web pages, corporate responsibility reports, press releases, articles, or news items. The data was col ected through online searches with F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 623 the company and service name, Web of Science and Scopus databases for academic publications on sustainable logistics or publications containing the exact service name. Method and analysis: Following Yin (2012), we used the case study method to investigate each value proposition in each company’s service business model. The analysis of each case is descriptive in that it provides details on specific dimensions of the value proposition as illustrated in Figure 1. We conducted a multiple-case study and cross-case synthesis of the qualitative data (Yin, 2012). All four authors contributed to the analysis to ensure consistency and triangulation. Cross-case synthesis initiated with a review of key-information on individual case studies. Later, after the authors searched for patterns across the four cases, the interpretation of findings has been organized according to the sustainable value proposition dimensions in the multiple-case study (Figure 1 and Table 1). 4 Results 4.1 Small, local & single mode case descriptions Posti Green Freight (Posti, n.d.) is a fossil-free land transportation service offered to business customers. The service is operated based on Book & Claim model (Centre, n.d.), where the customer purchases a share of fossil-free transportation service. Based on the purchase the share of fossil-free transportation is earmarked to the customer. The customer pays a premium for the green transportation service. The Green Freight service is produced with fossil-free energy sources (electricity, biogas or green hydrogen). The customer receives an emissions report four times a year. External verification is in place to ensure the use of fossil-free energy and prevent double-counting. Viking Line green marine transportation service (Viking Line LNG, 2024) has been available for passengers since the summer of 2023, and it has recently been extended to include the B2B segment. The service is based on biofuel surcharge and available only on routes with appropriate vessels and infrastructure in place for using biofuels. Customers opt to pay the biofuel surcharge when booking the transportation service. Viking Line has calculated the energy it takes to realize the purchased service and orders the equivalent amount of biofuel. The calculation of 624 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION the energy required is based on annual averages to even out the impact of severe weather conditions. The logic of this green service is that customer demand defines how much the service provider is purchasing biofuels. 4.2 Large, global & multi-modal case descriptions Maersk Eco Delivery (Maersk ECO Delivery, n.d.) is a multimodal decarbonization solution to green ocean logistics by replacing fossil fuels with biofuels or methanol. Maersk’s Eco Delivery uses “mass balance chain of custody” model to reduce emission with sustainable fuels even in specific shipments across its global network. Customers are not required to invest long-term and have the flexibility to apply Eco Delivery to their specific location and product need. The payment model for this service is based on greenhouse gas emission reduction per trade. In practice, customers ask sales representatives to add Eco Delivery in their contract and Maersk provides yearly CO2 savings certificates to their customers. DHL Go Green solution (DGF_2017_GoGreen-Solutions_EN, n.d.) is a portfolio of multimodal services to reduce CO2 emissions in the logistics transportation chain. The Go Green solution portfolio consists of three offerings: carbon transparency, carbon footprint optimization, and carbon emission offsetting. Carbon transparency is a digital toolkit consisting of a carbon report, carbon dashboard, and DHL quick scan. Carbon footprint optimization is a service that helps customers plan their transports more efficiently and sustainably by utilising modal shifts or consolidation of shipments. These services either provide above-industry average carbon efficient freight services or completely replace heavy fuel with biofuel. Carbon emission of set ing al ows customers to offset their carbon footprint with certified and verified climate protection projects. DHL Go Green solution customers receive annual reports and certified carbon reports. Customers need to pay a premium for the service. 4.3 Findings from the cross-case analysis of value propositions in logistics services The cases examined differ substantial y in service offerings and regarding the availability of public data on their services. For the global cases - Maersk Eco Delivery and DHL Go Green services - there was detailed information available and F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 625 hence our analysis was able to capture their services wholistical y. As for the local cases - Posti Green Freight and Viking Line Green Transportation Service - only a top-level value proposition was available from public resources. In practice, this means that even though these companies pursue carbon reduction for themselves and their customers, detailed value propositions in different dimensions (Figure 1) were not found at the time of the study. Posti Green Freight and Viking Line Green Transportation Service were only recently launched on the market, which may be one of the reasons for the scarcity of available information. Next, we turn to analyse the cases against the framework developed in Figure 1. 4.3.1 Financial value proposition Under the financial dimension, we observe that DHL and Maersk services, customers retain the power to govern their cost. For example, customers do not need to commit to the Maersk Eco Delivery service long-term, but they are able to obtain the service whenever needed. Moreover, customers always know the price they must pay and therefore they can circumvent price uncertainty and futureproof their logistics costs. Similarly, while DHL’s Go Green solution promises “cost optimization opportunities”, the means to secure that promise is different from Maersk. DHL provides a “carbon dashboard”, where customers can investigate cost improvement areas by themselves. Paired with their logistics consulting services, DHL offers “transport network optimization” expertise that wil ensure cost reduction and profit maximization potential. For Posti and Viking Line, no information on financial value proposition could be found. 4.3.2 Functional value proposition In the functional dimension, we identified several technology-driven performance improvements, which are likely to increase logistics sustainability (Fig.1). Waste reduction is observed in Maersk’s and Viking Line’s operation. Maersk reuses waste and residues for fuel feedstock, which in turn decreases waste. Likewise, in Viking Line’s liquified biogas used in their ships is produced from waste and residues from households, agriculture, and food industries. In addition to sourcing food waste from industries, Viking Line recycle the ship’s food waste into biogas production. Route optimization is found in DHL’s Go Green solution value proposition under the carbon dashboard service. There, customers can model their 626 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION entire supply chain, simulate multiple transportation modes, and find the optimal route with the smal est carbon footprint. Information or data sharing is another feature in DHL’s carbon dashboard report service. DHL offers an “end to end” supply chain visibility which integrates reports from all service providers in the value chain. Next, DHL guarantees carbon ef icient ships in the marine portfolio with the claim of “5% more carbon efficient than industry average”. This also fulfils the performance efficiency value proposition; however, the basis of such efficiency gain is not clear due to limited information. Posti and Viking Line replace fossil fuels with renewables and leverage data to define the amount of energy needed to produce the service bought by the customer. 4.3.3 Emotional value proposition The cross-case analysis revealed several types of value propositions connected to emotional responses employed by the companies to deliver messages of their environmental solution. These are “creating good feeling and positive image”, “convenience and flexibility of the service”, “commitment to green transition”, “trust and credibility”, “taking responsibility and being transparent”, “customizable service” and “adaptable service per external changes”. The first 4 were mentioned more often than others. Creating a good feeling for customers and establishing positive brand image are communicated by two of the cases, Posti’s Green Freight and Viking Line’s Green Transportation services. Posti and Maersk propose convenience and flexibility of their service offering, Viking Line and DHL do not communicate such propositions. For example, to choose Posti’s Green Freight service, no separate agreement is needed; with Maersk, customers can choose the Eco Delivery service flexibly according to their location, product preference, and sustainability agenda. Commitment to the green transition is found to be another example of value proposition in the case of Maersk’s Eco Delivery and DHL’s Go Green solution. Maersk’s commitment is visible through its promise to go beyond the legislative boundary to reduce emissions. On the other hand, DHL’s commitment starts from the top-level discussions on climate impact, which is operationalised through their core businesses. In relation to t ransparency, Maersk, for example, wants to be credible by providing certification of traceable feedstock source and proof of sustainability with CO2 certificate. They point out that they partner with trusted certification bodies. Similarly, Viking Line wants to “conduct business in a transparent and proper manner” to gain customers’ trust. While there are no explicit F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 627 value propositions on transparency in Posti and DHL’s case, there is implicit transparency as value proposition in DHL’s carbon dashboard solutions. In general, al case companies provide reports, which indicates their attempt in being transparent about their carbon emission data. Final y, two other value propositions are observed, that stand on their own. First is customisation: as part of DHL’s Go Green solution, DHL logistics consulting offers customizable environmental performance improvement recommendations. Second, adaptability is communicated by DHL as wel ; they promise to adjust their services as per the regulations changes so that their customers can reach their own environmental targets. 4.3.4 Digital technologies supporting sustainable value propositions From the customer’s perspective, our cross-case analysis reveals that digitalization and data support logistics transportation service provision at key touchpoints: 1. Pre-service purchase, when customers evaluate among and choose transportation alternatives that best meet their needs. 2. Tracking during service delivery 3. Post-transport, by offering visibility into the emission data generated and validated by third parties as wel as reporting for the customer. Based on the publicly available data, it is challenging to paint a complete picture of the role of data and digitalization in the service delivery process. Posti Green Freight uses data in allocating the share of fossil-free service to the customers that have paid for it. In a similar vein, Viking Line’s Green Transportation service uses yearly ship fuel consumption as a baseline to calculate the energy needed to transport the passenger or the cargo that purchased the green service and then, Viking Line purchases the equivalent amount of renewable fuel. Although Maersk offers an AI-driven platform cal ed “StarConnect” pooling vessels’ big data to forecast and thus improve fuel consumption need, energy efficiency, and reduce carbon emissions (A.P. Møller - Mærsk A/S, 2023, p. 29, 45), the company does not disclose whether StarConnect is related to the solution under investigation in the Eco Delivery case. 628 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 1: Summary of the value propositions in the four sustainable logistics service cases Value proposition dimensions Studied Cases Financial Functional Emotional Good feeling & Replace fossil fuels (with biogas and positive image; Posti Green electric transportation); Aid in n/a* Convenience & Freight achieving sustainability targets & flexibility compliance; Verified emission report Replace fossil fuels (with Bio-LNG); Good feeling & Viking Line Waste reduction; Aid in achieving positive image; Green Marine n/a* sustainability targets & compliance; Responsibility & Transportation Certified emission report/ transparency Service Sustainability certificate Convenience & Replace fossil fuels (with biofuel and flexibility; Maersk Eco methanol); Waste reduction; Aid in Cost reduction Commitment; Delivery achieving sustainability targets & Trust & compliance; Certified emission report credibility Cost reduction Replace fossil fuel (with biofuel); (through Carbon efficiency; Route optimization; Commitment; DHL Go Green optimization); Information sharing; Aid in achieving Customization; Solution Profit sustainability targets & compliance; Adaptability maximization Certified emission report Digital technologies: carbon dashboards, data analytics to calculate the share of fossil-free fuel provision, certified emission reporting, digital tools for route simulation and optimisation * Refers to the situation when a) when information could not be retrieved from public sources and/or b) at the time of writing, the value proposition does not include this dimension. There are notable differences between the cases in what kind of data is visible and reported to the customer. DHL is perhaps the most advanced in this respect, offering customers carbon reports, carbon dashboard, and DHL quick scan. customers can simulate and optimise routes with modal shifts or consolidation of shipments. The customers of Posti and Viking Line can opt for the green service when booking their journeys, typical y online. Posti offers its Green Freight customers four times a year an emission report that shows the share of fossil-free transportation al ocated to the company. Viking Line has been offering biofuel F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 629 surcharge to freight customers only recently and, as of now, is not communicating publicly how the emission reductions are reported to the freight customers. 5 Discussion, limitations, and conclusions While previous research has shown logistics providers’ failure to share cost and benefits with partners and differentiate themselves from competitors for sustainable logistics (Bask et al., 2018; Martins et al., 2019), our analysis suggests that digital solutions may help providers deliver both financial and functional value propositions and contribute to sharing costs and benefits of decreased emissions. Company size and scope (local vs global) also link to the value proposition capability: while the local cases do not offer financial value to their customers, the two global cases promise cost reductions through optimization to the customers., at least in the cases of global reach the global cases appear more mature compared to the local ones, also representing the transportation chain more comprehensively, even door-to-door, which means improved possibilities of multi-modal optimization. Digitalization was found to be a distinguishing factor among the cases: large global companies leverage more advanced digital technologies compared to smal er, recently established services. By examining how logistics service providers collate data on emissions, utilize and verify the data and deliver reports, the study expands our understanding of the role of data and digitalization in logistics sustainability. Digitalization and data are found to support or enable the green transportation services throughout the entire customer journey. This study also illustrate that digitally savvy firms can propose multiple value dimensions whereas the scope of new and cross dimensional value proposition for digitally young firms is limited. Digitalization facilitates the collection and calculation of emission data, which is a backend but central task for supporting sustainable logistics. For the local companies, providing emission reports to their B2B customers is functional value proposition, however, larger companies have moved further by incorporating digital elements (check table 1) in offerings and therefore both functional and financial value proposition are visible in collection and calculation of emission data. 630 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Furthermore, we observed interconnected relationship between functional and emotional value propositions. Al examined case companies communicated reduction of fossil fuel use as their core functional value proposition. In addition, all companies verified their carbon emissions reduction calculations through an independent third party. This independent certification process in turn not only adds credibility to the environmental claims of the services but also demonstrates their commitment to sustainability. By doing so, these companies establish trust and goodwil among consumers who prioritize environmental responsibility. Our findings are consistent with previous studies that businesses can address environmental issues through the value propositions in their products and services (Awan, 2020; Baldassarre et al., 2017; Chevrollier et al., 2023). In this study, we also uncovered various examples of emotional value propositions in the sustainable logistic services. The services analysed connect emotions with their offerings, demonstrating how sustainable logistics services assist customers in reaching their sustainability objectives. This communication builds trust in service providers as reliable partners. The number of emotional value propositions found in the case companies in comparison to their financial value proposition may indicate that companies are aware of the importance of communicating about their sustainability initiatives to their customers. On the other hand, the smaller service providers emphasize emotional value over functional or financial value, whereas the more mature service providers focus on functional and financial ones. Customers value services that solve their problems or meet their needs, which is an essential consideration for service providers in building a sustainable value proposition (Foss & Saebi, 2016; Schneider & Spieth, 2013; Chevrollier et al., 2023) as wel as services through which customers gain certain benefits (Patala et al., 2016; Osterwalder & Pigneur, 2010; Laukkanen & Tura, 2022). In the case of green logistics services, customers increasingly demand reporting and transparency in service providers’ operations. Customers greatly benefit from obtaining verified emission data: first in ensuring compliance, second establishing a baseline, and third following up on their own progress in emission reductions. From our cross-case analysis, we observed that all companies provide reports that contain emission reduction data, which becomes a necessary requirement for green logistics services. Differentiation among service providers is based on additional aspects, such as the inclusion of digital elements in offerings as discussed earlier. F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 631 This research has focused on exploring value propositions and the role of digital technologies in achieving sustainable transportation solutions from the viewpoint of service providers. This perspective, while valuable, presents certain limitations. First, our analysis is confined to the value propositions as presented by service providers, which does not encompass the customers' perspective, i.e., the value perception. Understanding how customers view and value these sustainable practices could significantly enrich the narrative around sustainable logistics and offer a more rounded perspective on the effectiveness of these services. Another limitation in our study is the reliance on publicly available data. Although we have opted for the service provider's perspective with an assumption that the quality of such data would suffice for customers to evaluate potential partners, this approach limits the scope of our analysis. Final y, our sample of four case companies inhibits the generalizability of our findings. As the relationship between sustainability and digitalization is expected to become more pronounced and complex, this twin transition provides a fertile ground for future research to delve into the role of digital technologies to help improve sustainability in logistics. By identifying and elucidating the synergies between sustainability and digitalization, this research contributes to the ongoing dialogue on twin transition and BM innovation, offering actionable insights for logistics companies and stakeholders aiming to advance towards more sustainable operations amidst global environmental chal enges. Acknowledgements The authors would like to thank two anonymous reviewers for their invaluable feedback in developing this study. This study is supported with funding from Business Finland for the Green Connect project (grant no. 3167/31/2023) 632 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION References A.P. Møller - Mærsk A/S. (2023). Annual report. Retrieved March 7, 2024, from https://investor.maersk.com/news-releases/news-release-details/annual-report-2023 Agarwala, N., & Guduru, S. S. (2021). The potential of 5G in commercial shipping. Maritime Technology and Research, 3(3), Manuscript. https://doi.org/10.33175/mtr.2021.248995 Agarwala, P., Chhabra, S., & Agarwala, N. (2021). Using digitalisation to achieve decarbonisation in the shipping industry. Journal of International Maritime Safety, Environmental Affairs, and Shipping, 5(4), 161–174. https://doi.org/10.1080/25725084.2021.2009420 Bähr, K., & Fliaster, A. (2022). The twofold transition: Framing digital innovations and incumbents’ value propositions for sustainability. Business Strategy and the Environment, 32(2), 920–935. https://doi.org/10.1002/bse.3082 Bask, A., Rajahonka, M., Laari, S., Solakivi, T., Töyli, J., & Ojala, L. (2018). Environmental sustainability in shipper-LSP relationships. Journal of Cleaner Production, 172, 2986–2998. https://doi.org/10.1016/j.jclepro.2017.11.112 Bektaş, T., Ehmke, J. F., Psaraftis, H. N., & Puchinger, J. (2019). The role of operational research in green freight transportation. European Journal of Operational Research, 274(3), 807–823. https://doi.org/10.1016/j.ejor.2018.06.001 Bhattacharya, M., Paramati, S. R., Öztürk, İ., & Bhattacharya, S. (2016). The effect of renewable energy consumption on economic growth: Evidence from top 38 countries. Applied Energy, 162, 733–741. https://doi.org/10.1016/j.apenergy.2015.10.104 Bocken, N., Rana, P., & Short, S. W. (2015). Value mapping for sustainable business thinking. Journal of Industrial and Production Engineering, 32(1), 67–81. https://doi.org/10.1080/21681015.2014.1000399 Boison, D. K., & Antwi-Boampong, A. (2020). Blockchain ready port supply chain using distributed ledger. Nordic and Baltic Journal of Information and Communications Technologies. https://doi.org/10.13052/nbjict1902-097x.2020.001 Borg, D. R., Mont, O., & Schoonover, H. A. (2020). Consumer Acceptance and Value in Use- Oriented Product-Service Systems: Lessons from Swedish Consumer Goods Companies. Sustainability, 12(19), 8079. https://doi.org/10.3390/su12198079 Brenner, B. (2018). Transformative Sustainable Business Models In the Light of the Digital Imperative—A Global Business Economics Perspective. Sustainability, 10(12), 4428. https://doi.org/10.3390/su10124428 Brinken, J., Behrendt, F., & Trojahn, S. (2023). Comparing decarbonization potential of digital and green technologies. Sustainable Futures, 6, 100125. https://doi.org/10.1016/j.sftr.2023.100125 Broccardo, L., Zicari, A., Jabeen, F., & Bhatti, Z. A. (2023). How digitalization supports a sustainable business model: A literature review. Technological Forecasting and Social Change, 187, 122146. https://doi.org/10.1016/j.techfore.2022.122146 Centre, S. F. (n.d.). Smart Freight Centre. https://smartfreightcentre.org/en/about-sfc/news/new- accounting-framework-to-address-barriers-to-transport-decarbonization/ Chevrol ier, N., Argyrou, A., Ainiwaer, N., & Nijhof, A. H. (2023). On the encroachment of sustainable value propositions: Business model innovation for impact. Journal of Cleaner Production, 382, 135341. https://doi.org/10.1016/j.jclepro.2022.135341 Completion of key ‘Fit for 55’ legislation. (2023, October 9). European Commission - European Commission. https://ec.europa.eu/commission/presscorner/detail/en/IP_23_4754 De Andres Gonzalez, O., Koivisto, H., Mustonen, J. M., & Keinänen-Toivola, M. M. (2021). Digitalization in Just-In-Time approach as a sustainable solution for maritime logistics in the Baltic Sea Region. Sustainability, 13(3), 1173. https://doi.org/10.3390/su13031173 De Giacomo, M. R., & Bleischwitz, R. (2020). Business models for environmental sustainability: Contemporary shortcomings and some perspectives. Business Strategy and the Environment, 29(8), 3352–3369. https://doi.org/10.1002/bse.2576 F. Khan et al.: Exploring Sustainable Value Propositions in Logistics Services – How Digitalization and Data Support Greener Services? 633 DGF_2017_GoGreen-Solutions_EN. (n.d.). [Video]. DHL. https://www.dhl.com/global- en/home/our-divisions/global-forwarding/special-expertise/gogreen-solutions.html Di Silvestre, M. L., Favuzza, S., Sanseverino, E. R., & Zizzo, G. (2018). How Decarbonization, Digitalization and Decentralization are changing key power infrastructures. Renewable & Sustainable Energy Reviews, 93, 483–498. https://doi.org/10.1016/j.rser.2018.05.068 Eriksson, T., Heikkilä, M., & Nummela, N. (2022). Business model innovation for resilient international growth. Smal Enterprise Research, 29(3), 205–226. https://doi.org/10.1080/13215906.2022.2092890 Felser, M., Rentschler, M., & Kleineberg, O. (2019). Coexistence standardization of operation technology and information technology. Proceedings of the IEEE, 107(6), 962–976. https://doi.org/10.1109/jproc.2019.2901314 Foss, N. J., & Saebi, T. (2016). Fifteen years of research on business model innovation. Journal of Management, 43(1), 200–227. https://doi.org/10.1177/0149206316675927 Gillström, H. (2023). Regional Electrified Logistics: Review of research projects, academic publications, and perspective of the industry. https://doi.org/10.3384/9789180750943 Gil y, M. C., Sheth, J. N., Newman, B. I., & Gross, B. L. (1992). Consumption Values and Market Choices: Theory and Applications. Journal of Marketing Research, 29(4), 487. https://doi.org/10.2307/3172719 Hiteva, R., & Foxon, T. J. (2021). Beware the value gap: Creating value for users and for the system through innovation in digital energy services business models. Technological Forecasting and Social Change, 166, 120525. https://doi.org/10.1016/j.techfore.2020.120525 Khan, S. a. R., & Dong, Q. (2017). Impact of green supply chain management practices on firms’ performance: an empirical study from the perspective of Pakistan. Environmental Science and Pollution Research, 24(20), 16829–16844. https://doi.org/10.1007/s11356-017-9172-5 Laukkanen, M., & Tura, N. (2022). Sustainable value propositions and customer perceived value: Clothing library case. Journal of Cleaner Production, 378, 134321. https://doi.org/10.1016/j.jclepro.2022.134321 Lüdeke-Freund, F. (2019). Sustainable entrepreneurship, innovation, and business models: Integrative framework and propositions for future research. Business Strategy and the Environment, 29(2), 665–681. https://doi.org/10.1002/bse.2396 Macguire, E. (2013, June 26). Maersk ‘Triple E’: Introducing the world’s biggest ship. CNN. https://edition.cnn.com/2013/06/26/business/maersk-triple-e-biggest-ship/index.html Maersk ECO Delivery. (n.d.). Low-emissions Shipping | Maersk. https://www.maersk.com/transportation-services/eco-delivery?exit=Eco+Delivery+Air Martins, V. W. B., Anholon, R., Quelhas, O. L. G., & Filho, W. L. (2019). Sustainable practices in logistics Systems: An overview of companies in Brazil. Sustainability, 11(15), 4140. https://doi.org/10.3390/su11154140 Mas, F. D., Dicuonzo, G., Massaro, M., & Del ’Atti, V. (2020). Smart contracts to enable sustainable business models. A case study. Management Decision, 58(8), 1601–1619. https://doi.org/10.1108/md-09-2019-1266 McKinnon, A. C. (2016). Freight Transport deceleration: Its possible contribution to the decarbonisation of logistics. Transport Reviews, 36(4), 418–436. https://doi.org/10.1080/01441647.2015.1137992 Mignon, I., & Bankel, A. (2022). Sustainable business models and innovation strategies to realize them: A review of 87 empirical cases. Business Strategy and the Environment, 32(4), 1357– 1372. https://doi.org/10.1002/bse.3192 Osterwalder, A., & Pigneur, Y. (2010). Business Model Generation : a handbook for visionaries, game changers, and challengers. https://academicjournals.org/journal/AJBM/article-ful -text-pdf/BA71B6427744.pdf Patala, S., Jalkala, A., Keränen, J., Väisänen, S., Tuominen, V., & Soukka, R. (2016). Sustainable value propositions: Framework and implications for technology suppliers. Industrial Marketing Management, 59, 144–156. https://doi.org/10.1016/j.indmarman.2016.03.001 634 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Plaza-Hernández, M., González, A. B. G., González, S. R., Prieto, J., & Corchado-Rodríguez, J. M. (2020). Integration of IoT technologies in the maritime industry. In Advances in intelligent systems and computing (pp. 107–115). https://doi.org/10.1007/978-3-030-53829-3_10 Posti. (n.d.). Green freight - For businesses - Posti. https://www.posti.fi/en/delivering-with-care/success- of-companies/freightservices/green-freight Psaraftis, H. N., & Kontovas, C. A. (2009). CO2 emission statistics for the world commercial fleet. WMU Journal of Maritime Affairs, 8(1), 1–25. https://doi.org/10.1007/bf03195150 Psaraftis, H. N., & Kontovas, C. A. (2014). Slow steaming in maritime transportation: fundamentals, trade-offs, and decision models. In International series in management science/operations research (pp. 315–358). https://doi.org/10.1007/978-3-319-11891-8_11 Reduced emissions | Viking Line Group. (n.d.). Viking Line. https://www.vikingline.com/sustainable-travel/lng/ Sánchez-Fernández, R., & Iniesta-Bonillo, M. Á. (2007). The concept of perceived value: a systematic review of the research. Marketing Theory, 7(4), 427–451. https://doi.org/10.1177/1470593107083165 Schaltegger, S., Lüdeke-Freund, F., & Hansen, E. G. (2012). Business cases for sustainability: the role of business model innovation for corporate sustainability. International Journal of Innovation and Sustainable Development, 6(2), 95. https://doi.org/10.1504/ijisd.2012.046944 Schneider, S., & Spieth, P. (2013). BUSINESS MODEL INNOVATION: TOWARDS AN INTEGRATED FUTURE RESEARCH AGENDA. International Journal of Innovation Management, 17(01), 1340001. https://doi.org/10.1142/s136391961340001x Sheth, J. N., Newman, B. I., & Gross, B. L. (1991). Why we buy what we buy: A theory of consumption values. Journal of Business Research, 22(2), 159–170. https://doi.org/10.1016/0148-2963(91)90050-8 Solakivi, T., Paimander, A., & Ojala, L. (2022). Cost competitiveness of alternative maritime fuels in the new regulatory framework. Transportation Research. Part D, Transport and Environment, 113, 103500. https://doi.org/10.1016/j.trd.2022.103500 Sweeney, J., & Soutar, G. N. (2001). Consumer perceived value: The development of a multiple item scale. Journal of Retailing, 77(2), 203–220. https://doi.org/10.1016/s0022-4359(01)00041-0 Tagliapietra, S., Häfner, M., & Noussan, M. (2020). The future of transport between digitalization and decarbonization. In SpringerBriefs in energy. https://doi.org/10.1007/978-3-030-37966-7 Teece, D. J. (2010). Business models, business strategy and innovation. Long Range Planning, 43(2–3), 172–194. https://doi.org/10.1016/j.lrp.2009.07.003 Towards a net-zero logistics sector. (2023, January 20). ISO. https://www.iso.org/contents/news/2023/01/a-net-zero-logistics-sector.html Trivel as, P., Μαλινδρέτος, Γ., & Reklitis, P. (2020). Implications of Green Logistics Management on Sustainable Business and Supply Chain Performance: Evidence from a Survey in the Greek Agri-Food Sector. Sustainability, 12(24), 10515. https://doi.org/10.3390/su122410515 Yin, R. K. (2012). Applications of case study research (3rd ed.). SAGE Publications, Inc. Zeithaml, V. A., Verleye, K., Hatak, I., Kol er, M., & Zauner, A. (2020). Three Decades of Customer Value Research: Paradigmatic roots and future research avenues. Journal of Service Research, 23(4), 409–432. https://doi.org/10.1177/1094670520948134 EUROPEAN COMPANIES' READINESS AND CHALLENGES IN DATA ECONOMY MARIKKA HEIKKILÄ, JUKKA HEIKKILÄ, FARHAN AHMAD University of Turku, Turku School of Economics, University of Turku, Finland marikka.heikkila@utu.fi, jups@utu.fi, farhan.ahmad@utu.fi This study investigates the readiness and chal enges European companies face in embracing a data-driven economy. By conducting a comprehensive survey of 1,200 European companies spanning various industries and sizes, we reveal a mixed outlook on the data economy's potential to offer a competitive advantage. We find that 20% have already leveraged data to gain competitive advantage, a majority see potential benefits, while 19% perceive no advantage. Importantly, our findings highlight the need for substantial business model transformations to capitalize on data-driven opportunities. The paper also identifies the obstacles faced by companies in adopting a data-driven approach, including legal complexity, lack of data, competence in data-driven business models and technological competence. Chal enges are most pronounced among micro-sized businesses and sectors like arts, Keywords: entertainment, leisure, and international NGOs. Overall, the data research suggests that if the company has the skil s, a data economy, business economy has the potential to drive innovation and growth. By model identifying the disparities in readiness and perception across changes, different sectors and company sizes, this study contributes to a competitive edge, more informed discourse on fostering a conducive environment challenges, for al companies to thrive in the data-driven economy. companies DOI https://doi.org/10.18690/um.fov.4.2024.37 ISBN 978-961-286-871-0 636 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The data economy is described as "a global digital ecosystem in which data is collected, structured, and shared to generate economic value" (Sestino et al., 2023). Organisations are increasingly dependent on analysing, exchanging, and utilizing digital information that has been col ected, cleansed, organized, and aggregated for multitude of purposes. This information is disseminated in business ecosystems and multi-sided markets for the collective advantage (Brynjolfsson et al., 2011; Jetzek et al., 2014; Heikkilä et al., 2023). The significance of data is anticipated to grow even further (European Commission, 2020), leading to a demand for the re-skilled labour force to enhance innovation within the European economic area (EC, 2023). While literature suggests that firms can leverage their Business Models (BMs) for innovations by digitalization and big data analytics capabilities (Bouwman et al., 2019; Ciampi et al., 2021; Sorescu, 2017), the experiences and outcomes of seizing these opportunities vary. Our earlier study (Heikkilä et al, 2023) found that companies’ data-driven innovation is boosted by participating to data ecosystems, which help them to establish practices for ethical sharing and using of data to innovate and sustain new business. In this paper we elaborate further our earlier findings by studying the role of BM changes in making use of data and by analysing which industries can keep up with the data economy development, what are the hindering factors for catching up or progressing faster, and eventually we discuss the importance and priorities in potential remedies to keep up with the development. Our analysis is based on a survey conducted among European companies, focusing on their perceptions on data economy and the changes they have made to their business models. The data are collected from 1,200 diverse companies in four European countries as part of Sitra’s IHAN project (2021), which focused on building a European data economy model and rule book for good conduct of data driven business (Sitra, 2022). M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 637 This research enhances the body of knowledge on BMs, particularly those of focusing on data-driven BMs where data forms a fundamental component of a company's business model (Trabucchi & Buganza, 2019; Heikkilä et al., 2023). The structure of this paper is laid out as follows: We begin with brief literature review elaborating on the research on data economy and business models. Next, we describe the research method and the data in more detail. Fol owing this, we disclose our findings concerning perceptions of benefits derived from the data economy, encountered obstacles, and alterations in business models - firstly examining all companies, then categorizing them by size, and final y concentrating on companies in selected industries. The paper concludes with a discussion, limitations of this study and suggestions for future research. 2 Data-based Business models A BM describes the logic of how a company creates, delivers, and captures value (Teece, 2010; Wirtz et al., 2016). Typical y, it consists of elements describing what kind of product or service is offered, who the customers are, how the production is organised, who the partners are and how the company generates income. Companies respond to continuously evolving environments by modifying their BMs (Marolt et al., 2018; Pucihar et al., 2019). These BM changes can range from modest refinements to some BM elements to a complete overhaul of the entire BM (Saebi et al., 2017; Eriksson et al., 2022). Data-driven innovation refers to innovation that utilizes data and information and communication technologies as its core ingredients. The scholarly work defines it as a type of business innovation that leverages data to yield positive economic and social outcomes (Jetzek et al., 2014). Firms employ data to enhance decision-making processes, optimize organizational operations, or augment value for customers (Brynjolfsson et al., 2011). For example, by collecting and analysing data from users or other sources (Trabucchi & Buganza, 2019; Jetzek et al., 2014), a firm can infuse the data (Schüritz & Satzger, 2016), and gain insights into how its customers value its products and services and how it could add even more value to the market. 638 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Several companies have realized the potential advantages of creating new BMs that utilize data as a crucial resource (Hartmann et al., 2014). Ciampi et al. (2021) assert that a company's competence in big data analytics positively affects its ability to innovate its BM. Bouwman et al. (2019) revealed that companies that actively utilize social media, big data, and information technology can increase their performance by transforming their BMs and strengthening their capacity for innovation. This however requires that the company has access to business and technical capabilities (Eriksson & Heikkilä, 2023). In their qualitative study Rashed & Drews (2021) noticed companies taking differing approaches to data utilization depending on the degree of understanding of their data and degree of self-incentive. Typically, external support in e.g. selecting technology, in MVP or in operating model design was provided by consultants. Surprisingly, no data ecosystems were mentioned, even though the EU sees data ecosystems as crucial for the commercialization and commodification of data services, products, and platforms, thereby stimulating economic growth and innovation (European union, 2023). The expectation is that in data economy, especial y with multi-sided markets and data ecosystems running on platforms (e.g., Hagiu, 2016; Tiwana, 2015; Giessman & Legner, 2016; Helfat & Raubitschek, 2018), the skil s needed for operating and being an active member of platform and data economy are on high demand (EC, 2023). Indeed, in our earlier study (Heikkilä et al, 2023) we found that companies’ data-driven business model innovation is boosted by participating to data ecosystems, which help them to establish practices for ethical sharing and using of data. Little is still known about differing companies’ perceptions on the benefits they can derive from the data economy, what are the obstacles they encounter, and how much do they change their business models. These are the topics of this study. 3 Methodology The study was executed in 2021 within four European countries: Germany, Finland, France, and the Netherlands, sponsored by Sitra, the Finnish Innovation Fund (www.sitra.fi). Sitra is distinguished by its unique role at the national level in Finland, given its accountability and direct reporting to the Finnish Parliament, with Sitra’s initial capital donated by the same body for the good of the nation. M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 639 Table 1: Profile of respondents’ companies Variable Category % Finland 25 France 25 Country Germany 25 The Netherlands 25 Micro (under 2 million) 28.1 Firm size Small (2 – 10 million) 26.6 (turnover €) Medium (10 – 50 million) 22.1 Large (over 50 million) 23.2 Professional, scientific, and technological operations 6,5 Administrative and support services 6,4 Information and communications 9.8 Mining and quarrying operations 0,3 Operations of international organisations and institutions 0,8 Real estate operations 2,4 Training 0,5 Transportation and warehousing 4,3 Farming, forestry, and fishing industries 1,8 Industry Hotel and catering 3,3 sector Other service operations 16,0 Finance and insurance 8,0 Construction 5,5 Electrical, gas, heat, and ventilation maintenance 1,2 Arts, entertainment, and leisure 2,3 Industry 11,3 Health and social services 8,1 Wholesale and retail; motor vehicle and motorcycle repairs 6,6 Water supply, sewerage, waste management and other 0,4 sanitation The survey respondents are B2B decision-makers, encompassing key management positions in charge of data, digitalization, information systems, strategy, marketing, and business development. The respondents were engaged on a voluntary basis, having previously indicated their willingness to participate in surveys. The process involved inviting respondents based on certain background criteria until stratum quotas of size in each country was met. The respondents retained the autonomy to accept or decline participation in the survey at their discretion. They were rewarded 640 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION for their contributions and time to the survey with incentives such as gift cards or airline miles. Table 1 outlines the final samples in all countries. They are equally distributed across the involved countries and almost equally in size categories (by company turnover in accordance with the EU size classification - large, medium, small, and micro, excluding sole proprietors). There were neither specific quotas for industry type (EU-sectoral classification), nor for relative position or role within the data economy (e.g. Hagiu, 2016). The questions, scales and the raw dataset is freely accessible through Sitra (2021), reflecting its commitment to transparency and public engagement (see also Saaristo & Heikkilä, 2022; Ulander et al. 2021). 4 Findings We will first present the overall findings at aggregated level, generalised over company sizes and industries. Then we analyse how the results differentiate between firm sizes, and last we take a deeper look at three industry groups: Information & Communication Technology, Finance, and Insurance (ICT-FIN); Transportation and warehousing (TRANSPORT), and Arts, entertainment, leisure and international organisations such as NGOs and charity organizations (ART_NGO), which differ in their expectations, skills and obstacles in engaging to data economy. 4.1 Overal Insights from the European Company Dataset Survey results indicate that the studied companies are wil ing and prepared to embrace a data-driven economy (Figure 1.): Nearly half (48%) of the total respondents stated that a data economy had created (21%) or would create (27%) a competitive edge for their company. About one in five were negative on the promise of data economy. M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 641 C O U L D D A T A E C O N O M Y C R E A T E A C O M P E T I T I V E E D G E F O R Y O U R C O M P A N Y ? Yes, it already has No 21% 19% Possibly Yes, in future 33% 27% Figure 1: Could data economy create a competitive edge for your company? Source: Own The companies were also asked if they had made any significant changes to their business model in seven key components (customers, channels, value proposition, activities, resources, partners, revenue models) during the last two years (see Figure 2. and Table 2). Three out of four companies had made at least some changes to at least one component of their business models during the last two years, nearly half of the changes coined as radical. HAS YOUR COMPANY MADE SIGNIFICANT CHANGES TO THE BUSINESS MODEL? Yes, some No 38% Yes 27% 73% Yes, radical 35% Figure 2: Has your company made significant changes to the business model during the last two years? Source: Own 642 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 2: Has your company made significant changes to the business model during the last two years? Frequency % Valid % No 325 27.1 27.3 Yes, some 449 37.4 37.7 Valid Yes, radical 418 34.8 35.1 Total 1192 99.3 100.0 Missing 8 0.7 Total 1200 100.0 Table 3., in turn, shows that there is a statistically significant association (<.001) between the degree of changes to the business model and believing that a data economy creates a competitive edge now or later versus possibly, or not at al . Those who do not believe in data economy, do little BM changes on the average, or vice versa. Especial y when compared with the believers the difference is striking (more than four changes vs one). Table 3: No of BM components changed & competitive edge How many BM Components were changed (max 7) Std. Mean N Deviation Could data Yes, it already has 4.3571 252 2.52782 economy Yes, it will in the create a 4.2747 324 2.53432 future competitive Possibly 2.5797 394 2.40148 edge for your company? No 1.0954 220 1.79928 Total 3.1427 1191 2.67163 The table 4. below illustrates the big and statistical y significant differences between two extreme groups of companies (those who already have created competitive edge and those who do not expect to gain competitive edge). While 54% to 66% of the first group has made changes to each BM component, in the second group the respective percentages vary between 11% and 23%. M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 643 Table 4: Changes in BM Components among two distinct groups: those who have achieved a competitive advantage through the data economy and those who perceive no competitive edge Could data economy create a competitive edge for your company? Changes in.. Yes, it already has, (N=252) No, (N=220) BM Customer component 54 % 14 % BM Value proposition 65 % 16 % BM Channels 64 % 13 % BM activities 64 % 17 % BM Resources 66 % 23 % BM Partners 66 % 18 % BM Revenue model 64 % 11 % Al the above pinpoints to the fact that data-driven business necessitates wide-scale changes in the business model of the company. As the survey did not cover the companies’ intentions to make changes to their business model, further modifications may stil be on the drawing board. 5.2 Analysis by Company Size Categories We asked whether the companies possess skil s to function as an active member in the data economy. Not surprisingly, the skill level is significantly correlated (<.001) with the size of the company (Figure 3.). The micro firms considered their skil s significantly lower compared to other companies while large companies claimed to have the best skills. Moreover, Figure 4 shows (<.001) that medium-sized companies hold the most optimistic expectations about gaining a competitive advantage through data utilization. On the other hand, for micro companies the outlook was less positive. This trend is hardly surprising and can largely be attributed to the disparity in resources. Micro companies, with their limited resources, face significant chal enges in collecting and utilizing data effectively – and as indicated earlier are also less skilled in participating to the data economy than bigger companies. Hence, the economic standpoint becomes especial y significant in this scenario, highlighting the critical role of resource availability in leveraging data for competitive gains. 644 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION O U R C O M P A N Y H A S T H E N E E D E D S K I L L S T O F U N C T I O N A S A N A C T I V E M E M B E R O F D A T A E C O N O M Y Large 3,55 Medium 3,47 Small 3,43 Micro 2,92 All industries 3,33 0,00 1,00 2,00 3,00 4,00 5,00 Figure 3: Data Economy Skil s per company size (<.001) COULD DATA ECONOMY CREATE COMPETITIVE EDGE FOR YOU COMPANY? Yes, it already has Yes, in future Possibly No L A R G E ( > 5 0 M I L L I O N € ) 25,10% 27,80% 33,30% 13,70% M E D I U M ( 1 0 - 5 0 M I L L I O N € ) 28,10% 31,00% 28,50% 12,40% S M A L L ( 2 - 1 0 M I L L I O N € ) 22% 30,70% 30,70% 16,90% M I C R O ( < 2 M I L L I O N € ) 9% 19,20% 41% 31,20% Figure 4: Company size and competitive edge (<.001). Similar story continues with regards BM changes (Figure 5.). Micro companies have made the least changes and medium sized companies the biggest changes to their BM. To the contrary it is the medium sized firms, which are best prepared for reaping the benefits from data economy by innovating their business model. M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 645 HAS YOUR COMPANY MADE SIGNIFICANT CHANGES TO THE BM? Radical changes At least one change No L A R G E ( > 5 0 M I L L I O N € ) 37,30% 39,60% 23,10% M E D I U M ( 1 0 - 5 0 M I L L I O N € ) 38,60% 43,20% 18,30% S M A L L ( 2 - 1 0 M I L L I O N € ) 45% 34,10% 21,00% M I C R O ( < 2 M I L L I O N € ) 21% 37,20% 42% Figure 5: Company size and extent of changes to the BM. (<.001). So, what are the main obstacles on the way taking advantage of the data driven economy. One could expect the size of the company being a determining factor for recognizing the obstacles – yet Table 5 shows, that the set of obstacles are rather similar in our sampled companies, although we can find some differences, too. Table 5: Obstacles on the way to data economy per company size Micro Small Medium Large Obstacles on the way to data economy: N=308 N=291 N=243 N=255 Complexity of the regulatory framework for businesses 26% 23% 33% 38% Lack of competence on data economy related business models 26% 31% 23% 22% Lack of available data 25% 33% 33% 37% Lack of competence on data economy related technical capabilities 18% 24% 26% 24% Customers do not recognize the potential of such new services 22% 22% 21% 24% Lack of funding available 17% 10% 17% 13% To our surprise, funding for data economy investments was not listed a major issue. Instead, lack of data is one of the most mentioned obstacles. The bigger the company the more chal enges seem to arise from lack of data – and obviously this has something to do with the complexity of regulatory environment, as it is similarly associated with the company size. And final y, the lack of competence in data 646 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION economy business models and lack of technical capabilities are also mentioned by many. 5.3 Analysis of specific industries For further analysis we decided to focus on three groups with different skil s: Information & Communications and Finance & Insurance sectors possess top skills in terms of being active participants in the data economy, so we combined them as one group (ICT-FIN). Transportation and Warehousing (TRANSPORT), in turn, have average skills. Contrary to ICT-FIN, which offers immaterial services mostly, TRANSPORT is lot about moving physical, material objects. The third group consists of Arts, Entertainment, Leisure and International Organisations (ART-INT) who self-evaluate themselves as having lowest skills for data economy (Figure 6). O U R C O M P A N Y H A S T H E N E E D E D S K I L L S T O F U N C T I O N A S A N A C T I V E M E M B E R O F D A T A E C O N O M Y ICT-FIN - Information and 3,72 communications; Finance and insurance TRANSPORT - Transportation and 3,45 warehousing ART-INT - Arts, entertainment, and 2,78 leisure; Operations of international… All industries 3,33 0,00 1,00 2,00 3,00 4,00 5,00 Figure 6: Industry sectors and skil s (<.001). Comparison of the groups in Figure 7. shows how ICT-FIN and TRANSPORT are rather similar in their view on getting competitive advantage from data economy. Over 30% of them report already benefiting from data. To the contrary, perceptions of ART-INT are less positive, only 13,5% of ART-INT have been able to gain competitive advantage from data. M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 647 COULD DATA ECONOM Y CREATE COM PETITIVE EDGE FOR YOU COM PANY? Yes, it already has Yes, in future Possibly No I C T - F I N - I N F O R M A T I O N A N D C O M M U N I C A T I O N S ; F I N A N C E A N D 30,10% 27,80% 31,10% 11,00% I N S U R A N C E T R A N S P O R T - T R A N S P O R T A T I O N A N D 33,30% 29,40% 27,50% 9,80% W A R E H O U S I N G A R T - I N T - A R T S , E N T E R T A I N M E N T , A N D L E I S U R E ; O P E R A T I O N S O F 13,50% 29,70% 37,80% 18,90% I N T E R N A T I O N A L O R G A N I S A T I O N S A L L I N D U S T R I E S 21% 27% 33% 18,50% Figure 7: Competitive edge in three industries. We dig deeper on how the companies perceive the data economy to financially benefit their business with the help of Kruskal-Wallis and Mann-Whitney tests. Results are presented in Table 7 and A1. Table 7: Three industry groups’ views on potential benefits from data How much potential you see in data sharing to Create additional revenue from Create new current business revenue streams Saving costs model from innovations ICT-FIN Mean 3.46 3.65 3.35 N 210 210 210 Std. Deviation .939 .863 .983 TRANSPOR Mean 3.53 3.63 3.67 T N 51 51 51 Std. Deviation 1.084 1.095 1.125 ART-INT Mean 3.00 3.00 3.24 N 37 37 37 Std. Deviation 1.202 1.106 1.278 All Industries Mean 3.22 3.27 3.31 N 1187 1183 1183 Std. Deviation 1.145 1.152 1.144 648 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION ICT-FIN and TRANSPORT sectors perceived a significant potential to increase their revenues in two ways: by optimizing their current business model and, even more importantly by innovations. ART-INT sector is significantly less convinced on the benefits of data economy altogether. ART-INT primarily sees the potential benefits of the data economy in terms of cost savings, an expectation that is uniformly shared across all industry groups. Perhaps surprisingly, almost half of the ART-INT mention complexity of regulation to be the biggest obstacle (46%, compared to 25% in TRANSPORT and 35% in ICT-FIN), followed by lack of competence in data-driven business models (32%, vs. TRANSPORT 37% and ICT- FIN 26%). Third obstacle is lack of data (16% vs. TRANSPORT 46% and ICT-FIN 33%). 6 Discussion and Conclusions This study finds that European companies perceive the data economy as offering a competitive advantage. Almost half of the surveyed companies stated that the data economy had provided, or would provide in future, a competitive edge for their business. 19% of the companies consider that the data economy wil not benefit them. The perceived economic potential from the data economy is significantly linked to the level of BM changes implemented by the companies: those that have already benefited from the data economy have made large-scale modifications to their BMs, in contrast to those companies that are unwil ing or unable to benefit from data, which have made minimal changes to their BMs. Furthermore, the study provides evidence that micro-sized firms (with a turnover of less than two mil ion euros) are particularly struggling to adapt to the data economy. These firms are lacking the necessary skills, and only 9% of micro companies believe they have been able to gain a competitive advantage from data. As the biggest obstacles, companies of all sizes cite a lack of data, the complexity of regulations, and a deficiency in data-driven BM and technical skills. Further analysis of three industry groups reveals that the ICT-FIN and TRANSPORT industries significantly outperform the ART-INT sector. The former two are able to leverage the benefits of the data economy by innovating and adapting their business models. This not only al ows them to increase revenues within their current business models but also to innovate new ones. We interpret this disparity M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 649 as reflecting the productivity and welfare losses in the ART-INT sector: despite anticipating some improvements in form of cost savings, it fails to engage fully in data-sharing ecosystems and thus cannot reap the ful benefits. Indirect evidence also suggests that the profitability of the ART-INT sector has not improved during the data economy era, despite a Compound Annual Growth Rate of 10-12% in industry revenue in the studied countries. Therefore, despite expected market growth from 2022 to 2027 (Statista, 2023), ART-INT is not benefiting from a data economy to enhance its business, unlike the more adept ICT-FIN and even the average TRANSPORT industries. The fact that the ART-INT industry has not caught up starts to show in the industry performance statistics (Figure 8, Eurostat, 2023). On the other hand, ICT-FIN sector is creating more value per employer. This may imply that inclination towards data economy is related with higher productivity. Figure 8: Percentage of value added and employment, EU 2021 Is there likely to be a significant disparity in the ability of different industries to benefit from data and engage in the data economy, potential y leading to creative destruction? This concern is particularly acute for industries such as ART-INT, which appear less capable of harnessing the advantages of data. We are less concerned on the evolution of ICT-FIN and even TRANSPORT, because the 650 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION utilization of data for changing their business keeps also profits up, the better the more involved the industry is into the data economy’s innovative traits. The results also raise similar concerns about level of data-driven innovation within micro-sized companies, which see themselves as significantly less skil ed with respect to the data economy. This study has some limitations that need to be taken into consideration when interpreting the results. Firstly, the data was col ected from four European countries only, and therefore, the results may not be generalizable to companies in other regions, such as the United States and China, which have different regulatory environments and attitudes towards fair and sustainable data practices. Secondly, the cross-sectional nature of the survey could introduce response bias, as firms at different stages of their innovation trajectories may hold varying views on data- driven innovation. However, the inclusion of a large sample size is expected to mitigate such variance. Moreover, our findings highlight the perceived low skills to benefit from data by the micro-sized companies. This discovery underscores an urgent need for more detailed research to understand the specific barriers and opportunities faced by these companies, which range from small retail stores to hairdressers and startups. Future studies should consider incorporating questions about company age to better distinguish between the unique chal enges faced by established micro-sized businesses and those encountered by startups. Such research is crucial for developing targeted strategies and support mechanisms to elevate the data economy competencies of micro-sized companies across various sectors. Acknowledgements We express our gratitude to Sitra for sharing their data, which was invaluable to our research. Their commitment to open data has greatly contributed to advancing knowledge in this area and has al owed us to draw meaningful insights from the data. References Bouwman, H., Nikou, S., & de Reuver, M. (2019). Digitalization, business models, and SMEs: How do business model innovation practices improve performance of digitalizing SMEs?. Telecommunications Policy, 43(9), 101828. Brynjolfsson, E., Hitt, L. M., & Kim, H. H. (2011). Strength in numbers: How does data-driven decision making affect firm performance? SSRN Electronic Journal, doi:10.2139/ssrn.1819486 M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 651 Ciampi, F., Demi, S., Magrini, A., Marzi, G., & Papa, A. (2021). Exploring the impact of big data analytics capabilities on business model innovation: The mediating role of entrepreneurial orientation. Journal of Business Research, 123, 1-13. EC (2023). European Year of Skills - Skills shortages, recruitment and retention strategies in small and medium-sized enterprises, European Commission, Employment Society, Culture and Demography 2994/FL529, Sept 2023. Available at https://europa.eu/eurobarometer/surveys/detail/2994 Eriksson, T., & Heikkilä, M. (2023). Capabilities for data-driven innovation in B2B industrial companies. Industrial Marketing Management, 111, 158-172. Eriksson, T., Heikkilä, M. & Nummela, N. (2022). Crafting SME Business Model for International Expansion with Data-Driven Services, in Business Models and Firm Internationalisation, Christian Nielsen, Svetla T. Marinova, Marin A. Marinov (eds.) in the series of Routledge Frontiers in the Development of International Business, Management and Marketing. European Union (2023). Navigating European data ecosystems: unveiling the Data Spaces Blueprint, 04 Dec, 2023. https://data.europa.eu/en/news-events/news/navigating-european-data-ecosystems-unveiling-data-spaces-blueprint Eurostat, (2023). Analysis of value added and employment within the business economy (%, EU, 2021), Eurostat (sbs_sc_ovw) Giessmann, A., & Legner, C. (2016). Designing business models for cloud platforms. Information Systems Journal, 26(5), 551–579. https://doi.org/10.1111/isj.12107 Hagiu, A. (2014). Strategic Decisions for Multisided Platforms. MIT Sloan Management Review, 55(2), 71–80. Hartmann, P. M., Zaki, M., Feldmann, N., & Neely, A. (2014). Big data for big business? A taxonomy of data-driven business models used by start-up firms. Cambridge service al iance, 1-29. Helfat, C. E., & Raubitschek, R. S. (2018). Dynamic and integrative capabilities for profiting from innovation in digital platform-based ecosystems. Research Policy, 47(8), 1391–1399. https://doi.org/10.1016/j.respol.2018.01.019 Heikkilä M., Ahmad F., & Heikkilä J., (2023). Unlocking the Potential of Data-driven Business Models: An empirical investigation into the role of ecosystems and fair data use. In the proceedings of the 36th Bled eConference Digital Economy and Society: The Balancing Act for Digital Innovation in Times of Instability, June 2023, Bled, Slovenia, pp. 85-103. Jetzek, T., Avital, M., & Bjorn-Andersen, N. (2014). Data-driven innovation through open government data. Journal of Theoretical and Applied Electronic Commerce Research, 9(2), 15-16. doi:10.4067/S0718-18762014000200008 Marolt, M., Lenart, G., Borstnar, M. K., Vidmar, D., & Pucihar, A. (2018). SMEs Perspective on Business Model Innovation. 31th Bled eConference Digital Transformation – Meeting the Challenges (June 17 - 20, 2018, Bled, Slovenia). Pucihar, A., Lenart, G., Kljajić Borštnar, M., Vidmar, D., & Marolt, M. (2019). Drivers and outcomes of business model innovation—Micro, small and medium-sized enterprises perspective. Sustainability, 11(2), 344. Rashed, F., & Drews, P. (2021). Pathways of Data-Driven Business Model Design and Realization: A Qualitative Research Study. https://doi.org/10.24251/HICSS.2021.689) Saaristo, A., & Heikkilä, M. (2022). A fair data-driven economy - The future of Europe, CCR Insights 2/2022, Turku School of Economics, Retrieved from: https://www.utu.fi/sites/default/files/public%3A//media/file/ccr_insights_2_2022_afaird atadriveneconomy_thefutureofeurope_v3_saavutettava%281%29.pdf Saebi, T., Lien, L. & Foss, N. J. (2017). What drives business model adaptation? The impact of opportunities, threats and strategic orientation. Long Range Planning, 50 (5), pp. 567-581. Schüritz, R., & Satzger, G. (2016). Patterns of Data-Infused Business Model Innovation, in Proc. 18h Conf. Bus. Informatics, Paris, France, pp. 133–142. 652 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Sestino, A., Kahlawi, A., and De Mauro, A. (2023). Decoding the data economy: a literature review of its impact on business, society and digital transformation. European Journal of Innovation Management. https://doi.org/10.1108/EJIM-01-2023-0078. Sitra (2021). Raw data (excel). Retrieved from : https://www.sitra.fi/en/publications/the-future-of- european-companies-in-data-economy-2021/ Sitra (2022), Rulebook for a Fair Data Economy, version 2.0. Available with editable templates at https://www.sitra.fi/en/publications/rulebook-for-a-fair-data-economy/#1-1-why-and- when-you-should-use-a-rulebook-for-data-sharing Sorescu, A. (2017). Data-Driven Business Model Innovation. Journal of Product Innovation Management 34(5). 691-696. Statista (2023). On-line industry statistics available at https://www.statista.com/outlook/dmo/app/entertainment/ Teece, D.J., (2010). Business models, business strategy and innovation. Long Range Plan ning. 43 (2–3), 172–194. https://doi.org/10.1016/j.lrp.2009.07.003. Tiwana, A. (2015). Evolutionary Competition in Platform Ecosystems. Information Systems Research, 26(2), 266–281. https://doi.org/10.1287/isre.2015.0573 Trabucchi, D. and Buganza, T. (2019). Data-driven innovation: switching the perspective on Big Data. European Journal of Innovation Management, 22(1), 23-40. https://doi.org/10.1108/EJIM- 01-2018-0017 Ulander, M.; Vierula, T.; Ahomäki, M.; Kultanen, H. & Polifke, A. (2021). The Future of European Companies in Data Economy. Sitra. https://www.sitra.fi/app/uploads/2021/05/sitra-thefuture-of-european-companies-in-data-economy-2021.pdf Wirtz, B.W., Pistoia, A., Ul rich, S. & Göttel, V. 2016. Business Models: Origin, Development and Future Research Perspectives. Long Range Planning 49: 36−54. M. Heikkilä, J. Heikkilä, F. Ahmad: European Companies' Readiness and Chal enges in Data Economy 653 Appendix Table A1: Mann-Whitney test presenting pairwise differences between industries ICTFin – ICTFin - ArtInt - How much potential you see in data sharing to ArtInt Transport Transport Create additional revenue from current business model -2.066 -0.866 -2.142 Create new revenue streams from innovations -3.288* -0.125 -2.504* Saving costs -0.127 -2.449* -1.676 Notes: values in the cells are z scores. * represents 2-talied significance 654 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION CUSTOMER PERCEPTIONS OF IN-STORE IDENTIFICATION TIINA KEMPPAINEN,1 LAURI FRANK2 1 University of Jyväskylä, School of Business and Economics, Jyväskylä, Finland tikemppa@jyu.fi 2 University of Jyväskylä, Faculty of Information Technology, Jyväskylä, Finland lauri.frank@jyu.fi Customer identification is essential for all businesses; however, brick-and-mortar (B&M) retailers often face chal enges capturing shopping behaviors in physical stores where customers are not accustomed to being identified, especial y outside checkout. Nevertheless, B&M retailers are increasingly pressured to improve their identification and data collection methods to stay competitive with online retailers. Additionally, businesses are under pressure to incorporate digital elements into physical environments to keep them engaging and inspiring for the increasingly digitized customers. Hence, this study describes customer perceptions of in-store identification, focusing on B&M store entry and checkout-related identification. The data were col ected from interviews with 18 customers of two Finnish B&M retailers. The findings show that while generally viewed positively, there is some reluctance towards identification initiatives. Customers cautiously accept entrance- and in-store identification, yet there are chal enges in grasping the associated Keywords: customer benefits. Additional y, concerns about privacy infringement and identification, the extent of tracking were identified. While commonplace, B&M checkout identification raises privacy concerns, highlighting the stores, retail, importance of nimble and discreet processes and preserving qualitative anonymity. study DOI https://doi.org/10.18690/um.fov.4.2024.38 ISBN 978-961-286-871-0 656 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Customer identification is an essential process for all businesses, involving the recognition of individual customers within a business setting, often achieved through the gathering and analysis of personal information or behavioral data. Identification practice is essential for businesses as it al ows the evaluation of the effectiveness of business strategies. By analyzing shopping behavior data, companies can better understand how customers interact with their products and stores, al owing them to make informed decisions to optimize their marketing and merchandising efforts (Zhou et al., 2017). Ultimately, leveraging shopping behavior data al ows retailers to enhance customer satisfaction, increase sales, and maintain a competitive edge in the marketplace. While retailers can readily utilize data analytics in online channels to analyze click streams and customer shopping carts, among other things, they often face chal enges in capturing comprehensive shopping behaviors in traditional brick-and-mortar (B&M) stores (Saßnick et al., 2023; Zhou et al., 2017). These chal enges are related to in-store customer identification and behavior tracking. Physical service environments general y possess fewer resources and technological capabilities for monitoring and analyzing customer behavior in contrast to online platforms, which can amass substantial data through digital interactions. In B&M stores, customers are typically identified at the checkout through loyalty programs and methods like loyalty cards. Even though there are technologies available for the identification and tracking of customers (such as Bluetooth/Wi-Fi tracking, RFID tags on products, and biometric scanners like fingerprint readers), the adoption of these technologies remains restricted and there is a deficiency in comprehension regarding customer behaviors within the service environment, such as entering and navigating the store. However, gathering customer data from physical environments is increasingly vital for B&M retailers seeking to compete with online counterparts as it provides retailers with valuable insights into customer behavior, preferences, and trends. The data can be used to enable personalized marketing efforts, improve the overal shopping experience, enhance operational efficiency, and facilitate targeted marketing campaigns to shoppers. In addition, the data can be used in fraud prevention and security. Moreover, retail innovations such as automated checkout systems, personal shopping assistants, and omnichannel services drive a profound T. Kemppainen, L. Frank: Customer Perceptions of In-Store Identification 657 evolution of conventional retail outlets into smart stores (Hauser et al., 2019). As businesses transition towards omnichannel and metaverse environments, where digital and physical elements are increasingly integrated through various technologies, the role of data collection and customer identification is poised to become even more significant. Whereas today's customers commonly enjoy unhindered mobility within retail environments without formal identification, the prospective retail landscapes potential y necessitate operations such as pre-entry authentication protocols in order to personalize the customer experience and to prevent instances such as shoplifting or other undesirable behavior by customers. Therefore, the purpose of this qualitative study is to describe customer perceptions of in-store identification at B&M stores focusing on store entrance and checkout identification. The data of this study were col ected from interviews with 18 customers of two Finnish B&M retailers. The study aims to enhance the understanding of how customers perceive identification, along with exploring the positive and negative viewpoints on identification and associated data collection. Although there has been limited research on customers’ perceptions of identification, especial y concerning store entrance, understanding customer viewpoints is essential due to the rising need for customer identification and digital technologies in various physical service environments. 2 In-store identification of customers Customer identification refers to the process of recognizing and verifying the identity of individuals who interact with a business or utilize its services. Customer identification encompasses different methods in retail settings and B&M stores (Knof et al., 2023). However, the most prevalent methods primarily target identifying customers at the checkout counter, with loyalty programs frequently utilized, relying on different types of loyalty cards for customer identification. Standard loyalty programs primarily center around purchase data and often mandate customers to provide personal information in exchange for discounts, rewards, or tailored offers. (Shirai, 2022). In recent years, the traditional methods, including physical membership documents, have been increasingly supplanted by diverse mobile solutions. Retailers offer mobile apps and other solutions such as QR-codes that enable customers to check in. The transaction data captured at checkout offers 658 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION valuable insights into customer behavior, encompassing details like purchased items, payment methods, and time of purchase. While retailers are typically quite advanced in collecting customer data at checkout, the other phases of the customer journey during the B&M store visit often remain less investigated and monitored. Unlike with online shopping, where each click and interaction can be monitored during the customer journey, tracking customer movements within a physical store can be chal enging. However, due to technological advancements, retailers are increasingly integrating smart technologies into their retail environments to compete with e-commerce. This enables them to analyze customer behavior and provide personalized shopping experiences (Knof et al., 2023). The growing prevalence of smartphone ownership has provided companies with the means to monitor individuals' movements and track their customers through wireless technologies such as Wi-Fi and Bluetooth tracking (Knof et al., 2023). Many retailers offer free Wi-Fi in their stores, which allows them to track the movements of customers who connect to the network. This data can be used to analyze foot traffic patterns and understand how customers navigate the store. Customer movements in stores can also be monitored using tags attached to products. RFID (Radio-Frequency Identification) tags are smal electronic devices that can be embedded in products or attached to merchandise or shopping carts (Ali et al., 2022; Hui et al., 2009). Retailers can use RFID technology to track specific items as they are moved within the store (Choi et al., 2015). In the current retail environment, the capability to detect and understand customer behavior can provide a significant competitive advantage (Landmark & Sjøbakk, 2017.) Nevertheless, there exists substantial potential for the further development of diverse technologies within the retail industry (Knof et al., 2023). Integrating various technologies and consolidating the gathered data can enable a comprehensive analysis of in-store customer behavior (Knof et al., 2023). To stay appealing, B&M stores must optimize their operations efficiently while engaging customers through innovative means. From the customer's perspective, encountering customer identification or data col ection while entering or moving around brick-and-mortar stores is currently rare. Typically, customers only need to provide identification during checkout, and this process is voluntary. Therefore, the adoption of different and unconventional modern methods for customer identification and detailed data gathering in a B&M store environment may raise concerns among shoppers. Examining these concerns T. Kemppainen, L. Frank: Customer Perceptions of In-Store Identification 659 and delving into customers' viewpoints presents a significant topic from both managerial and research perspectives. The traditional methods of monitoring shopper behavior within stores, such as surveys and observational studies (Nordfält & Ahlbom, 2024), do not adequately address customers' thoughts and concerns. Hence, to gain a deeper insight into the issue, it is imperative to examine the thoughts and reactions that B&M store identification and data collection evoke among customers. 3 Methodology Since this study aimed to enhance understanding of customers' perspectives on identification, the study was conducted as a qualitative study. The data were col ected in April and November 2022 through Zoom interviews. Interview durations varied from approximately 36 to 64 minutes. Participants were recruited through newsletters that were sent to the loyal customers of two different Finnish retailers: participants 1-9 were sourced from a textile-sel ing company's registry, and participants 10-18 were sourced from a natural products and supplements -selling company's registry. As compensation, participants received gift cards. In total, 18 individuals participated in the interviews, comprising 16 females and two males. The participants' ages ranged from 27 to 69 years. The participant demographics are summarized in Appendix 1. The interviews were themed around issues related to customer loyalty and what motivates participants to remain loyal customers. During the interviews, participants were asked to share their customer history, purchasing behavior related to the company both online and offline, as wel as thoughts on marketing communications and customer identification and data col ection. The participants were asked about their perspectives on loyalty programs, their views on the diverse identification and data col ection methods employed by companies online and offline, the kinds of information they are open to sharing, and their sentiments regarding identification upon arrival at B&M stores, such as through mobile phone location data, QR codes, or alternative means. The interviews were recorded and transcribed. Subsequently, the data was analyzed using NVivo 12 Pro qualitative research analysis software. The analysis was data- driven. During the process, participants' thoughts regarding customer in-store 660 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION identification were initialy extracted from the data and subsequently categorized into two primary sections: perspectives on 1) identification upon entering a B&M store and 2) identification at the B&M store checkout. The subsequent section will provide a more detailed exploration of the findings. 4 Findings Identifying customers in B&M stores elicited various opinions and perspectives among study participants. Identification was generally viewed positively, but some reported avoiding loyalty programs and customer identification. 4.1 Identification Upon Entry Participants had not personal y encountered or noticed the process of being identified upon entering a B&M store. The concept sparked both support and opposition. While some expressed hesitation, a significant portion found it acceptable. Identification upon arrival was viewed favorably due to its perceived potential for enhancing personalized service, thereby making customers feel valued. It was deemed crucial that recognition be voluntary, al owing customers the option to shop without identification. Additional y, identification was deemed acceptable if it was wel -explained and easily declined. Moreover, recognition was seen as bolstering the security of physical service environments by ensuring knowledge of who is present in stores. Why not? It would be nice if someone recognized you; it wouldn't bother me. It would make me feel appreciated. But you should be able to get into the store without having to identify yourself. – P8 It wouldn't bother me to be recognized if it's clearly communicated, and it's an action that can be turned off if I don't want to be tracked. – P6 It's modern, for sure; I have nothing against it. [.. ] Also, for the store's safety, it is good to have identification. – P5 T. Kemppainen, L. Frank: Customer Perceptions of In-Store Identification 661 Identifying a customer's location was considered beneficial if it could offer targeted promotions and reminders about the store. When close to the store, location-based marketing communication could incentivize a visit more effectively than, for instance, e-mail advertisements. It could be used if you are in a shopping center and have al owed the location data from a certain distance. Then you could get a notification: "Hey, visit store X; you have a ten percent discount." – P7 Conversely, the notion of customers being identified upon arrival prompted negative sentiments due to the perceived lack of benefits associated with the identification. The traditional customer service models were deemed effective enough. Thus, there was no perceived need for more personalized attention or digital enhancements while browsing the store. Recognition of customers in the store was also seen as going "too far," which led to reluctance to share one’s location data with businesses. This was explained by concerns about privacy and loss of anonymity: a company should not know where a customer moves, as it “feels like having a stalker” (P11). Absolutely not. I would never agree to that. [. .] I don't like the idea of some business having that data. – P1 I would not like that. It may go too far. [. .] There is some psychological thing there: someone is following you. – P3 Participants who initially perceived identification and tracking positively also had their suspicions about companies' operating models. There were suspicions about the limits of tracking: whether it was limited to movement in the store or if customers could be tracked elsewhere. Recognition outside the store environment raised negative thoughts because it was seen as violating customer privacy. It is okay that they identify me there. But I have a suspicion that they might fol ow me when I'm somewhere else. – P6 On the other hand, identification was also seen as providing the company with too much detailed information, which could affect how customers are treated in the store. One perceived threat was the loss of equal treatment of customers and more 662 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION active sales efforts toward those customers who have purchased actively from the company. Also, a situation where the company recognizes a customer but the customer does not purchase anything was considered potential y embarrassing. I like to be anonymous. It would be like wearing a shirt saying: “I've bought from here and spent X amount of money." [. .] Benefits can be given through e-mail, but elevating or diminishing the customer based on the amounts spent is not justified in public. – P7 It would feel a bit embarrassing if you do not buy anything. – P9 Customer identification and tracking were seen as part of contemporary operational models, which customers may not necessarily be able to influence. It was seen as a current trend that consumers cannot do much about, and they must submit to the use of digital solutions. Data collection and tracking of individuals were suspected to happen regardless of whether the person was aware of it. Well, they col ect data all the time. […] I do not see it as harmful if some company does it because it's being done al the time anyway. – P2 Surely, the phone is always listening to what we're talking about. That's how we're being tracked. – P3 In addition to suspicions of mobile phones eavesdropping on their users, monitoring was observed through monthly reports provided by Google. 4.2 Identification at Checkout Unlike identification upon arrival, checkout identification was a routine procedure familiar to all study participants. It was primarily perceived positively. However, some participants reported avoiding all identification and loyalty programs. These individuals perceived identification as an intrusive attempt to gather information, and the loyalty programs were not seen as providing sufficient benefits to the customer. Additionally, identification was viewed as bothersome since it is frequently requested in many B&M stores. The participants emphasized the importance of smooth identification processes and highlighted privacy-related T. Kemppainen, L. Frank: Customer Perceptions of In-Store Identification 663 concerns. Identification should be carried out as discreetly as possible to prevent the situation from feeling overwhelming to the customer. Checkout identification should not be too personal or personalized, as anonymity was preferred during transactions. For example, being addressed by one's name was mentioned as an example of identification going too far. Moreover, participants expressed a preference for not receiving reminders from the company regarding customer data col ection. (It's okay) if it doesn't become creepy. If they were like, "Hey Mary" (name changed) at the checkout, I might be a little scared at that point. There's no need to remind that they're col ecting data. – P15 Identification at the checkout was considered useful when the customer benefits from it, such as gets loyalty points or a discount. Identification was also seen as applicable if it allows for better-targeted marketing on topics that interest the customer. The context of the purchase also determined the meaningfulness of identification. For example, in electronics purchases, identification was seen as useful because it can provide benefits to the customer: the product warranty is saved in the loyalty system, which facilitates the customer's actions in case of problems. Electronics are the best because the warranty goes straight to your account. You don't have to save the receipts. I can go to the store and say: “The phone I bought doesn't work." – P1 Determining the best method for identification proved chal enging as different methods were associated with both advantages and disadvantages. The main types of identification methods discussed by the participants, including 1) tangible and 2) digital methods, and 3) personal information, are reported next. Tangible Identification Methods Most participants found traditional tangible loyalty cards inconvenient because these cards often accumulate, making them difficult to fit in wallets. Additionally, they are prone to being forgotten at home, rendering them unavailable when needed. Because of this, the loyalty program can easily be forgotten. Plastic cards were deemed obsolete, with suspicions mounting that they would slowly phase out from the available options. 664 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION It annoys me that you always must have a card. It’s a bit too much if you need a backpack (for the cards) just to go to the stores. – P8 Whenever you go to a store and they ask if you have that card, you are like: "No, I don't have it." Then, eventually, your membership automatically gets forgotten. – P7 On the other hand, some participants also considered tangible cards a suitable means of identification because they are familiar and easy to use. Conventional identification methods were favored, particularly for vital and frequently utilized cards like those for grocery stores. Everything is changing so rapidly into this electronic form. I feel that those old methods were better in some respects. I kind of miss them. I think it used to be easier, clearer. – P14 Combining customer information with other cards, such as a driver's license, was seen as an easy option, as the driver's license is usual y always with the customer, easily accessible, and on the other hand, the customer avoids extra plastic cards. It's good if identification is available easily, for example, through an identity document. […] Something that goes as smoothly as possible, that's nice. – P6 Conversely, concerns were raised about combining customer data with official documents like driver's licenses and credit/debit cards, as those were not preferred to be associated with commercial entities or other purposes. In some companies, they put that identifier on the driver's license. That made me think a little; is it wise? Somehow, it's so official that you wouldn't want to put it there, or on bank cards. – P4 T. Kemppainen, L. Frank: Customer Perceptions of In-Store Identification 665 Identification based on Personal Information Participants also pondered various personal "intangible pieces of information," such as phone numbers, names, and customer numbers, as means of identification. Using a phone number for identification garnered mixed opinions. Some found identifying with a phone number easy to authenticate because it is simple and quick to recite. Phone number was also perceived as less personal than disclosing one's name at the cashier. Using a phone number was considered a suitable authentication method, especial y for those without smartphones. Some, however, reacted negatively to reciting their phone number, as it was perceived as slow and cumbersome. Privacy concerns related to the phone number were also discussed. Other customers in the checkout line can hear the phone number, so potential repercussions were considered. Telling your phone number is very easy. But it's not very nice if there's a long line, and you're reciting your phone number, and the cashier is confirming your name and address. – P1 Privacy concerns related to phone numbers were also adressed, considering work and residency issues. For example, working in small towns and public professions were seen as potential problematic issues. For instance, a secret phone number doesn't fit wel with identifying with a phone number. Some did not see serious threats associated with reciting a name or phone number since potential wrongdoers were perceived to find other methods if they wanted to cause harm. My daughter is a doctor, so her information is confidential. It might feel strange for her to recite a phone number at a cashier, where there might be a patient. – P3 Strangers hear it at the cashier. But it may not cause much damage. Because if someone wants to cause trouble, they can also follow me home from the store. – P4 666 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Digital Identification Digital identification methods such as QR codes, barcodes, and applications were general y considered a good way to identify a customer, often seen as better than tangible authentication methods such as loyalty cards, as they are conveniently stored on the phone, which is typical y always with the customer. QR codes were general y seen as a good way for identification. QR codes do not require disclosing any personal information, and their use was general y considered effortless. The phone is usual y always with me. For example, Subway has a nice system, they have that QR code that you show to the reader, and it identifies your account. It's pretty cool. – P9 However, potential usage-related problems were also identified with QR codes, such as difficulty opening the code and losing one’s customer account when changing phones. On the other hand, QR codes exclude loyalty from those who do not have a smartphone. They would be quite handy, but how do you make them easy to show at the register? […] There is always the problem that when you change phones, they disappear. Then you must get them again. – P4 Another identified problem with digital identification methods was their company- specific nature, which typical y means the customer must download several company-specific apps to identify themselves. Apps where multiple loyalty cards can be loaded were seen as a good and functional alternative. 5 Discussion The study contributes to the existing literature by providing insights into how customers perceive customer identification in B&M stores, shedding light on both the positive and negative viewpoints regarding identification and associated data collection. Despite the limited research on customers' perceptions of identification, particularly regarding store entrance identification, comprehending it is vital due to the increasing utilization of customer identification and digital technologies in diverse physical service settings. T. Kemppainen, L. Frank: Customer Perceptions of In-Store Identification 667 The findings of this study revealed that customer in-store identification evokes a range of opinions and viewpoints among customers. While it is recognized as standard practice in today's business landscape, there are differing perceptions towards these practices. While identification was primarily viewed positively, there is also reluctance, characterized by actively avoiding participation in loyalty programs and customer identification initiatives. Checkout identification emerged as a routine yet contentious procedure. Privacy concerns were paramount, with emphasis on the importance of discreet identification processes and the preservation of anonymity during identification. Tangible identification methods, such as loyalty cards, were deemed inconvenient and outdated, leading to cal s for more streamlined digital solutions. Digital identification methods, including QR codes and mobile apps, were general y favored for their convenience and accessibility. However, concerns were raised about the company-specific nature of these methods and their potential exclusion of non-smartphone users, as well as customers with low technical skills. Also, previous studies have indicated that consumers prefer coalition loyalty programs over single-firm programs (Shirai, 2022); therefore, it is worth considering how partnerships and services that combine various operators can be utilized in in- store identification and loyalty programs. As the findings indicate, platforms that combine different loyalty cards are a practical solution from the customer perspective. Customers' identification during the entrance and store visits was cautiously received with positive perceptions. This finding is pleasant from the merchants' perspective, as ultimately, retailers should aim to gather comprehensive information about customers as soon as they step into the store, enabling them to customize marketing messages based on individual needs (Landmark & Sjøbakk, 2017). However, grasping the associated benefits was regarded as chal enging, and there were suspicions related to identification, including concerns about privacy infringement, loss of anonymity, and the extent of tracking and data collection. The traditional customer service models were general y deemed sufficient, with no perceived need for additional digital enhancements or personalized attention during the B&M store visit. Moreover, the potential negative implications of customer recognition, such as unequal treatment based on the loyalty level were emphasized. By evaluating such apprehensions, retailers can evaluate the potential advantages and drawbacks of incorporating identification technologies, enabling them to make better-informed decisions regarding investment and implementation. Based on the findings, for 668 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION example RFID tags attached to shopping carts or baskets can be a good option for tracking customer movements, as they are not as personal as, for instance, mobile phone location data. The findings underscore the delicate balance between identification and privacy protection in the retail environment. As businesses continue to embrace digital solutions and data-driven strategies, it is imperative to address consumer concerns and uphold ethical data practices. Transparency, consent, and data security must be prioritized to foster trust and accountability. Furthermore, retailers must navigate evolving consumer preferences and technological advancements to ensure a seamless and secure shopping experience for al customers. More research is needed on how these issues can be further investigated and considered in the planning of identification and data collection in B&M environments. References Allmér, H. (2018). Servicescape for digital wellness services for young elderly. Åbo Akademi University Press, Turku, Finland. Attig, C., Franke, T. (2020). Abandonment of personal quantification: a review and empirical study investigating reasons for wearable activity tracking attrition. Computers in Human Behavior, 102, 223-237. Ali, K., Liu, A. X., Chai, E., & Sundaresan, K. (2022). Monitoring Browsing Behavior of Customers in Retail Stores via RFID Imaging. IEEE Transactions on Mobile Computing, 21(3), 1034– 1048. https://doi.org/10.1109/TMC.2020.3019652 Choi, S. H., Yang, Y. X., Yang, B., & Cheung, H. H. (2015). Item-level RFID for enhancement of customer shopping experience in apparel retail. Computers in Industry, 71, 10–23. https://doi.org/10.1016/j.compind.2015.03.003 Hauser, M., Günther, S. A., Flath, C. M., & Thiesse, F. (2019). Towards Digital Transformation in Fashion Retailing: A Design-Oriented IS Research Study of Automated Checkout Systems. Business & Information Systems Engineering, 61(1), 51–66. https://doi.org/10.1007/s12599-018-0566-9 Hui, S. K., Bradlow, E. T., & Fader, P. S. (2009). Testing Behavioral Hypotheses Using an Integrated Model of Grocery Store Shopping Path and Purchase Behavior. Journal of Consumer Research, 36(3), 478–493. https://doi.org/10.1086/599046 Knof, M., Stock-Homburg, R., & Schurer, J. (2023). How in-store sensor technologies can help retailers to understand their customers: Overview on two decades of research. The International Review of Retail, Distribution and Consumer Research, 0(0), 1–18. https://doi.org/10.1080/09593969.2023.2273256 Landmark, A. D., & Sjøbakk, B. (2017). Tracking customer behaviour in fashion retail using RFID. International Journal of Retail & Distribution Management, 45(7/8), 844–858. https://doi.org/10.1108/IJRDM-10-2016-0174 Nordfält, J., & Ahlbom, C.-P. (2024). Utilising eye-tracking data in retailing field research: A practical guide. Journal of Retailing. https://doi.org/10.1016/j.jretai.2024.02.005 Saßnick, O., Zniva, R., Schlager, C., Horn, M., Kozlica, R., Neureiter, T., Kranzer, S., Mül ner, V., & Nöbauer, J. (2023). Analyzing Customer Behavior In-Store: A Review of Available T. Kemppainen, L. Frank: Customer Perceptions of In-Store Identification 669 Technologies. In F. J. Martínez-López (Ed.), Advances in Digital Marketing and eCommerce (pp. 243–252). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-31836-8_25 Shirai, M. (2022). Which loyalty program do customers prefer: A coalition program or a single-firm program? Journal of Services Marketing, 37(5), 563–573. https://doi.org/10.1108/JSM-04-2022-0139 Zhou, Z., Shangguan, L., Zheng, X., Yang, L., & Liu, Y. (2017). Design and Implementation of an RFID-Based Customer Shopping Behavior Mining System. IEEE/ACM Transactions on Networking, 25(4), 2405–2418. https://doi.org/10.1109/TNET.2017.2689063 670 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix 1: Participant’s background information Participant Gender Age Occupation Interview Duration Childcare Assistant/Personal P1 Female 29 01:04:18 Assistant P2 Female 51 Procurement Specialist 00:46:44 P3 Female 59 Lecturer 00:50:23 P4 Female 39 Student 01:02:22 P5 Female 69 Retired (Physiotherapist) 00:31:19 P6 Female 34 Lawyer 00:49:23 P7 Female 35 Nurse 01:01:43 P8 Male 41 Electrician 00:50:48 P9 Female 54 Hourly Teacher 00:36:30 P10 Female 28 Unemployed/Part-time Worker 00:48:52 P11 Female 44 Entrepreneur/Influencer 00:46:37 P12 Female 66 Secretary 00:49:00 P13 Female 35 Speech Therapist 00:50:59 P14 Male 45 Nursing Assistant & Student 00:48:48 P15 Female 27 Musician, Freelancer 00:42:21 P16 Female 54 Entrepreneur, Designer 00:43:11 P17 Female 54 Entrepreneur, Homeopath 00:55:14 P18 Female 35 Unemployed 00:55:41 GENERATIVE AI IN ASSISTING PATIENTS WITH SYRINGOMYELIA AND HYDROCEPHALUS: A PRELIMINARY COMPARISON OF CHATGPT AND GEMINI PAVLINA KRÖCKEL1,2 1 Friedrich-Alexander Universität Erlangen-Nürnberg, Nuremberg, Germany pavlina.kroeckel@fau.de 2 Universitäres Hochschulinstitut Schaffhausen, Schaffhausen, Switzerland pavlina.kroeckel@fau.de Patients have been relying on online resources for more information on their symptoms and diagnosis. Existing research has examined Generative AI (GenAI), mostly via ChatGPT as a way of providing more information or even comfort to patients. However, research is scarce on the appropriateness of GenAI to support and inform patients with rare conditions. These patients often find themselves in a difficult-to-navigate maze especial y because they have never heard of their condition before the diagnosis and the information online can be overwhelming. In this pre-study, we demonstrate the potential of GenAI to provide support to patients concerning their diagnoses. We work with a patient who has two rare neurological conditions – syringomyelia and hydrocephalus. Utilizing a qualitative and quantitative methodology, including the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) and the Flesch Kincaid Reading Ease (FRE) score, we Keywords: ChatGPT, analyzed the patient’s feedback to a series of prompts derived from Gemini, their medical reports. The results demonstrate the potential of GenAI, GAI, generative AI as a valuable support for patients, with preliminary syringomyelia, hydrocephalus, findings indicating that Gemini excels in certain aspects such as rare understandability, actionability, readability, and is more trustworthy, condition, making it an effective tool for patient education in this context. patient DOI https://doi.org/10.18690/um.fov.4.2024.39 ISBN 978-961-286-871-0 672 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Patients with rare disorders, such as syringomyelia and hydrocephalus, have specialized information needs owing to the complexity of their conditions. These needs include access to high-quality, comprehensible information, technologies that facilitate communication among various stakeholders, and patient portals that centralize resources to enhance usability and empower decision making (Litzkendorf et al., 2016; Long et al., 2022; Stanarević Katavić, 2019). Since its launch in 2022, ChatGPT has significantly influenced AI's role in healthcare, particularly in patient education and medical decision making (Campbell et al., 2023; Chiesa-Estomba et al., 2023; Gabriel et al., 2023; Gordon et al., 2024). However, the application of digital tools such as ChatGPT and Gemini for patients with syringomyelia or hydrocephalus is still not well documented. Our study investigates how these Generative AI tools assist patients in understanding and managing their health conditions amidst the overwhelming amount of online information and limited resources from healthcare providers. Hydrocephalus is characterized by abnormal circulation of cerebrospinal fluid, leading to symptoms such as headaches and disorientation, with shunting as the primary treatment for draining excess fluid (Bristol, 2014; Hydrocephalus Association, 2024). Syringomyelia involves a fluid-fil ed cyst within the spinal cord, causing varying symptoms and potential nerve damage, which complicates both diagnosis and treatment (Milhorat, 2000; Zheng et al., 2023). It is rare, but possible for a patient to be diagnosed with both conditions. Our study focuses on a 38-year-old female initially diagnosed with hydrocephalus at the age of 33 years, required emergency shunt surgery, and later developed syringomyelia due to complications from spinal anesthesia. Unfamiliar with their conditions, they turned to online sources to seek clarity, comfort, and alternative treatments, as wel as to understand the complex medical jargon in the reports. This research aims to answer the question: " Can Generative AI tools like ChatGPT or Gemini help patients with syringomyelia and hydrocephalus understand their condition and navigate their healthcare journey?" By addressing this question, we intend to explore the potential of these tools in simplifying complex medical information and aiding treatment decision-making, thus fil ing a gap in the current research on these conditions. P. Kröckel: Generative AI in Assisting Patients with Syringomyelia and Hydrocephalus: A Preliminary Comparison of Chatgpt and Gemini 673 2 Methodology We initiated our study by interviewing the patient to gather essential information about their diagnosis and treatment, forming questions based on prior research and online patient forum discussions. After identifying their main challenge as “understanding their medical reports”, we used an MRI report from July 7, 2023, filled with complex medical terminology, to generate specific questions. These questions, listed in Table 1, were categorized into four themes: 1) Interpretation – this prompt referred to interpreting the results from the radiology report; 2) Explanation – which included additional information for the chatbot, that is, that the patient has both hydrocephalus and syringomyelia (to test whether this affects the interpretation); 3) Treatment – we asked for suggestions on how to slow down the worsening of the condition, especial y alternative medicine treatments; and 4) Reliability & Trustability – evaluating the answers in terms of how trustable and reliable they seem. A full list of questions is provided in the supplementary file1. Both ChatGPT and Gemini were tested using these prompts to determine their effectiveness in helping the patient navigate their health information. The patient used both AI tools via their account, and detailed results are available in the supplementary file. Table 1: Terms extracted from the radiology report Original sentences from the report Translated in English Chondrotische Veränderungen mit Chondrotic changes with protrusions at Protrusionen C3 bis C7 C3 to C7 Signalverlust des Nucleus pulposus der Loss of signal intensity of the nucleus Bandscheiben in der T2-Wichtung mit pulposus of the intervertebral discs in T2- diskreten Bandscheibenvorwölbungen weighted imaging with discrete disc C3 bis C7 zum Teil auch leicht protrusions at C3 to C7, partly also paramedian betont slightly paramedian accentuated Sklerosen der kleinen Sclerosis of the smal vertebral facet Wirbelkörpergelenke joints Eintauchen der Kleinhirntonsil en in Displacement of the cerebellar tonsils das Foramen magnum into the foramen magnum 1 The file can be downloaded here. 674 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION We conducted a qualitative evaluation by using methods similar to those described by Pate et al. (2021). Quantitatively, we assessed the results using the Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P), as referenced in studies by Ayoub et al. (2023), Coskun et al. (2023), McCarthy et al. (2023). For readability, we utilized the WebFX tool to calculate the Flesch Kincaid Reading Ease (FRE) score, a method established in 1975 for the US Navy (Kincaid et al., 1975) and commonly used to measure the readability of medical texts (Bernard et al., 2018; Szmuda et al., 2020). 3 Preliminary Results Understandability and actionability The patient used the PEMAT Tool and assigned scores for both criteria. Several areas were evaluated within these criteria. The results are presented in Table 2. Table 2: PEMAT score comparison for ChatGPT & Gemini Criteria Area ChatGPT Gemini Content 2 2 Word choice & Style 1 3 Understandability Organization 4 4 Layout & Design 1 1 Action identification 1 1 Actionability Direct user address 0 1 Action breakdown 0 1 Note: In the PEMAT tool, it is possible to assign either 0 (disagree) or 1 (agree), or for some areas, an NA (not applicable). To assess the understandability and actionability of the information provided by ChatGPT and Gemini, several criteria were evaluated. Both tools scored 2/2 on content, indicating that the information was understandable to the patient. However, there was a significant difference in word choice and style; Gemini achieved a perfect score of 3/3, suggesting its use of a simpler, more patient-appropriate language, whereas ChatGPT scored only 1/3, reflecting its use of more complex language that might hinder patient comprehension. Both tools excel ed in organization, each scoring 4/4, which signifies that the information was wel structured and thus P. Kröckel: Generative AI in Assisting Patients with Syringomyelia and Hydrocephalus: A Preliminary Comparison of Chatgpt and Gemini 675 facilitated understanding. In addition, the layout and design of both tools were deemed satisfactory, with each scoring 1/1. Regarding actionability, both ChatGPT and Gemini demonstrated effectiveness by scoring 1/1 in action identification, indicating that they provided clear actionable advice for the patient. However, in terms of direct user address, Gemini outperformed ChatGPT with a score of 1/1 compared to ChatGPT's 0, showing Gemini’s superior ability to engage directly with the user. Furthermore, Gemini proved more effective in breaking down actions into clear, manageable steps, scoring 1/1, whereas ChatGPT did not score in this area. Gemini’s capability enhances both the understandability and execution of suggested actions, making it a more effective tool for patient education in this context. Readability As previously mentioned, we used the FRE score to evaluate this criterion. A comparison of the readability scores is shown in Figure 1. 46,7 33 22,5 27,9 24,3 1 21,9 29,3 Interpretation Explanation Treatment Reliability Gemini ChatGPT Figure 1: FRE scores for our pre-defined prompt themes The Flesch Kincaid Reading Ease (FRE) score, a tool used to assess text readability on a scale from 0 to 100, where higher scores denote easier reading, was employed to evaluate the outputs from ChatGPT and Gemini across four different prompts. The analysis revealed that Gemini general y provided more accessible text than ChatGPT did. In the 'Interpretation' category, Gemini scored 46.7, which was markedly higher than ChatGPT’s 24.3, indicating a clearer articulation of information. Similarly, in the 'Explanation' prompt, while both scores suggested difficulty, Gemini’s 22.5 was significantly more comprehensible than ChatGPT’s 676 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION notably low 1.0. In assessing 'Treatment' options, Gemini again outperformed with a score of 33 against ChatGPT’s 21.9, underscoring a consistent trend in readability. The 'Reliability' scores were more comparable, with ChatGPT scoring 29.3 and Gemini 27.9, both indicating challenging texts yet showing slight ease with ChatGPT. This evaluation underscores a consistent pattern in which Gemini’s outputs are easier to understand than those of ChatGPT, thus enhancing its utility in delivering patient education. For qualitative evaluation, we defined five open questions to obtain direct feedback from the patient following the use of both chatbots. The questions and their answers are also included in the supplementary file. The feedback from the patient confirmed the quantitative results. The patient appreciated the clearer explanations and structured responses from Gemini, which helped demystify medical jargon more effectively than ChatGPT. Both tools motivated the patient to pursue further medical consultations, but Gemini was preferred for its use of trusted sources and ability to break down information. 4 Summary and next steps Our study assessed the effectiveness of ChatGPT and Gemini in supporting a patient with rare conditions such as syringomyelia and hydrocephalus, focusing on providing clear, actionable, and reliable information. Evaluation results indicate that Gemini excels in delivering accessible and actionable information through a structured approach, including detailed term-by-term explanations and a concluding "Outlook." In contrast, ChatGPT, while matching in content quality and organization, lacks in actionability and requires improvements in language simplification. The patient feedback was positive, particularly for Gemini, suggesting that Generative AI can empower patients by effectively bridging the information gap. However, it is important to acknowledge that these findings are based on the experience of a single patient, highlighting the need for further research. Future studies will involve a larger cohort of patients with these conditions to validate and expand the initial findings. P. Kröckel: Generative AI in Assisting Patients with Syringomyelia and Hydrocephalus: A Preliminary Comparison of Chatgpt and Gemini 677 In conclusion, this study highlights the potential of AI tools in patient education and healthcare management of rare diseases. Future research should extend to more patients and incorporate diverse evaluation metrics, such as patient engagement and information retention rates, to ful y assess the impact of AI in patient care. This work underscores the necessity of developing patient-centered AI tools that complement traditional healthcare methods, fostering a wel -supported patient community. References Ayoub, N. F., Lee, Y.‑J., Grimm, D., & Divi, V. (2023). Head-to-Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition. Otolaryngology - Head and Neck Surgery. Advance online publication. https://doi.org/10.1002/ohn.465 Bernard, S., Cooke, T., Cole, T., Hachani, L., & Bernard, J. (2018). Quality and readability of online information about type 2 diabetes and nutrition. JAAPA, 31(11), 41–44. https://doi.org/10.1097/01.JAA.0000546481.02560.4e Bristol, R. E. (2014). Hydrocephalus. In Encyclopedia of the Neurological Sciences (pp. 636–639). Elsevier. https://doi.org/10.1016/B978-0-12-385157-4.00753-3 Campbel , D. J., Estephan, L. E., Sina, E. M., Mastrolonardo, E. V., Alapati, R., Amin, D. R., & Cottrill, E. E. (2023). Evaluating ChatGPT Responses on Thyroid Nodules for Patient Education. Thyroid : Official Journal of the American Thyroid Association. Advance online publication. https://doi.org/10.1089/thy.2023.0491 Chiesa-Estomba, C. M., Lechien, J. R., Vaira, L. A., Brunet, A., Cammaroto, G., Mayo-Yanez, M., Sanchez-Barrueco, A., & Saga-Gutierrez, C. (2023). Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. European Archives of Oto-Rhino-Laryngology : Official Journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : Affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery. Advance online publication. https://doi.org/10.1007/s00405-023-08104-8 Coskun, B., Ocakoglu, G., Yetemen, M., & Kaygisiz, O. (2023). Can ChatGPT, an Artificial Intel igence Language Model, Provide Accurate and High-quality Patient Information on Prostate Cancer? Urology, 180, 35–58. https://doi.org/10.1016/j.urology.2023.05.040 Gabriel, J., Shafik, L., Alanbuki, A., & Larner, T. (2023). The utility of the ChatGPT artificial intelligence tool for patient education and enquiry in robotic radical prostatectomy. International Urology and Nephrology, 55(11), 2717–2732. https://doi.org/10.1007/s11255- 023-03729-4 Gordon, E. B., Towbin, A. J., Wingrove, P., Shafique, U., Haas, B., Kitts, A. B., Feldman, J., & Furlan, A. (2024). Enhancing Patient Communication With Chat-GPT in Radiology: Evaluating the Efficacy and Readability of Answers to Common Imaging-Related Questions. Journal of the American Col ege of Radiology : JACR, 21(2), 353–359. https://doi.org/10.1016/j.jacr.2023.09.011 Hydrocephalus Association. (2024). Shunt Systems. https://www.hydroassoc.org/shunt-systems/ Kincaid, J. P., Fishburne, R. P., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count, and Flesch reading ease formula) for Navy enlisted personnel. Chief of Naval Technical Training: Naval Air Station Memphis. Research Branch Report. 678 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION https://stars.library.ucf.edu/istlibrary/56?utm_source=stars.library.ucf.edu%2Fistlibrary%2F 56&utm_medium=PDF&utm_campaign=PDFCoverPages Litzkendorf, S., Babac, A., Rosenfeldt, D., Schauer, F., Hartz, T., Lührs, V., Schulenburg, J.‑M. G. v., & Frank, M. (2016). Information Needs of People with Rare Diseases - What Information Do Patients and their Relatives Require? Journal of Rare Disorders: Diagnosis & Therapy, 2(2:40). https://raredisorders.imedpub.com/articles/information-needs-of-people-with-rare-diseases--what-information-do-patients-and-their-relatives-require.pdf Long, J. C., Best, S., Nic Giol a Easpaig, B., Hatem, S., Fehlberg, Z., Christodoulou, J., & Braithwaite, J. (2022). Needs of people with rare diseases that can be supported by electronic resources: A scoping review. BMJ Open, 12(9), e060394. https://doi.org/10.1136/bmjopen-2021-060394 McCarthy, C. J., Berkowitz, S., Ramalingam, V., & Ahmed, M. (2023). Evaluation of an Artificial Intelligence Chatbot for Delivery of IR Patient Education Material: A Comparison with Societal Website Content. Journal of Vascular and Interventional Radiology : JVIR, 34(10), 1760-1768.e32. https://doi.org/10.1016/j.jvir.2023.05.037 Milhorat, T. H. (2000). Classification of syringomyelia. Neurosurgical Focus, 8(3). https://doi.org/10.3171/foc.2000.8.3.1 Pate, J. W., Tran, E., Radhakrishnan, S., & Leaver, A. M. (2021). The Importance of Perceived Relevance: A Qualitative Evaluation of Patient's Perceptions of Value and Impact Following a Low-Intensity Group-Based Pain Management Program. Medicina (Kaunas, Lithuania), 57(1). https://doi.org/10.3390/medicina57010046 Stanarević Katavić, S. (2019). Health information behaviour of rare disease patients: Seeking, finding and sharing health information. Health Information and Libraries Journal, 36(4), 341–356. https://doi.org/10.1111/hir.12261 Szmuda, T., Özdemir, C., Ali, S., Singh, A., Syed, M. T., & Słoniewski, P. (2020). Readability of online patient education material for the novel coronavirus disease (COVID-19): A cross-sectional health literacy study. Public Health, 185, 21–25. https://doi.org/10.1016/j.puhe.2020.05.041 WebFX. Readability Test [Computer software]. https://www.webfx.com/tools/read-able/ Zheng, Y.‑C., Liu, Y.‑T., Wei, K.‑C., Huang, Y.‑C., Chen, P.‑Y., Hsu, Y.‑H., & Lin, C.‑L. (2023). Outcome predictors and clinical presentation of syringomyelia. Asian Journal of Surgery, 46(2), 705–711. https://doi.org/10.1016/j.asjsur.2022.06.150 REPEAT AFTER ME: DYNAMIC WORKOUTS FOR THE WELLBEING OF ELDERLY INDIVIDUALS WITH DEMENTIA ANNEMAE VAN DE HOEF, SAM LEEWIS, KOEN SMIT, MARK SLINGERLAND, LARISSA KAISER, KATARIINA SAUKKO, RUBEN VAN GEND, MIRJAM DE HAAS HU University of Applied Sciences Utrecht, Digital Ethics, Utrecht, Netherland annemae.vandehoef@hu.nl, sam.leewis@hu.nl, koen.smit@hu.nl, mark.slingerland@hu.nl, larissa.kaiser@hu.nl, katari na.saukko@hu.nl, ruben.vangend@hu.nl, mirjam.dehaas@hu.nl In the past two decades, social robots have been utilized in the healthcare of elderly individuals with dementia to assist in exercising and companionship. In this research, we investigated whether elderly individuals with dementia were more engaged when receiving varying mobility sessions from a social robot than non-varying mobility sessions. Groups of elderly individuals with dementia (N=18) of differing dementia levels participated in three exercise sessions either with varying sessions (n=9) or with non-varying sessions (n=9). Our results show a smal Keywords: dementia, improvement in engagement in varied sessions compared to elderly non-varied sessions, however, the difference was not significant. care, Additionally, the study gives insights into the application of this social robots, technology and the chal enges involved in conducting this type engagement, of experiment. mobility DOI https://doi.org/10.18690/um.fov.4.2024.40 ISBN 978-961-286-871-0 680 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction As society faces an ever-increasing aging population (Kanasi et al., 2016; Luchsinger, 2023), different systems are under greater strain (Lee & Mason, 2017). These strains are particularly evident in elderly care, where there is a shortage of caregivers (Resnick et al., 2022) while care must be provided to an increasing influx of elderly patients. That is, by 2023, there were 55 million people worldwide with dementia, most of them elderly people, with 10 mil ion new cases added every year (World Health Organization, 2023). This development has inspired research into the application of technological innovations to improve the quality of life for this ever-increasing aging population (Fares et al., 2021; Penno & Gauld, 2017; Stone & Harahan, 2010). Further research is needed because in the US, for example, only half of the geriatricians needed for elderly care are available because of a financial disadvantage (e.g., there is a decrease in income when becoming a geriatrician due to the fee-for-service system) (Rowe et al., 2016). This exacerbates loneliness among the elderly population (Gardiner et al., 2020; James Alexander Crewdson, 2016), which can affect physical and mental health (James Alexander Crewdson, 2016). Especial y when it comes to elderly individuals with dementia, who experience even more social loneliness than those without dementia (Holmén et al., 2000). Due to these concerns, different methods and technologies are being explored for the healthcare system. One example of the exploration of technology is the use of social robots (SRs), which proofs to be applicable in healthcare (Aymerich-Franch & Ferrer, 2021). That is, SRs can become a tool for a caregiver, to ease or support care for the elderly (Kachouie et al., 2014). Previous research has shown the potential of SRs in healthcare. For example, the companion robot Paro provided a positive experience when it was used as a comfort buddy for the elderly (Chen et al., 2022). Indeed, Paro is a robotic seal used in elderly care for companionship, emotional support, and as a distraction for patients exhibiting chal enging behavior (Wada et al., 2003). Moreover, Paro has also been shown to have positive effects on elderly individuals with dementia in particular (Kang et al., 2020; Kelly et al., 2021). Another example of using the application of SRs to assist elderly individuals with dementia is the use of the Tessa robot from tinyBots, which can help elderly individuals with dementia establish a daily structure using reminders (Smit et al., 2021). A. van de Hoef et al.: Repeat After Me: Dynamic Workouts for The Wel being of Elderly Individuals with Dementia 681 Offering multicomponent interventions, which include physical activity, to older people with dementia can al eviate the negative effects of dementia (Blankevoort et al., 2010). Moreover, exercise can be effective in maintaining and improving the health of elderly people (Dawe & Moore‐Orr, 1995; McMurdo & Rennie, 1993). A SR could be useful in this regard, as it can be used as an on-demand exercise program (Lotfi et al., 2018). With this in mind, the SR NAO can be of particular interest due to its humanoid form and the wide range of motion possible with this robot (Robaczewski et al., 2021). Furthermore, to ensure that participants remain motivated and focused during the exercises, the session should remain engaging for them. Since there are different stages of dementia (Reisberg et al., 1982), a later stage of dementia may affect engagement due to the novelty effect (Suchy et al., 2011). In this context, the novelty effect can occur again because elderly individuals with dementia cannot remember a previous session and therefore experience the same exercise as new. However, this novelty effect may not apply to the elderly with early- onset dementia, who might remember the previous, and therefore, the repetition of exercises may negatively impact engagement. To explore the effect of repeating exercises on the engagement of elderly individuals with dementia, we beg to answer the following main research question in this paper: “How does introducing a new exercise routine through three sessions with an NAO robot af ect the engagement of elderly individuals with dementia? ” 2 Background and Related Work With a limited number of caregivers, there is a need to address the shortage so that the elderly get the care they need (Resnick et al., 2022). Besides, elderly individuals with dementia need even more support with daily activities, i.e., elderly individuals with dementia often need help with traveling or become incontinent, depending on the stage of dementia (Reisberg et al., 1982). Moreover, it is important for the quality of life that elderly individuals with dementia in long-term care facilities have the opportunity to participate in meaningful physical activities (Sampaio et al., 2021; Telenius et al., 2022). This is because physical activities are associated with a reduced risk of dementia (Kirk-Sanchez & McGough, 2013) and have been shown to have a significant positive treatment effect on elderly individuals with dementia (Heyn et al., 2004). It introduces benefits such as reducing depression, improving cognitive skills, and consequently improving the quality of life (Zhou et al., 2022). Regarding physical activity, a greater positive effect on cognition in people with dementia was 682 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION found with physical activity of less than 30 minutes (Jia et al., 2019). Therefore, arranging a physical activity for the elderly can take up to 30 minutes, but as caregivers have little time, a solution is needed. One solution could be to use SRs that help support activities for elderly individuals with dementia. For example, one study demonstrated the suitability of telepresence robots for remote research because they evoke more engagement in people with dementia compared to Skype or phone cal s (Hung et al., 2023). Another example is that elderly individuals with dementia have benefited from a SR pet which was often used on a daily basis (Harris-Gersten et al., 2023). Moreover, caregivers found that SRs can improve physical activity by, e.g., motivating and guiding elderly individuals to exercise (Zuschnegg et al., 2022). For example, research included dance sessions led by the SRs Pepper and NAO, in which the robots invited elderly individuals (without dementia) to dance (Li et al., 2022). These dance sessions were positively received by both the elderly and the caregivers (Li et al., 2022). Such an approach could help maintain physical functionality without increasing the burden on caregivers. Moreover, it also provides opportunities for conversations between people with dementia by coming together in a common room to engage with a SR (Blindheim et al., 2023). For example, research showed that using a SR led to conversation and joking in the common room of a long-term care facility (Blindheim et al., 2023). Besides physical activities, SRs can help people with dementia by communicating with them, which also improves their quality of life through engagement (Mordoch et al., 2013). In addition, SRs can provide assistance by communicating in group sessions, allowing caregivers to provide individual support when needed (Raß et al., 2023). Lastly, it was found that caregivers felt that a SR could help them avoid danger, e.g., by preventing injury through assistance in walking (Zuschnegg et al., 2022). Research, on the other hand, shows that even though SRs can be used to help in healthcare, there are stil some areas for improvement. For example, research with a robot used to comfort children before a blood draw showed that caregivers do not have al the time or experience to use such a SR (Smakman et al., 2021). In addition, functionalities of a SR are also missed due to, e.g., not being aware of certain keyboard shortcuts when using a SR (Blindheim et al., 2023). Moreover, when AI is built into a SR that monitors elderly individuals with dementia, recognizing speech patterns can be an issue because the speech of a person with dementia is sometimes A. van de Hoef et al.: Repeat After Me: Dynamic Workouts for The Wel being of Elderly Individuals with Dementia 683 unclear. This can lead to failure to summon help, for example, because the robot does not pick up on the user’s behavior properly (Felzmann, 2020). This is due to the problem of SRs being used to help vulnerable individuals and not being able to understand their vulnerability themselves (Ragno et al., 2023). Furthermore, Misselhorn et al. (2013) argue that a SR should not replace human- human interaction, as it is important to be empathetic towards the person with dementia. Moreover, caregivers also expressed concerns about the usability of SRs for people with dementia, as computer interfaces would be inaccessible to these people (Pino et al., 2015). Finally, caregivers expressed concern about SRs looking too much like humans, as they might confuse individuals with dementia (Pino et al., 2015). On the other hand, an elderly individual with dementia in research suggested that the SR used looked too mechanical (Moyle et al., 2016). In conclusion, research has shown that SRs can help elderly individuals with dementia in many ways, through physical activity and sociability, both one-on-one and in group settings, despite there being areas for improvement. However, to the knowledge of the authors, no research has yet been conducted on the engagement of older people with dementia with SRs and whether the difference in type of exercises affects engagement. 3 Research Method The purpose of this research was to determine the extent to which varying exercise sessions improve the engagement of elderly individuals with dementia compared to non-varying exercise sessions. Therefore, the same experiment was conducted twice with two separate groups. In addition, a short interview with caregivers was conducted. In the first session, both groups included the same exercise session. In the second session, the non-varied group included the same exercise session, while the varied group had a different one. In the third session, the non-varied group again included the same exercise session, while the varied group, once again, received a different exercise session along with additional human-voiced instructions through the NAO robot. The robot executed various exercises based on gym workouts and dances, as these were observed to be enjoyable to other elderly individuals (Li et al., 2022). The exercises by the robot were developed after interviews with several caregivers about 684 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION their strategies for motivating the elderly to engage in physical activity. These exercise sessions were created using the ‘Robots in de klas’ and ‘Choreographe’ platforms. In addition, the robot movements were designed to al ow the elderly to perform an entire exercise session in a seated position for safety. To complement the exercise sessions, NAO had sporty wrist and headbands, as shown in Figure 1. Figure 1: NAO Robot and Observer Source: Own To consistently measure and evaluate participant engagement, an adapted version of the ZIKO evaluation instrument (Self-evaluation Instrument for Care Settings) was used (Laevers, 2005). The adapted version of this evaluation instrument for robot interactions has more often been used with children in human-robot interactions (de Haas et al., 2022). Although this form is originally designed for children of 5 years old, it is also applicable to the elderly with dementia because the decline in cognitive abilities over time can reach a point where it intersects with the cognitive development of children (Rubial-Álvarez et al., 2012). Additionally, the engagement instrument emphasizes the interaction that the elderly with dementia had with both the tasks and the robot. Given that the exercises provided by the robot were relatively simple, the engagement instrument had criteria for the specific moments of interest related to the robot’s behavior. Therefore, we argue that ZIKO is an appropriate measurement instrument in this study. The form includes an evaluation scale ranging from 1 (extremely low) to 5 (extremely high), with specific criteria for A. van de Hoef et al.: Repeat After Me: Dynamic Workouts for The Wel being of Elderly Individuals with Dementia 685 each level of engagement. Using this form, the overal engagement of the elderly could be consistently and accurately assessed. To ensure the validity of the instrument, each elderly individual was coded twice by two researchers and the mean of the scores was used. The experiment took place in two different retirement homes in the Netherlands that were visited three separate times with a week between sessions. The participants were 65 years and older and consisted of both men and women. A purposive sample was used to ensure that only elderly people with different stages of dementia participated, detemined by their caregivers, excluding those with early-onset dementia. Only participants who were able to give informed consent were approached and asked if they wanted to participate in the experiment. Participants signed a consent form and if at any time they decided not to participate, they could opt out. The experiment took place in two retirement homes, where caregivers selected the participants and divided the participants randomly into four groups of four/five people. This resulted in two non-varied groups (NV) of a total of nine participants and two varied groups (V) of a total of nine participants. During each visit, group NV performed the same exercise session, while group V was presented with a different session each time. In each 10-minute exercise session, participants were brought into a room and placed on stools in a semicircle around a table on which the NAO robot stood. By placing NAO on a table, its instructions were easier for participants to see. The robot gave verbal instructions on what to do and demonstrated the instructions through movements. In each session, the robot introduced itself at the beginning and informed participants about what would happen during the session. Then the 10- minute practice session begins with exercises such as twisting the wrists, opening and closing the fingers, and forward punches. Each of these exercise sessions consists of a program that the robot wil execute. After each session, the robot expressed thanks for the participation and wished the participants goodbye. During the exercise sessions, the engagement of each participant with NAO was graded by two researchers at two 5-minute intervals as described above. After the three sessions, the caretakers were interviewed to reflect on the robot interaction. 686 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4 Data Analysis and Results Before the data was analyzed, engagement scores were calculated. To ensure a non- biased engagement score, the average score was used during the data analysis of two researchers who individual y scored the participants. Moreover, a mean score per session was calculated for each participant, leading to three scores for each participant in total. Not every participant was able to attend every session (session 1: n = 18, session 2: n = 15, session 3: n =15). The average engagement scores of each group of participants per session are shown in Figure 2. Because of the smal sample size, we were not able to perform any statistical analyses. Figure 2: Average engagement score of al participants per session Source: Own Figure 2 shows that the scores for the elderly were approximately the same across each session and there are only smal differences. That is, there is a slight increase in engagement between the first and third sessions in both groups (for group NV, session one: M = 2.61 and session three: M = 3.21, for group V, session one: M = 2.87 and session three: M = 3.37). Despite the scores in both groups increasing slightly each session, there is no statistical y significant difference, due to the overall scores for the two conditions being roughly similar. A. van de Hoef et al.: Repeat After Me: Dynamic Workouts for The Wel being of Elderly Individuals with Dementia 687 5 Discussion and Future Research The goal of this research was to evaluate whether introducing varied exercise sessions increases the engagement of elderly individuals with dementia compared to non-varied sessions. The data showed that there is no difference in engagement between the varied and non-varied groups. The lack of difference could be explained by several factors that occurred during the experiment. First, the different stages of dementia of the participants may have influenced the data, as this would explain the wide range of engagement scores. However, the decline is not present at al stages. For this reason, it is therefore important to conduct future research validating the results of this study. Second, it is possible that the ZIKO form, developed for children, might not have been entirely accurate because there could have been a difference in children's cognitive skills and elderly with dementia in an earlier stage of dementia. Third, an interview with caregivers present during the experiment revealed that participants could benefit more from a higher grade of individualization of the robot’s exercise program. For example, addressing a participant by name or responding differently to elderly individuals with dementia, depending on their engagement in the exercises. Therefore, future research is needed on personalizing exercises by, e.g., increasing length or difficulty, and personally addressing the elderly. In addition, future research is needed on how exercises can be adapted to different stages of dementia, combined with using a SR. Fourth, it was also mentioned by the caregivers that participants found it more difficult to follow smal er motor skil movements made by the robot such as opening and closing hands, compared to larger motor skil movements such as stretching the arms. Fifth, the different timing of the sessions may also have played a role. It was not possible to schedule al sessions at the same time of day. Some sessions took place in the morning and others in the afternoon. The different times of the sessions may have also affected engagement scores, as some participants were more active in the morning and others in the afternoon. Sixth,, the experiment included 18 participants. This makes it impossible to generalize our conclusions to a larger population and to perform analyses to confirm our hypotheses statistical y. Future research is needed in which this experiment is repeated with a larger sample size of elderly individuals with dementia, thereby increasing the overal statistical relevance and generalizability of the findings. However, elderly with dementia are a particularly difficult group to recruit, since not al of them have the power of attorney. As a result, recruitment must be done through their immediate relatives, which lengthens and complicates 688 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION the recruitment period. Besides that, the sessions took place in December, which also reduced participant recruitment due to national holidays. Moreover, it was also difficult to recruite participants because of the study's timeframe. During the experiment, the robot also drew the attention of other residents and caregivers at the care homes. Although many of them found the robot interesting, they did not want to participate in the study and commit to it for three sessions. This resulted in these extra participants in the background not being graded and not being included in the research data. Future studies should take this into account. Final y, the retirement homes could also already have offered exercise activities, which were not checked for or taken into account. The introduction of the unfamiliar robot sessions, which were focused on physical exercise, could have been made less interesting by this. 6 Conclusion With the growing aging population, the number of elderly with dementia is constantly increasing. However, the time caregivers have to provide individual care and attention is limited, requiring new solutions for the care of elderly individuals with dementia. SRs may be a key solution, helping with the daily activities of elderly individuals with dementia and thereby improving their quality of life. This paper investigated whether the engagement of the elderly with dementia would change by conducting an experiment that involved exercise sessions with the SR NAO. To do so, we tried to answer the fol owing research question: “How does introducing a new exercise routine through three sessions with an NAO robot af ect the engagement of elderly individuals with dementia? ” The results revealed that there indeed was a smal improvement in engagement in varied sessions compared to non-varied sessions, however, the difference was smal . Despite that, the elderly with dementia often showed an interest in the robot, even if they did not perform the movements instructed by the robot. From a theoretical point of view, this research contributed to the body of knowledge of social robotics, providing research on the engagement of elderly with dementia in exercise sessions given by a robot. The difference between exercise variation and no exercise variation shows that future research is needed to validate this further. From a practical point of view, this research contributes to the practical application of SRs in healthcare, specifical y in helping the elderly with dementia and relieving and assisting caregivers. Care homes can use A. van de Hoef et al.: Repeat After Me: Dynamic Workouts for The Wel being of Elderly Individuals with Dementia 689 technology, such as SR, to encourage movement in elderly individuals with dementia, which seems to have a smal effect, though this has not yet been significantly demonstrated. References Aymerich-Franch, L., & Ferrer, I. (2021). Socially assistive robots’ deployment in healthcare settings: a global perspective. CoRR, abs/2110.07404. https://arxiv.org/abs/2110.07404 Blankevoort, C. G., van Heuvelen, M. J. G., Boersma, F., Luning, H., de Jong, J., & Scherder, E. J. A. (2010). Review of Effects of Physical Activity on Strength, Balance, Mobility and ADL Performance in Elderly Subjects with Dementia. Dementia and Geriatric Cognitive Disorders, 30(5), 392–402. https://doi.org/10.1159/000321357 Blindheim, K., Solberg, M., Hameed, I. A., & Alnes, R. E. (2023). Promoting activity in long-term care facilities with the social robot Pepper: a pilot study. Informatics for Health and Social Care, 48(2), 181–195. https://doi.org/10.1080/17538157.2022.2086465 Chau, S. A., Herrmann, N., Eizenman, M., Chung, J., & Lanctôt, K. L. (2015). Exploring Visual Selective Attention towards Novel Stimuli in Alzheimer’s Disease Patients. Dementia and Geriatric Cognitive Disorders Extra, 5(3), 492–502. https://doi.org/10.1159/000442383 Chen, S.-C., Davis, B. H., Kuo, C.-Y., Maclagan, M., Chien, C.-O., & Lin, M.-F. (2022). Can the Paro be my Buddy? Meaningful experiences from the perspectives of older adults. Geriatric Nursing, 43, 130–137. https://doi.org/10.1016/j.gerinurse.2021.11.011 Dawe, D., & Moore‐Orr, R. (1995). Low‐intensity, range‐of‐motion exercise: invaluable nursing care for elderly patients. Journal of Advanced Nursing, 21(4), 675–681. https://doi.org/10.1046/j.1365-2648.1995.21040675.x de Haas, M., Vogt, P., van den Berghe, R., Leseman, P., Oudgenoeg-Paz, O., Wil emsen, B., de Wit, J., & Krahmer, E. (2022). Engagement in longitudinal child-robot language learning interactions: Disentangling robot and task engagement. International Journal of Child- Computer Interaction, 33, 100501. https://doi.org/10.1016/j.ijcci.2022.100501 Fares, N., Sherratt, R. S., & Elhajj, I. H. (2021). Directing and Orienting ICT Healthcare Solutions to Address the Needs of the Aging Population. Healthcare, 9(2), 147. https://doi.org/10.3390/healthcare9020147 Felzmann, H. (2020). Can a Robot Provide the Answer? Ethical Considerations in Using Social Robots for Assessment and Monitoring in Healthcare. In Developing and Utilizing Digital Technology in Healthcare for Assessment and Monitoring (pp. 29–44). Springer International Publishing. https://doi.org/10.1007/978-3-030-60697-8_3 Gardiner, C., Laud, P., Heaton, T., & Gott, M. (2020). What is the prevalence of loneliness amongst older people living in residential and nursing care homes? A systematic review and meta-analysis. Age and Ageing, 49(5), 748–757. https://doi.org/10.1093/ageing/afaa049 Harris-Gersten, M. L., Davagnino, J. M., Alcorn, E. R., & Hastings, S. N. (2023). Usability and Acceptability of Social Robot Pets Among Community-Dwelling Veterans Living With Dementia and Their Caregivers. American Journal of Alzheimer’s Disease & Other Dementias®, 38. https://doi.org/10.1177/15333175231200973 Heyn, P., Abreu, B. C., & Ottenbacher, K. J. (2004). The effects of exercise training on elderly persons with cognitive impairment and dementia: A meta-analysis11No commercial party having a direct financial interest in the results of the research supporting this article has or wil confer a benefit upon the author(s) or upon any organization with which the author(s) is/are associated. Archives of Physical Medicine and Rehabilitation, 85(10), 1694–1704. https://doi.org/10.1016/j.apmr.2004.03.019 690 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Holmén, K., Ericsson, K., & Winblad, B. (2000). Social and emotional loneliness among non-demented and demented elderly people. Archives of Gerontology and Geriatrics, 31(3), 177– 192. https://doi.org/10.1016/S0167-4943(00)00070-4 Hung, L., Lake, C., Hussein, A., Wong, J., & Mann, J. (2023). Using telepresence robots as a tool to engage patient and family partners in dementia research during COVID-19 pandemic: a qualitative participatory study. Research Involvement and Engagement, 9(1), 12. https://doi.org/10.1186/s40900-023-00421-w Ince Yenilmez, M. (2015). Economic and Social Consequences of Population Aging the Dilemmas and Opportunities in the Twenty-First Century. Applied Research in Quality of Life, 10(4), 735–752. https://doi.org/10.1007/s11482-014-9334-2 James Alexander Crewdson. (2016). The Effect of Loneliness in the Elderly Population: A Review. Healthy Aging & Clinical Care in the Elderly, 8, 1–8. https://doi.org/10.4137/HACCE.S35890 Jia, R., Liang, J., Xu, Y., & Wang, Y. (2019). Effects of physical activity and exercise on the cognitive function of patients with Alzheimer disease: a meta-analysis. BMC Geriatrics, 19(1), 181. https://doi.org/10.1186/s12877-019-1175-2 Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human-Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278 Kanasi, E., Ayilavarapu, S., & Jones, J. (2016). The aging population: demographics and the biology of aging. Periodontology 2000, 72(1), 13–18. https://doi.org/10.1111/prd.12126 Kang, H. S., Makimoto, K., Konno, R., & Koh, I. S. (2020). Review of outcome measures in PARO robot intervention studies for dementia care. Geriatric Nursing, 41(3), 207–214. https://doi.org/10.1016/j.gerinurse.2019.09.003 Kel y, P. A., Cox, L. A., Petersen, S. F., Gilder, R. E., Blann, A., Autrey, A. E., & MacDonel , K. (2021). The effect of PARO robotic seals for hospitalized patients with dementia: A feasibility study. Geriatric Nursing, 42(1), 37–45. https://doi.org/10.1016/j.gerinurse.2020.11.003 Kirk-Sanchez, N., & McGough, E. (2013). Physical exercise and cognitive performance in the elderly: current perspectives. Clinical Interventions in Aging, 51. https://doi.org/10.2147/CIA.S39506 Laevers, F. (2005). Well-being and Involvement in care settings. A process-oriented Self-evaluation Instrument (SIC’s). https://api.semanticscholar.org/CorpusID:203489187 Lee, R., & Mason, A. (2017). Cost of Aging. Finance & Development. Li, Y., Liang, N., Effati, M., & Nejat, G. (2022). Dances with Social Robots: A Pilot Study at Long-Term Care. Robotics, 11(5), 96. https://doi.org/10.3390/robotics11050096 Lotfi, A., Langensiepen, C., & Yahaya, S. (2018). Social y Assistive Robotics: Robot Exercise Trainer for Older Adults. Technologies, 6(1), 32. https://doi.org/10.3390/technologies6010032 Luchsinger, G. (2023). World social report 2023: Leaving no one behind in an ageing world. United Nations: Department of Economic and Social Affairs. McMurdo, M. E. T., & Rennie, L. (1993). A Control ed Trial of Exercise by Residents of Old People’s Homes. Age and Ageing, 22(1), 11–15. https://doi.org/10.1093/ageing/22.1.11 Misselhorn, C., Pompe, U., & Stapleton, M. (2013). Ethical Considerations Regarding the Use of Social Robots in the Fourth Age. GeroPsych, 26(2), 121–133. https://doi.org/10.1024/1662-9647/a000088 Mordoch, E., Osterreicher, A., Guse, L., Roger, K., & Thompson, G. (2013). Use of social commitment robots in the care of elderly people with dementia: A literature review. Maturitas, 74(1), 14–20. https://doi.org/10.1016/j.maturitas.2012.10.015 Moyle, W., Jones, C., Sung, B., Bramble, M., O’Dwyer, S., Blumenstein, M., & Estivil -Castro, V. (2016). What Effect Does an Animal Robot Cal ed CuDDler Have on the Engagement and Emotional Response of Older People with Dementia? A Pilot Feasibility Study. International Journal of Social Robotics, 8(1), 145–156. https://doi.org/10.1007/s12369-015-0326-7 A. van de Hoef et al.: Repeat After Me: Dynamic Workouts for The Wel being of Elderly Individuals with Dementia 691 Natarajan, M., & Gombolay, M. (2020). Effects of Anthropomorphism and Accountability on Trust in Human Robot Interaction. Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 33–42. https://doi.org/10.1145/3319502.3374839 Penedo, F. J., & Dahn, J. R. (2005). Exercise and wel -being: a review of mental and physical health benefits associated with physical activity. Current Opinion in Psychiatry, 18(2), 189–193. https://doi.org/10.1097/00001504-200503000-00013 Penno, E., & Gauld, R. (2017). Change, Connectivity, and Chal enge: Exploring the Role of Health Technology in Shaping Health Care for Aging Populations in Asia Pacific. Health Systems & Reform, 3(3), 224–235. https://doi.org/10.1080/23288604.2017.1340927 Pino, M., Boulay, M., Jouen, F., & Rigaud, A.-S. (2015). “Are we ready for robots that care for us?” Attitudes and opinions of older adults toward social y assistive robots. Frontiers in Aging Neuroscience, 7. https://doi.org/10.3389/fnagi.2015.00141 Ragno, L., Borboni, A., Vannetti, F., Amici, C., & Cusano, N. (2023). Application of Social Robots in Healthcare: Review on Characteristics, Requirements, Technical Solutions. Sensors, 23(15), 6820. https://doi.org/10.3390/s23156820 Raß, E., Unbehaun, D., Wulf, V., Lüssem, J., Eilers, H., Lenz, G., Tandler, J., Afzali, S. N., & Eroglu, B. (2023). Investigating the Potential and Impacts of Social Robots to Engage People with Advanced Dementia and their Caregivers: Early Insights from an Exploratory Ethnographic Study within a Protected Care Environment. Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, 272–278. https://doi.org/10.1145/3594806.3594826 Reisberg, B., Ferris, S. H., de Leon, M. J., & Crook, T. (1982). The Global Deterioration Scale for assessment of primary degenerative dementia. American Journal of Psychiatry, 139(9), 1136– 1139. https://doi.org/10.1176/ajp.139.9.1136 Resnick, B., Young, H. M., Fick, D. M., & Kagan, S. H. (2022). Making care for older people the choice of nurses today, tomorrow, and forever. Geriatric Nursing, 46, A1–A2. https://doi.org/10.1016/j.gerinurse.2022.04.014 Robaczewski, A., Bouchard, J., Bouchard, K., & Gaboury, S. (2021). Social y Assistive Robots: The Specific Case of the NAO. International Journal of Social Robotics, 13(4), 795–831. https://doi.org/10.1007/s12369-020-00664-7 Rowe, J. W., Fulmer, T., & Fried, L. (2016). Preparing for Better Health and Health Care for an Aging Population. JAMA, 316(16), 1643. https://doi.org/10.1001/jama.2016.12335 Rubial-Álvarez, S., de Sola, S., Machado, M.-C., Sintas, E., Böhm, P., Sánchez-Benavides, G., Langohr, K., Muñiz, R., & Peña-Casanova, J. (2012). The Comparison of Cognitive and Functional Performance in Children and Alzheimer’s Disease Supports the Retrogenesis Model. Journal of Alzheimer’s Disease, 33(1), 191–203. https://doi.org/10.3233/JAD-2012- 121123 Sampaio, A., Marques-Aleixo, I., Seabra, A., Mota, J., & Carvalho, J. (2021). Physical exercise for individuals with dementia: potential benefits perceived by formal caregivers. BMC Geriatrics, 21(1), 6. https://doi.org/10.1186/s12877-020-01938-5 Scoglio, A. A., Reil y, E. D., Gorman, J. A., & Drebing, C. E. (2019). Use of Social Robots in Mental Health and Wel -Being Research: Systematic Review. Journal of Medical Internet Research, 21(7), e13322. https://doi.org/10.2196/13322 Smakman, M. H. J., Smit, K., Buser, L., Monshouwer, T., van Putten, N., Trip, T., Schoof, C., Preciado, D. F., Konijn, E. A., van der Roest, E. M., & Tiel Groenestege, W. M. (2021). Mitigating Children’s Pain and Anxiety during Blood Draw Using Social Robots. Electronics, 10(10), 1211. https://doi.org/10.3390/electronics10101211 Smit, K., Smakman, M., Bakker, S., Blokhuis, J., Evertzen, G., & Polman, L. (2021). Hello, is Someone There? A Case Study for Using a Social Robot in Dementia Care. 34th Bled EConference Digital Support from Crisis to Progressive Change: Conference Proceedings, 529–546. https://doi.org/10.18690/978-961-286-485-9.38 692 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Stone, R., & Harahan, M. F. (2010). Improving The Long-Term Care Workforce Serving Older Adults. Health Affairs, 29(1), 109–115. https://doi.org/10.1377/hlthaff.2009.0554 Suchy, Y., Kraybil , M. L., & Franchow, E. (2011). Practice Effect and Beyond: Reaction to Novelty as an Independent Predictor of Cognitive Decline Among Older Adults. Journal of the International Neuropsychological Society, 17(01), 101–111. https://doi.org/10.1017/S135561771000130X Suttanon, P., Hil , K. D., Said, C. M., LoGiudice, D., Lautenschlager, N. T., & Dodd, K. J. (2012). Balance and Mobility Dysfunction and Falls Risk in Older People with Mild to Moderate Alzheimer Disease. American Journal of Physical Medicine & Rehabilitation, 91(1), 12–23. https://doi.org/10.1097/PHM.0b013e31823caeea Telenius, E. W., Tangen, G. G., Eriksen, S., & Rokstad, A. M. M. (2022). Fun and a meaningful routine: the experience of physical activity in people with dementia. BMC Geriatrics, 22(1), 500. https://doi.org/10.1186/s12877-022-03149-6 Wada, K., Shibata, T., Saito, T., & Tanie, K. (2003). Psychological and social effects of robot assisted activity to elderly people who stay at a health service facility for the aged. 2003 IEEE International Conference on Robotics and Automation, 3996–4001. https://doi.org/10.1109/ROBOT.2003.1242211 World Health Organization. (2023, March 15). Dementia. https://www.who.int/news-room/fact- sheets/detail/dementia Zhou, S., Chen, S., Liu, X., Zhang, Y., Zhao, M., & Li, W. (2022). Physical Activity Improves Cognition and Activities of Daily Living in Adults with Alzheimer’s Disease: A Systematic Review and Meta-Analysis of Randomized Control ed Trials. International Journal of Environmental Research and Public Health, 19(3), 1216. https://doi.org/10.3390/ijerph19031216 Zuschnegg, J., Paletta, L., Fel ner, M., Steiner, J., Pansy-Resch, S., Jos, A., Koini, M., Prodromou, D., Halfens, R. J. G., Lohrmann, C., & Schüssler, S. (2022). Humanoid socially assistive robots in dementia care: a qualitative study about expectations of caregivers and dementia trainers. Aging & Mental Health, 26(6), 1270–1280. https://doi.org/10.1080/13607863.2021.1913476 THE USE OF NEUTRALISATION TECHNIQUES IN THE CONTEXT OF RESPONSIBLE ONLINE SHOPPING: A LATENT PROFILE ANALYSIS MARKUS MAKKONEN,1, 2 LAURI FRANK,2 MATILDA HOLKKOLA,2 TIINA PAANANEN2 1 Tampere University, Faculty of Management and Business, Tampere, Finland markus.makkonen@tuni.fi 2 University of Jyvaskyla, Faculty of Information Technology, Jyvaskyla, Finland markus.v.makkonen@jyu.fi, lauri.frank@jyu.fi, matilda.i.holkkola@jyu.fi, ti na.e.paananen@jyu.fi Although many consumers use various neutralisation techniques to eliminate the anticipated guilt that results from not engaging in responsible consumption, the use of such techniques in the context of responsible online shopping has attracted little attention in prior research. In this study, we aim to address this gap by examining (1) whether it is possible to segment consumers in terms of their use of neutralisation techniques to eliminate the Keywords: anticipated guilt that results from not engaging in responsible responsible consumption, online shopping and (2) how these segments potential y differ responsible from each other in terms of demographics (e.g., gender, age, and online income), online shopping frequency, and anticipated guilt. The shopping, neutralisation examination is based on 478 responses from Finnish consumers techniques, that were col ected in spring 2023 and are analysed with latent anticipated profile analysis. Our findings suggest the existence of four guilt, latent distinct consumer segments with several differences between profile them in terms of demographics and anticipated guilt. analysis DOI https://doi.org/10.18690/um.fov.4.2024.41 ISBN 978-961-286-871-0 694 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Today, more and more consumers are engaging in responsible consumption, which refers to consumption that has a less negative or more positive impact on the environment, society, self, and others (Ulusoy, 2016). Because of this, it is not surprising that responsible consumption has attracted more and more attention also in academic research (Nangia et al., 2024) and has been predicted to remain a prominent research topic also concerning the consumption environments of tomorrow, such as the novel metaverse marketplaces (Pellegrino et al., 2023) that can be seen as digitally mediated spaces that immerse users in shared, real-time experiences (Hadi et al., 2024). According to prior studies (e.g., Onwezen et al. 2013, 2014a, 2014b; Antonetti & Maklan, 2014a, 2014b; Theotokis & Manganari, 2015; Lindenmeier et al., 2017), one main driver for consumers to engage in responsible consumption is their anticipated guilt, which refers to the feelings of guilt that arise from contemplating a potential deviation from one’s standards (Rawlings, 1970), such as engaging in consumer behaviour that cannot be considered responsible. However, despite this driver, there are stil many consumers who do not commonly engage in responsible consumption, for which one explanation may be the various neutralisation techniques suggested in the neutralisation theory by Sykes and Matza (1957) that consumers may use to eliminate their anticipated guilt. Prior studies (e.g., Strutton et al., 1994; Chatzidakis et al., 2007; McGregor, 2008; Antonetti & Maklan, 2014b; Gruber & Schlegelmilch, 2014) have shown the use of such techniques among consumers to be relatively common. However, their use in the specific context of online shopping has attracted little attention in prior information systems (IS) and marketing research. In this study, we aim to address the aforementioned gap in prior research. More specifically, in order to differentiate the study from prior studies on the topic, our objective is to focus less on the potential effects of the use of neutralisation techniques on other constructs, such as anticipated guilt (cf. Makkonen et al., 2023), and more on the precise use patterns of neutralisation techniques among consumers. As such, we examine (1) whether it is possible to segment consumers in terms of their use of neutralisation techniques to eliminate the anticipated guilt that results from not engaging in responsible online shopping and (2) how these segments potential y dif er from each other in terms of demographics (e.g., gender, age, and income), online shopping frequency, and anticipated guilt. The examination is based on 478 responses from Finnish consumers that were M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 695 collected in spring 2023 and are analysed by using latent profile analysis (cf. Ferguson et al., 2020) as the main analysis method. 2 Theoretical Foundation Table 1: Neutralisation techniques examined in the study Name Reference Description Denial of responsibility Sykes & Claiming not to be responsible for the deviant (DOR) Matza (1957) behaviour Denial of Sykes & Claiming that the deviant behaviour caused no injury (DOI) Matza (1957) injury Condemnation of Claiming that those who condemn the deviant the condemners Sykes & behaviour (COC) Matza (1957) engage themselves in similar behaviour Appeal to higher Sykes & Claiming that the deviant behaviour was due to loyalties (AHL) Matza (1957) actualising a higher-order ideal or value Metaphor of the Klockars Claiming that the previous good behaviour ledger (MOL) (1974) counterbalances the present bad behaviour Defence of necessity (DON) Minor (1981) Claiming that the deviant behaviour was necessary Claim of relative Claiming that the deviant behaviours of others acceptability Henry & are even (CRA) Eaton (1999) worse than my deviant behaviour Claim of Claiming not to care about what others think of individuality Henry & me or (COI) Eaton (1999) my behaviour Justification by Cromwell & Claiming that the deviant behaviour is still better comparison (JBC) Thurman in (2003) comparison to some other behaviours Claim of Coleman Claiming to have the right to engage in the entitlement (COE) (2005) deviant behaviour and to gain the benefits from it The theoretical foundation of the study is based on the neutralisation theory by Sykes and Matza (1957), which suggests that when individuals engage in deviant behaviour, they may try to eliminate the resulting feelings of guilt or shame by using various justifications for the deviant behaviour that are referred to as neutralisation 696 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION techniques. Although originally developed for the context of juvenile delinquency, the neutralisation theory has later been applied to also other contexts, such as inappropriate consumer behaviour (Strutton et al., 1994), fair trade (Chatzidakis et al., 2007), immoral and unethical consumption (McGregor, 2008), employee IS security policy violations (Siponen & Vance, 2010), software piracy (Siponen et al., 2012), music piracy (Riekkinen & Frank, 2014), sustainable consumption (Antonetti & Maklan, 2014b; Gruber & Schlegelmilch, 2014), shadow IT use (Silic et al., 2017), employee unauthorised computer access (Lin et al., 2018), and responsible online shopping (Makkonen et al., 2023). Of the various neutralisation techniques proposed in prior literature, this study focuses specifically on the ten neutralisation techniques in Table 1. These have all been found to be used by consumers in the context of sustainable consumption by Gruber and Schlegelmilch (2014), which is why we assume them to be relevant for consumers also in the closely connected context of responsible online shopping. 3 Methodology The data for the study was col ected from Finnish consumers between February and March 2023 with an online survey conducted by using the LimeSurvey service. The survey respondents were recruited by promoting the survey on social media and via the communication channels of Finnish universities and student associations. As an incentive for responding, al the respondents who completed the survey were able to take part in a prize drawing of ten gift boxes worth about 25 € each. In the survey questionnaire, the use of the ten neutralisation techniques was measured with two items each. These were developed for the study based on the studies by Siponen and Vance (2010) as wel as Gruber and Schlegelmilch (2014). In turn, anticipated guilt was measured with three items. These were adapted from the guilt inventory by Kugler and Jones (1992) as exemplified by Onwezen et al. (2013, 2014a, 2014b). The wordings of these 23 items are reported in Appendix A, and before presenting them to the respondents, we also provided a brief definition of responsible online shopping as “making consumption choices that take various ecological and ethical values (e.g., sustainable development and fair trade) into account while shopping online”. The measurement scale of al the aforementioned items was the traditional five-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree). In contrast, gender, age, income, and online shopping frequency were measured with only one item each, with age being M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 697 measured on a continuous scale and the other variables on a categorical scale. To avoid forced responses, the respondents also had the option to skip any item in the survey. The col ected data was analysed in three phases. First, we calculated a composite score for each neutralisation technique construct and the anticipated guilt construct by averaging the scores of the individual items that were measuring them as wel as assessed their reliability in terms of internal consistency by using Cronbach’s alphas and their validity in terms of discriminant validity by using disattenuated correlations as suggested by Rönkkö and Cho (2022). Second, we used the Mplus 8.8. statistical software (Muthén & Muthén, 2024) to conduct a latent profile analysis for the neutralisation technique constructs by estimating multiple models with a varying number of profiles and assessing their goodness of fit with the data. To estimate the models, we used the robust maximum likelihood (MLR) estimator, with the full information maximum likelihood (FIML) estimator used for handling the potential missing values. In turn, to assess model fit, we used four information criteria and two likelihood ratio tests recommended in recent methodological literature (e.g., Nylund-Gibson & Choi, 2018; Ferguson et al., 2020; Weller et al., 2020). The four information criteria were the consistent Akaike information criterion (CAIC) by Bozdogan (1987), the Bayesian information criterion (BIC) by Schwarz (1978), the sample-size adjusted Bayesian information criterion (SABIC) by Sclove (1987), and the approximate weight of evidence (AWE) by Banfield and Raftery (1993). In the case of these al , a lower value suggests a better fitting model, thus typically resulting in the selection of the model with the lowest value. Or, if the values continue to decrease while increasing the number of profiles, then the model after which the improvements in model fit become only marginal may also be selected (Nylund- Gibson & Choi, 2018). In turn, the two likelihood ratio tests were the Vuong-Lo- Mendel -Rubin adjusted likelihood ratio test (VLMR-LRT) by Vuong (1989) and Lo et al. (2001) as wel as the bootstrapped likelihood ratio test (BLRT) by McLachlan and Peel (2000). These are used to compare a model with k profiles against a model with k – 1 profiles to see whether the additional profile provides a statistically significant improvement in model fit or whether the model with fewer profiles is sufficient. In addition, we also diagnosed the quality of the estimated models by examining their entropy (Celeux & Soromenho, 1996), in the case of which a value that is greater than 0.8 is commonly considered to suggest sufficient differentiation between the profiles (Nylund-Gibson & Choi, 2018). Third, we used the IBM SPSS 698 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Statistics 28 software to conduct post-hoc analyses of the potential differences between the members of each profile in terms of their gender, age, income, online shopping frequency, and anticipated guilt based on the most likely profile membership. In the case of gender, income, and online shopping frequency, this was done by using cross-tabulation analysis, whereas in the case of age and anticipated guilt, this was done by using one-way analysis of variance. 4 Results In total, we received 478 valid responses to the conducted online survey. The descriptive statistics of this sample in terms of the gender, age, yearly personal taxable income, socioeconomic status, and average online shopping frequency of the respondents are reported in Table 2. As can be seen, most of the respondents were women and students as well as had a relatively low income, which was not surprising when considering how they were recruited. The age of the respondents ranged from 19 to 75 years, with a mean of 28.3 years and a standard deviation of 9.0 years. Most of the respondents (68.8%) were also relatively active online shoppers who shopped online at least monthly on average. Table 2: Sample statistics (N = 478) N % N % Gender Socioeconomic status Man 88 18.4 Student 341 71.3 Woman 364 76.2 Employee or self- employed 132 27.6 Other 26 5.4 Unemployed or unable to work 10 2.1 Age Pensioner 4 0.8 Under 25 years 206 43.1 Other 4 0.8 25–49 years 253 52.9 Online shopping frequency 50 years or over 19 4.0 At least weekly 31 6.5 Yearly personal taxable income At least monthly 298 62.3 Under 15,000 € 286 59.8 At least yearly 139 29.1 15,000–29,999 € 71 14.9 Less frequently than yearly 8 1.7 30,000 € or over 98 20.5 Has never shopped online 1 0.2 No response 23 4.8 No response 1 0.2 M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 699 4.1 Construct Reliability and Validity Table 3 reports for each neutralisation technique construct and the anticipated guilt construct the mean (M) and standard deviation (SD) of its composite score as wel as its Cronbach’s alpha (on-diagonal) and disattenuated correlations (off-diagonal). Of them, Cronbach’s alphas of at least 0.7 are commonly considered to suggest sufficient construct reliability in terms of internal consistency (Nunally & Bernstein, 1994). This criterion was met by al the constructs except for the claim of individuality, which was also so close to meeting the criterion that we decided not to drop it. In turn, disattenuated correlations of less than 0.85 are commonly considered to suggest sufficient construct validity in terms of discriminant validity (Rönkkö & Cho, 2022). This was met by al the constructs. Table 3: Construct statistics Cronbach’s alphas and disattenuated correlations N M SD DOR DOI COC AHL MOL DON CRA COI JBC COE AG DOR 478 2.134 1.118 0.932 DOI 476 1.913 0.943 0.476 0.889 COC 460 2.548 1.164 0.554 0.455 0.870 AHL 478 4.271 0.794 0.274 0.228 0.311 0.882 MOL 477 1.932 0.901 0.418 0.486 0.543 0.285 0.786 DON 475 4.024 0.931 0.149 0.028 0.056 0.355 0.138 0.847 CRA 469 2.457 1.090 0.553 0.444 0.754 0.414 0.657 0.196 0.720 COI 478 2.690 1.058 0.399 0.627 0.509 0.332 0.434 0.031 0.428 0.695 JBC 478 2.522 1.088 0.607 0.638 0.663 0.315 0.522 0.088 0.713 0.611 0.833 COE 476 2.532 1.159 0.350 0.572 0.514 0.365 0.368 0.005 0.457 0.716 0.591 0.881 AG 470 3.310 1.057 -0.322 -0.446 -0.245 -0.279 -0.196 0.012 -0.210 -0.520 -0.366 -0.428 0.838 4.2 Latent Profile Analysis Table 4 reports the log-likelihood (LL) value, the values of the four information criteria (i.e., CAIC, BIC, SABIC, and AWE), the p-values of the two likelihood ratio tests (i.e., VLMR-LRT and BLRT), and the entropy value of the estimated models in which the number of profiles (k) ranged from one to seven. The values of the four information criteria are also plotted graphical y in Appendix B. The four information criteria all suggested the selection of the four-profile model because both CAIC and AWE reached their lowest value in the case of this model and also 700 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION the values of BIC and SABIC showed only marginal decreases when increasing the number of profiles beyond four. This suggestion was also supported by VLMR- LRT, which showed that increasing the number of profiles from four to five would not result in a statistically significant improvement in model fit (p = 0.232). Despite the lack of support from BLRT, we thus decided to proceed with the four-profile model. This model also had a very high entropy value of 0.927, which suggests good differentiation between the profiles. Table 4: Fit and entropy of the estimated models k LL CAIC BIC SABIC AWE VLMR- LRT BLRT Entropy 1 -5,994.406 12,447.667 12,383.667 12,180.539 12,479.667 < 0.001 < 0.001 – 2 -5,890.131 12,317.983 12,242.983 12,004.942 12,355.483 0.002 < 0.001 0.933 3 -5,813.364 12,243.315 12,157.315 11,884.361 12,286.315 0.046 < 0.001 0.929 4 -5,764.002 12,223.456 12,126.456 11,818.590 12,271.956 0.232 < 0.001 0.927 5 -5,727.538 12,229.394 12,121.394 11,778.615 12,283.394 0.354 < 0.001 0.928 6 -5,686.565 12,226.314 12,107.314 11,729.622 12,285.814 0.601 < 0.001 0.925 7 -5,662.880 12,257.809 12,127.809 11,715.205 12,322.809 0.508 < 0.001 0.918 Table 5: Estimation results of the four-profile model Mean score (from 1 to 5) Result of the Wald test (p-value) LP1 LP2 LP3 LP4 LP1 vs. LP1 vs. LP1 vs. LP2 vs. LP2 vs. LP3 vs. (62.8%) (23.4%) (7.5%) (6.3%) LP2 LP3 LP4 LP3 LP4 LP4 DOR 1.634 3.917 1.616 1.389 < 0.001 0.902 0.046 < 0.001 < 0.001 0.193 DOI 1.794 2.522 1.665 1.314 < 0.001 0.440 < 0.001 < 0.001 < 0.001 0.071 COC 2.342 3.384 2.552 1.637 < 0.001 0.424 < 0.001 0.002 < 0.001 0.001 AHL 4.387 4.583 4.229 2.157 0.007 0.354 < 0.001 0.050 < 0.001 < 0.001 MOL 1.816 2.400 1.879 1.511 < 0.001 0.709 0.038 0.006 < 0.001 0.082 DON 4.253 4.257 1.922 3.416 0.973 < 0.001 0.004 < 0.001 0.005 < 0.001 CRA 2.325 3.171 2.364 1.609 < 0.001 0.862 < 0.001 < 0.001 < 0.001 0.003 COI 2.590 3.145 2.858 1.899 < 0.001 0.239 < 0.001 0.244 < 0.001 < 0.001 JBC 2.315 3.395 2.204 1.873 < 0.001 0.621 0.007 < 0.001 < 0.001 0.200 COE 2.428 2.989 2.791 1.669 < 0.001 0.211 < 0.001 0.499 < 0.001 < 0.001 Table 5 reports the estimation results of the four-profile model in terms of the mean scores of the neutralisation technique constructs in each of the four latent profiles (LP1–LP4) and the p-values of the Wald test for the pairwise comparisons of the differences in the mean scores between the profiles. The mean scores are also plotted M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 701 graphically in Appendix C, with each line representing a particular profile. In addition, Table 5 and Appendix C report the relative sizes of the profiles based on the most likely profile membership. Here, LP1 was found as the largest of the four profiles with a 62.8% share of the respondents, followed by LP2 with a 23.4% share, LP3 with a 7.5% share, and LP4 with a 6.3% share. Of the profiles, LP2 consisted of the most active users of neutralisation techniques, with the mean scores of al the constructs being either the highest or at least equal y high as in the other three profiles. The highest mean scores concerned the appeal to higher loyalties and the defence of necessity constructs as wel as the denial of responsibility construct, of which the latter was a unique feature of this particular profile. In contrast, LP4 consisted of the least active users of neutralisation techniques, with the mean scores of al the constructs being either the lowest or at least equal y low as in the other three profiles except for the defence of necessity construct, in the case of which the mean score was higher than in LP3 but lower than in LP1 and LP2. Final y, LP1 and LP3 were situated between these two extremes. These two profiles were practical y identical to each other except for the defence of necessity construct. That is, LP1 was characterised by the high mean scores of both the appeal to higher loyalties and the defence of necessity constructs but relatively low mean scores of al the other constructs. In contrast, LP3 was characterised by the high mean score of only the appeal to higher loyalties construct, whereas the mean scores of the defence of necessity construct and al the other constructs remained relatively low. 4.3 Post-Hoc Analyses Table 6 reports the relative distributions of gender, income, and online shopping frequency in the four profiles and in the entire sample as wel as the results of the χ2 test for testing the statistical significance of the differences in these distributions between the profiles. The χ2 test suggested statistical y significant differences in the case of gender (p = 0.002) and income (p = 0.003) but not in the case of online shopping frequency (p = 0.797). To examine these differences more closely, Table 6 also reports (in parenthesis) the adjusted standardised residuals of which values higher than 1.960 or lower than -1.960 (in bold) suggest a statistically significant difference between the distribution of a particular profile and the distribution of the entire sample at the level of p < 0.05 (Agresti, 2012). Here, in the case of gender, LP1 was found to have a higher proportion of women and a lower proportion of men than the entire sample, whereas LP2 was found to have a higher proportion of 702 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION men and a lower proportion of women than the entire sample. In addition, LP4 was found to have a higher proportion of individuals who did not identify themselves as either men or women, although this finding must be taken with caution because of the low number of such individuals in the entire sample. In turn, in the case of income, LP1 was found to have a lower proportion and LP3 and LP4 were found to have a higher proportion of respondents with a yearly personal income of 30,000 € or over than the entire sample, and LP3 was also found to have a lower proportion of respondents with a yearly personal income of under 15,000 € than the entire sample. Table 6: Results of cross-tabulation analysis Relative distributions Result of the χ2 test Variable Category LP1 LP2 LP3 LP4 Sample χ2 df p Man 14.0% 25.9% 30.6% 20.0% (-3.230) (2.335) (1.955) (0.232) 18.4% Gender 68.8% 69.4% 63.3% (N = 478) Woman 81.0% 21.111 6 0.002 (3.230) (-2.100) (-0.982) (-1.702) 76.2% Other 5.0% 5.4% 0.0% 16.7% (-0.550) (-0.044) (-1.496) (2.801) 5.4% Under 66.1% 66.4% 37.1% 48.1% 15,000 € (1.853) (0.856) (-3.277) (-1.631) 62.9% Income 15,000– 16.4% 11.2% 25.7% 11.1% (N = 455) 20.185 6 0.003 29,999 € (0.634) (-1.431) (1.715) (-0.663) 15.5% 30,000 € 17.5% 22.4% 37.1% 40.7% or over (-2.738) (0.256) (2.337) (2.503) 21.5% At least 7.0% 4.5% 5.6% 10.0% Online weekly (0.602) (-0.999) (-0.239) (0.804) 6.5% shopping At least 61.2% 67.0% 66.7% 53.3% frequency 3.098 6 0.797 monthly (-0.742) (1.122) (0.540) (-1.068) 62.5% (N = 477) Less frequently 31.8% 28.6% 27.8% 36.7% than monthly (0.456) (-0.642) (-0.438) (0.690) 31.0% Table 7 reports the means (M) and standard deviations (SD) of age and anticipated guilt in the four profiles and in the entire sample as wel as the results of one-way analysis of variance (ANOVA) for testing the statistical significance of the differences in these means between the profiles. More specifical y, we employed Welch’s (1951) one-way ANOVA because Levene’s (1960) test did not support the hypothesis on equal variances across the profiles in the case of either age (p = 0.011) or anticipated guilt (p = 0.038). Welch’s one-way ANOVA suggested statistical y significant differences in the case of both age (p = 0.031) and anticipated guilt (p < M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 703 0.001). These differences were examined more closely with multiple comparisons conducted by using the Games-Howell (1976) test because of the unequal variances across the profiles. Here, in the case of age, the multiple comparisons initial y suggested no statistical y significant differences between any of the profiles, although the mean age seemed to be lower in LP1 and LP2 than in LP3 and LP4. Thus, we repeated Welch’s one-way ANOVA after merging LP1 with LP2 and LP3 with LP4, and its result confirmed that the observed difference in the mean age between the two former and two latter profiles was indeed statistically significant (p = 0.003). In contrast, in the case of anticipated guilt, the multiple comparisons immediately suggested several statistical y significant differences between the profiles. More specifically, anticipated guilt was found to be higher in LP4 than in LP1 (p < 0.001), LP2 (p < 0.001), and LP3 (p = 0.003) as well as higher in LP1 than in LP2 (p < 0.001). Table 7: Results of one-way ANOVA Welch’s one-way ANOVA Variable Profile N M SD W df1 df2 p LP1 300 27.827 8.634 LP2 112 27.339 7.629 Age (N = 478) LP3 36 32.028 11.277 3.099 3 78.944 0.031 LP4 30 32.367 12.107 Sample 478 28.314 8.998 LP1 295 3.411 0.991 Anticipated LP2 112 2.850 1.083 guilt LP3 36 3.278 1.137 18.463 3 80.153 < 0.001 (N = 470) LP4 27 4.148 0.742 Sample 470 3.310 1.057 5 Discussion and Conclusion In this study, we examined (1) whether it is possible to segment consumers in terms of their use of neutralisation techniques to eliminate the anticipated guilt that results from not engaging in responsible online shopping and (2) how these segments potential y dif er from each other in terms of demographics (e.g., gender, age, and income), online shopping frequency, and anticipated guilt. In answer to the first question, we were able to identify four consumer segments (or latent profiles, as they are called in latent profile analysis), each with its characteristic profile for using neutralisation techniques. Of these, the second segment consisted 704 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION of the most active users of neutralisation techniques, the fourth segment consisted of the least active users of neutralisation techniques, and the first and third segments were situated between these two extremes with only smal differences between each other. Overal , we made two interesting findings concerning these segments. On one hand, we found that, in each segment, there was at least one actively used neutralisation technique, meaning that the use of neutralisation techniques is very universal and practical y al consumers use them in one way or another. On the other hand, we also found that, in al the segments, the use focused on only a few neutralisation techniques: the appeal to higher loyalties and the defence of necessity in the first segment, the appeal to higher loyalties, the defence of necessity, and the denial of responsibility in the second segment, the appeal to higher loyalties in the third segment, and the defence of necessity in the fourth segment. In other words, consumers most often try to eliminate the anticipated guilt that results from not engaging in responsible online shopping with justifications based on the actualisation of some higher-order ideal or value (e.g., choosing a cheaper but less responsible alternative to be able to provide for one’s family), the lack of responsible alternatives, and the fact that they cannot real y change anything with their own consumption choices. In answer to the second question, we found several differences between the segments in terms of gender, age, income, and anticipated guilt, but not in terms of online shopping frequency. In terms of demographics, we found that the active use of neutralisation techniques was most strongly associated with being a man instead of a woman as wel as being younger, whereas the inactive use of neutralisation techniques was most strongly associated with being older and having a higher income. These findings are not surprising when considering that prior studies have found men to engage in sustainable consumption less likely than women (e.g., Isenhour & Ardenfors, 2009) and that limited financial resources likely force younger consumers with a lower income to resort to less responsible alternatives more often than older consumers with a higher income. Thus, also the use of neutralisation techniques to eliminate the resulting anticipated guilt is likely to be more common among men and younger consumers with a lower income than among women and older consumers with a higher income. In turn, in terms of online shopping frequency, it was interesting to find no association with the use of neutralisation techniques, suggesting that the mere frequency of having to make consumption choices does not, per se, seem to make one a more or less active user M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 705 of neutralisation techniques. Finally, in terms of anticipated guilt, our findings suggest a strong negative association with the use of neutralisation techniques because anticipated guilt was found to be lowest in the second segment with the most active users of neutralisation techniques and highest in the fourth segment with the least active users of neutralisation techniques. This finding is not only consistent with the neutralisation theory by Sykes and Matza (1957) but also supports the findings of the prior study by Makkonen et al. (2023) concerning the negative effect of using neutralisation techniques on the anticipated guilt that results from not engaging in responsible online shopping. To our knowledge, this study is the first to focus on segmenting individuals based on their use of neutralisation techniques, which is why its findings can be seen to contribute to a better understanding of the use of neutralisation techniques not only in the specific context of responsible online shopping but also more general y. This better understanding of the most used neutralisation techniques and the specific segments in which they are being used can be seen as highly valuable in not only theoretical but also practical respects. For example, by better understanding which consumers are most likely to use which neutralisation techniques, the retailers in the traditional online and offline as wel as in the novel omnichannel and metaverse environments can better target their actions for limiting the use of neutralisation techniques among consumers, thus nudging them away from less responsible and towards more responsible consumption practices. This, in turn, can be seen as critical for the future survival of our whole planet. 6 Limitations and Future Research We see this study to have three main limitations. First, our sample consisted only of Finnish consumers and was also not ful y representative of all Finnish consumers in terms of variables like gender and age. This obviously limits the generalisability of our findings, particularly in terms of the relative sizes and precise compositions of the identified segments, and urgently calls for future replications of this study in other countries and by using more representative samples. Second, in the post-hoc analyses, we focused only on a very limited set of variables that were used to examine the potential differences between the identified segments. Future studies could extend this set with numerous other variables, such as the personality (e.g., Bosnjak et al., 2007), individual values (e.g., Makkonen et al., 2019a), as well as online 706 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION shopping skilfulness and self-efficacy (e.g., Makkonen et al., 2022) of consumers and the emotions that consumers experience during online shopping (e.g., Makkonen et al., 2019b). Third, in the study, we focused on the use of neutralisation techniques only in the context of responsible online shopping in general. Future studies could focus on their use also in some more specific contexts in which responsible consumption has been found to play an important part, such as fashion retailing (e.g., Kemppainen et al., 2021, 2022). References Agresti, A. (2012). Categorical Data Analysis (3rd ed.). Hoboken, NJ: Wiley. Antonetti, P., & Maklan, S. (2014a). Exploring postconsumption guilt and pride in the context of sustainability. Psychology & Marketing, 31(9), 717–735. Antonetti, P., & Maklan, S. (2014b). Feelings that make a difference: How guilt and pride convince consumers of the effectiveness of sustainable consumption choices. Journal of Business Ethics, 124(1), 117–134. Banfield, J. D., & Raftery, A. E. (1993). Model-based Gaussian and non-Gaussian clustering. Biometrics, 49(3), 803–821. Bosnjak, M., Galesic, M., & Tuten, T. (2007). Personality determinants of online shopping: Explaining online purchase intentions using a hierarchical approach. Journal of Business Research, 60(6), 597–605. Bozdogan, H. (1987). Model selection and Akaike’s Information Criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370. Celeux, G., & Soromenho, G. (1996). An entropy criterion for assessing the number of clusters in a mixture model. Journal of Classification, 13(2), 195–212. Chatzidakis, A., Hibbert, S., & Smith, A. P. (2007). Why people don’t take their concerns about fair trade to the supermarket: The role of neutralisation. Journal of Business Ethics, 74(1), 89–100. Coleman, J. W. (2005). The Criminal Elite: Understanding White-Collar Crime (6th ed.). New York, NY: Worth Publishers. Cromwell, P., & Thurman, Q. (2003). The devil made me do it: Use of neutralizations by shoplifters. Deviant Behavior, 24(6), 535–550. Ferguson, S. L., Moore, E. W. G., & Hul , D. M. (2020). Finding latent groups in observed data: A primer on latent profile analysis in Mplus for applied researchers. International Journal of Behavioral Development, 44(5), 458–468. Games, P. A., & Howell, J. F. (1976). Pairwise multiple comparison procedures with unequal n’s and/or variances: A Monte Carlo study. Journal of Educational Statistics, 1(2), 113–125. Gruber, V., & Schlegelmilch, B. B. (2014). How techniques of neutralization legitimize norm- and attitude-inconsistent consumer behavior. Journal of Business Ethics, 121(1), 29–45. Hadi, R., Melumad, S., & Park, E. S. (2024). The Metaverse: A new digital frontier for consumer behavior. Journal of Consumer Psychology, 34(1), 142–166. Henry, S., & Eaton, R. (1999). Degrees of Deviance: Student Accounts of Their Deviant Behavior (2nd ed.). Salem, WI: Sheffield Publishing. Isenhour, C., & Ardenfors, M. (2009). Gender and sustainable consumption: Policy implications. International Journal of Innovation and Sustainable Development, 4(2–3), 135–149. Kemppainen, T., Frank, L., & Luhtanen, V. (2022). What is meaningful for responsible shoppers in online fashion retail? In P. Bednar, A. S. Islind, H. Val o Hult, A. Nolte, M. Rajanen, F. Zaghloul, A. Ravarini & A. M. Braccin (Eds.), Proceedings of the 8th International Workshop on Socio-Technical Perspective in IS Development. CEUR Workshop Proceedings 3239. M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 707 Kemppainen, T., Frank, L., Makkonen, M., & Hyvönen, O.-I. (2021). Barriers to responsible consumption in e-commerce: Evidence from fashion shoppers. In A. Pucihar, M. Kljajić Borštnar, R. Bons, H. Cripps, A. Sheombar & D. Vidmar (Eds.), Proceedings of the 34th Bled eConference (pp. 323–335). Maribor, Slovenia: University of Maribor Press. Klockars, C. B. (1974). The Professional Fence. New York, NY: Free Press. Kugler, K., & Jones, W. H. (1992). On conceptualizing and assessing guilt. Journal of Personality and Social Psychology, 62(2), 318–327. Levene, H. (1960). Robust tests for equality of variances. In I. Olkin (Ed.), Contributions to Probability and Statistics (pp. 278–292). Palo Alto, CA: Stanford University Press. Lin, T.-C., Hsu, J. S.-C., Wang, Y.-C., & Wu, S. (2018). Examining the antecedents of employee unauthorized computer access. Journal of Statistics & Management Systems, 21(3), 493–517. Lindenmeier, J., Lwin, M., Andersch, H., Phau, I., & Seemann, A.-K. (2017). Anticipated consumer guilt: An investigation into its antecedents and consequences for fair-trade consumption. Journal of Macromarketing, 37(4), 444–459. Lo, Y., Mendell, N. R., & Rubin, D. B. (2001). Testing the number of components in a normal mixture. Biometrika, 88(3), 767–778. Makkonen, M., Frank, L., & Kemppainen, T. (2019a). The effects of individual values on online shopping spending. In A. Pucihar, M. Kljajić Borštnar, R. Bons, J. Seitz, H. Cripps & D. Vidmar (Eds.), Proceedings of the 32nd Bled eConference (pp. 969–994). Maribor, Slovenia: University of Maribor Press. Makkonen, M., Frank, L., & Paananen, T. (2023). The role of anticipated guilt and its neutralisation in explaining responsible online shopping. In A. Pucihar, M. Kljajić Borštnar, R. Bons, G. Ongena, M. Heikkilä & D. Vidmar (Eds.), Proceedings of the 36th Bled eConference (pp. 595–613). Maribor, Slovenia: University of Maribor Press. Makkonen, M., Nyrhinen, J., Frank, L., & Karjaluoto, H. (2022). The effects of general and mobile online shopping skilfulness and multichannel self-efficacy on consumer showrooming behaviour. In A. Pucihar, M. Kljajić Borštnar, R. Bons, A. Sheombar, G. Ongena & D. Vidmar (Eds.), Proceedings of the 35th Bled eConference (pp. 479–494). Maribor, Slovenia: University of Maribor Press. Makkonen, M., Riekkinen, J., Frank, L., & Jussila, J. (2019b). The effects of positive and negative emotions during online shopping episodes on consumer satisfaction, repurchase intention, and recommendation intention. In A. Pucihar, M. Kljajić Borštnar, R. Bons, J. Seitz, H. Cripps & D. Vidmar (Eds.), Proceedings of the 32nd Bled eConference (pp. 931–953). Maribor, Slovenia: University of Maribor Press. McGregor, S. L. T. (2008). Conceptualizing immoral and unethical consumption using neutralization theory. Family and Consumer Sciences Research Journal, 36(3), 261–276. McLachlan, G., & Peel, D. (2000). Finite Mixture Modeling. New York, NY: Wiley. Minor, W. W. (1981). Techniques of neutralization: A reconceptualization and empirical examination. Journal of Research in Crime and Delinquency, 18(2), 295–318. Muthén, L. K., & Muthén, B. O. (2024). Mplus Home Page. Available at https://www.statmodel.com Nangia, P., Bansal, S., & Thaichon, P. (2024). Doing more with less: An integrative literature review on responsible consumption behaviour. Journal of Consumer Behaviour, 23(1), 141–155. Nunnal y, J. C., & Bernstein, I. H. (1994). Psychometric Theory (3rd ed.). New York, NY: McGraw-Hill. Nylund-Gibson, K., & Choi, A. Y. (2018). Ten frequently asked questions about latent class analysis. Translational Issues in Psychological Science, 4(4), 440–461. Onwezen, M. C., Antonides, G., & Bartels, J. (2013). The Norm Activation Model: An exploration of the functions of anticipated pride and guilt in pro-environmental behaviour. Journal of Economic Psychology, 39, 141–153. Onwezen, M. C., Bartels, J., & Antonides, G. (2014a). Environmental y friendly consumer choices: Cultural differences in the self-regulatory function of anticipated pride and guilt. Journal of Environmental Psychology, 40, 239–248. 708 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Onwezen, M. C., Bartels, J., & Antonides, G. (2014b). The self-regulatory function of anticipated pride and guilt in a sustainable and healthy consumption context. European Journal of Social Psychology, 44(1), 53–68. Pel egrino, A., Wang, R., & Stasi, A. (2023). Exploring the intersection of sustainable consumption and the Metaverse: A review of current literature and future research directions. Heliyon, 9(9), e19190. Rawlings, E. I. (1970). Reactive guilt and anticipatory guilt in altruistic behavior. In J. Macaulay & L. Berkowitz (Eds.), Altruism and Helping Behavior (pp. 163–177). New York, NY: Academic Press. Riekkinen, J., & Frank, L. (2014). Music piracy neutralization and the youth of the 2010’s. In A. Pucihar, C. Carlsson, R. Bons, R. Clarke & M. Kljajić Borštnar (Eds.), Proceedings of the 27th Bled eConference (pp. 44–54). Kranj, Slovenia: Moderna organizacija. Rönkkö, M., & Cho, E. (2022). An updated guideline for assessing discriminant validity. Organizational Research Methods, 25(1), 6–47. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6(2), 461–464. Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333–343. Silic, M., Barlow, J. B., & Back, A. (2017). A new perspective on neutralization and deterrence: Predicting shadow IT usage. Information & Management, 54(8), 1023–1037. Siponen, M., & Vance, A. (2010). Neutralization: New insights into the problem of employee information systems security policy violations. MIS Quarterly, 34(3), 487–502. Siponen, M., Vance, A., & Wil ison, R. (2012). New insights into the problem of software piracy: The effects of neutralization, shame, and moral beliefs. Information & Management, 49(7–8), 334–341. Strutton, D., Vitel , S. J., & Pelton, L. E. (1994). How consumers may justify inappropriate behavior in market settings: An application on the techniques of neutralization. Journal of Business Research, 30(3), 253–260. Sykes, G. M., & Matza, D. (1957). Techniques of neutralization: A theory of delinquency. American Sociological Review, 22(6), 664–670. Theotokis, A., & Manganari, E. (2015). The impact of choice architecture on sustainable consumer behavior: The role of guilt. Journal of Business Ethics, 131(2), 423–437. Ulusoy, E. (2016). Experiential responsible consumption. Journal of Business Research, 69(1), 284–297. Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2), 307–333. Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38(3/4), 330–336. Wel er, B. E., Bowen, N. K., & Faubert, S. J. (2020). Latent class analysis: A guide to best practice. Journal of Black Psychology, 46(4), 287–311 M. Makkonen et al.: The Use of Neutralisation Techniques in the Context of Responsible Online Shopping: A Latent Profile Analysis 709 Appendix A: Item Wordings Item Wording I find that is OK for me not to make responsible consumption choices when shopping online because… DOR1 … one person cannot real y trigger any change with his or her choices. DOR2 … one person cannot real y change anything with his or her choices. DOI1 … it causes no actual harm to anybody. DOI2 … it caused no actual damage to anybody. COC1 … people who cal for responsibility from others sometimes do the same. COC2 … people who cal for responsibility from others do not always themselves make responsible choices. AHL1 … I have to consider also other values or criteria (e.g., price) when making my choices. AHL2 … I have to take into account also other values or criteria (e.g., price) when making my choices. MOL1 … I have already made enough responsible choices earlier in my life. MOL2 … the responsible choices that I have made earlier in my life compensate for it. DON1 … the lack of responsible alternatives sometimes makes it necessary. DON2 … responsible alternatives are not always available. CRA1 … many other people fail to make them even more often than me. CRA2 … I stil fail to make them less often than many other people. COI1 … I do not care what other people think about my choices. COI2 … my choices do not belong to other people. JBC1 … there are far worse things in the world. JBC2 … it is not a very bad thing compared to many other things. COE1 … I am entitled to do so if I want to. COE2 … I have the right to do so if I wish. If I do not make responsible consumption choices when shopping online, I feel… AG1 … guilty. AG2 … remorseful. AG3 … bad. 710 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix B: Information Criteria of the Estimated Models 12.500 12.400 12.300 12.200 12.100 12.000 11.900 11.800 11.700 1 2 3 4 5 6 7 CAIC BIC SABIC AWE Appendix C: Estimation Results of the Four-Profile Model 5,000 4,000 3,000 2,000 1,000 DOR DOI COC AHL MOL DON CRA COI JBC COE LP1 (62.8%) LP2 (23.4%) LP3 (7.5%) LP4 (6.3%) MAPPING THE LANDSCAPE OF DIGITAL HEALTH USAGE IN INFORMATION SYSTEMS RESEARCH SARA ARZIDEH, JOZSEF MEZEI Åbo Akademi University, Faculty of Social Sciences, Business and Economics, and Law, Southwest, Finland sara.arzideh@abo.fi, jozsef.mezei@abo.fi This study investigates the use of digital healthcare in information systems (IS) research, emphasizing the need for a nuanced understanding of the conflation of related terms. The lack of an agreement on the definition of "digital healthcare usage" in research within this domain complicates assessing its impact. A conceptual framework is essential to clarify these terms and facilitate further investigation of digital health in IS. Through a combined quantitative and qualitative analysis of 5510 carefully identified articles from the IS literature, we outlined the landscape of digital healthcare usage. This groundwork is a crucial stepping stone for understanding technology integration and users’ engagement, pivotal for sustainable digital health development. The analysis revealed evolving trends in digital Keywords: health research, shifting from utility, usability, and user-centric digital design to sustainability, privacy, and security considerations. The healthcare, proposed framework not only provides clarity in terminology but information systems, also serves as a foundation for future research. This study is usability, instrumental in guiding future IS research. sustainability DOI https://doi.org/10.18690/um.fov.4.2024.42 ISBN 978-961-286-871-0 712 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction A significant segment of individuals now has access to internet via smartphones and tablets, leading to an increased interaction with digital platforms to meet various needs, such as health-related information seeking (Hollis et al. 2015). This increase in digital interaction is part of a larger trend referred to as the digital transformation. Digital health is broadly defined as the understanding and application of digital technologies to enhance health outcomes (Yao et al. 2022; Saukkonen et al., 2022; Adjekum, Blasimme, and Vayena, 2018). Digital health includes electronic health (E- health) and mobile health (m-health) (Adjekum, Blasimme, and Vayena, 2018). These terms refer to healthcare provided online using technology such as mobile phones and remote monitoring equipment (Hollis et al. 2015). Achieving synergy between clinical information and communication technology (ICT) solutions and advanced computer science is vital to realising digital health’s potential and improving healthcare management and care delivery (Kostkova 2015). The digital health sector, focusing on innovations in public health and ICT, promotes multidisciplinary research, and advocates collaboration among stakeholders to enact meaningful change (Kostkova 2015). However, several chal enges remain. Improving patient outcomes while lowering costs is a global chal enge for healthcare providers. Furthermore, the effective use of digital health is hindered by factors such as outdated systems, the absence of standardized data sources, technology-related debt, security concerns, and privacy issues (Kruse et al. 2017; Gopal et al., 2019). The use of digital healthcare solutions has been the topic of a significant amount of literature. However, a shared understanding of the term “usage” is missing, resulting in sometimes conflicting interpretations in the academic discourse. Based on Jakob Nielsen’s definitions (1993), if a system can be used to accomplish a desired goal, it is useful, and usefulness is defined by a pair of usability and utility. While utility measures whether the system's functioning can meet the required needs, usability refers to how wel that function can be utilized by users. When health information systems are considered, usability is primarily defined by the factors impacting the likelihood of their usage, such as response time or the user's ability to figure out the necessary actions to achieve their desired activity (Overhage 2003). The S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 713 development of this discipline has been shaped by its initial foundations in statistics and psychology towards a focus on user-centric design (Lewis 2014). Despite existing definitions of the usage of systems, an understanding of what defines "usage" in this field is required. This research aims to differentiate between the descriptor terms utilized for characterizing the use of digital health solutions, such as utility and usability. This misconception can lead to chal enges when assessing the impact of digital healthcare solutions, resulting in overlooking broader implications in addressing users' needs and sustainable development elements. This research aims to provide an in-depth understanding of the terminology and principles through the following research objectives: • RO1: Identifying and categorizing the dominant key-terms related to the “use of digital healthcare” in the field of IS to understand how researchers in this field have conceptualized and approached the use of digital healthcare. • RO2: Understanding the evolution of key concepts of digital health utilization over the years in IS, to reveal the most prominent research areas and gaps • RO3: Mapping the evolution of the “use of digital healthcare” in Information Systems research and the interconnections with related concepts. This article presents a conceptual framework to define the landscape of research terms associated with the “use of digital healthcare” in the field of information systems. This groundwork can play an important role as the background of future studies to discern effective factors influencing the sustainable use of digital healthcare solutions. 2 Methodology In this study we present the analysis of 5510 articles focused on the use of digital healthcare within IS. By categorizing terms, analysing trends, and visualizing interconnections, this research aims to effectively map out this fast-evolving field. 714 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 2.1 Data Collection In identifying relevant articles, we used the Scopus database. The research term query included a combination of terms from two groups: (i) related to digital healthcare, and (ii) related to its usage. The term digital health can be used interchangeably with digital medicine, electronic health, mobile health (mHealth), telecare, and telehealth (Adjekum, Blasimme, and Vayena 2018). To ensure coverage of al related terms, their synonyms and alternative spel ings were verified in EBSCOhost (Academic Search Complete), Cambridge dictionary thesaurus, and Mesh terms. The database search was completed on 1.3.2024 using the research term presented in Table1. The screening of articles was done in several stages, as shown in Table 1. The initial search based on Article title, Abstract, and Keywords in Scopus resulted in 67,795 articles. In the subsequent screening phases, the records were filtered based on language, document type, the Scopus subject area (Information Systems). The final number of records included in our research was 5510. Table 1: Data col ection table Research-terms Databases Findings Scopus ALL=(("digital health*" OR "e-health" OR "ehealth" Title, Abstract, OR "telemedic*" OR "health information system*" Keywords 67795 OR "telehealth*" OR "tele-health*" OR "mobile Refined By: health*" OR "mhealth*" OR "m-health*" OR Languages: English "online health*" OR "virtual*medicine" OR "virtual 41599 health*" OR "tele*care" OR "remote health*" OR Document Types: "telemonitoring*" OR "teleconsult*") AND Article ("usage*" OR "utili?ation*" OR "usabilit*" OR Refined By Scopus "utilit*" OR "use" )) subject area: Computer 5510 Science 2.2 Bibliometric analysis Bibliometric analysis refers to a quantitative method to study scientific publications (Lazarides 2023). Bibliometric analysis is recognised for its capacity to handle vast amounts of scientific data (Donthu et al. 2021). The advancement and availability of bibliometric software like VOSviewer, scientific databases such as Scopus and Web S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 715 of Science, and the cross-disciplinary use of bibliometric methodology have contributed to the growing popularity of bibliometric analysis in research (Donthu et al. 2021). Bibliometric analysis tools can assist researchers in uncovering domain trends, gaining perspective and identifying knowledge gaps (Li and Zhou 2021). In this study, the col ected data was analysed to identify the spectrum of terminologies employed to describe the use of digital health technologies, employing the keyword co-occurrence analysis method. Following this initial data analysis, the data was further investigated using in two steps: (i) trend analysis and (ii) network analysis. These steps aimed to find and visualise concepts’ trends over time, and to seek a way to place the key-terms in the correct sections of the conceptual framework. This was achieved through the application of Pandas for data manipulation, Openpyxl to read the data files, Matplotlib and Seaborn for visualization, and NetworkX for network analysis and visualisation. 2.3 Co-occurrence analysis At first step of data analysis, the data file extracted from Scopus database was utilised for author keywords co-occurrence data mapping applying VOSviewer. The minimum number of occurrences was set at 10. The thesaurus file, including the 1823 most frequently co-occurring words, was created after data cleansing. The data was cured manual y and by using Excel to merge spel ing differences, ignore irrelevant terms, remove repetitions in order to have the most occurrent terms focused on use of digital healthcare in the field of Information Systems. This process is depicted in Figure 1. Figure 1: Process of data analysis by VOSviewer Source: Own 716 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 2.4 Data Curing Numerous procedures were taken into account during the curation and cleansing of the data to guarantee accuracy and clarity. The issue of spel ing differences is addressed by merging different spel ings of the same terms, such as "user center design" and "user centered design" and "user-center design", or "access to healthcare" and "access to health care". Singular and plural forms were merged, for example, "health outcome" and "health outcomes". Abbreviations were expanded and merged into their complete forms, for instance, "tam" and "technology acceptance model". This process seeks to reduce duplication and prevent the scattering of data with similar values, thereby enabling more efficient data analysis. Final y, certain terms that were considered too general or irrelevant to the focus of the study were excluded, like "people", "health", "diseases", "nurse", "hospital", "alcohol", "Wi-Fi", and "student". The steps of this process are summarized in Figure 2. After the data cleaning process, VOSviewer was utilized to calculate and visualize the co-occurrence of keywords in order to understand how “usage” is conceptualized. In this study, the words that had most occurrence with the author keywords, cal ed "key-terms”. The list of key-terms presented in Appendix 1. Figure 2: Workflow diagram Source: Own S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 717 2.5 Thematic analysis In qualitative research, thematic analysis is an approach for extracting significant themes and patterns from unstructured data (Thompson 2022). Rather than just summing or categorising codes, themes are deliberate patterns or concepts derived from data gathering that address a study issue (Kiger and Varpio 2020). The study’s conceptual framework was formed utilising a thematic approach to identify dominant themes and patterns in the dataset. During this process, patterns and themes are derived from textual data of articles’ title, abstract, and keywords. The elements of this framework are defined by using Jakob Nielsen’s definitions on usability and utility as the starting point, and shifting focus to user-centric design as mentioned by Lewis in 2014. The framework was expanded by incorporating two other most frequent concepts, resulting in a framework with five core concepts: “user-centred design”, “usability”, “utility”, “sustainability”, and “considerations”. These categories wil serve as elements of the conceptual framework. In the following analysis phases, the most co-occurring terms used in included documents on “use of digital healthcare” in IS area, categorised into these five sections, based on their connections in the co-occurrence network. 2.6 Trend analysis To further explore the co-existing conceptualizations regarding the notion of “use” of digital healthcare, content analysis was performed on the included articles. This step involved doing trend analysis on included articles to track the citations of the main concepts (“usability”, “utility”, and “user-centred") over time to understand the impact and evolution of these concepts. The extracted data file was cleaned by eliminating rows with missing values and by standardizing terms with thesaurus. The main analysis involved identifying articles that address the concepts of “usability”, “utility”, and “user-centred design” through string matching in the “Title”, “Abstract”, and “Author Keywords” columns, the “Concepts” column appended to the data frame in order to categorize each article effectively. The final data frame was careful y curated to retain only pertinent data, included “Title”, “Year”, “Cited by”, “Author Keywords”, “Abstract”, and “Concepts”. This structured approach enabled a comprehensive and detailed 718 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION analysis of the thematic trends within the dataset. The citation trends of “utility”, “usability”, “user-centred design”, “Sustainability”, and “Considerations” visualized over time to provide insights into understanding changes in the key concepts of digital health utilization over the years along with their citation. Figure 3 presents this plot. Figure 3: 3D view of concepts' changes over time Source: Own 2.7 Network analysis The data created in the previous phase was used for network analysis. A new thesaurus was defined to address alternative representations of identified key-terms, to accurately identify and normalize terms in the dataset. A network was constructed using NetworkX package in Python. It involved checking co-occurrence, extracting and normalising thesaurus terms and their variants within the “Title”, “Abstract”, and “Author Keywords” of the articles. Nodes and weighted edges were generated in the graph based on the co-occurrences these terms with the concepts in each article. This network was visualised, using Matplotlib and shown in Figure 4. In this figure, size of nodes represents their degrees and edges are weighted to represent the strength of connections. S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 719 Figure 4: Network visualization of use of digital health Source: Own Figure 5: Conceptual Framework Source: Own 720 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The edge data of the network, was saved in a table, including source, target, and weight. This data frame, which can be found in Appendix 2, provides detailed insights into the connections between different nodes in the network, and was applied to categorise the “use of digital healthcare” most co-occurrent key-terms, to the appropriate sections of the conceptual framework according to their connection to each concept. The conceptual framework is shown in Figure 5. 3 Findings The findings of this study provide insights into the prevalent research areas and terms associated with the use of digital healthcare in IS. The key-terms in the context of “use of digital healthcare” in the IS field, have been presented as a density visualisation of co-occurring key terms in Figure 6 and components of the framework in Figure 5. These results, linked with the first study objective, indicate that the dominant studied concepts include utility, usability, and user-centric design. The most studied key-terms, related to the use of digital healthcare, which are applied by researchers in IS, are usability, technology acceptance, technology adoption, human-computer interaction, and user experience, as wel as privacy and security. These key-terms can be seen in Figure 6. Figure 6: Density visualisation of author keywords co-occurrence Source: Own S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 721 In order to investigate the use of digital healthcare, most of the IS studies have initially dealt with “usability”, and then “utility”, as demonstrated in Figure 4. Articles in the IS field that have addressed these concepts, focused primarily on assessing various aspects of health technology, including user experience, accessibility, technology adoption, reliability, or technology acceptance (perceived ease of use, perceived usefulness). Additional y, further frequently used key terms include security, privacy, and ethics. While they are more recent in their evolution (as shown in Figure 3), these terms have surpassed usability and utility in terms of frequency in recent years as wel as in cumulative numbers. This demonstrates the increased significance and focus placed on these ethical considerations in the past few years as further supported by Figure 7. The concept of "user-centered" care, which emerged as a newer and less prominent concept, was expected to encompass key terms like user participation, user-centered care, and human factors. However, these terms were found to have lower frequency and co-occurrence in the included articles, as wel as not being sufficiently associated with other nodes in the network. As a result, they did not appear in the network graph shown in figure 4. Figure 7: Overlay visualisation of keyword co-occurrence Source: Own 722 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The visualisation results for keyword co-occurrence, displayed in Figures 3 and 6, reveal that past studies mostly concentrated on three main concepts: utility, usability, and user-centric design. More recent research has emphasised the significance of privacy and security, sustainability, healthcare disparities, digital divide, digital literacy, and health literacy. Regarding the second research objective, analysing the evolution and impact of research contributions on these concepts over time, as depicted in Figure 3, it is evident that prior to 2000, the concept of “utility” was a dominant topic and received significant focus from IS researchers. Subsequently, it experienced a period of neglect before regaining attention and recognition (measured by citations), particularly leading up to 2020. Study on the concept of “usability” has accelerated after 2000 and has gradual y become the trendiest research concept related to use of digital healthcare in the field of IS. The notion of “user-centred design” in this field is a newer concept and has not yet gained as much prominence. The concept of "sustainability" is a new trend in this field of study and is gradual y gaining increased attention from academics. Although the notion of consideration, has gained more attention and recognition in recent years, it has emerged as the most current trending concept in the field. Figure 4 and the Appendix 2, effectively addressed the requirement to map out the evolution of key-terms based on their interconnections, as outlined in the third study’s objective. The results show that ethical “considerations” have strong connections with dominant concepts of “usability” and “utility”. It indicates the importance of these considerations in academic research through the use of main concepts of digital healthcare. The two main concepts of “usability” and “utility”, have almost similar edges weight, when it comes to the “sustainability” concept in use of digital healthcare studies. Because the concept of “user-centered design” is newer and less prevalent in this domain, there may be a reason to weaker interconnection of this concept with ethical “considerations” and “sustainability”. These two concepts seem to have a contextual impact in the field, and can be positioned as contextual section in framework, because of having significant connections with two of three core concepts. S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 723 4 Discussion and Conclusions The basic objective of digital health, as highlighted by Yao et al. (2022), Saukkonen et al. (2022), Adjekum et al. (2018), and Gopal et al. (2019), is to enhance health outcomes and elevate quality of life. The simultaneous occurrence of these words in key-terms and in definition of digital healthcare, could demonstrate their importance in the research terminology of this field. Although these key-terms, "health outcomes" and "quality of life", appeared at the first stage of this data analysis as some of the most co-occurring terms in this field, but they were less prevalent in IS research and require further investigation. Kruse et al. (2017) expressed concerns over the security of health information systems that store sensitive patient information and diagnostic data, pointing out that technological advancements are raising the dangers of threatening the privacy and security of these systems. The ethical concerns, raised by Kruse et al. (2017), Gardiyawasam Pussewalage and Oleshchuk (2016), and Gopal et al. (2019) regarding the use of digital healthcare have been recognised by IS researchers as wel . Recent trends on this topic, as revealed by the findings, support this. In order to establish confidence and guarantee the protection of sensitive patient data, it is essential that digital health solutions prioritise security and privacy. Yao et al. (2022) highlighted the significance of user- centered care in digital healthcare research. However, the patterns revealed a lack of attention and recognition of this concept in IS research. This trend may be attributed to the novelty of this concept or the transition of focus to a more recent concept. Disparities in digital health technologies reveal unequal access to the use of healthcare, resulting in different health outcomes. Age, eHealth literacy, and geographic location can also impact health disparities across different groups (Yao et al. 2022). These terms have lower frequency and co-occurrence in the findings, and do not have sufficient interconnection with other nodes to appear in the network graph. Terms that meet these criteria, along with similar terms, were manual y reviewed and categorised in sections of the framework, indicated by a fader colour. This fading indicates that they received less attention in IS studies and can be considered more in future studies. The most important output of this study is its conceptual framework, that serves as a foundation for further exploration in the use of digital healthcare, al owing for a more shared understanding and agreed-upon terminology, as well as identification of research, highlighting research gaps such as unequal access, ethical challenges, and 724 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION evolving trends in technology adoption. Moving forward, it will be vital to analyse the concepts of the included articles to deepen understanding and create frameworks that align with practical and theoretical development in digital health usage. Providing defined terminology and key-points, researchers may better design studies that contribute to the sustained development of digital healthcare in society. It is important to mention some of the limitations of the study. Thematic analysis, a qualitative analysis method, is accurate yet susceptible to the researcher's interpretation bias, which may impact the classification and comprehension of key- terms and concepts. Filtering the documents by English language, article type, and computer science area (Information Systems), might result in to missing significant data. Acknowledgements This research was supported by the grant from Åbo Akademi University’s Fabian Klingendahl fond. References Adjekum, Afua, Alessandro Blasimme, and Effy Vayena. 2018. ‘Elements of Trust in Digital Health Systems: Scoping Review’. Journal of Medical Internet Research 20 (12): e11254. https://doi.org/10.2196/11254. Demir, G., P. Chatterjee, and D. Pamucar. 2024. ‘Sensitivity Analysis in Multi-Criteria Decision Making: A State-of-the-Art Research Perspective Using Bibliometric Analysis’. Expert Systems with Applications 237. https://doi.org/10.1016/j.eswa.2023.121660. Donthu, Naveen, Satish Kumar, Debmalya Mukherjee, Nitesh Pandey, and Weng Marc Lim. 2021. ‘How to Conduct a Bibliometric Analysis: An Overview and Guidelines’. Journal of Business Research 133 (September): 285–96. https://doi.org/10.1016/j.jbusres.2021.04.070. Eck, Nees Jan van, and Ludo Waltman. 2010. ‘Software Survey: VOSviewer, a Computer Program for Bibliometric Mapping’. Scientometrics 84 (2): 523–38. https://doi.org/10.1007/s11192- 009-0146-3. Gardiyawasam Pussewalage, Harsha S., and Vladimir A. Oleshchuk. 2016. ‘Privacy Preserving Mechanisms for Enforcing Security and Privacy Requirements in E-Health Solutions’. International Journal of Information Management 36 (6, Part B): 1161–73. https://doi.org/10.1016/j.ijinfomgt.2016.07.006. Garg, Shivank, Noelle L. Williams, Andrew Ip, and Adam P. Dicker. 2018. ‘Clinical Integration of Digital Solutions in Health Care: An Overview of the Current Landscape of Digital Technologies in Cancer Care’. JCO Clinical Cancer Informatics, June. https://doi.org/10.1200/CCI.17.00159. Gopal, Gayatri, Clemens Suter-Crazzolara, Luca Toldo, and Werner Eberhardt. 2019. ‘Digital Transformation in Healthcare – Architectures of Present and Future Information Technologies’. Clinical Chemistry and Laboratory Medicine (CCLM) 57 (3): 328–35. https://doi.org/10.1515/cclm-2018-0658. S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 725 Hol is, Chris, Richard Morriss, Jennifer Martin, Sarah Amani, Rebecca Cotton, Mike Denis, and Shôn Lewis. 2015. ‘Technological Innovations in Mental Healthcare: Harnessing the Digital Revolution’. The British Journal of Psychiatry 206 (4): 263–65. https://doi.org/10.1192/bjp.bp.113.142612. Kiger, Michelle E., and Lara Varpio. 2020. ‘Thematic Analysis of Qualitative Data: AMEE Guide No. 131: Medical Teacher’. Medical Teacher 42 (8): 846–54. https://doi.org/10.1080/0142159X.2020.1755030. Kostkova, Patty. 2015. ‘Grand Challenges in Digital Health’. Frontiers in Public Health 3 (May). https://doi.org/10.3389/fpubh.2015.00134. Kruse, Clemens Scott, Brenna Smith, Hannah Vanderlinden, and Alexandra Nealand. 2017. ‘Security Techniques for the Electronic Health Records’. Journal of Medical Systems 41 (8): 127. https://doi.org/10.1007/s10916-017-0778-4. Lazarides, Miltos K. 2023. ‘Bibliometric Analysis: Bridging Informatics With Science’. International Journal of Lower Extremity Wounds, 15347346231153538–15347346231153538. https://doi.org/10.1177/15347346231153538. Lewis, James R. 2014. ‘Usability: Lessons Learned … and Yet to Be Learned’ 30: 663–84. https://doi.org/10.1080/10447318.2014.930311. Li, Yuhua, and Wei Zhou. 2021. ‘Knowledge Mapping of Global Value Chain: A Visual Analysis Using VOSviewer and CiteSpace’. In 2021 International Conference on Management Science and Software Engineering (ICMSSE), 77–85. https://doi.org/10.1109/ICMSSE53595.2021.00025. Lockl, Jannik, Doreen Schick, Jens-Christian Stoetzer, and Katrin Huff. 2022. ‘A Model to Assess the Impact of Digital Technologies on the Health-Related Quality of Life’. International Journal of Technology Assessment in Health Care 38 (1): e81. https://doi.org/10.1017/S0266462322003245. Nielsen, Jakob. 1993. ‘Chapter 2 - What Is Usability?’ In Usability Engineering, edited by Jakob Nielsen, 23–48. San Diego: Morgan Kaufmann. https://doi.org/10.1016/B978-0-08-052029-2.50005-X. Overhage, J. Marc. 2003. ‘Understanding and Implementing Computerized Physician Order Entry’. In Healthcare Information Systems, 2nd ed. Best Practices. Boca Raton, FL: Auerbach Publications. https://doi.org/10.1201/9781420031409. Saukkonen, Petra, Marko Elovainio, Lotta Virtanen, Anu-Marja Kaihlanen, Janna Nadav, Tinja Lääveri, Jukka Vänskä, Johanna Vi tanen, Jarmo Reponen, and Tarja Heponiemi. 2022. ‘The Interplay of Work, Digital Health Usage, and the Perceived Effects of Digitalization on Physicians’ Work: Network Analysis Approach’. Journal of Medical Internet Research 24 (8): e38714. https://doi.org/10.2196/38714. Thompson, Jamie. 2022. ‘A Guide to Abductive Thematic Analysis’. The Qualitative Report 27 (5): 1410–21. https://doi.org/10.46743/2160-3715/2022.5340. Yao, Rui, Wenli Zhang, Richard Evans, Guang Cao, Tianqi Rui, and Lining Shen. 2022. ‘Inequities in Health Care Services Caused by the Adoption of Digital Health Technologies: Scoping Review’. Journal of Medical Internet Research 24 (3): e34144. https://doi.org/10.2196/34144. 726 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix 1: List of key-terms No. Key-term 1 Human factors 2 Human-computer interaction 3 User centered care 4 User participation 5 Perceived ease of use 6 Perceived usefulness 7 Reliability 8 Success 9 Usability 10 Usability assessment 11 User experience 12 User satisfaction 13 Interoperability 14 Intention to use 15 Efficiency 16 Health behaviour change 17 Health outcomes 18 Medication adherence 19 Quality of life 20 Technology acceptance 21 Technology adoption 22 User empowerment 23 User engagement 24 Utility 25 Adaptation 26 Ethics 27 Privacy 28 Security 29 Trust 30 Accessibility 31 Digital divide 32 Digital health literacy 33 Health literacy 34 Healthcare disparities 35 Sustainability S. Arzideh, J. Mezei: Mapping the Landscape of Digital Health Usage in Information Systems Research 727 Appendix 2: Network edge data table Source Target Weight user experience user-centred 22 technology acceptance user-centred 17 user satisfaction user-centred 8 perceived usefulness user-centred 6 technology adoption user-centred 4 reliability user-centred 4 trust user-centred 3 privacy user-centred 3 user-centred accessibility 2 user-centred adaptation 2 medication adherence user-centred 1 human-computer interaction user-centred 1 ethics user-centred 1 user-centred healthcare disparities 1 usability user experience 172 usability security 143 usability user satisfaction 96 technology acceptance usability 88 usability accessibility 64 usability technology adoption 52 usability medication adherence 52 usability privacy 50 usability reliability 48 perceived usefulness usability 35 usability ethics 23 usability trust 13 usability adaptation 11 usability human-computer interaction 9 usability healthcare disparities 7 utility privacy 90 utility security 49 utility user experience 42 utility technology acceptance 37 utility accessibility 21 utility technology adoption 19 utility user satisfaction 14 utility trust 13 utility ethics 10 utility perceived usefulness 9 utility medication adherence 8 utility reliability 6 utility healthcare disparities 5 728 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Source Target Weight utility adaptation 3 sustainability trust 30 security sustainability 27 technology acceptance sustainability 23 technology adoption sustainability 22 user experience sustainability 20 sustainability adaptation 6 perceived usefulness sustainability 5 sustainability privacy 5 sustainability ethics 5 reliability sustainability 4 user satisfaction sustainability 3 sustainability medication adherence 3 sustainability accessibility 3 sustainability healthcare disparities 2 security considerations 216 considerations privacy 172 considerations ethics 75 technology acceptance considerations 31 user experience considerations 28 technology adoption considerations 17 considerations trust 17 considerations accessibility 16 reliability considerations 8 considerations human-computer interaction 3 considerations adaptation 2 perceived usefulness considerations 1 user satisfaction considerations 1 considerations medication adherence 1 PREPARING FOR DISEASE X: PREDICTING ICU ADMISSIONS USING TIME SERIES FORECASTING WITH DECODER-ONLY TRANSFORMER NEURAL NETWORKS NEJC ČELIK, ANDREJ ŠKRABA University of Maribor, Faculty of Organizational Sciences, Cybernetics & Decision Support Systems Laboratory, Kranj, Slovenia nejc.celik1@um.si, andrej.skraba@um.si The COVID-19 pandemic has underscored the critical importance of predictive model ing in managing healthcare resources and shaping public health policies. This paper explores the application of advanced Artificial Intelligence (AI) techniques, specifical y decoder-only transformer neural networks (DOTNN), in forecasting weekly Intensive Care Unit (ICU) admissions. Our research is driven by the necessity to enhance preparedness for potential future pandemics, referred to as "Disease X", by leveraging large datasets of publicly available information. A prediction model has been developed that incorporates several key indicators, such as new cases, ICU admissions, and testing rates. Our DOTNN architecture, Keywords: inspired by the Generative Pre-trained Transformer (GPT), ICU admission focuses on time series forecasting without the necessity for predictions, encoder components, thereby streamlining the prediction decoder-only process. Despite limited data availability, the proposed method transformer neural can achieve notable accuracy, with Mean Absolute Percentage networks, Error (MAPE) values below 15% for a significant number of disease X, predictions. This performance highlights the potential of pandemic preparedness, DOTNNs in forecasting ICU admissions, which is crucial for COVID-19 healthcare planning and resource al ocation during pandemics. forecasting DOI https://doi.org/10.18690/um.fov.4.2024.43 ISBN 978-961-286-871-0 730 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The COVID-19 pandemic spurred a wide range of model ing efforts to predict the spread of the virus, inform public health policies, and manage healthcare resources. While numerous methodologies were employed, many of these efforts were indeed grounded in classical epidemiological models like the SIR (Susceptible-Infected- Removed), SEIR (Susceptible-Exposed-Infected-Removed), SEIUR (Susceptible- Exposed-Infected-Uncertain-Removed), and agent-based models. These models have a long history in epidemiology for their utility in understanding the dynamics of infectious diseases. In contrast, advanced Artificial Intel igence (AI) methods constituted a smal er proportion of the research landscape during the initial stages of the pandemic (Adam, 2020; Kucharski et al., 2020). In the aftermath of the COVID-19 era, one might question the need to continue refining pandemic prediction methods. However, it's essential to consider the concept of “Disease X”—a potential unknown threat that underscores the critical importance of preparing for yet-undiscovered pathogens capable of sparking future pandemics. Proactive stance in research of prediction methods for pandemics can markedly improve prepares for future pandemics. Such forward-thinking strategies are vital for saving lives, preventing the recurrence of past errors, and ensuring an effective response to novel infectious threats. (Banerjee et al., 2023; Prioritizing Diseases for Research and Development in Emergency Contexts, n.d.; What Is Disease X | Johns Hopkins | Bloomberg School of Public Health, n.d.) Our study aims to apply cutting-edge AI methodologies to a vast dataset for the purpose of forecasting ICU admissions. Rather than refining prediction algorithms, we seek to evaluate the efficacy of employing transformer neural networks for this task. Specifical y, we are interested in predicting weekly ICU patient counts based on historical data, mirroring real-world scenarios where daily infection rates are recorded. This predictive capability holds promise for optimizing resource allocation and preparedness in managing COVID-19 patients. While existing research predominantly focuses on predicting ICU requirements for confirmed cases (Lorenzen et al., 2021) (Subudhi et al., 2021) (Chadaga et al., 2024) (Dipaola et al., 2023), our methodology adopts a novel approach, utilizing time series forecasting with readily accessible data and a customized transformer architecture. However, our analysis is constrained by the limited availability of comprehensive weekly ICU N. Čelik, A. Škraba: Preparing for Disease X: Predicting ICU Admissions Using Time Series Forecasting with Decoder-Only Transformer Neural Networks 731 admission data from fewer than 20 countries, highlighting the necessity for enhanced data acquisition protocols, particularly in anticipation of future outbreaks. Despite potential chal enges arising from data scarcity, precise forecasting of ICU occupancy remains pivotal for ensuring sufficient capacity to accommodate all critically ill patients. Development of Covid-19 models or “digital twins” should incorporate the real time sensing. In our early efforts in this regard, we have developed the hardware interface (Stojanovic et al., 2020) to monitor Covid-19 patients and provide the input to the predictive simulation models. In the next stage of our previous attempts to simulate the Covid-19 spread we have applied Bass diffusion model (Škraba et al., 2021). Agent-based approach has also been applied (Škraba & Vavtar, 2022) addressing the spatial distribution of the infected population. In order to improve the accuracy of the predictions the set of models was used: SI, Bass diffusion, SIR, SEIR and SEIUR (Stanovov et al., 2022) Parametrization was performed on paral el computer stack with the Differential Evolution (DE) methods. Combination of standard models with DE enabled us to confirm the hypothesis of latent spread mechanisms of Covid-19 which are important for the epidemics prediction. In present paper we strive to extend the predictability accuracy with the transformer neural network model. 2 Methodology 2.1 Data 2.1 To develop our prediction model, we used publicly available dataset on COVID-19 infections ( Data on COVID-19 (Coronavirus) by Our World in Data, n.d.) which includes data of number of confirmed cases, tests, intensive care units (ICU) admissions, number of hospitalizations and more per country. Data for number of weekly hospitalizations and intensive care patients for at least 102 weeks is only available for 17 countries. For input data of our model, we used some of the data from dataset (new_cases_smoothed_per_million, weekly_icu_admissions_per_million, date (as number YYMMDD), aged_65_older (fixed per country), aged_70_older (fixed per country), gdp_per_capita (fixed per country), cardiovasc_death_rate (fixed per country), diabetes_prevalence (fixed per country), female_smokers (fixed per country), male_smokers (fixed per country), population (in mil ions) (fixed per 732 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION country), weekly_hosp_admissions_per_million (updated weekly), population_density (fixed per country), median_age (fixed per country), life_expectancy (fixed per country), human_development_index (fixed per country), new_deaths_smoothed_per_mil ion(updated weekly), new_tests_per_thousand (updated weekly)). Figure 1 shows five time series per country that were used in the training phase. Here the timeseries for Chile (CHL) are shown as an example in order to il ustrate the complexity of addressed task. The graph includes the following time series data: • new cases, smoothed per mil ion people, which exhibits several peaks that correspond to waves of infections over time. • weekly intensive care unit (ICU) admissions per million people, which shows notable spikes that are usual y correlated with the waves of new cases. • new deaths, smoothed per mil ion people, again showing peaks which typical y fol ow the trend in new cases with a certain lag. • weekly hospital admissions per mil ion people, closely following the trends of the ICU admissions, but typically at higher levels, suggesting that not all hospital admissions require ICU care. • new tests conducted per thousand people which has visible drops on weekends, highlighting the consistency of testing throughout the pandemic except for lower testing rates on weekends. To represent the dynamic character of the input data, the timeseries were normalized on the interval [0, 1] (y-axis), which makes it easier to compare the different scales of data. The normalization process transformed the raw data so that each metric fits within the same range for comparative purposes. The "waves" of the pandemic are evident from the recurring peaks in new cases, hospitalizations, ICU admissions, and deaths, with testing showing periodicity Figure 1 shows the time series for only one of the seventeen countries which were used to train the neural network models. N. Čelik, A. Škraba: Preparing for Disease X: Predicting ICU Admissions Using Time Series Forecasting with Decoder-Only Transformer Neural Networks 733 Figure 1: Normalized time series that were used as the input for Chile (CHL) Source: Own 2.1.1 Model 2.1.1 Our architecture leverages a decoder-only transformer neural network (DOTNN) model inspired by GPT (Generative Pre-trained Transformer) as detailed by Radford et al. (2018). This design choice is a deviation from the original transformer architecture proposed by Vaswani et al. (2017), focusing on sequence generation task without the need for an encoder. This decision stems from the nature of our forecasting task, which primarily involves generating future values from past data rather than transforming one sequence into another. Following the methodology outlined by Wen et al. (2022) and Zhou et al. (2020), we implement zero-mean normalization on both model input and target data. . During inference, we apply the inverse process (de-normalization) using the training data's mean and variance to transform model outputs back to their original scale, ensuring that our predictions are interpretable and comparable to real-world values. In our 734 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION model we employ learned positional embeddings (Wen et al., 2022; Zhou et al., 2020). In our model before decoder blocks, we expand inputs dimensionality to 512 and combine it with positional embeddings, each decoder block is equipped with two masked multi-head attention layers featuring four attention heads. The attention mechanism is complemented by a feed-forward network with 512 neurons. As we started predictions after 20 weeks context length of 20 weeks was chosen and was then kept through all subsequent weeks. We have also developed a smaller model with only one decoder block and only one attention head to expedite the experimental phase since model training takes considerable computer time. For training, we utilized a cosine learning rate decay strategy (Loshchilov & Hutter, 2016), starting with an initial rate of 3e-4 and gradually reducing it to 1e-4 across the epochs. Coupled with the AdamW optimizer, which introduces a decay component to the weights (Kingma & Ba, 2014) (Loshchilov & Hutter, 2017) All examples were feed into network in a single batch (1 step per epoch). Figure 2 illustrates our network architecture, adapted from Vaswani et al. (2017), detailing the composition and workflow of our model. Our experimental setup involves training the model on a dataset covering 20 weeks of data, aiming to predict ICU patient numbers in the subsequent week. This process is iteratively repeated, incorporating the data from the newly predicted week into the training set for subsequent predictions. This iterative retraining strategy simulates a real-world application where the model adapts to new information over time, aiming to provide accurate forecasts. During development we used up to 102 weeks of data. To accommodate the growing dataset, we proportional y increase the number of training epochs based on the added examples, opting for a pragmatic approach over hyperparameter optimization for each iteration. N. Čelik, A. Škraba: Preparing for Disease X: Predicting ICU Admissions Using Time Series Forecasting with Decoder-Only Transformer Neural Networks 735 Figure 2: Network architecture Source: adapted from (Vaswani et al., 2017) As the measure to determine the accuracy of a model's predictions the Mean Absolute Percentage Error (MAPE) (Oliva & Oliva, 1995; Sterman, 2000) was used, defined as: 1 𝑛𝑛 𝐶𝐶� 𝑀𝑀𝐴𝐴𝑀𝑀𝑀𝑀 = 𝑖𝑖 − 𝐶𝐶𝑖𝑖 𝐶𝐶 � � 𝐶𝐶 � × 100% 𝑖𝑖 𝑖𝑖=1 736 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION where 𝐶𝐶 is the number of observations, 𝐶𝐶�𝑖𝑖 is the predicted value and 𝐶𝐶𝑖𝑖 is the actual value. While, due to the absolute value, the term 𝐶𝐶�𝑖𝑖 − 𝐶𝐶𝑖𝑖 is sometimes swapped which might contribute to the confusion, the proper order as is written enables us to understand the meaning of MAPE. There is also plethora of other possibilities to determine the accuracy of the predictions (Batagelj & Bren, 1995) however, we have used MAPE due to its intuitive interpretability, as it directly expresses error as a percentage of the actual values. 3 Results With proposed method of using DOTNN for predicting the number of ICU patients the MAPE values for 6 out of the 17 countries were below 15% which might be considered good in terms of ICU prediction. Observing the results we saw that in some countries weekly ICU admissions were logged daily for those countries our model made daily predictions (Table 1, marked with asterisk *) and performed better in general. We have tested our method with two different model sizes. While the larger model exhibited a superior average MAPE, the smal er model demonstrated better performance in countries where the MAPE was below 15%. Bolded values in Table 1 indicates better, i.e. lower MAPE values indicating better model. The same goes for the MSE (mean squared error) while for the r2 (Wright, 1921) higher values are better. Asterisk * marks the countries where daily data was used for training. In examining the accuracy of predictions, t-test was conducted to compare the mean accuracy of predictions with a MAPE below 25% (M = 8.70, SD = 3.09) against those with a MAPE above 25% (M = 87.81, SD = 58.45). Results indicated a statistically significant difference between the two groups, t(11) = -4.68, p < .001, two-tailed. The significant statistical difference in prediction accuracy, as evidenced by the t-test comparing MAPE values below 25% to those above this threshold, highlights a potential for enhancing the prediction method as wel as good accuracy in the <25% group. N. Čelik, A. Škraba: Preparing for Disease X: Predicting ICU Admissions Using Time Series Forecasting with Decoder-Only Transformer Neural Networks 737 Table 1: Prediction scores MAPE MAPE ISO CODE (bigger MSE r2 (smaller MSE r2 model) model) CHL* 7.36 8.31 0.90 4.69 5.56 0.93 CYP 64.78 31.69 0.60 113.31 38.11 0.55 CZE* 112.03 0.36 0.84 152.98 0.25 0.88 EST 52.51 45.32 0.70 112.88 52.16 0.53 FRA* 11.44 0.51 0.99 13.00 0.52 0.99 DEU* 12.77 2.03 0.82 5.50 0.32 0.98 GRC 34.36 16.23 0.85 30.28 17.38 0.86 IRL 137.52 19.35 0.35 230.85 16.72 0.12 ISR* 244.66 1.21 0.84 191.91 0.72 0.90 ITA* 7.59 4.22 0.64 4.97 2.10 0.86 LVA 50.42 380.02 0.73 49.15 697.01 0.43 LUX 75.45 46.59 0.59 84.28 57.49 0.60 NLD* 8.86 2.40 0.80 7.23 1.80 0.85 NOR 79.01 1.94 0.77 246.66 2.81 0.72 SVK 105.28 13.17 0.81 163.89 14.27 0.82 SVN 36.30 73.48 0.81 53.57 93.41 0.76 ESP* 4.16 1.21 0.98 4.84 1.41 0.98 AVERAGE 61.44 38.12 0.77 86.47 58.94 0.75 Figure 3 represents the overal best forecasting accuracy for ICU admissions which is for Spain (ESP) over a span of 102 days, using a daily scale. The blue line signifies the actual reported ICU admissions per million individuals, while the red arrows illustrate our model's daily predictions. The accurate alignment between the blue line and the red arrows demonstrates the high precision of the predictions, with the MAPE being exceptional y low at 4.16%, indicating a close correspondence to the real data. The model's capacity to closely follow the trend lines throughout the entire 738 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION time highlights its robustness and the potential for reliable future forecasting in similar scenarios. Figure 3: Best performing country (ESP bigger model) Source: Own Figure 4 presents a comparative analysis of ICU admission predictions for the three countries where both the smal er model (left column) and the bigger, more accurate model (right column) performed the least accurately. Each row represents one country, with Israel (ISR), Norway (NOR), and Ireland (IRL) from top to bottom. The time series data spans 102 days/weeks and shows weekly ICU admissions per million. The blue lines indicate the actual data for ICU admissions, while the red lines depict the predictions made by the respective model. Figure 4 reveals the limitations and challenges faced by the models, particularly in the countries where predictions did not align as closely with the actual data. Some of the error can be explained by extreme errors when predictions are close to 0 which is known disadvantage of MAPE. We observed the smal er model does not adequately account for al the information contained in the data. As the model gets larger, the predictions in our case improved on average. Model might be also missing some key data for making best predictions as for example amount and strictness of COVID measures and restrictions and country’s location and size. N. Čelik, A. Škraba: Preparing for Disease X: Predicting ICU Admissions Using Time Series Forecasting with Decoder-Only Transformer Neural Networks 739 Figure 4: worst performing countries (left smal er model, right bigger model) Source: Own To explore relationship between amount of data and prediction accuracy we selected the smal er set of countries to compare the results between smal er and larger data set. In the left part of table 2 (smal er model separate data), we have only one country in the training set and not al of seventeen selected countries. The set of input data is correspondingly smal er in this part. The right side of the table (smal er model al data) shows the results with al countries in the training set. 740 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 2: Results for smal er (left) vs bigger (right) training set ISO CODE MAPE MSE r2 MAPE MSE r2 GRC 48.54 24.04 0.80 30.28 17.38 0.86 NOR 74.83 5.02 0.40 246.66 2.81 0.72 SVK 77.70 16.90 0.77 163.89 14.27 0.82 SVN 327.03 81.97 0.82 53.57 93.41 0.76 ITA 3.15 0.75 0.96 4.97 2.10 0.86 In the case of smoother data input, where there was smal volatility in data the model trained on less data turns out to be better. With harder to predict countries we observed better performance if we included al countries in the training dataset. Considering better MSE and r2 scores for Norway and Slovakia predictions one could hypothesise that better MAPE might be explained by lower number of extreme errors when predictions are near zero. Figure 5 ~ Left shows predictions for Greece when trained with only Greece’s data and Figure 5 ~ Right shows predictions when trained on al data. Both cases used smal er model for prediction. Figure 5 Left ~ GRC trained only on GRC data | Right ~ GRC trained on al data (smal er model) Comparatively, the data in Figure 5 ~ Right may represent a more stable and possibly more accurate model for predicting ICU admissions, incorporating a diverse set of training data. However, Figure 5 ~ Left, with its higher resolution of Greece-specific fluctuations, might be more sensitive to local variations and potentially overfit to N. Čelik, A. Škraba: Preparing for Disease X: Predicting ICU Admissions Using Time Series Forecasting with Decoder-Only Transformer Neural Networks 741 Greece's patterns. With better performance observed with more data and a bigger model, gathering detailed data for regions or municipalities could potentially increase accuracy of predictions while adding additional value for planning. Pretraining the model on different diseases is also worth exploring. Main possible advantage of proposed method of predicting number of ICU admissions is the ability to put any data as an input and then let the model to learn if the data is valuable to making accurate predictions or if it should be ignored. Additionally, adding additional inputs to the model does not increase the complexity of model ing. Previous methods such us Google Flu Trends (Dugas et al., 2013)and Skraba (Škraba & Vavtar, 2022) were based on rather different principles as one was monitoring user searches and behaviours in Google search tools and the other was predicting epidemic dynamics for a single wave of pandemic. Our methodology could also provide ability to easily include predictions of other methods as inputs with zero additional model ing complexity increase besides gathering the data. 4 Conclusion We've shown that the DOTNN method could offer high accuracy in predicting ICU admissions numbers, though occasional higher prediction errors can occur. Future research should focus on data preparation and identifying factors influencing inaccurate predictions. Despite some discrepancies, many predictions were satisfactory, with MAPE below 15%. Our aim wasn't to dive into prediction accuracy but to explore a novel method, unused in the context of COVID-19. This approach prepares for potential future pandemics, like "Disease X," where AI methods could be pivotal. While our methodology shows promise, its usability remains unclear due to data limitations. We observed better accuracy predicting ICU admissions a day ahead compared to a week ahead which indicates that predicting for less than week in advance may be worth exploring. We believe that for future research data preparation is critical, with need for manual inspection and filtering due to potential errors. Adapting these methods for real-time processing is vital, particularly for future pandemics. Despite chal enges in model ing pandemic waves, our method shows potential, although comparing it with others is difficult due to data complexity. 742 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Acknowledgements The authors acknowledge the financial support from the Slovenian Research and Innovation Agency (research core funding No. P5‑0018) and the Ministry of Higher Education, Science, and Innovation of the Republic of Slovenia as part of the Next Generation EU National Recovery and Resilience Plan (Grants No. 3330‑22‑3515; NOO No: C3330‑22‑953012). References Adam, D. (2020). Special report: The simulations driving the world’s response to COVID-19. Nature, 580(7803), 316–318. https://doi.org/10.1038/D41586-020-01003-6 Banerjee, S., Sultana, Q., Mukherjee, D., Agrawal, V., Jaiswal, V., & Paudel, K. (2023). Disease‐X: Accounting for the unknown. Health Science Reports, 6(4). https://doi.org/10.1002/HSR2.1173 Batagelj, V., & Bren, M. (1995). Comparing resemblance measures. Journal of Classification, 12(1), 73–90. https://doi.org/10.1007/BF01202268/METRICS Chadaga, K., Prabhu, S., Sampathila, N., Chadaga, R., Umakanth, S., Bhat, D., & G S, S. K. (2024). Explainable artificial intelligence approaches for COVID-19 prognosis prediction using clinical markers. Scientific Reports, 14(1). https://doi.org/10.1038/S41598-024-52428-2 Data on COVID-19 (coronavirus) by Our World in Data. (n.d.). Retrieved February 24, 2024, from https://github.com/owid/covid-19-data/tree/master/public/data Dipaola, F., Gatti, M., Giaj Levra, A., Menè, R., Shiffer, D., Faccincani, R., Raouf, Z., Secchi, A., Rovere Querini, P., Voza, A., Badalamenti, S., Solbiati, M., Costantino, G., Savevski, V., & Furlan, R. (2023). Multimodal deep learning for COVID-19 prognosis prediction in the emergency department: a bi-centric study. Scientific Reports 2023 13:1, 13(1), 1–10. https://doi.org/10.1038/s41598-023-37512-3 Dugas, A. F., Jalalpour, M., Gel, Y., Levin, S., Torcaso, F., Igusa, T., & Rothman, R. E. (2013). Influenza forecasting with Google Flu Trends. PloS One, 8(2). https://doi.org/10.1371/JOURNAL.PONE.0056176 Kingma, D. P., & Ba, J. L. (2014). Adam: A Method for Stochastic Optimization. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings. https://arxiv.org/abs/1412.6980v9 Kucharski, A. J., Russel , T. W., Diamond, C., Liu, Y., Edmunds, J., Funk, S., Eggo, R. M., Sun, F., Jit, M., Munday, J. D., Davies, N., Gimma, A., van Zandvoort, K., Gibbs, H., Hel ewel , J., Jarvis, C. I., Clifford, S., Quilty, B. J., Bosse, N. I., … Flasche, S. (2020). Early dynamics of transmission and control of COVID-19: a mathematical model ing study. The Lancet. Infectious Diseases, 20(5), 553–558. https://doi.org/10.1016/S1473-3099(20)30144-4 Lorenzen, S. S., Nielsen, M., Jimenez-Solem, E., Petersen, T. S., Perner, A., Thorsen-Meyer, H. C., Igel, C., & Sillesen, M. (2021). Using machine learning for predicting intensive care unit resource use during the COVID-19 pandemic in Denmark. Scientific Reports 2021 11:1, 11(1), 1–10. https://doi.org/10.1038/s41598-021-98617-1 Loshchilov, I., & Hutter, F. (2016). SGDR: Stochastic Gradient Descent with Warm Restarts. 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings. https://arxiv.org/abs/1608.03983v5 Loshchilov, I., & Hutter, F. (2017). Decoupled Weight Decay Regularization. 7th International Conference on Learning Representations, ICLR 2019. https://arxiv.org/abs/1711.05101v3 Oliva, R., & Oliva, © R. (1995). A Vensim® Module to Calculate Summary Statistics for Historical Fit D-4584 D-4584. http://metasd.com/wp-content/uploads/2010/03/D4584theil.pdf Prioritizing diseases for research and development in emergency contexts. (n.d.). Retrieved March 14, 2024, from https://www.who.int/activities/prioritizing-diseases-for-research-and-development-in-emergency-contexts?gad_source=1 N. Čelik, A. Škraba: Preparing for Disease X: Predicting ICU Admissions Using Time Series Forecasting with Decoder-Only Transformer Neural Networks 743 Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training. https://gluebenchmark.com/leaderboard Škraba, A., & Vavtar, B. (2022). Razvoj simulacijskega modela širjenja epidemije SI po principih agentnega modeliranja in sistemske dinamike. 945–956. https://doi.org/10.18690/UM.FOV.3.2022.69 Škraba, A., Vavtar, B., Stanovov, V., Semenkin, E., & Stojanović, R. (2021). Parametrization of bass diffusion model on COVID-19 first wave data. IOP Conference Series: Materials Science and Engineering, 1047(1), 012084. https://doi.org/10.1088/1757-899X/1047/1/012084 Stanovov, V., Grabljevec, S., Akhmedova, S., Semenkin, E., Stojanović, R., Rozman, Č., & Škraba, A. (2022). Identification of COVID-19 spread mechanisms based on first-wave data, simulation models, and evolutionary algorithms. PloS One, 17(12). https://doi.org/10.1371/JOURNAL.PONE.0279427 Sterman, J. (2000). Business Dynamics, System Thinking and Modeling for a Complex World. Http://Lst-Iiep.Iiep-Unesco.Org/Cgi- Bin/Wwwi32.Exe/[In=epidoc1.in]/?T2000=013598/(100), 19. Stojanovic, R., Skraba, A., & Lutovac, B. (2020). A Headset Like Wearable Device to Track COVID-19 Symptoms. 2020 9th Mediterranean Conference on Embedded Computing, MECO 2020. https://doi.org/10.1109/MECO49872.2020.9134211 Subudhi, S., Verma, A., Patel, A. B., Hardin, C. C., Khandekar, M. J., Lee, H., McEvoy, D., Stylianopoulos, T., Munn, L. L., Dutta, S., & Jain, R. K. (2021). Comparing machine learning algorithms for predicting ICU admission and mortality in COVID-19. NPJ Digital Medicine, 4(1). https://doi.org/10.1038/S41746-021-00456-X Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 2017-December, 5999–6009. https://arxiv.org/abs/1706.03762v7 Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2022). Transformers in Time Series: A Survey. IJCAI International Joint Conference on Artificial Intelligence, 2023-August, 6778–6786. https://doi.org/10.24963/ijcai.2023/759 What is Disease X | Johns Hopkins | Bloomberg School of Public Health. (n.d.). Retrieved March 14, 2024, from https://publichealth.jhu.edu/2024/what-is-disease-x Wright, S. (1921). Correlation and causation. Journal of Agricultural Research, 20(7), 557–585. Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., & Zhang, W. (2020). Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. 35th AAAI Conference on Artificial Intelligence, AAAI 2021, 12B, 11106–11115. https://doi.org/10.1609/aaai.v35i12.17325 744 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION RESEARCH IN PROGRESS STRENGTHENING RESILIENCE IN A DANISH HEALTH SYSTEM – A LONGITUDINAL STUDY JARL TUXEN, MORTEN BOESEN, ERIK LETH MØLLER Copenhagen School of Design & Technology, Copenhagen, Denmark jart@kea.dk, mobo@kea.dk, erlm@kea.dk This study evaluates the enhancement and preservation of resilience initiatives within the Danish health IT platform, sundhed.dk1, used for managing COVID-19 test results and vaccine certifications. Employing a longitudinal approach, the research builds on qualitative interviews with IT personnel directly involved in the system's design and implementation. It assesses which measures implemented during the pandemic Keywords: persisted, which were discarded, and how these strategies resilience, adapted over time. Results categorize enduring resilience- health enhancing strategies into optimization of organisational information systems, structures, refinement of frontend processes, and improvement rapid of backend systems. The research underscores the importance of scaling, re-evaluation over time to establish best practices for managing sensitive data, resilience in public sector IT infrastructure handling sensitive longitudinal citizen data. study 1 Sundhed.dk is both the name of organisation behind the Danish health platform and the name of the system. DOI https://doi.org/10.18690/um.fov.4.2024.44 ISBN 978-961-286-871-0 746 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The focus on resilience in the development, operation, and maintenance of IT systems has recently gained even greater importance. Society at large depends on the existence of a secure, robust, and resilient digital infrastructure. This is particularly crucial in the context of health crises, climate emergencies, and the current geopolitical climate. When considering IT systems, resilience - as defined by Liu et al. (2010) - involves the capability of an information system to maintain its function with minimal service degradation in face of unexpected workloads or disturbances that may lead to partial failures. Furthermore, resilience entails the system's ability to swiftly recover to its normal service level once such unexpected situations cease. In essence, resilience encompasses both the sustained functionality amidst adversity and the subsequent seamless return to normal operations following the resolution of unforeseen disruptions. In a prior study. the scalability of the Danish health platform, responsible for managing test results and vaccine certifications, was examined (Frederiksen et al., 2023). Specifical y, it explored how the platform managed to scale from a few to many users within a mere two weeks. The aim of the present study is to fol ow up and evaluate which of the initiatives, conceived during a crisis, have been sustained, discarded, or potential y expanded upon to strengthening resilience. The research question is: What initiatives were sustained, discarded, or expanded upon by sundhed.dk to strengthening resilience in the Danish IT health platform before, during and after the Covid-19 crisis? 2 Related literature The unprecedented COVID-19 pandemic has necessitated a rapid and effective response from health systems worldwide. A critical aspect of this response has been the scalability and resilience of health platforms. Resilience in public health systems, particularly during crises such as the COVID-19 pandemic, is a subject of considerable interest. Haldane et al. (2021) provide a comprehensive analysis of health systems' resilience in managing the COVID-19 pandemic, drawing lessons from 28 countries. J. Tuxen, M. Boesen, E. Leth Møl er: Strengthening Resilience in a Danish Health System – A Longitudinal Study 747 The architectural perspective of resilience, as discussed by Liu et al. (2010), provides a foundational understanding of how system designs contribute to resilience. Further, Pan et al. (2023) propose critical sub-attributes of resilience used to evaluate software architecture resilience: reliability, restoration, availability, safety, robustness, and rapidity. The role of business capabilities and organisational structures in resilience is explored by Müller et al. (2013), Duchek (2020), and Annarel i & Nonino (2016). They emphasize the importance of strategic and operational management in building resilient systems. Gardner LeGars et al. (2023) introduce frameworks for Information Technology System Resilience and Organisational Resilience. The literature underscores the multifaceted nature of resilience in health systems, particularly in the face of global crises like the COVID-19 pandemic. The Danish health system's rapid scaling serves as a case study in resilience, illustrating the importance of architectural, organisational, and information systems perspectives in managing unforeseen chal enges. 3 Methodology The objective of this study was to investigate the measures implemented by sundhed.dk in response to the chal enges encountered during the phased reopening of society following the COVID-19 pandemic, with a focus on the strategies employed to enhance or preserve the resilience of the information system responsible for managing test results and immunity certification. We argue that resilience is not just about immediate responses, but also about the ability to sustain and develop effective initiatives over time. Therefore, the study adopted a longitudinal approach to examine the strategies that sundhed.dk developed and implemented before, during, and after the COVID-19 pandemic. To investigate the developments before, during, and after the pandemic, the study draws on qualitative interviews with IT architects, operation leads, and developers who participated in developing the system. The selection of interview participants was guided by their level of direct involvement and active engagement in the design, development, and implementation processes. This criterion ensured that the chosen individuals possessed comprehensive and critical insights into the timeline of events, the decision-making rationales, and the strategic responses to the emerging 748 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION chalenges. By focusing on those with first-hand experience, the study aimed to draw upon a depth of knowledge and expertise that would il uminate the complexities and nuances of the system's evolution and resilience during the pandemic. We decided not to interview the users as the primary aim was to evaluate the system’s resilience measures from a technical and organisational standpoint rather than on the user experience or satisfaction. The interviews were conducted with inspiration from the critical incident technique (Chell, 1998; Flanagan, 1954), which aims to reveal the most crucial situations and initiatives that contributed to enhancing or sustaining the system's resilience from the perspectives of the individuals. The longitudinal aspects were empirical y studied using a snapshot approach, with data col ected at two distinct time points. The initial stage of the interviews targeted a significant two- week timeframe, beginning with the Danish Prime Minister's announcement regarding the partial reopening of the country, and ending with the expected date for the system to be completely operational. The initial four interviews were geared towards understanding the extraordinary measures undertaken during this period of heightened urgency. A subsequent interview took place sixteen months later, with the aim of evaluating the enduring impact of those measures and exploring additional steps taken to further reinforce the system's resilience and thus examine which best practices emerged from this extraordinary effort. We analysed the data, aligning it with the themes and initiatives delineated by Frederiksen et al. (2023). Our goal was to elucidate the progression and maturation of these initiatives since the preceding series of interviews. In tandem with this analysis, we adopted an inductive approach to uncover emergent and prospective initiatives, those which have been set into motion or proposed subsequently. 4 Preliminary results Upon analysis, the initiatives identified for enhancing resilience can be categorised into three primary domains: optimising the organisation, refining the frontend processes, and improving the backend systems. These three thematic areas are presented below. We refrain from ful y unfold the details of the before phase as this is already addressed by the preceding study (Frederiksen et al., 2023). Rather, our focus lies in the following phases, examining alterations and continuities of the initiatives applied. J. Tuxen, M. Boesen, E. Leth Møl er: Strengthening Resilience in a Danish Health System – A Longitudinal Study 749 4.1 Optimising the organisation Before the two-week period, sundhed.dk faced a comprehensive process for making changes, requiring approval from multiple internal stakeholders and management layers. This bureaucratic system slowed decision-making significantly. The operation lead provides a clear picture of the situation: “Before, if we should have made such changes, we had to inform and get approval from various internal stakeholders. As a minimum, we had to ask the change advisory board and an architect. And the management should nod and agree that it is a good idea and approve the costs”. However, a pivotal initiative during the two weeks streamlined operations by breaking down barriers between departments. An empowered task force gained authority to interact directly with top management, reducing approval times from days to hours as commented by the operation lead: “So, if we had a good idea before noon, we could implement it the same afternoon”. This change al owed for rapid implementation of ideas, bolstering the organisation's adaptability and resilience. Although the task force disbanded afterward, the organisation retained insights gained from the experience. It initially embraced DevOps principles with operations and development working together in that same department, but later changed to merely integrate a representative from operations into development meetings to maintain collaborative gains. Discussions continue for a broader adoption of DevOps, reflecting sundhed.dk's commitment to building on positive changes made during the transformative period. During the task force, documentation was scarce. After the task force documentation has improved as stated by the IT Architect: “We have become much bet er at writing documentation”. Sundhed.dk is currently looking into automating part of the documentation by using tools like Swagger/OpenAPI. The goal is to ease onboarding of new employees, limit knowledge loss and thus improving organisational resilience. 4.2 Refining the frontend processes As traffic to sundhed.dk's digital services began to swell, the organisation faced a series of network and infrastructure chal enges. The team proactively expanded their incoming and outgoing bandwidth and strengthened their existing setup with additional hardware. The firewall-interface policy was reconfigured to support the influx of users. However, a noticeable imbalance in traffic volume became apparent. 750 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The operation lead clarifies: “An incoming request is not very large, but an outgoing request is” highlighting the disproportionate size of data being processed in each direction. The task force swiftly responded to these imbalances by integrating a content delivery network (CDN), thereby optimizing the delivery of static, less sensitive content to users while ensuring that secured services managed the more sensitive data. This strategic al ocation of resources proved effective, as the operation lead reveals, " The same afternoon, we had access to 40 Mbit CDN" underscoring the task force's ability to rapidly enhance network capacity. In tandem with these efforts, the team introduced a queue on the front page. This tool was critical for strengthening system robustness and guaranteeing reliability during times of peak load. The IT architect describes the mechanism's utility: “We can limit the number of users we let in”, a measure to prevent system overloads. Presently, the CDN continues to be utilized for serving static content to some degree, with the IT architect noting, “We have that to some extent”. While the queuing system implemented during the peak period is no longer active, its architecture allows for it to be swiftly reactivated should the need arise. Sundhed.dk has also embraced server-side rendering using web components, a move that not only facilitates the distribution of more assets via the CDN but also accelerates the development process. The IT architect reflects on this technological evolution with satisfaction: " We should arrive at a scenario where we could develop our solutions significantly faster. That's the experience we've had so far with the web component concept. And our experience so far is one solution. Things move faster". This adaptability and speed mark a new era for sundhed.dk's digital service infrastructure, one marked by increased efficiency and readiness for future demands. 4.3 Improving the backend systems Sundhed.dk faced chal enges with their security measures, which significantly slowed down the application performance. As the IT architect explains, some security services were not equipped for many users at the same time: " And the problem is that some of the security services we have running, they cannot serve many concurrent users". The issue was compounded by excessive encryption for each user, which the IT architect describes as " completely overkil in this context. It real y sucks up performance". To address the concerns raised, the team during the peak project period focused on optimizing data transfers between services. This involved reevaluating the need for encryption, J. Tuxen, M. Boesen, E. Leth Møl er: Strengthening Resilience in a Danish Health System – A Longitudinal Study 751 particularly for data sensitive to security breaches, such as man-in-the-middle attacks. The IT architect notes the necessity of encryption for inter-service communication: " There's a service that needs to cal another service. It needs to transport some data that's personal y sensitive, so therefore we have to encrypt it". The combination of access control and encryption was crucial for ensuring the security of these transfers: " There's an access control element, and at the same time, it's encrypted" (The IT architect). To further enhance performance, the task force considered switching off message encryption and relying solely on HTTPS, a less resource-intensive option, but as indicated by the IT architect they ended up with message encryption: " Turn of HTTPS and run with message encryption". Following the peak period, the task force's work resulted in work to re-introduce HTTPS on the backend and eliminate resource-intensive elliptic encryption. This step, alongside a more holistic view of the scalability chal enges, particularly concerning external dependencies like health data and NemID1 login mechanisms, marked an evolution in sundhed.dk's approach. The IT architect highlights the necessity for external components to scale in tandem: " It's not just sundhed.dk. . there are some external factors that need to scale with us". The implementation of short timeouts for synchronous cal s was an interim solution to ensure service restoration in case of failures. The task force planned and later implemented a circuit-breaker pattern to manage these issues, although its complexity led to a reduction in its use. A shift to lazy loading for a less-frequently used service led to faster response times, a change that has been maintained post task force, as the IT architect confirms, " It's in operation unchanged". The IT architect explains that prior to the task force, an exploration into replacing old servers took place. During the peak, services were consolidated onto virtual machines, which allowed for local calls, removal of service discovery service and thus reducing communication overhead and enabling horizontal scaling with the addition of hardware and a load balancer. The IT architect details this scalable solution: " So, the web service and gateway (service) were put on the same server.. If we need to support 1000 simultaneous users, we simply make 10 of those machines". Post-task force, the practice of virtualisation was continued, and service discovery was reintroduced. 1 Personal key to digital services in Denmark 752 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Work is in progress to split a monolithic application into smaler, independent, and scalable microservices, enhancing reliability and failover capabilities. The microservices chassis pattern and service template pattern are being utilized to speed up development and ensure consistency across services. The IT architect expresses optimism for the new direction: " We implement chassis. The goal is faster development. The experiences from the first solution are good". Overal , the changes done by the task force has been refined to ensure resilience of sundhed.dk going forward. 5 Conclusion and further research Our initial investigation revealed several strategies that enhanced the resilience of Sundhed.dk. However, as this subsequent study indicates, not every seemingly promising concept withstands the scrutiny of time. The circuit-breaker pattern was ultimately discarded, as it led to an overly complex architecture without effectively addressing the core issue. Nevertheless, this architectural pattern remains robust and should be considered for future implementations. A selection of initiatives has proven enduring: The use of a Content Delivery Network (CDN) for the static components of the frontend has been successful. The backend remains virtualised, yet there is ongoing effort to streamline the architecture, progressively moving towards a microservices approach. A service discovery mechanism, removed in the task force, has now been reintegrated, enhancing the architectural design with minimal additional burden. Furthermore, the utilisation of web components, lazy loading of proxy components, and the microservices chassis pattern are considered to be advantageous. In the realm of security, the transition from dual to singular encryption was made early in the process. We anticipate adopting web standards for encryption as they are deemed sufficiently secure. J. Tuxen, M. Boesen, E. Leth Møl er: Strengthening Resilience in a Danish Health System – A Longitudinal Study 753 The organisation has undergone several changes. Under favourable conditions, the re-establishment of a DevOps team is recognised as potential y beneficial. Further work is going on at sundhed.dk to improve documentation to prevent knowledge loss. Through our study, we have identified the specific initiatives that were sustained, discarded, or expanded upon by sundhed.dk in strengthening resilience within the Danish IT health platform across the phases before, during, and after the COVID- 19 crisis. The insights gained may serve as a foundation for subsequent research in public entities that manage highly sensitive citizen data and thus have stringent privacy requirements. Acknowledgements We would like to extend our deepest gratitude to Nicklas Frederiksen for his invaluable contribution to this research. His efforts in identifying the case of Sundhed.dk and facilitating our access to it have been instrumental in the success of this study. References Annarelli, A., & Nonino, F. (2016). Strategic and operational management of organizational resilience: Current state of research and future directions. Omega, 62, 1–18. Chell, E. (1998). Critical incident technique. In G. Symon & C. Cassell (Eds.), Qualitative Methods and Analysis in Organizational Research (pp. 51–72). SAGE Publications. Duchek, S. (2020). Organizational resilience: a capability-based conceptualization. Business Research, 13(1), 215–246. Flanagan, J. C. (1954). The critical incident technique. Psychological Bul etin, 51(4), 327–358. Frederiksen, N., Møl er, E. L., Tuxen, J., O’Neil , S. E., & Boesen, M. (2023). Rapid Scaling of a Danish Public Health System Under COVID-19. 36th Bled EConference - Digital Economy and Society: The Balancing Act for Digital Innovation in Times of Instability, Bled, Slovenia, 723–730. Gardner LeGars, J., Simonin, J., Waldeck, R., & Puentes, J. (2023). A dual perspective of organizational resilience (OR) and information technology systems resilience (ITSR): an analysis of interdependencies and tensions. ARPHA Conference Abstracts, 6. Haldane, V., De Foo, C., Abdal a, S. M., Jung, A. S., Tan, M., Wu, S., Chua, A., Verma, M., Shrestha, P., Singh, S., Perez, T., Tan, S. M., Bartos, M., Mabuchi, S., Bonk, M., McNab, C., Werner, G. K., Panjabi, R., Nordström, A., & Legido-Quigley, H. (2021). Health systems resilience in managing the COVID-19 pandemic: lessons from 28 countries. In Nature Medicine (Vol. 27, Issue 6, pp. 964–980). Nature Research. Liu, D., Deters, R., & Zhang, W. J. (2010). Architectural design for resilience. Enterprise Information Systems, 4(2), 137–152. 754 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Müler, G., Koslowski, T. G., & Accorsi, R. (2013). Resilience - A New Research Field in Business Information Systems? In W. Abramowicz (Ed.), Business Information Systems Workshops. BIS 2013. Lecture Notes in Business Information Processing (Vol. 160). Springer. Pan, J., Liu, Z., Li, D., Wang, L., & Li, B. (2023). An empirical study of software architecture resilience evaluation methods. The Journal of Systems & Software, 202, 1–11. RESEARCH IN PROGRESS UNRAVELLING THE USE OF DIGITAL TWINS TO ASSIST DECISION- AND POLICY-MAKING IN SMART CITIES LUCY TEMPLE,1 GABRIELA VIALE PEREIRA,1 LUKAS DANIEL KLAUSNER 2 1 University for Continuing Education Krems, Department for E-Governance and Administration, Krems, Austria lucy.temple@donau-uni.ac.at, gabriela.viale-pereira@donau-uni.ac.at 2 St. Pölten University of Applied Sciences, Department of Computer Science and Security, St. Pölten, Austria mail@l17r.eu This short paper represents a systematic literature review that sets the basis for the future development of a framework for digital twin-based decision support in the public sector, specifical y for the smart city domain. The final aim of the research is to model context-specific digital twins for aiding the decision-making processes in smart cities and devise methods for defining the policy agenda. Overall, this short paper provides a Keywords: decision foundation, based on the main concepts from existing literature, support, for further research in the role and applications of urban digital digital twin, twins to assist decision- and policy-making in smart cities. The policy existing literature analyses common applications of digital twins process, in smart city development with a focus on supporting decision- simulation, smart and policy-making. Future work wil centre on developing a city digital-twin-based sustainable smart city and defining different governance, scenarios concerning chal enges of good governance, especial y smart city so-called wicked problems, in smaller-scale urban and non-urban policy contexts. domain DOI https://doi.org/10.18690/um.fov.4.2024.45 ISBN 978-961-286-871-0 756 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The importance of an information systems (IS) focus on smart-city-related studies is highlighted by Ismagilova et al. (2019), especially regarding the implementation and use of IS to design, develop and plan smart cities. Following the approach by Ismagilova et al. (2019, p. 90): Smart cities use an IS-centric approach to the intel igent use of information and communication technologies (ICT) within an interactive infrastructure to provide advanced and innovative services to its citizens, impacting quality of life and sustainable management of natural resources. Emerging technologies that serve to support human decision-making are transforming existing government arrangements and are very promising for decision support in the context of smart cities, both in the policy process and in operative decision-making (König & Wenzelburger, 2021). Digital twins are considered an emerging technology in smart city research (Hämäläinen, 2021). Although digital twin technology originally emerged as a tool for designing virtual replicas in manufacturing, it has now evolved into a concept that can be applied to different domains, from engineering through automobile manufacturing to energy supply systems (White et al., 2021), and existing research has shown a growing interest in their application to facilitate decision-making for urban planning and policy decisions alike (Lei et al., 2023; White et al., 2021). Digital twins of cities can assist policymakers in the process of making strategic long-term decisions regarding urban planning (Lohman et al., 2023), as wel as supporting the planning and management of cities in urban and peri-urban areas (Lei et al., 2023). According to Lohman et al. (2023), digital twins connect data, analytics and visualisation, enabling policymakers to simulate what-if scenarios in a scenario-based analysis. While “analytics” means the growth of data that leads to new technological and scientific developments, “policy analytics” consists in the use of analytics to support public policy decision-making (Daniel et al., 2016, p. 7). Bringing this to a public sector perspective, digital technologies can be used to support different government functions, e. g. legislative (for policymaking), management (for policy advice and administrative tasks) and service delivery through employees and service providers. In the case of local government, according to Clement and Crutzen (2021), a local political system and objectives are supported by the so-cal ed smart city policy domain, which defines the smart city policy agenda as the list of topics to address local problems that are influenced by the local context. The agenda is set when a given problem aligns with an appropriate smart city solution and is also aligned with policy priorities at the local level. Yossef L. Temple, G. Viale Pereira, L. Daniel Klausner: Unravel ing the Use of Digital Twins to Assist Decision-and Policy-Making in Smart Cities 757 Ravid and Aharon-Gutman (2023) show the potential of digital twins to also incorporate social aspects into the decision-making process. We use this idea as the basis for analysing the role and applications of digital twins to support urban planning, city management, smart city initiatives and policy in smart cities. When designing urban planning policies, such as deciding where to create a new park, data retrieved from the physical smart city such as air pollution, noise pollution, traffic flow and sunlight can be used to simulate various park placement options in the digital twin. By using smart sensors, the flow of people in the area can also be simulated to help decide where benches, fitness equipment, paths etc. need to be planned (Ramu et al., 2022; White et al., 2021). The main objective of this ongoing research is to review the existing literature on the intersection of digital twins and smart cities with a focus on decision and policy- making support and to answer the research question: “What are the existing applications of digital twins for smart cities for aiding decision-and policy-making?” This work in progress forms part of the baseline research conducted within the project Smart Cities and Digital Twins in Lower Austria (SCiNDTiLA), which explores how the concept of smart cities can be transferred to smal er-scale urban and non-urban contexts and how the use of digital twins and algorithms can aid decision support and policy-making towards developing smart sustainable solutions. A variety of use cases wil be defined within Lower Austria and digital twins will be simulated to provide scenarios to be used for local decision-making. 2 Systematic Literature Review Methodology This short paper analyses the existing literature at the intersection of digital twins and smart cities with a focus on the application of this technology as a tool to support decision- and policy-making for city managers. For the systematic literature review, we explored the existing literature on digital twins and smart cities, using the following search in the Scopus database: “⟨title OR abstract OR author-specified keywords: “smart cit*” AND “digital twin*” AND ⟨“decision support” OR “decision-making”⟩⟩ AND ⟨publication year: >2013⟩“. We found 94 relevant records; we then scanned the titles and abstracts and selected 47 papers which had a clear focus on supporting decision-making. After a full-text screening of these papers, the preliminary analysis included 25 papers. The full analysis of the selected articles focused on the variety of uses of digital twins in smart cities, the tools 758 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION involved to gather the relevant data needed, the simulations these can produce and the chal enges and enablers identified. The following section presents the relevant findings from the literature and is subdivided into two subsections: data and tools as wel as chal enges and enablers. These themes were derived by grouping the findings of the selected articles, al owing us to synthesise and interpret the literature to provide a comprehensive understanding of the current state of research in the field. 3 Literature Analysis A digital twin, as defined in literature (White et al., 2021), is a virtual representation of a physical process, person, place, system or device, initial y developed to enhance manufacturing processes through precise simulations featuring highly accurate models of individual components. The overarching aim of digital twins is to simulate the behaviour of the targeted object and enable real-time decision-making based on reasonable predictions (Shi et al., 2023). This entails capturing real-time characteristics and status to facilitate proactive actuation orders. A more comprehensive definition, as proposed by Zhou et al. (2022), underscores the creation of a virtual representation for a dynamic physical object or system, spanning multiple life cycle stages and facilitating decision-making through the application of data analysis methods. 3.1 Data and Tools With the rapid digitisation of cities, there is more and more data available for use and analysis. However, often the raw data are insignificant without being embedded in the right context. Moreover, the amount of data created exceeds human capabilities for simple analysis and prediction. These data can be interlinked and placed in simulated environments, and with the right tools, a Smart City Digital Twin (SCDT) can be model ed. As cities are complex systems, digital twins of smart cities should focus on the interdependencies between networks and city infrastructure to be able to accurately represent the physical object in its specific usage context (Mohammadi et al., 2020). The variety of components within a digital twin work together to create a virtual representation or urban portrait of the physical city. This allows for real-time monitoring, but also analysis and optimisation. The availability of spatial and non-spatial data alongside evolving simulation technologies allows for the digital representation of such intricate entities as cities (Jeddoub et al., 2023). L. Temple, G. Viale Pereira, L. Daniel Klausner: Unravel ing the Use of Digital Twins to Assist Decision-and Policy-Making in Smart Cities 759 Relevant types of data and technologies used in the context of a physical smart city include the internet of things (IoT), geographic information systems (GIS), building information model ing (BIM), natural language processing (NLP) and machine learning (ML) (Shi et al., 2023; White et al., 2021; Yaqoob et al., 2023). Data used for simulating digital twins of physical counterparts must be of high quality, accurate, valid, complete and consistent. This will allow producing more valuable and reliable scenarios for policy makers to base their decisions on (Li & Tan, 2023). A SCDT is expected to be used as a means to achieve evidence-based decisions, not as an end, leading to better outcomes and more informed policy processes (Wan et al., 2019). Petrova-Antonova and Ilieva (2019) believe such simulations allow for discussion between stakeholders to select the best outcome and get insights regarding the possible effects after deployment. The design and implementation of a SCDT can provide a broader vision for future planning and city improvements by using computer model ing and information technologies to diagnose and map aspects present in the physical world, al owing for high efficiency and intelligent decision-making (Lyu et al., 2022). In order to do so, the SCDT requires high-quality, long-term data for decision-making (Ramu et al., 2022). Weil et al. (2023) state that all digital infrastructure and sensors in a smart city can be used more efficiently for decision-making processes thanks to the advances in SCDT, al owing for simulation models and predictions. By using real-time data to simulate the system’s performance, it al ows for real-time monitoring, to forecast and optimise the physical counterpart, but also for analysis and streamlining to allow for faster and more accurate predictions, better decision-making, and quicker response time (Li & Tan, 2023; Wang et al., 2023). The SCDT can be used as a decision support tool, but the agency of the actors is stil necessary for decision- making. The technology supports the decision process by linking the actors with the resources and information (West et al., 2021). As digital twins are able to store and process more historical and real-time data, they become capable of predicting and forecasting variations, visualising what-if scenarios that can help city decision-makers make proactive decisions (such as for disaster prevention) (Li & Tan, 2023; Mavrokapnidis et al., 2021). SCDT can enable an overview of heterogeneous data instead of isolated interpretations of specific datasets (Raes et al., 2021). This holistic analysis and visualisation approach allows for policymakers to process and use heterogeneous city data, integrating their 760 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION domain expertise with the information provided by the digital twin and thus supporting integrated management decisions, macro decision-making and evaluation (Lyu et al., 2022; Mohammadi et al., 2020). 3.2 Chal enges and Enablers Ramu et al. (2022) state that digital twins are still in a very early stage of usage for smart city applications; the main reason for this is the lack of trust and privacy issues of sharing sensitive data. SCDT are commonly used in the planning, design and development of a city, its core policy agenda and for both short- and long-term planning (Mendula et al., 2022; Weil et al., 2023). Weil et al. (2023) highlight the importance of cooperation between decision-makers in order for the SCDT to become a useful tool – data-sharing, joint planning, political support and dialogue are needed. Despite the increased popularity of SCDT, there stil exists a series of bottlenecks for their implementation. First, it is crucial to have consistency between the physical and the virtual representation. There is a lack of centralised, unified and defined frameworks regarding how to make the data connections between the digital and the physical (Lei et al., 2023; Ma et al., 2024; Raes et al., 2021). Challenges identified in the literature range from technical (interoperability and semantic standards) to non-technical (the need to be purposeful, trustworthy and functional, or the lack of business models) (Lei et al., 2023; Weil et al., 2023). In order to be able to represent the physical city and support decision- and policy-making, a digital twin must be able to break through data silos while at the same time providing secure information (Shi et al., 2023; Yaqoob et al., 2023). Moreover, the data quality is key for presenting policymakers with accurate data and reliable results and measurements (Weil et al., 2023). Early SCDT platforms relied on IoT implementations throughout smart cities, but these lacked the scale and suitability to be useful in making policy decisions (Raes et al., 2021). Many authors still claim that digital twins are lacking the ability to integrate the socio-economic and human dynamics of cities (Mohammadi & Taylor, 2019). White et al. (2021) propose stakeholder and citizen involvement in the SCDT. Citizens identify problem areas within the city and give feedback to proposed policies; this information is fed back into the SCDT to create additional data. Not all SCDT are designed for citizen engagement, but when users are able to participate, comment and request changes by assessing the virtual environment, they are able to single out existing issues. To adequately include citizens’ voice in decision-making, a SCDT needs to receive L. Temple, G. Viale Pereira, L. Daniel Klausner: Unravel ing the Use of Digital Twins to Assist Decision-and Policy-Making in Smart Cities 761 human input for al owing human participation in decision-making (Abdeen et al., 2023; Ramu et al., 2022). It is important for SCDT to consider changes in organisational culture, processes and structures allowing for a clear digital data flow between the cities’ information systems and the entities within the cities (Hämäläinen, 2021).The use of a digital twin allows for simulating a series of what-if scenarios, which can be immensely helpful in such contexts. Time and computational capabilities remain key chal enges for SCDT; decision support scenarios ideally need to be evaluated in a series of workshops or focus groups, and to that end, computational times need to be manageable, with synchronicity and the rate at which data can flow key for the practical usefulness of SCDT (Jeddoub et al., 2023; Lohman et al., 2023; Wang et al., 2023; Weil et al., 2023). 4 Conclusions and Future Work This literature review analyses common applications of digital twins in smart city development for assisting decision support. This research in progress highlights the significance of information systems in smart-city-related studies and the potential of digital twin technology to assist in designing, planning and developing smart cities, specifically how digital-twin-based decision support can aid policymakers to make better decisions. The empirical research wil be conducted in Lower Austria, which is characterised by the growing number of policies that have been developed to support “digitalisation of the public sector” and, especially, “smart initiatives” at the regional and local level according to the Digitalization Strategy of Lower Austria. Future work includes developing a SCDT and defining different scenarios concerning chal enges of good governance in smal er-scale urban and non-urban contexts as wel as implementing this proof of concept in an exemplary region and using the insight gained to draft a roadmap, highlighting methodologies, guidelines and policy recommendations on how smart and sustainable solutions in cities and regions shape inhabitants’ perception of local governance. 762 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Acknowledgements This research was funded by the Geselschaft für Forschungsförderung Niederösterreich (GFF NÖ) project GLF21-2-010 “Smart Cities and Digital Twins in Lower Austria”. The financial support by the Gesel schaft für Forschungsförderung Niederösterreich is grateful y acknowledged. References Abdeen, F. N., Shirowzhan, S., & Sepasgozar, S. M. E. Citizen-Centric Digital Twin Development with Machine Learning and Interfaces for Maintaining Urban Infrastructure. Telemat. Inform., 84 (2023). https://doi.org/10.1016/J.TELE.2023.102032. Clement, J., & Crutzen, N. How Local Policy Priorities Set the Smart City Agenda. Technol. Forecast. Soc. Change, 171 (2021). https://doi.org/10.1016/J.TECHFORE.2021.120985. Daniel , K. A., Morton, A., & Ríos Insua, D. Policy Analysis and Policy Analytics. Ann. Oper. Res., 236(1), 1–13 (2016). https://doi.org/10.1007/S10479-015-1902-9. Hämäläinen, M. Urban Development with Dynamic Digital Twins in Helsinki City. IET Smart Cities, 3(4), 201–210 (2021). https://doi.org/10.1049/SMC2.12015. Ismagilova, E., Hughes, L., Dwivedi, Y. K., & Raman, K. R. Smart Cities: Advances in Research—An Information Systems Perspective. Int. J. Inf. Manag., 47, 88–100 (2019). https://doi.org/10.1016/J.IJINFOMGT.2019.01.004. Jeddoub, I., Nys, G.-A., Hajji, R., & Billen, R. Digital Twins for Cities: Analyzing the Gap Between Concepts and Current Implementations with a Specific Focus on Data Integration. Int. J. Appl. Earth Obs. Geoinf., 122 (2023). https://doi.org/10.1016/J.JAG.2023.103440. König, P. D., & Wenzelburger, G. The Legitimacy Gap of Algorithmic Decision-Making in the Public Sector: Why It Arises and How to Address It. Technol. Soc., 67 (2021). https://doi.org/10.1016/J.TECHSOC.2021.101688. Lei, B., Janssen, P., Stoter, J., & Biljecki, F. Chal enges of Urban Digital Twins: A Systematic Review and a Delphi Expert Survey. Autom. Constr., 147 (2023). https://doi.org/10.1016/J.AUTCON.2022.104716. Li, B., & Tan, W. A Novel Framework for Integrating Solar Renewable Source into Smart Cities Through Digital Twin Simulations. Sol. Energy, 262 (2023). https://doi.org/10.1016/J.SOLENER.2023.111869. Lohman, W., Cornelissen, H., Borst, J., Klerkx, R., Araghi, Y., & Walraven, E. Building Digital Twins of Cities Using the Inter Model Broker Framework. Future Gener. Comput. Syst., 148, 501– 513 (2023). https://doi.org/10.1016/J.FUTURE.2023.06.024. Lyu, Z., Shang, W.-L., & Guizani, M. Impact of Digital Twins and Metaverse on Cities: History, Current Situation, and Application Perspectives. Appl. Sci., 12(24) (2022). https://doi.org/10.3390/APP122412820. Ma, X., Qi, Q., & Tao, F. An Ontology-Based Data-Model Coupling Approach for Digital Twin. Robot. Comput. Integr. Manuf., 86 (2024). https://doi.org/10.1016/J.RCIM.2023.102649. Mavrokapnidis, D., Mohammadi, N., & Taylor, J. E. Community Dynamics in Smart City Digital Twins: A Computer Vision-Based Approach for Monitoring and Forecasting Col ective Urban Hazard Exposure. Proceedings of the 54th Hawai International Conference on System Sciences, 1810–1818 (2021). http://hdl.handle.net/10125/70832 Mendula, M., Bujari, A., Foschini, L., & Bel avista, P. A Data-Driven Digital Twin for Urban Activity Monitoring. Proceedings of the 2022 IEEE Symposium on Computers and Communications, 1–6 (2022). https://doi.org/10.1109/ISCC55528.2022.9912914. Mohammadi, N., & Taylor, J. E. Devising a Game Theoretic Approach to Enable Smart City Digital Twin Analytics. Proceedings of the 52nd Hawai International Conference on System Sciences, 1995–2002 (2019). https://hdl.handle.net/10125/59639 L. Temple, G. Viale Pereira, L. Daniel Klausner: Unravel ing the Use of Digital Twins to Assist Decision-and Policy-Making in Smart Cities 763 Mohammadi, N., Vimal, A., & Taylor, J. E. Knowledge Discovery in Smart City Digital Twins. Proceedings of the 53rd Hawai International Conference on System Sciences, 1656–1664 (2020). https: //hdl.handle.net/10125/63943 Petrova-Antonova, D., & Ilieva, S. Methodological Framework for Digital Transition and Performance Assessment of Smart Cities. Proceedings of the 4th International Conference on Smart and Sustainable Technologies, 536–542 (2019). https://doi.org/10.23919/SPLITECH.2019.8783170. Raes, L., Michiels, P., Adolphi, T., Tampere, C., Dalianis, A., McAleer, S., & Kogut, P. DUET: A Framework for Building Interoperable and Trusted Digital Twins of Smart Cities. IEEE Internet Comput., 26(3), 43–50 (2021). https://doi.org/10.1109/MIC.2021.3060962. Ramu, S. P., Boopalan, P., Pham, Q.-V., Maddikunta, P. K. R., Huynh-The, T., Alazab, M., Nguyen, T. T., & Gadekal u, T. R. Federated Learning Enabled Digital Twins for Smart Cities: Concepts, Recent Advances, and Future Directions. Sustain. Cities Soc., 79 (2022). https://doi.org/10.1016/J.SCS.2021. 103663. Shi, J., Pan, Z., Jiang, L., & Zhai, X. An Ontology-Based Methodology to Establish City Information Model of Digital Twin City by Merging BIM, GIS and IoT. Adv. Eng. Inform., 57 (2023). https://doi.org/10.1016/J.AEI.2023.102114. Wan, L., Nochta, T., & Schooling, J. M. Developing a City-Level Digital Twin – Propositions and a Case Study. Proceedings of the 2019 International Conference on Smart Infrastructure and Construction, 187–194 (2019). https://doi.org/10.1680/ICSIC.64669.187. Wang, H., Chen, X., Jia, F., & Cheng, X. Digital Twin-Supported Smart City: Status, Chal enges and Future Research Directions. Expert Syst. Appl., 217 (2023). https://doi.org/10.1016/J.ESWA.2023.119531. Weil, C., Bibri, S. E., Longchamp, R., Golay, F., & Alahi, A. Urban Digital Twin Chal enges: A Systematic Review and Perspectives for Sustainable Smart Cities. Sustain. Cities Soc., 99 (2023). https://doi.org/10.1016/J.SCS.2023.104862. West, S., Stoll, O., Meierhofer, J., & Züst, S. Digital Twin Providing New Opportunities for Value Co-Creation Through Supporting Decision-Making. Appl. Sci., 11(9) (2021). https://doi.org/10.3390/APP11093750. White, G., Zink, A., Codecá, L., & Clarke, S. A Digital Twin Smart City for Citizen Feedback. Cities, 110 (2021). https://doi.org/10.1016/J.CITIES.2020.103064. Yaqoob, I., Salah, K., Jayaraman, R., & Omar, M. Metaverse Applications in Smart Cities: Enabling Technologies, Opportunities, Chal enges, and Future Directions. Internet Things, 23 (2023). https://doi.org/10.1016/J.IOT.2023.100884. Yossef Ravid, B., & Aharon-Gutman, M. The Social Digital Twin: The Social Turn in the Field of Smart Cities. Environ. Plan. B Urban. Anal. City Sci., 50(6), 1455–1470 (2023). https://doi.org/10.1177/23998083221137079. Zhou, J., Zhang, S., & Gu, M. Revisiting Digital Twins: Origins, Fundamentals, and Practices. Front. Eng. Manag., 49(4), 668–676 (2022). https://doi.org/10.1007/S42524-022-0216-2. 764 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION RESEARCH IN PROGRESS OVERVIEW OF ASTHMA RELATED SMARTPHONE APPLICATIONS ON GOOGLE PLAY AND APPLE APP STORES ZAID ZEKIRIA SAKO, NILMINI WICKRAMASINGHE La Trobe Uiversity, School of Computing, Engineering and Mathematical Sciences, Victoria, Australia z.sako@latrobe.edu.au, n.wickramasinghe@latrobe.edu.au Climate change, as a result of rising levels of greenhouse gases, is causing havoc around the world. This affects respiratory systems thus causing unprecedented rise in asthma and chronic obstructive pulmonary disease (COPD) cases. Making an Keywords: individual aware of the surrounding climate conditions enables asthma, pollen, them to take preventative measures. One way to deliver this type air of information is through smartphone applications. Thus, this quality, paper surveyed the two major Australian application stores, smartphone applications, Google Play and Apple App, for asthma related smartphone app applications. stores DOI https://doi.org/10.18690/um.fov.4.2024.46 ISBN 978-961-286-871-0 766 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Climate change is closely associated with the rise in the number of al ergic respiratory diseases due to the changes in weather, pollen levels and air pollution (D’Amato et al., 2020). The fast-changing climate around the world is the result of increased levels of Carbon Dioxide (CO2) emissions produced by the burning of fossil fuels, industrial revolution, agriculture and forestry, bushfires, and other sources of CO2 (Pacheco et al., 2021). As the CO2 levels continue to climb, long exposure to such harmful element and other air pollutants is linked to adverse effects on the respiratory system (Eguiluz‐Gracia et al., 2020). Two of the major non- communicable respiratory diseases linked to such changes in climate conditions are asthma, and chronic obstructive pulmonary disease (COPD) (Tran et al., 2023). Self- management of asthma using smartphone applications is one of the methods people with asthma are adhering to for managing their condition. With such reliance on applications, this serves as the motivation for this short paper to analyze the types of applications available for people with asthma, examine the information delivered through the applications, and understand their limitations. The focus of this paper is around applications that provide weather, pollen, and air quality information. 2 Literature Review Asthma is considered a chronic non-communicable condition where the airways carrying the air to the lungs are irritated thus causing shortness of breath or wheezing (Porsbjerg et al., 2023). Whereas COPD is a chronic lung disease and is a result of long-term exposure to cigarette smoke, air pollution, and occupational dust (Tran et al., 2023). The association between climate change and respiratory diseases is due to the events that are created by the changing weather which produces various conditions that trigger respiratory reactions. With the ever-increasing fluctuations in climate conditions, they pose a threat to individuals with asthma as frequency of unpredictable and often changing weather conditions can exacerbate asthma (Kel y et al., 2023). Chronic respiratory diseases require appropriate and on-time diagnosis, response, and therapy (Gurbeta et al., 2018). Appropriate information related to weather that is delivered in a timely manner can render great aid for people with asthma (Johnston et al., 2018). One of the options available for people with asthma to self-care and management is smartphone applications developed for people with asthma. Asthma patients with access to mobile phones demonstrated the wil ingness Z. Zekiria Sako, N. Wickramasinghe: Overview of Asthma Related Smartphone Applications on Google Play and Apple App Stores 767 to use the technology to access care for asthma and receive information using social networks and smartphone applications (Nabovati et al., 2020). The availability of weather, pollen levels, and air quality information are key to people with asthma in taking preventative measures when managing their condition. Google Play and Apple App stores are the two primary key players in the smartphone applications market, and they provide a variety of applications. 3 Data Collection The term ‘asthma’ was used to search the Australian Google Play and Apple App store across both online and mobile stores to obtain a sense of how the search engine operates. Subsequently, multiple searches were performed on both versions of the application stores for both platforms for Australia using terms from the study that covered ‘asthma’, ‘pollen’, and ‘air quality’. Once the search was completed, a list of applications was collected and extracted from both platforms for examination. The search was conducted in November 2023. After filtering and reviewing the list of collected applications, a total of 159 applications were excluded from further analysis for the following reasons: • Applications developed in languages other than English as the study focused on applications developed in English. • Applications not updated for more than one year (one year and above), as this demonstrated that the application is not maintained. • Applications developed for specific countries such as Indonesia, Africa, England, Europe, or Wales as the focus of this study is applications available in Australia. • Applications available through invite only from medical professionals. • Applications not written nor developed in English. All other applications were included in the review and for further examination. 4 Data Analysis The first step of the analysis involved reviewing each application using the application description and the accompanied screenshots to develop a list of categories that best describe the purpose of the application. Once an application was 768 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION categorized, a SWOT analysis followed and was applied against each application. However, applications that exhibited features that covered weather, air quality, and pollen information were further examined to understand: • Count of how many weather-related features the app provided (Weather, Air Quality, and Pollen Information). • Source of the information (satel ites, local pollen monitoring stations, local air quality monitoring stations). • Relevance of the information, and whether the application focused on one feature than the other. • Geographical region that the information covered. • Information timeliness, specifically checking if the information was forecast or real time based. The examination of the applications was conducted by instal ing each of the shortlisted applications on an iPhone and a Samsung phone to test the features. 5 Results 5.1 Unique Applications At the end of the comprehensive search and analysis on both Google Play and Apple App stores, a total of 74 applications were found to be related to asthma and provided information that assisted people with asthma. Upon close analysis of the applications names, the total number of unique applications was 70. As illustrated in Figure 1, a smal number of applications exhibited features that covered weather, pollen, and air quality measures in one single application. Z. Zekiria Sako, N. Wickramasinghe: Overview of Asthma Related Smartphone Applications on Google Play and Apple App Stores 769 Figure 1: Count of Applications with features covering Weather, Air Quality and Pol en 5.2 Types of Applications A list of themes was developed during the analysis of the applications that aided in generating the categories that best describe the intended use for the applications. These are defined below: • Add on: These are applications developed for use in conjunction with Bluetooth connected smart attachments such as Bluetooth enabled asthma puffers that synchronize with the application to collect information about the patient use of the puffers. • Diary: Applications developed for assisting people taking notes of their conditions that can be later retrieved for discussion with their medical professional. • Educational: Applications developed to educate people about asthma and how to best manage the condition. • Game: Applications developed to make learning about asthma and adherence to asthma medications more fun. • Management: To learn how to best manage asthma, including usage of asthma puffers, and breathing techniques. • Tracker: Applications developed to assist asthma patients with tracking their asthma condition, medications and noting things about their condition that can be later retrieved for discussion with medical professionals. 770 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • Weather Tracker: Applications specifical y designed to educate people with asthma about their surrounding climate conditions including weather conditions, air quality measures, and pollen levels. This is to prevent asthma exacerbation. Table 1 provides a breakdown of the number of applications discovered for each category. Table 1: Application Categories Category Number of Applications Addon 14 Diary 5 Educational 7 Game 6 Management 5 Tracker 8 Weather Tracker 29 5.3 SWOT and TOWS Results During the analysis of the shortlisted applications, a key critical factor in understanding the quality of the information provided by the applications was based on the source of the information. Weather information was provided by the official government agency Bureau of Meteorology, pollen was provided by a limited number of pollen sensors installed around the country, and air quality information was provided by Environment Protection Authority (EPA) agency that has limited number of sensors instal ed around the country too. As such, the analysis provided in Table 2 presents the results of SWOT analysis coupled with TOWS method of the shortlisted applications. The specific focus on those applications is led by the underlying research background which focuses on understanding solutions available to people with asthma that can help people avoid asthma exacerbation due to fluctuating climate conditions. Z. Zekiria Sako, N. Wickramasinghe: Overview of Asthma Related Smartphone Applications on Google Play and Apple App Stores 771 Table 4: SWOT & TOWS Analysis Strengths Weaknesses - Not clinically endorsed. - Near realtime - Applications available that information. provide weather, pol en and - Limited number of air air quality information. quality and pollen sensors - Air quality applications around the country. SWOT/TOWS available with information - Each state presents its sourced from Environment own set of information Protection Authority. limited to their sensors. - Pollen levels applications - Most applications are the available with information result of a research activity, sourced from limited local not official y endorsed. pollen sensors. - Limited spatial coverage of air quality and pollen monitoring. Opportunities SO – Strategies WO - Strategies - Integrate realtime weather. - Leverage other weather - Combine new sources of sources such as satellites, - Co-design the data to provide greater and street maps. applications with medical coverage. - Automate sourcing of professionals. - Automate the col ection information to provide more - Seek new data sources for and distribution of national coverage. air quality and pollen information in realtime. - Build an application that information. - Seek new input source for provides al 3 key weather air quality and pollen data. information elements at a national level. Threats ST – Strategies WT - Strategies - Seek backup source of data - Sensors going offline, thus to cover for when sensors impacting sensor readings. - New smarter technology go offline. - Telecommunications to faciliate better - Build a mirror site for outages or maintenace infrastructure for building applications to provide end taking down sensors. better platform. users with alternative ways to access the information. 772 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 6 Discussion Applications provide an exceptional advantage for delivering tailored information to individuals with specific needs. The tailored custom experience is a way of encouraging end users to make use of the application which can potential y influence their behaviour towards managing their medical condition. Despite the benefits, a common weakness is around the coverage. Applications that provided climate information were based on the locally available Pollen and Air quality sensors. As the population continues to grow, the reliance on physical limited number of sensors becomes inadequate as the population outweighs the number of required sensors (Li et al., 2022). These sensors are limited in number thus limiting the area of coverage. New sources such as satel ites have new capabilities where pollen and air quality can be measured from space (Bechle et al., 2013). Satellites have the ability of measuring different air pollutants and particulate matter (Lin et al., 2021). This requires further exploration and research to understand the quality of the data from space, the measuring distance it covers, the frequency of the data, and spatial coverage. 7 Conclusion and Future Work The Google Play and Apple App stores contained a large collection of various smartphone applications, including health applications designed to cover a range of medical services. The analysed mobile applications il ustrate how they can be an effective tool for creating and delivering custom tailored information specific to a medical condition or group. In the context of asthma, they can be an effective method of enhancing peoples’ knowledge of their condition and providing climate information surrounding the patients to ensure a timely management of their asthma. However, with climate information (weather, pollen, and air quality) lacking a larger spatial coverage, this proves to be a key weakness that requires further investigation. As such, the issue of coverage should be investigated and studied further to explore different ways Air Quality and Pol en data can be col ected and processed so that it can be transmitted to people with asthma in almost real time. Z. Zekiria Sako, N. Wickramasinghe: Overview of Asthma Related Smartphone Applications on Google Play and Apple App Stores 773 References Bechle, M. J., Milet, D. B., & Marshal, J. D. (2013). Remote sensing of exposure to NO2: Satelite versus ground-based measurement in a large urban area. Atmospheric Environment, 69, 345-353. D’Amato, G., Chong‐Neto, H. J., Monge Ortega, O. P., Vitale, C., Ansotegui, I., Rosario, N., Haahtela, T., Galan, C., Pawankar, R., Murrieta‐Aguttes, M., Cecchi, L., Bergmann, C., Ridolo, E., Ramon, G., Gonzalez Diaz, S., D’Amato, M., & Annesi‐Maesano, I. (2020). The effects of climate change on respiratory al ergy and asthma induced by pollen and mold allergens. Allergy (Copenhagen), 75(9), 2219-2228. https://doi.org/10.1111/all.14476 Eguiluz‐Gracia, I., Mathioudakis, A. G., Bartel, S., Vijverberg, S. J. H., Fuertes, E., Comberiati, P., Cai, Y. S., Tomazic, P. V., Diamant, Z., Vestbo, J., Galan, C., & Hoffmann, B. (2020). The need for clean air: The way air pollution and climate change affect allergic rhinitis and asthma. Allergy (Copenhagen), 75(9), 2170-2184. https://doi.org/10.1111/all.14177 Gurbeta, L., Badnjevic, A., Maksimovic, M., Omanovic-Miklicanin, E., & Sejdic, E. (2018). A telehealth system for automated diagnosis of asthma and chronical obstructive pulmonary disease. Journal of the American Medical Informatics Association : JAMIA, 25(9), 1213-1217. https://doi.org/10.1093/jamia/ocy055 Johnston, F. H., Wheeler, A. J., Wil iamson, G. J., Campbel , S. L., Jones, P. J., Koolhof, I. S., Lucani, C., Cooling, N. B., & Bowman, D. M. J. S. (2018). Using smartphone technology to reduce health impacts from atmospheric environmental hazards. Environmental research letters, 13(4). https://doi.org/https://doi.org/10.1088/1748-9326/aab1e6 Kel y, G., Idubor, O. I., Binney, S., Schramm, P. J., Mirabel i, M. C., & Hsu, J. (2023). The Impact of Climate Change on Asthma and Al ergic-Immunologic Disease. Current al ergy and asthma reports, 23(8), 453-461. https://doi.org/10.1007/s11882-023-01093-y Li, L., Hao, D., Li, X., Chen, M., Zhou, Y., Jurgens, D., Asrar, G., & Sapkota, A. (2022). Satel ite-based phenology products and in-situ pol en dynamics: A comparative assessment. Environmental research, 204, 111937. Lin, H., Li, S., Xing, J., He, T., Yang, J., & Wang, Q. (2021). High resolution aerosol optical depth retrieval over urban areas from Landsat-8 OLI images. Atmospheric Environment, 261, 118591. Nabovati, E., Farzandipour, M., Heidarzadeh Arani, M., Akbari, H., Sharif, R., & Anvari, S. (2020). The use of mobile phone functionalities by patients with asthma and their desire to use for self-care purposes. BMC Medical Informatics and Decision Making, 20(1), 1-8. Pacheco, S. E., Guidos-Fogelbach, G., Annesi-Maesano, I., Pawankar, R., D' Amato, G., Latour-Staffeld, P., Urrutia-Pereira, M., Kesic, M. J., & Hernandez, M. L. (2021). Climate change and global issues in allergy and immunology. Journal of allergy and clinical immunology, 148(6), 1366-1377. https://doi.org/10.1016/j.jaci.2021.10.011 Porsbjerg, C., Melén, E., Lehtimäki, L., & Shaw, D. (2023). Asthma. The Lancet, 401(10379), 858-873. https://doi.org/https://doi.org/10.1016/S0140-6736(22)02125-0 Tran, H. M., Chuang, T.-W., Chuang, H.-C., & Tsai, F.-J. (2023). Climate change and mortality rates of COPD and asthma: A global analysis from 2000 to 2018. Environmental research, 233, 116448-116448. https://doi.org/10.1016/j.envres.2023.116448 774 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION RESEARCH IN PROGRESS A VALUE SENSITIVE DESIGN APPROACH TO EQUITABLE REMUNERATION OF THE DUTCH MUSIC COPYRIGHT SYSTEM IN THE DIGITAL AGE NERKO HADŽIARAPOVIĆ,1 MARLIES VAN STEENBERGEN,2 PASCAL RAVESTEIJN1 1 University of Applied Sciences, Research Center Digital Business & Media , Utrecht, Netherland nerko.hadziarapovic@hu.nl, pascal.ravesteijn@hu.nl 2 University of Applied Sciences, Research group Digital Ethics, Utrecht, Netherland marlies.vansteenbergen@hu.nl There is a need for modernizing the Dutch col ective management system of music copyright to match the rapidly changing digital music industry. Focusing on the often-neglected human values aspect, this study, part of a larger PhD research, examines the value preferences of music rights holders: creators and publishers. It aims to advise on technological redesign for music copyright management system and contribute to Keywords: discussions on equitable collective management. Building upon equitable remuneration, prior research, which comprehensively analyzed the Dutch music digital copyright system and identified key stakeholders, this paper music copyright analyses 24 interviews with those key stakeholders to identify system, their values and potential value tensions. Initial findings establish value a set of shared values, crucial for the next phases of the study – sensitive design, values operationalization. This research makes a academic digital contribution by integrating the Value Sensitive Design (VSD) music approach with Distributive Justice Theory, enriching VSD's industry, economics of application and enhancing our understanding of the Economics collective of Col ective Management (ECM). management DOI https://doi.org/10.18690/um.fov.4.2024.47 ISBN 978-961-286-871-0 776 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction The existing system of col ective management of music copyright urgently requires modernization to align with the rapidly evolving digital landscape of the music industry (Singer and Rosenblatt, 2023). The current system does not adequately account for the diverse and dynamic values of music creators, who may have different preferences, expectations, and motivations for their work (Bulayenko et al., 2018; WIPO, 2022). This paper presents a study that explores these value preferences of music creators and publishers, in this study referred to as rights holders. While a considerable body of work exists on equitable remuneration for music creators, it predominantly examines legal, economic, and technological dimensions, often overlooking human values (Nagel and Kranz, 2021). According to Van de Poel (2013), Value Sensitive Design (VSD) method typical y emphasizes product development, offering the opportunity to incorporate values in technology right from the beginning. However, in the case of the digital music industry, which is an already existing socio-technological system, starting from scratch is not feasible. Instead, the focus shifts to designing interventions that adapt and integrate with what is already in place, modifying and enhancing the current system (Van de Poel, 2013). Our study integrates Distributive Justice Theory (DJT) within VSD approach for the Dutch music copyright system adding a normative layer to VSD using DJT and enhancing our understanding of the Economics of Collective Management (ECM) core constructs, as detailed in section 3 of this paper. The study is guided by two primary research objectives, each building upon the other: 1) To identify the values and possible value tensions within the context of collective management of music copyrights; 2) To operationalize these identified values into a value-sensitive system of collective music copyright management. This paper begins with a review of the literature on ECM and VSD in section 2. Section 3 outlines the research method. This research-in-progress paper concludes with preliminary findings, what work remains to complete the paper, and conclusions in Section 4. N. Hadžiarapović, M. van Steenbergen, P. Ravesteijn: A Value Sensitive Design Approach to Equitable Remuneration of the Dutch Music Copyright System in the Digital Age 777 2 Literature review The digital era has transformed the music industry, requiring Collective Management Organizations (CMOs) to navigate new challenges in digital music distribution and consumption (Hesmondhalgh et al., 2021; Priest, 2021). European Directive 2014/26/EU aims to streamline online music licensing, enhancing CMOs' ability to manage and distribute copyrights efficiently (European Union, 2019). VSD is a research approach aimed at embedding values into technologies (Friedman et al., 2013). Within VSD, the definition of value is what matters to people in their lives, focusing on ethics and morality (Friedman and Hendry, 2019). The goal of VSD is twofold: 1) it supports critical analyses of existing technologies concerning values, and according to Simon (2016) 2) it provides a concrete methodology to embed these values into new technologies and technology implementations. For the first goal, the critical analysis of existing technologies, VSD can be used to assess whether desired values, such as privacy and/or justice, are achieved by the technology design, and to identify the absence of values by dismantling biases within the technology (Friedman and Hendry, 2019). The methodology involves three main types of investigations: conceptual, empirical, and technical. These investigations interact with each other and iterate in a process that helps to identify stakeholders, values, and potential value conflicts (Friedman et al. 2013, Friedman and Hendry, 2019). The conceptual phase includes the identification of relevant values and direct and indirect stakeholders (Simon, 2016). In empirical investigations, social science methods are used to revise these findings with a focus on the opinions of stakeholders, as wel as anticipated usage contexts (Manders-Huits, 2011). The technical phase, as described by Friedman et al. (2013), consists of two parts. 1) role that values play in existing technologies with a pronounced focus on the technology itself; 2) the proactive design of systems to support values identified in the conceptual and empirical research phases (Friedman and Hendry, 2019). VSD's method for identifying key stakeholder values often employs a bottom-up, descriptive strategy, relying on stakeholder and user surveys (Van Wynsberghe and Robbins, 2014), yet the prioritization of these values often lacks a comprehensive evaluation system (Jenkins et al., 2020). Therefore we argue to incorporate the DJT as a normative ethical framework within VSD for our study. Firstly, DJT, with its focus on human values, aligns with the VSD's goals to embed values in technologies. 778 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Secondly, the DJT’s adaptability to various contexts and viewpoints makes it particularly suitable for the dynamic and diverse field of music copyright. Furthermore, the principle of proportionality relative to position, as articulated by Rawls (1999), is also considered. This principle posits that individuals who are more privileged or less vulnerable should contribute more, while those who are less privileged or more vulnerable should receive more support, thereby ensuring a fair balance of benefits and burdens in society. Next to the abovementioned solution to the lacking of an normative framework within VSD-approach, there is a research gap in the empirical investigation of relevant stakeholder values and the potential value conflicts that arise from the digital transformation of the music copyright industry, particularly in the context of its collective rights management. Understanding these dynamics and their derived implications for both design and research is critical for the future development of an equitable system of collective music copyright management. 3 Methodology The conceptual investigation was the starting point of the tripartite iterative VSD approach conducted in preliminary research1. At the conceptual level, analyses were undertaken to identify the key stakeholders impacted by the technology, including their roles and their mutual relations. Empirical investigations, in this research in progress, wil contribute to a deeper understanding and clear definition of the perspectives and experiences of the identified stakeholders, employing qualitative interviews with key stakeholders as a method (Friedman et al., 2013). Qualitative interviews provide a robust method for comprehensively exploring and understanding stakeholders’ values, offering insights that may not be captured in a structured survey (Merriam and Tisdell, 2015). The study involved participants selected through purposive sampling, as described by Merriam and Tisdell (2015). Some were contacted directly per email or phone and were approached through our own network, some through LinkedIn or their organizations’ email. All were selected based on the insights of the stakeholder analysis and involved both stakeholders directly and indirectly affected by digital transformation of the music copyright industry and its further technological development. We restricted the scope to key 1 kept anonymous to ensure a blind review process. N. Hadžiarapović, M. van Steenbergen, P. Ravesteijn: A Value Sensitive Design Approach to Equitable Remuneration of the Dutch Music Copyright System in the Digital Age 779 stakeholders that might be most significantly affected (Friedman et al., 2013). Eventual y we interviewed twenty individuals form five stakeholders groups. Four of these individuals were interviewed twice, due to time restrictions during the first interview, what resulted in total of twenty four interview transcripts with: six composers (direct key stakeholders), six publisher (direct key stakeholders), three representatives of Dutch CMO Buma Stemra (direct key stakeholders), 2 copyright lawyers (indirect key stakeholders), one editor in chief of a major Dutch radio station (indirect key stakeholder), one Digital Service Provider (DSP) representative (indirect key stakeholders) and 1 book publisher (indirect key stakeholders). These individuals spanned a wide age range from 20 to 59 years and brought diverse educational backgrounds, predominantly holding Bachelor's or Master's degrees. Their experience in the music industry varied, ranging from 10 to 35 years. Interviews were conducted comprehensively, lasting from approximately 36 minutes to over two hours, and took place between November 2020 and December 2022. The format of these interviews varied, with some conducted face-to-face and others via video call. Al interviews were audio-recorded and transcribed verbatim, with the ful consent of the participants. In our research, we designed open-ended, flexible interview questions al owing participants to express their views and experiences fully. The data analysis utilized a thematic approach, with open and thematic coding to identify main themes and subthemes, following Merriam and Tisdell (2015). To ensure reliability and validity, three coders and one controller coder were involved, acknowledging the chal enges in using multiple coders such as increased time, resources, and coordination. After transcribing the interviews, the next phase is to conduct a qualitative content analysis. Our developing thematic categories draw both deductively from Distributive Justice Theory literature, and inductively from empirical data. Two primary categories identified are 'Value' and 'Value Conflict,' each with various subcategories being defined. In the next research phase, these categories wil be detailed in a codebook, complete with coding rules and examples. The transcripts are coded using Atlas.ti software, which, while not providing specific intercoder reliability statistics, offers valuable support for coding comparison and visual analysis. This software aids in identifying coder agreement and disagreement and al ows for data export for further analysis. This process, including the potential revision of the codebook, enhances intercoder agreement and the overal reliability of our qualitative research. Additionally, the analysis wil undergo peer debriefing, 780 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION where scholars and experts external to the project wil review the research approach and outcomes. 4 Findings and conclusions 4.1 Preliminary findings and work to be completed In the initial conceptual phase of the investigation, key stakeholder groups impacted by the digital transformation within the music copyright domain were identified. The current analysis of interviews has identified all eight values as defined in section 2. It also acknowledges potential value tensions. One such tension is for example between fairness and efficiency, particularly how efficiency-oriented algorithms might introduce bias, possibly marginalizing less popular music genres and affecting fair income distribution among creators. Another tension exists between autonomy and security, highlighting the complex balance between protecting creators' rights and the potential restrictions that might limit their creative and, for example, distributional freedoms. These areas present intricate chal enges and opportunities for further exploration. This research-in-progress has certain limitations. 1) its focus is primarily on the Dutch system, which may not ful y represent the global complexities of music copyright management; 2) our emphasis on rightsholders means other key stakeholders like users, labels, and performing artists are not directly included in the analysis; 3) given the iterative nature of the VSD approach, additional research is required to ensure the identified values are accurately defined and captured. While our current research adopts a qualitative methodology, future study will incorporate a mixed-methods approach to enrich the depth and breadth of our findings. Final y, although we have engaged directly with stakeholders, there is a scope for deeper and more active involvement of these stakeholders in future studies, enhancing the practical relevance and applicability of our research in improving the collective management system. N. Hadžiarapović, M. van Steenbergen, P. Ravesteijn: A Value Sensitive Design Approach to Equitable Remuneration of the Dutch Music Copyright System in the Digital Age 781 4.2 Conclusions This paper presents the findings from qualitative interviews with key stakeholders of music copyrights system in The Netherlands. The analysis offers a comprehensive collection and exploration of the values held by rights holders. These initial findings contribute specifical y to the overarching goals of the PhD study and provide a foundational set of shared values that serve as a starting point for operationalizing these values. Subsequent papers wil explore how these values can be operationalized as part of a technical investigation in VSD. Advice for future research is to implement an iterative and participatory design process, involving stakeholders at its every stage to continually refine and evaluate design solutions through stakeholder feedback. Despite its ongoing nature and inherent limitations, this paper lays a foundation with its comprehensive synthesis and dive into stakeholders' values and value tensions, paving the way for future research and design of an equitable system for col ective music copyright management. References Bulayenko, O., van Gompel, S., Handke, C., Peeters, R., Poort, J., Quintais, J. P., and Regeczi, D. (2018). Study on emerging issues on collective licensing practices in the digital environment. Final Report for the Directorate-General for Communications Networks, Content and Technology (DG CNECT) of the European Commission, SMART, 69. Davis, J., and Nathan, L. P. (2015). Value Sensitive Design: Applications, Adaptations, and Critiques, in: van den Hoven, J., Vermaas, P. E. and van de Poel, I. (eds.), Handbook of Ethics, Values, and Technological Design: Sources, Theory, Values and Application Domains. Springer Netherlands. Deutsch, M. (1975). Equity, equality, and need: What determines which value wil be used as the basis of distributive justice?. Journal of Social Issues, 31, 137-149. European Union. (2019). Directive (EU) 2019/790 of 17 April 2019 on copyright and related rights in the Digital Single Market and amending Directives 96/9/EC and 2001/29/EC. Official Journal of the European Union. Forsyth, D. R. (2006). Conflict In Group Dynamics. Wadsworth Cengage Learning, 5, 388–389. Friedman, B., Kahn, P. H., Jr., Borning, A., Huldtgren, A. (2013). Value sensitive design and information systems. In: Doorn, N., Schuurbiers, D., Van de Poel, I., Gorman, M. E. (Eds.). Early Engagement and New Technologies: Opening up the Laboratory. Springer, Dordrecht, The Netherlands. Friedman, B., Hendry, D. G. (2019). Value sensitive design: Shaping technology with moral imagination. Mit Press. Handke, C., Towse, R. (2007). Economics of copyright col ecting societies. International Review of Intellectual Property and Competition Law, 38(8), 937-957. Hesmondhalgh, D., Osborne, R., Sun, H., Barr, K. (2021). Music creators’ earnings in the digital era. Intel ectual Property Office Research Paper Forthcoming. 782 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION International Labour Organization. (2023). Decent work. URL: https://www.ilo.org/global/topics/decent-work/lang–en/index.htm (visited on November 12, 2023) Jenkins, K. E., Spruit, S., Milchram, C., Höffken, J., Taebi, B. (2020). Synthesizing value sensitive design, responsible research and innovation, and energy justice: A conceptual review. Energy Research & Social Science, 69, 101-727. Lomerson, W., Wingreen, S. (2009). Improving End User Value in Information Technology Projects: Exploring The Benefits of Q-sort Analysis. In: SAIS 2009 Proceedings, 24. Manders-Huits, N. (2011). What Values in Design? The Chal enge of Incorporating Moral Values into Design. Science and Engineering Ethics, 17 (2), 271–287. Merriam, S. B., Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation. John Wiley & Sons. Nagel, E., Kranz, J. (2021). Is' Chained' Property the Future? A Review and Synthesis of Literature on Blockchain Technology and Property Rights. In: ECIS 2021 Research Papers. Priest, E. (2021). The Future of Music Copyright Collectives in the Digital Streaming Age. In: Colum. JL & Arts, 45 (1). Rawls, J. (1999). A Theory of Justice, Revised Edition. Harvard University Press. Simon, J. (2016). Value-Sensitive Design and Responsible Research and Innovation. In: Hansson, S.- O. (eds.) The Ethics of Technology - Methods and Approaches. London: Rowman & Littlefield International Singer, H., Rosenblatt, B. (2023). Key Changes: The Ten Times Technology Transformed the Music Industry. Oxford: University Press. Van de Kaa, G., Rezaei, J., Taebi, B., Van de Poel, I., Kizhakenath, A. (2019). How to weigh values in value sensitive design: A best worst method approach for the case of smart metering. Science and Engineering Ethics, 25(6), 1657–1680. Van de Poel, I. (2013). Translating values into design requirements. In: Michelfelder, D. P., McCarthy, N. and Goldberg, D. E. (eds.) Philosophy and Engineering: Reflections on Practice, Principles and Process. Dordrecht: Springer. Van Wynsberghe, A., Robbins, S. (2014). Ethicist as designer: A pragmatic approach to ethics in the lab. Science and Engineering Ethics, 20, 947-961. WIPO. (2022). Collective Management of Copyright and Related Rights. 3rd edition. Geneva: WIPO. Yakubova, I.B. (2022). Understanding col ective management of copyright in historical context. E Conference Zone, 260–263. DOCTORAL CONSORTIUM DESIGNING A PROCESS MODEL OF INTEGRATED LIFELONG PROVISION OF HEALTHCARE TO PATIENTS ŽIVA RANT, TOMAŽ KERN University of Maribor, Faculty of Organizational Sciences, Kranj, Slovenia ziva.rant@student.um.si, tomaz.kern@um.si During a period of treatment, patients come into contact with a variety of health providers at different levels of the health system. Gaps can form between individual treatments. Judging from experiences in other sectors, such as industry, viewing patient care as a process could be one of the factors in a successful solution. The basic research method wil be the Design Science Keywords: Research approach. The research wil explore the intersection of healthcare business processes, healthcare provision and digital pr ovision, transformation. The result will be an artefact - a conceptual business process, organizational process model of the lifelong integration of organisational patient care. We would like to demonstrate that understanding process, healthcare provision as a lifelong organisational process has a organisational gap, significant positive effect on reducing organisational and information information gaps between different instances of treatment. gap DOI https://doi.org/10.18690/um.fov.4.2024.48 ISBN 978-961-286-871-0 784 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Healthcare is in crisis. On the one hand, we have an ageing population (Prebivalstvo - Slovenske regije in občine v številkah, 2023), which also means an increase in the number of patients needing healthcare. This requires an increasing number of medical treatments. While new treatment methods and new drugs help people live longer and enjoy better treatment outcomes, they also require ever greater financial investment. On the other hand, we are faced with limited resources: human (shortage of doctors and other healthcare workers), spatial and financial (Strategija razvoja zdravstvene dejavnosti na primarni ravni zdravstvenega varstva do leta 2031, 2024). We cannot influence demand or the limitations in any meaningful way. We therefore need to find greater efficiency within the health system. Our previous research (Rant, 2001) and other sources (Dumas et al., 2018; Hammer, 2015; Hammer & Champy, 1995, 2003; Keen, 1997; Keen & Knapp, 1995; Kern, 2022; Vila, 1994, 2000; Vila & Kovač, 2006) show that the organisational and information gaps that form between consecutive activities in traditional functional organisations could be reduced or even eliminated by a process-based form of organisation. We note a similar situation, and thus an opportunity, in healthcare. 2 Problem definition During a period of treatment, patients come into contact with a variety of health providers at different levels of the health system. Over the course of their lifetime, a patient can receive treatment from the fol owing: • at the primary level: general practitioner, family doctor or paediatrician, dentist, gynaecologist, physiotherapist, home nursing service, • at the secondary level: specialists in specialist clinics and hospitals (a patient may also move between hospital departments), health resorts, • at the tertiary level: the University Medical Centre (UKC) (where the patient may move between clinics and departments), and in these contexts: home help, day centres for the elderly, care homes, hospices. Ž. Rant, T. Kern: Designing a Process Model of Integrated Lifelong Provision of Healthcare to Patients 785 Organisational gaps and information gaps (hereinafter: gaps) can form between individual treatments. 1 On the basis of the reviewed literature, we wil focus on the fol owing problems in our research: P1 Organisational and informational gaps occur between treatments with different healthcare providers. (Amelung et al., 2021; Bürkle et al., 2017). P2 Attending health professionals are not always apprised of the activities and results of previous treatments (Amelung et al., 2021, p. 11; Bürkle et al., 2017). P3 Attending health professionals are not always apprised of the drugs that have been prescribed in the course of previous treatments or the drugs that the patient is currently taking. (Bürkle et al., 2017; Žerovnik et al., 2018). P4 The fragmented nature of healthcare promotes duplication and the inefficient use of resources, which leads to gaps in care for patients with multimorbidities and reduces the general capacity of the health sector, since it forces the best health professionals to focus on specific conditions (WHO Global Strategy on People-Centred and Integrated Health Services Interim Report, 2015). 2.1 Thesis Understanding healthcare provision as a lifelong organisational process has a significant positive effect on reducing organisational and information gaps between different instances of treatment. 2.2 Expected results The main result of the research wil be an artefact: Conceptual organisational process model of integrated lifelong provision of healthcare to patients. 1 In our research we understand gaps as organisational and information gaps. 786 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION In constructing this artefact, we asked the following research questions (RQs): RQ 1 Do organisational and information gaps occur between treatments with different providers? RQ 2 How do organisational and information gaps between treatments influence the effectiveness of patient care? RQ 3 How do organisational and information gaps between treatments influence the quality of patient care? RQ 4 How does process organisation influence (organisational and information) gaps between treatments? The doctoral thesis wil explore the intersection of business processes, healthcare provision and digital transformation (Fig. 1). Figure 1: The intersection of business processes, healthcare provision and digital transformation Source: Own The thesis’s original contribution to science is represented by a new comprehensive conceptual organisational process model of integrated lifelong provision of healthcare to patients, which represents an innovation in the field of organisational sciences and, at the same time, contributes to in-depth understanding of lifelong Ž. Rant, T. Kern: Designing a Process Model of Integrated Lifelong Provision of Healthcare to Patients 787 healthcare provision. The expected original scientific contribution made by the thesis wil be proof that changing our view of lifelong healthcare provision, as a process, has a significant positive effect on reducing or eliminating organisational and information gaps between different instances of treatment and, consequently, improves the effectiveness and quality of treatment, and thus of the health system. 3 Methodology The basic research method wil be the design science research methodology (DSRM) or design science research (DSR) (Hevner et al., 2004; Hevner, 2007, 2022; Kljajić Borštnar, 2022; Kuechler & Vaishnavi, 2008; Peffers et al., 2007, 2007; vom Brocke et al., 2020a, 2020a). The result of the doctoral thesis wil be an artefact – a conceptual organisational process model of lifelong integrated healthcare provision for patients. On the basis of the finding of the theoretical sources cited, we wil follow the design science research process model, adapted from (Kljajić Borštnar, 2022; Kuechler & Vaishnavi, 2008; Peffers et al., 2007; vom Brocke et al., 2020a), Fig. 2. The planning and development research process consists of the following phases: • identification of the problem and motivation, • definition of objectives and proposed solution, • design and development, • demonstration, • evaluation, • communication. 788 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Figure 2: The design science research process model, adapted from Peffers et al. (2007); vom Brocke et al. (2020b); Kuechler & Vaishnavi (2008). 3.1 Identification of the problem and motivation We wil begin by reviewing the relevant literature and in this way study theoretical starting points. This will allow us to identify problems from the literature. We will summarise findings in three fields. The first is patient care from a process point of view. The second is business process management (BPM). And the third is digital transformation. We will address the identified problems in the context of Slovenia’s health system, using the case study research methodology (Kljajić Borštnar, 2021; Yin, 2018). We wil research the accessibility and use of data on treatments and output documents Ž. Rant, T. Kern: Designing a Process Model of Integrated Lifelong Provision of Healthcare to Patients 789 on treatments with the help of a study of documentation and the use of real data within Slovenia’s health system. 3.2 Definition of objectives and proposed solution To construct the model we wil use the systems development life cycle methodology (Dennis et al., 2014; Valacich et al., 2017), Fig. 3. Figure 3: The systems development life cycle (Valacich et al., 2017) We wil compare the number of hospital treatments with the number of discharge letters in the Central Register of Patient Data (CRPD). We wil then compare these data with data on views of discharge letters. We wil also compare data on prescriptions. We wil propose solutions to identify problems. In this way we wil prove that the principles of system theory and business process management apply to the integrated lifelong provision of healthcare – and demonstrate how they do. 3.3 Design and development On the basis of the findings from the previous step, we wil develop, as the key part of the research, a conceptual organisational process model of lifelong continuity of patient care, using the design science research approach (Hevner, 2007; Hevner et al., 2004; vom Brocke et al., 2020b), Fig. 2. 790 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3.4 Demonstration We wil present the developed model and provide answers to the research questions. We wil show an example of the use of the model in the case of gaps between hospital treatments and the continuation of treatment following discharge from hospital. 3.5 Evaluation We wil evaluate the model through a process of confirmation by experts from relevant domains. We will use structured interviews in this phase. 3.6 Communication We will publish the results as a scholarly article in a reputable journal, give presentations at scientific conferences and present the results to key stakeholders. 4 Preliminary/expected results The results of the research to date are presented below. 4.1 Identification of the problem and motivation We reviewed relevant literature on patient care from a process point of view, business process management and digital transformation. We identified the following problems from the literature: P1 Organisational and informational gaps occur between treatments with different healthcare providers (Amelung et al., 2021; Bürkle et al., 2017). P2 Attending health professionals are not always apprised of the activities of previous treatments (Amelung et al., 2021, p. 11; Bürkle et al., 2017). P3 Attending health professionals are not always apprised of the drugs that have been prescribed in the course of previous treatments or the drugs that the patient is currently taking (Bürkle et al., 2017; Žerovnik et al., 2018). Ž. Rant, T. Kern: Designing a Process Model of Integrated Lifelong Provision of Healthcare to Patients 791 P4 The fragmented nature of healthcare promotes duplication and the inefficient use of resources, which leads to gaps in care for patients with multimorbidities and reduces the general capacity of the health sector, since it forces the best health professionals to focus on specific conditions ( WHO Global Strategy on People-Centred and Integrated Health Services Interim Report, 2015). We identified gaps between different instances of treatment in the literature. Through our research, we wil analyse these gaps and show, through the development of a conceptual model, how it is possible to reduce or eliminate these gaps. We intend to transfer findings from the theory of business processes, which are general y applicable, to the field of the organisation of health systems. The treatment of a patient can be viewed as a process, while individual treatments can be viewed as phases and activities in this process. Here it is also necessary to define the process owner. If we observe the integrated lifelong provision of healthcare to patients, we can understand the individual elements in the above definition as follows: • The following can be understood as inputs: o health professionals – GPs, specialists, nurses, home care nurses, physiotherapists, care workers; o information – referrals, results, discharge letters, recommendations, prescriptions, procedures, treatment results. • In this case, work activities are instances of treatment at different levels of healthcare – at the primary level, in specialist clinics and in hospital. • The end result is the outcome of treatment. • The client is the patient – newborn, patient, person taking part in preventive treatment. The process is managed by the patient, who decides on their own treatment within the relevant legal and professional frameworks. The process is led by the process owner. We treat the general practitioner, in connection with the patient and the patient’s relatives, as the process owner. Here, the general practitioner operates 792 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION according to the “case manager” principle and leads the overal treatment process. For more in-depth activities, the GP authorises specialists in other fields (with a referral, order form, work order). 4.2 Definition of objectives and proposed solution We assume that organisational and information gaps exist between different instances of treatment. We intend to confirm this. We intend to prove that this is bad for the patient and a weakness of the health system. We also intend to identify how gaps form and show how they could be reduced or even eliminated. Judging from experiences in other sectors, such as industry (Dumas et al., 2018; Hammer, 2015; Hammer & Champy, 1995, 2003; Keen, 1997; Keen & Knapp, 1995; Kern, 2018, 2022; Krhač & Kern, 2018; Urh et al., 2022), viewing patient care as a process could be one of the factors in the effective organisation and efficiency of the health system. It could increase quality of treatment and reduce unnecessary treatments, duplication of tests and total treatment time, thereby improving treatment outcomes, reducing the number of hospitalisations and, last but not least, bringing down costs. In order to analyse the state of the problem, we obtained real data from Slovenia’s health system. We found that analysis of col ected data on treatments wil make a significant contribution to proving the existence of gaps between treatments in practice. We obtained real anonymised data which, however, were not col ected for the purpose of the research, so there is a possibility that some data wil not be useful for the research and wil therefore not be used. We obtained real anonymised data on hospital treatments from the National Institute of Public Health (NIJZ) and real anonymised data on hospitalisations from the Health Institution Institute of Slovenia (ZZZS). We obtained real anonymised data on documents of the discharge letter type from the administrative module of the CRPD in the context of the eHealth system and data on prescriptions from the ePrescription database within the eHealth system. In order to analyse data on views, we will obtain audit track data from the administrative modules of the CRPD and ePrescription. In all cases we will use data for 2022. Ž. Rant, T. Kern: Designing a Process Model of Integrated Lifelong Provision of Healthcare to Patients 793 We wil conduct the analysis by comparing the number of hospital treatments with the number of discharge letters in the CRPD. We wil then compare these data with data on views of discharge letters (Fig. 4). Figure 4: Comparison of data on discharge letters The effectiveness of the treatment process and, consequently, the successfulness of treatment in a selected period, is frequently dependent on previous treatments and the information on these treatments that is accessible and used in the selected period (Bürkle et al., 2017; Stevens et al., 2022). Access to data and information on previous treatments must be guaranteed in every instance of treatment. This is technical y facilitated by established and functioning IT infrastructure and a single up-to-date repository. Data and information must be provided by the healthcare provider responsible for each treatment. Available data and information must also be used by the provider of the next treatment. Information on previous treatments must be: • accessible (if it is not accessible it cannot be used), • used (information is not used despite being accessible and therefore available to be used). 794 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4.3 Design and development On the basis of the findings to date, we prepared a draft model. The model as we now understand it represents an il ustration of the desired state (TO-BE) (Fig. 5). In it, we show a sequence of treatment instances as an organisational process from birth to death. In the real world this represents the life of an individual and treatments in the health system throughout their life. Figure 5: Conceptual organisational process model of integrated lifelong provision of healthcare to patients (Rant, 2020) During our research we established that, in order to design a new organisational model, it is important to research the information system associated with it. We find that: • For successful implementation of the treatment process, access to data and information about past treatments is necessary. • Access to relevant data and information about previous treatments must be provided to those providing current treatment at whatever level. • Treatment providers must use data and information about previous treatments. • A single repository is necessary, i.e. a central EHR. An example of this is Slovenia’s Central Register of Patient Data. This is shown in graphic form in Fig. 6. Ž. Rant, T. Kern: Designing a Process Model of Integrated Lifelong Provision of Healthcare to Patients 795 Figure 6: Integration and access to health data in a single repository, the central HER Source: Own Such a repository already exists in Slovenia. It is the Central Register of Patient Data (CRPD), in the context of the eHealth system. 4.5 Evaluation We wil evaluate the process model of integrated lifelong provision of healthcare to patients from the point of view of experts from relevant domains. The evaluation wil make use of structured interviews. The experts/interviewees are expected to be doctors and nurses from the fields covered by the model. We plan to divide them into three groups: • experts in the public health field, who generally have a comprehensive view of treatment provision. • experts in the field of hospital care and specialists, who in principle provide data, information and documents on treatments, • experts in family medicine, who use information and documents from prior treatments. We wil conduct 3–5 interviews with representatives of each group of experts. Outline interview content: • we wil explain the basic concepts, purpose, progress and results of the quantitative part of the research to the interviewees, 796 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • we wil verify whether they perceive gaps between successive instances of treatment, • we wil ask their opinion on whether medical documentation from previous treatments is needed when providing treatment, • wil verify whether they use medical documentation from previous treatments and in what form, • we wil verify whether they forward medical documentation on treatments that they provide, and in what form, • we wil ask them what they think about the integrated lifelong provision of healthcare and ask for their comments on the process model of integrated lifelong provision of healthcare to patients, • we wil verify whether they believe that the organisational model we have presented reduces gaps between different instances of treatment, • we wil verify whether they believe that a single repository helps reduce gaps between different instances of treatment. 5 Future development Our research is focused on the organisational field. 1. We wil model and analyse the system and processes of the existing situation (AS-IS). We wil do so using the business process management (BPM) methodology and Aris 10 software. 2. We will analyse real data obtained from the Slovenia’s health system. 3. We wil prove the existence of organisational and information gaps between hospital treatments and primary care treatments. 4. We wil design an artefact – a conceptual organisational process model of lifelong integrated healthcare provision for patients – that wil be based on theoretical findings, process analysis, analysis of data obtained, the existence of organisational and information gaps, and practical experience. 5. As well as researching activities and transitions between activities in the process in the organisational sense, we wil carry out detailed research of the information system associated with them. 6. We wil show how the proposed model affects organisational and information gaps between instances of treatment. Ž. Rant, T. Kern: Designing a Process Model of Integrated Lifelong Provision of Healthcare to Patients 797 7. We will publish our findings in a reputable scientific journal. The results obtained from the research wil contribute to an in-depth understanding of the integrated lifelong provision of healthcare to patients. References Amelung, V., Stein, V., Suter, E., Goodwin, N., Nolte, E., Ran, ·, & Editors, B. (2021). Handbook Integrated Care Second Edition. Bürkle, T., Denecke, K., Lehmann, M., Zetz, E., & Holm, J. (2017). Integrated care processes designed for the future healthcare system. Studies in Health Technology and Informatics, 245, 20– 24. https://doi.org/10.3233/978-1-61499-830-3-20 Dennis, A., Wixom, B. H., & Roth, R. M. (2014). Systems analysis and design. John Wiley & Sons. http://www.saigontech.edu.vn/faculty/huynq/SAD/Systems_Analysis_Design_UML_5th ed.pdf Dumas, M., La Rosa, M., Mendling, J., & Reijers, H. A. (2018). Fundamentals of business process management: Second Edition. Fundamentals of Business Process Management: Second Edition, 1– 527. https://doi.org/10.1007/978-3-662-56509-4/COVER Hammer, M. (2015). What is business process management? Handbook on Business Process Management 1: Introduction, Methods, and Information Systems, 3–16. https://doi.org/10.1007/978-3-642-45100-3_1 Hammer, M., & Champy, J. (1995). Preurejanje podjetja : manifest revolucije v poslovanju. Gospodarski vestnik. Hammer, M., & Champy, J. (2003). Reengineering the Corporation: A Manifesto for Business Revolution (Col ins Business Essentials). 272. http://www.amazon.com/Reengineering-Corporation-Manifesto-Revolution-Essentials/dp/0060559535 Hevner, A. R. (2007). A Three Cycle View of Design Science Research. Scandinavian Journal of Information Systems, 19(2). Hevner, A. R. (2022). Design science research. Computing Handbook: Two-Volume Set, 1–23. https://doi.org/10.1201/B16768-26 Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly: Management Information Systems, 28(1), 75–105. https://doi.org/10.2307/25148625 Keen, P. G. W. (1997). The Process Edge : Creating Value Where It Counts. Harward Business School Press. Keen, P. G. W., & Knapp, E. M. (1995). Every manager’s guide to business processes : a glossary of key terms & concepts for today’s business leader : a glossary of key terms & concepts for today’s business leader. Harvard. Kern, T. (2018). Management projektov : specialistični program Organizacija in management. Fakulteta za organizacijske vede. Kern, T. (2022). Management poslovnih procesov in projektov. Kljajić Borštnar, M. (2021). Raziskovanje informacijskih sistemov. Kljajić Borštnar, M. (2022). Znanstveno proučevanje družboslovnih pojavov. Krhač, E., & Kern, T. (2018). Kazalniki strukturne učinkovitosti za ovrednotenje procesov v zdravstveni oskrbi = Structural efficiency indicators for evaluating processes in healthcare. In O. Arsenijević (Ed.), Organizacija in negotovosti v digitalni dobi (p. Str. 505-525). Univerzitetna založba Univerze. http://press.um.si/index.php/ump/catalog/book/326 798 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Kuechler, B., & Vaishnavi, V. (2008). On theory development in design science research: anatomy of a research project. European Journal of Information Systems, 17(5), 489–504. https://doi.org/10.1057/ejis.2008.40 Peffers, K., Tuunanen, T., Rothenberger, M. A., & Chatterjee, S. (2007). A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, 24(3), 45–77. https://doi.org/10.2753/MIS0742-1222240302 Prebivalstvo - Slovenske regije in občine v številkah. (2023). https://www.stat.si/obcine/sl/Theme/Index/PrebivalstvoIndeks Rant, Ž. (2001). Kontinuirano učenje kot stalnica v procesni organizaciji : magistrska naloga. [Ž. Rant]. Rant, Ž. (2020). Tehnologija je tu, kaj zdaj ? In P. Šprajc (Ed.), 39th International Conference on Organizational Science Development (pp. 629–643). University Press. https://press.um.si/index.php/ump/catalog/view/503/613/1097-1 Stevens, G., Hantson, L., Larmuseau, M., & Verdonck, P. (2022). A human-centered, health datadriven ecosystem. Discover Health Systems 2022 1:1, 1(1), 1–12. https://doi.org/10.1007/S44250-022-00011-9 Strategija razvoja zdravstvene dejavnosti na primarni ravni zdravstvenega varstva do leta 2031. (2024). Ministrstvo za zdravje. Urh, B., Kern, T., & Krhač Andrašec, E. (2022). Model ocenjevanja učinkov organizacijskih sprememb. 1011– 1025. https://doi.org/10.18690/UM.FOV.3.2022.73 Valacich, J. S., George, J. F., Columbus, B., New, I., San, Y., Cape, F. A., Dubai, T., Madrid, L., Munich, M., Montréal, P., Delhi, T., São, M. C., Sydney, P., Kong, H., Singapore, S., & Tokyo, T. (2017). Modern Systems Analysis and Design 8th Edition. www.pearsoned.com/permissions/. Vila, A. (1994). Organizacija in organiziranje. Moderna organizacija. Vila, A. (2000). Organizacija v postmoderni družbi. Moderna organizacija. Vila, A., & Kovač, J. (2006). Osnove organizacije in managementa (1. dopolnjena izd.). Moderna organizacija. vom Brocke, J., Hevner, A., & Maedche, A. (2020a). Introduction to Design Science Research. 1–13. https://doi.org/10.1007/978-3-030-46781-4_1 vom Brocke, J., Hevner, A., & Maedche, A. (2020b). Introduction to Design Science Research. In J. vom Brocke, A. Hevner, & A. Maedche (Eds.), Design Science Research. Cases (pp. 1–13). Springer International Publishing. https://doi.org/10.1007/978-3-030-46781-4_1 WHO global strategy on people-centred and integrated health services Interim Report. (2015). www.who.int Yin, R. K. (2018). Case Study Research and applications, 6th edition. Paper Knowledge . Toward a Media History of Documents, 414. Žerovnik, Š., Locatelli, I., & Kos, M. (2018). Brezšivna skrb pri zdravljenju z zdravili v Sloveniji. Farmacevtski vestnik, 69(3). DOCTORAL CONSORTIUM MEASURING SMART PUBLIC GOVERNANCE MATURITY IN PUBLIC ADMINISTRATION INSTITUTIONS: A MULTI-ATTRIBUTE APPROACH PETRA VUJKOVIĆ1,2 University of Ljubljana, Faculty of Public Administration, Ljubljana, Slovenia petra.vujkovic@fu.uni-lj.si University of Rijeka, Faculty of Economic and Business, Rijeka, Croatia petra.vujkovic@fu.uni-lj.si The paper conceptualizes a multi-attribute model based on the design science approach for measuring smart public governance maturity in public administration institutions. This is achieved on the basis of a systematic literature review (i.e. content analysis) of Keywords: Web of Science and Scopus records. The SPG attributes or design science criteria elicited from these two databases are integrated into a research; decision support model, thus setting the layout of a multi- public attribute model for measuring smart public governance maturity administration institutions; in public administration institutions. The final model maturity; conceptualized here consists of 29 attributes or criteria grouped multi-attribute into four categories: (ICT)-enabled governance innovation in the model; smart public sector, inter-organisational changes, changes in citizen- public government interaction, and outcome–public value creation. governance DOI https://doi.org/10.18690/um.fov.4.2024.49 ISBN 978-961-286-871-0 800 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction and Problem Definition In this paper, we present a design of a multi-attribute model based on the design science approach for measuring smart public governance maturity in public administration institutions. The need for such a model emerges in a context where various governments and societies across the globe see a possibility to address rapidly changing socio-economic pressures and complex public policy problems by adopting the 'smart public governance' (SPG) concept (Šiugždiniene et al., 2017). However, the label SPG is a fuzzy concept (Lin, 2018; cf. Table 1 in Appendix for a selection of various SPG definitions), and the absence of a commonly accepted SPG definition makes measuring SPG difficult. For the purpose of this paper, the working definition of SPG is as follows: smart public governance is a process based on the use of ICT's or in today's times adoption of various smart "disruptive" technologies (e.g., big data, Internet of Things, and artificial intelligence) (Zhao & Zou, 2021; Zhu & Kou, 2019; Jiang et al., 2019; Pereira et al., 2018) to upgrade (modernise) traditional (bureaucratic or hierarchical) administrative systems (e.g., institutional change – dismantling old department/institutional silos) (Meuleman, 2021; Scott & Gong, 2021; Hansen, 2014; Bannister, 2001), involving new forms of multi-actor collaboration and participation (Popova & Popovs, 2023; Šiugždiniene et al., 2017; Bolivar & Meijer, 2016; Willke, 2007) in decision-making processes (Jiang et al., 2022; Örselli et al., 2022; Demirel & Mülazımoglu, 2021; Scholl & AlAwadhi, 2016), with a focus on outcomes (e.g., creating public value) (Criado & Gil-Garcia 2019; Webster & Leleux, 2018; Albino et al., 2015; Batagan 2011; Gil- Garcia, 2012). Such a holistic definition affirms that SPG must be seen as a transformative and socio-technical governance approach (Jucevicius & Juceviciene, 2018) and not solely techno-centric focused. Over the last decade, scholars in the smart city (SC) research domain have increasingly turned their attention to SPG, referring to smart city governance (SCG), smart urban governance, and/or smart local governance (e.g., Jiang et al., 2022; Criado & Gil-Garcia, 2019; Pereira et al., 2018; Scholl & AlAwadhi, 2016; Meijer & Bolivar, 2015). However, SPG does not follow the same maturity process as the SC (Anthopoulos et al., 2021). Therefore, there's a gap in research regarding its application and maturity in broader governance contexts – especial y in the context of public administration institutions. And, despite the sizeable opportunity of the SPG concept, chal enges for public governance systems to become or be smart P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 801 exceed the scope of their current capacities. A limited quantity of either theoretical frameworks, toolboxes or roadmaps and models to measure SPG have been put forward in literature by scholars (notably Ruijer et al., 2023; Lin, 2018; Šiugždiniene et al., 2017; Bolivar & Meijer, 2016; Scholl & Scholl, 2014). However, they are not directly useful (applied or practical) approaches for measuring the maturity of SPG. Therefore, a more holistic approach that identifies attributes (criteria) of SPG (as discussed in this paper) appears to be lacking. To provide the conceptual design for measuring SPG maturity in public administration institutions, this paper is guided by the following research question: which at ributes (also criteria) correspond to the aspect (i.e. the subject/area) should be considered when measuring the maturity of SPG in public administration institutions? By answering the research question, the paper endeavours to add clarity and rigor to the ongoing debate by proposing a practical tool, the multi-attribute model, designed to measure the maturity of SPG in public administration institutions. To provide an as clear as possible elaboration of our model's conceptual design, we have structured the paper into four sections. In the next section, we present the methodological framework based on content analysis of Web of Science and Scopus records. In addition, in the third section, we present the results of the content analysis studies – the list of SPG attributes (criteria), which are used for the structure of the multi- attribute model for measuring smart public governance maturity in public administration institutions. Finally, in the last section, we discuss open questions that need to be taken into consideration by future research. 2 Methodology 2.1 SPG attributes (criteria) 2.1.1 Content analysis of the Web of Science (WoS) and Scopus records In the search for an answer to the research question, a content analysis of the WoS and Scopus records was conducted in order to identify SPG attributes (criteria). The list of attributes (criteria) for the SPG multi-attribute model was gathered from: 802 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • A structured literature review, i.e., content analysis of WoS and Scopus papers. The selection of papers has been performed on the basis of the following criteria: o time-span of the records: 8 years, between 2015 and 2023, o including terms (in title, abstract and keywords): "smart public governance" OR "smart governance" OR "smart city governance" OR "smart urban governance" OR "smart local governance" OR "smart public administration" AND "indic*" OR "meas*" OR "defin*" OR "tool" OR "empirical analysis" OR "model" OR "framework", o type: article, o written in the English language. The content analysis focused on SPG and its associated/related terms. Despite representing different scopes of governance (e.g., state versus city or regional), these terms are often used synonymously (Vujković & Jukić, 2023). Restricting the analysis exclusively to records specifical y referencing SPG would have excluded significant data about SPG attributes (criteria). Using this criteria, 427 records were initially identified. After a thorough review to eliminate duplicates, 242 records remained. A screening process, which involved examining the introduction, literature review, and conclusions of each record, determined that 80.9% did not meet two crucial inclusion criteria – the paper needs to: • have definitions or interpretations of the following essential terms: smart public governance, smart city governance, smart urban governance, smart local governance, smart public administration, OR • coverage essential foundational components (building blocks), including attributes (criteria), indicators, elements, dimensions, and measurement tools (models, theoretical and conceptual frameworks). Ultimately, 46 records (10.7%) qualified for in-depth analysis. This stage involved a second, detailed review of the ful papers to extract definitions/interpretations of terms and a comprehensive list of SPG attributes for a multi-attribute model. On this basis, 29 subordinate and 13 single-parent SPG attributes (criteria) were P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 803 identified. Table 1 presents a detailed list of the SPG attributes (criteria) and information about the source of each attribute. Table 1: List of SPG attributes (criteria) according to their source Attribute (criteria) Source Lindgren & Veenstra (2018); Pereira et al. (2018); Lin 1. (ICT)-enabled governance innovation in the public sector (2018); Bolivar & Meijer (2016) 1.1. Present IT infrastructure – the basis for the use of emerging DESI (2023); Lin (2018); Bolivar & Meijer (2016); technologies in PA Scholl & Scholl (2014) Ruijer et al. (2023); Ronzhyn et al. (2019); Brennan et al. (2019); Lindgren & Veenstra 1.2. The use of emerging technologies in PA (2018); Lin (2018); Bolivar & Meijer (2016); Schol & Schol (2014) Mergel et al. (2023); Van Noordt & Misuraca (2022); 1.2.1. Artificial intelligence in PA – impact, use and presence Newman et al. (2022); Misuraca & Van Noordt (2020); Choi et al. (2021); Craglia et al. (2018) Savignon et al. (2024); Giest & Klievink (2024); Van Noordt & Misuraca (2022); Newman et al. (2022); Kolkman (2020); Misuraca & Van Noordt (2020); 1.2.1.1. The impact of artificial intelligence to automate routine Veale & Brass (2019); Mikalef et al. (2019); Brennan processes et al. (2019); Arntz et al. (2016); Kostoff (2004); Danziger & Andersen (2002); Bovens & Zouridis (2002) Savignon et al. (2024); Giest & Klievink (2024); Wil ems et al. (2023); Van Noordt & Misuraca (2022); Misuraca & Van Noordt (2020); Pencheva et 1.2.1.2. Using artificial intelligence to improve public services al. (2020); Okuyucu & Yavuz (2020); Veale & Brass, (2019); Mikalef et al. (2019); Veenstra et al. (2019); Kim & Cho (2017) Van Noordt & Misuraca (2022); Misuraca & Van Noordt (2020); Pencheva et al. (2020); Okuyucu & 1.2.1.3. Using artificial intelligence for decision-making Yavuz (2020); Veale & Brass (2019); Mikalef et al. (2019); Veenstra et al. (2019); Vydra & Klievink (2019); Kim & Cho (2017) 1.2.1.4. Use of virtual assistants (chatbots/talkbot/interactive Misuraca & Van Noordt (2020); Androutsopoulou et agent) to support the provision of information al. (2019); Cooper et al. (2018) 1.2.2. Cloud computing utilization in PA organisations – data lakes Ramos et al. (2023); Miloslavskaya & Tolstoy (2016) for data collection Favoreu et al. (2024); Bartelt et al. (2020); Tõnurist et 1.2.3. Presence of innovation labs for testing emerging technologies al. (2017) 1.3. Using data for the transformation of traditional PA to smart Ruijer et al. (2023); Newman et al. (2022); PA Fridriksson (2018); Schol in Schol (2014) 1.3.1. Big data in PA – state of art, use and effects Schol in Schol (2014) 1.3.1.1. Transformation of (traditional) data warehouses into an Dibouliya (2023); Bouaziz et al. (2017) efficient data warehouse Abuljadail et al. (2023); Campion et al. (2022); 1.3.1.2. Data administrators' skil s and expertise in handling big Kreuter et al. (2019); Sarker et al. (2018); Fridriksson data (2018) 1.3.1.3. The impact of big data on the policy cycle – from Van Noordt & Misuraca (2022); Pencheva et al. process orientation toward performance orientation (2020); Veale & Brass (2019); Mikalef et al. (2019); 804 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Attribute (criteria) Source Vydra & Klievink (2019); Höchtl et al. (2018); Lin (2018); Bolivar & Meijer (2016) Valle-Cruz & Garcia-Contreras (2023); Van Noordt & Misuraca (2022); Okuyucu & Yavuz (2020); Pencheva et al. (2020); Vydra & Klievink (2019); 1.3.1.4. The use of data analytics as support in decision-making Fridriksson (2018); Höchtl et al. (2018); Janssen et al. (2017); Šiugzdiniene et al. (2017); Becker (2016); Margel et al. (2016); Bolivar & Meijer (2016); Desouza & Jacob (2014) 1.3.2. Open and accessible data in PA Schol in Schol (2014) 1.3.2.1. Country open data policies and strategies Page et al. (2023) 1.3.2.2. Monitoring and measuring open data reuse and impact Page et al. (2023) 1.3.2.3. Assessing portal functions and features that enable users Page et al. (2023) to access open data 1.3.2.4. The quality of the (meta)data Page et al. (2023) Wil ems et al. (2023); Ruijer et al. (2023); Campion et al. (2022); Romansky & Noninska (2020); Lin (2018); 1.4. Privacy and cyber security Angelopoulos et al. (2017); Bolivar & Meijer (2016); Margel et al. (2016); Janssen & Hoven (2015); Schol & Schol (2014); Bertot & Choi (2013) Giest & Klievink (2024); Newman et al. (2022); 2. Inter-organisational changes Wimmer et al. (2020); Lin (2018); Bolivar & Meijer (2016); Schol & Schol (2014); Andersen et al. (2010) Ruijer et al (2023); Ruhlandt (2018); Lin (2018); 2.1. Renewal of structural arrangements Bolivar & Meijer (2016); Scholl & AlAwadhi (2016); Milakovich (2011) Giest & Klievink (2024); Newman et al. (2022); Meuleman (2021); Scott & Gong (2021); Scott 2.1.1. Establishment of connected organisational structure and (2020); Šiugzdiniene et al. (2017); Hansen (2014); dismantling old structures – institutional/department silos Navarra & Cornford (2005); Marche & McNiven (2003); Bovens & Zouridis (2002); Bannister (2001) Ruijer et al. (2023); Meuleman (2021); Tett (2014); 2.1.2. Col aborative culture – passage of mental silos ʺstate of mindʺ Cil iers & Greyvenstein (2012) Ruijer et al. (2023); Yahia et al. (2021); Ruhlandt (2018); Söderström et al. (2018); Lin (2018); 2.2. Streamlining intra-organisational processes – inter- Šiugzdiniene et al. (2017); Pereira et al. (2017); Schol departmental and inter-institutional col aboration & AlAwadhi (2016); Nam & Pardo (2014); Scholl & Schol (2014); Xiao et al. (2013); Chun et al. (2012) Interoperable Europe Academy (IOPEU Academy); 2.2.1. Established interoperable digital environment (or platform) for Ruijer et al. (2023); Šiugzdiniene et al. (2017); Bolivar inter-institutional and inter-departmental collaboration & Meijer (2016); Schol & Schol (2014) 2.2.2. Training and education opportunities for civil servants to Ruijer et al. (2023); Šiugzdiniene et al. (2017); Schol develop col aborative skil s & Schol (2014); Bouckaert et al. (2010) Rackwitz et al. (2024); Ruijer et al. (2023); Sørensen 2.2.3. Facilitate leadership – leaders act like policy entrepreneurs – et al. (2021); Giulio & Vecchi (2021); Meerkerk they promote new ideas, encourage innovations, and build trust in the (2019); Sørensen & Torfing (2019); Šiugzdiniene et team al. (2017); Torfing & Ansell (2016); Ansell & Gash (2012, 2008) Wimmer et al. (2020); Lin (2018); Šiugzdiniene et al. 3. Changes in citizen-government interaction (2017); Bolivar & Meijer (2016); Scholl & Scholl (2014); Andersen et al. (2010) Ruijer et al. (2023); Meerkerk (2019); Ruhlandt 3.1. Streamlining of external processes – collaboration and (2018); Lin (2018); Šiugzdiniene et al. (2017); Pereira participation et al. (2017); Bolivar & Meijer (2016); Schol & Schol P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 805 Attribute (criteria) Source (2014); Nam & Pardo (2014); Cano (2014); Chun et al. (2012) 3.1.1. Established collaboration tools for participation with external Ruijer et al. (2023); Šiugzdiniene et al. (2017); Bolivar stakeholders & Meijer (2016); Schol & Schol (2014) Riduan (2024); Guil aumie et al. (2024); Van Noordt & Misuraca (2022); Meerkerk (2019); Cardul o & 3.1.2. Col aborative decision-making –taking into account citizens' Kitchin (2019); Chen & Aitamurto (2019); Lin opinions and proposals (2018); Šiugzdiniene et al. (2017); Bolivar & Meijer (2016); Schol & Schol (2014); Cunha et al. (2013) Meynhardt (2022); Hartley et al. (2019); Neumann et al. (2019); Ruhlandt (2018); Lin (2018); Faulkner & Kaufman (2017); Scott et al. (2016); Bolivar in Meijer 4. Outcome – public value creation (2016); Pang et al. (2014); Benington & Moore (2011); Wil iams & Shearer (2011); Meynhardt (2009); Moore & Khagram (2004); Moore (1995) 4.1. Public service quality provision capability – improved Šiugzdiniene et al. (2017); Scott et al. (2016); Pang et efficiency and effectiveness of public service production al. (2014); Alford & Hughes (2008); Moore (1995) Rasmussen & Rehe (2023); Wilson & Knighton 4.2. Citizen engagement capability – public administrations (2021); Faulkner & Kaufman (2017); Pang et al. identify and respond more quickly to citizens' aspirations (2014); Talbot & Wiggan (2010); Stoker (2006) Rasmussen & Rehe (2023); Wilson & Knighton (2021); Faulkner & Kaufman (2017); Šiugzdiniene et 4.2.1. Citizen participation in policy-making and improved democracy al. (2017); Pang et al. (2014); Talbot & Wiggan (2010); Stoker (2006) Twizeyimanaa & Andersson (2019); Castro & Lopes 4.2.2. Increased transparency of public administration operations – (2022); Lin (2018); Šiugzdiniene et al. (2017); Bolivar citizens have better access to government information & Meijer (2016) Regal et al. (2024); Li & Shang (2023); Vrbek & Jukić (2023); Jukić & Vrbek (2023); Jukić et al. (2021); Torfing et al. (2021); Sørensen et al. (2021); Meerkerk 4.2.3 Co-creation capability delivering more inclusive public services (2019); Torfing et al. (2019); Twizeyimanaa & that are citizen-centred and tailored to citizens' needs Andersson (2019); Lindsay et al. (2018); Voorberg et al. (2014); Bovaird et al. (2014); Alves (2013); Hellang & Flak (2012); Jansen (2012); Bason (2010) Castro & Lopes (2022); Wilson & Knighton (2021); Twizeyimanaa & Andersson (2019); Pereira et al. 4.2.4. Trust and legitimacy – increase citizens' trust in public (2017); Šiugzdiniene et al. (2017); Faulkner & administration operations and recognition of legitimacy Kaufman (2017); Talbot & Wiggan (2010); Stoker (2006) Once the list of SPG attributes (criteria) was completed, we identified the common denominator of attributes (criteria), namely the subject/area (e.g., the role of ICT and emerging technologies, the evolving nature of data, internal structural and procedural adjustments, changes in public administration interactions with citizens, and outcomes). Hence, this aspect (i.e. the subject/area) was taken as the key criterion for the categorization of the SPG attributes (criteria) in the following four categories: 806 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • (ICT)-enabled governance innovation in the public sector; • inter-organisational changes; • changes in citizen-government interaction; • outcome – public value creation. 3 Preliminary Results and Discussion 3.1 Summary of attributes based on four categories 3.1.1 (ICT)-enabled governance innovation in the public sector In the general discourse on public administration (PA) transformation, the development of information and communication technology (ICT) is seen as an enabler or even a driver of (digital) transformation (Lindgren & Veenstra, 2018). To discuss what this means in practice in the case of PA, 'current IT infrastructure, the use of new emerging "disruptive" technologies, data, privacy and cyber security' have been identified as attributes (criteria) of the first category '(ICT)-supported innovation in public sector'. In the past, the public sector has lagged behind the private sector in ICT adoption (Wimmer et al., 2020; Ndou, 2004). Therefore, despite some technological advances in PA (both in research and in practice), knowledge of the actual effects of digitisation on changes in PA (resulting from the development and use of ICT) remains scarce. In this regard, Meijer (2014) noted that most scholars from the social science domain do not pay enough attention to the role of ICTs in PA. They seem to be rather reserved in this context, and research on the impact of ICTs on PA remained limited. But the development of new emerging "disruptive" technologies – which have emerged in recent years – has forced PA to transform (Ronzhyn et al., 2019). Examples of technologies that "disrupt" 1 the traditional or bureaucratic (hierarchical) approach of public sector operations most commonly includes artificial intelligence (AI), machine learning and big data (Brennan et al., 2019). For 1 "disrupt" means causing major technology-related shifts and, therefore, interrupting established processes and operations. This may be caused because of a new combination of existing technologies or entirely new technologies that are becoming integrated into PA (Brennan et al., 2019; Kostoff et al., 2004). P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 807 public sector organisations, this means changes in internal structures and processes and changes in the way PA interact with citizens (Wimmer et al., 2020; Andersen et al., 2010). Changes in the public sector are, therefore, very much linked to the development of ICT. Although the use of ICT in PA is becoming more common, from the relatively simple automation of routine work by civil servants (screen-level bureaucrats) to more sophisticated and complex applications that support the performance of administrative tasks (Danziger & Andersen, 2002; Bovens & Zouridis, 2002), both scholars and practitioners commonly agree that the evidence of actual improvements in the performance of PA remains stil rather scarce (Misuraca & Viscusi, 2015). An important task of al PA organisations is the collection, processing, storage and sharing of information (Janssen et al., 2017; Janssen & Hoven, 2015). And, while public sector information is the main source of big data, PA is the main storehouse of such data. Citizens, in turn, play the role of big data generators (Abuljadail et al., 2023). It is important to note that the public sector had been collecting, storing, and processing data for at least a decade before the development of ICT. However, the rapid development of ICT (and the rise of emerging "disruptive" technologies) has increased the amount of this data, bringing new chal enges to the public sector (Fredriksson et al., 2017). And while the private sector is making significant progress in the use and analysis of big data, the public sector seems to be fal ing behind once again (Rogge et al., 2017; Munne, 2016; Desouza & Jacob, 2014). However, public sector stakeholders have already recognised that better use of big data would bring many benefits to the public sector (Klievink et al., 2017; Munne, 2016). Key ones include improving the efficiency and effectiveness of PA organisations, improving public service delivery and better support in data-driven decision-making (Pencheva et al., 2020; Okuyucu & Yavuz, 2020; Veenstra et al., 2019; Kim & Cho, 2017). However, the major problem that big data brings to the public sector seems to be its governance (Chen & Hsieh, 2014). PA organisations face the chal enges of a lack of analysts who know how to process and analyse information in real-time and outdated technological equipment for processing and storing big data (Abuljadail et al., 2023; Sarker et al., 2018). In addition, the huge amounts of data col ected within the public sector are typical y fragmented (or localised) within PA institutions and their departments. And because PA organisations operate as separate departmental and functional bureaucratic units 808 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION (silos), the fragmentation of data sources (and the related lack of data sharing) hinders the use of big data for model ing, real-time problem analysis and support for data-driven decision-making (Okuyucu & Yavuz, 2020; Janssen et al., 2017; Becker, 2016; Margel et al., 2016; Desouza & Jacob, 2014). In addition, the rapid growth of ICT has changed (i.e., transformed) access to information. Whereas in the past, public records were available and not exactly accessible to the general public, today, we live in an era where data is freely accessible (Margel et al., 2016; Janssen & Hover, 2015). This shift towards open data enables new ways of collaboration between governmental and non-governmental stakeholders, as wel as civil society (citizens), which in the digital age expects public services to be more efficient and PA organisations to be more responsive and transparent in their operations. However, despite the public sector opening up its databases to the public and thereby contributing to democratic principles, understanding the impact of open data on the public sector remains surprisingly narrow (Meijer et al., 2014). Recently, breakthroughs in machine learning and the amount and availability of data have encouraged PA organisations to consider integrating (or adopting) AI into their processes and activities (Mergel et al., 2023; Craglia et al., 2018). Yet, although several fields are involved in AI research, very little is known about the use of AI in PA (Bailey & Barley, 2019). As Van Noordt and Misuraca (2022) note, a negligible number of public sector organisations in Europe have already started to use AI. And while this is partly due to the chal enges of introducing new "disruptive" technologies into established work processes (Kolkman, 2020), Margetts and Dorobantu (2019), on the other hand, argue that PAs have not even begun to engage with AI in a comprehensive and consistent way. The situation is different in the academic realm, where "optimism" prevails about the impact of AI on PA, as its implementation is expected to bring tangible improvements in the functioning of PA organisations. The biggest advances are expected to be in the large-scale automation of routine processes, more efficient delivery of public services, increased efficiency in data-driven decision-making and related improvements in public policymaking (Van Noordt & Misuraca, 2022; Veale & Brass, 2019; Mikalef et al., 2019). Indeed, AI technology in public policymaking follows the traditional public policy-making cycle, with the difference that public P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 809 policymaking is data-based and policy decision-making, therefore, becomes more data-driven and based on better analytics that is accurate and less uncertain (Van Noordt & Misuraca, 2022; Vydra & Klievink, 2019). Key benefits of using AI in this context include not only increased efficiency and effectiveness of PA organisations but also increased legitimacy in public policymaking processes (Van Noordt & Misuraca, 2022; Pencheva et al., 2020). As AI also enables new ways of collaboration (participation) between government stakeholders and civil society (citizens), its use is expected to contribute to a more open and participatory policymaking process. This would, therefore, facilitate the analysis and consideration of public opinions, views and demands of citizens, al owing policymakers to better address societal needs and preferences (Van Noordt & Misuraca, 2022; Cardullo & Kitchin, 2019; Chen & Aitamurto, 2019). Although we have discussed some promising theoretical perspectives on AI in PA, we should point out that there is currently stil no solid empirical basis to support these theories (Van Noordt & Misuraca, 2022; Kuziemski & Misuraca, 2020). The problems and chal enges associated with using AI often overshadow its benefits (Zuiderwijk et al., 2021). Theoretical studies on AI that dominate the public sector governance literature (Mergel et al., 2023), therefore, often focus on specific AI applications, such as the use of virtual assistants (e.g., chatbots/talkbot/interactive agent) that provide information and assistance to users of government platforms (Androutsopoulou et al., 2019). Or on discussions about the potential challenges that surround the adoption of AI, including gaps in the skills required and risks to data privacy and security (Campion et al., 2022). Therefore, with the obvious development of ICT and the associated benefits, privacy and security concerns should not be overlooked (Margel et al., 2016; Bertot & Choi, 2013). Privacy and security are constantly in motion due to the development of emerging "disruptive" technologies (Romansky & Noninska, 2020). While privacy represents a fundamental value for maintaining democracy, unrestricted data collection can threaten democratic principles (Romansky & Noninska, 2020; Janssen & Hoven, 2015). In light of this, PA organisations must follow privacy and security policies that align with legislation when processing data col ected from citizens (Angelopoulos et al., 2017). And if the fragmentation (or localisation) of the PA into separate departmental and functional units (silos), on the one hand, hinders the use of big data for model ing, real-time problem analysis and (policy) decision support 810 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION in terms of privacy and security, it allows storing data in isolated departments – commonly referred to as silos. This prevents unauthorised access by civil servants to all the information collected (i.e., citizen's data), further enhancing data security (Janssen & Hoven, 2015). Table 2: Attributes (criteria) of the model based on the first category, '(ICT)-enabled governance innovation in the public sector' Attributes deriving from the (ICT)-enabled governance innovation Attribute in the public sector number Present IT infrastructure – the basis for the use of emerging technologies 1.1. in PA) The use of emerging technologies in PA) 1.2. Artificial intelligence in PA – impact, use and presence) 1.2.1. The impact of artificial intelligence to automate routine processes) 1.2.1.1. Using artificial intelligence to improve public services) 1.2.1.2. Using artificial intelligence for decision-making) 1.2.1.3. Use of virtual assistants (chatbots/talkbot/interactive agent) to support 1.2.1.4. the provision of information) Cloud computing utilization in PA organisations – data lakes for data 1.2.2. collection) Presence of innovation labs for testing emerging technologies) 1.2.3. Using data for the transformation of traditional PA to smart PA) 1.3. Big data in PA – state of art, use and effects) 1.3.1. Transformation of (traditional) data warehouses into an efficient data 1.3.1.1. warehouses) Data administrators' skil s and expertise in handling big data) 1.3.1.2. The impact of big data on the policy cycle – from process orientation 1.3.1.3. toward performance orientation) The use of data analytics as support in decision-making) 1.3.1.4. Open and accessible data in PA) 1.3.2. Country open data policies and strategies) 1.3.2.1. Monitoring and measuring open data reuse and impact) 1.3.2.2. Assessing portal functions and features that enable users to access open 1.3.2.3. data) The quality of the (meta)data) 1.3.2.4. Privacy and cyber security) 1.4. P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 811 It is true, however, that we live in a digital age where data can be accessed from many sources. The volume of information collected about individuals and organisations is changing "traditional" understandings of privacy and security. As a result, privacy and security are being transformed, with younger generations now revealing much more about their lives than older generations, which in turn leads to a changed perception of what is considered private and secure in the digital age (Janssen & Hoven, 2015). 3.1.2 Inter-organisational changes 'Renewal of structural arrangements' and 'streamlining intra-organisational processes – inter-departmental and inter-institutional collaboration' have been identified as attributes (criteria) of the second category of 'intra-organisational change' (Ruhlandt, 2018; Scholl & AlAwadhi, 2016). Many PA organisations typically operate as traditional bureaucracies, organising their tasks by breaking down complex issues into simpler sub-problems. These sub- problems are then handled by separate departmental and functional units (silos). Civil servants usually remain confined within a single silo, which, while providing a structured work environment, can hinder effective collaboration both internal y and external y. The latter is why the word 'silo' general y has a negative connotation (Meuleman, 2021; Scott & Gong, 2021; Hansen, 2014; Bannister, 2001). It is important to note that 'silo' is not limited only to physical structures (department) but can also refer to a mental attitude (Tett, 2014), leading to tunnel vision and resistance to change among civil servants who believe their approach is not only the best but also the sole solution to the problem (Meuleman, 2021). In Europe, recognising the issues arising from silos has been a driving force behind PA reforms (Scott, 2020; Navarra & Cornford, 2005). Therefore, structural change represents a shift from the traditional 'silo' approach, fostering the creation of connected organisational structures and promoting a culture of col aboration (passage of mental silos ʺstate of mindʺ). 812 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Intra-organisational change is also linked in the literature to the streamlining (or improvement) of processes. It il ustrates the response of public sector organisations to modernisation, including digitisation (i.e. changes, transformations or improvements implemented within the public organisation itself) and its consequences (i.e. innovations in the way public services are delivered, as well as changes in the way citizens interact with public sector organisations). Table 3: Attributes of the model based on the second category, 'inter-organisational change' Attribute Attributes deriving from the inter-organisational change number Renewal of structural arrangements 2.1. Establishment of connected organisational structure and dismantling old 2.1.1. structures – institutional/department silos Collaborative culture – passage of mental silos ʺstate of mindʺ 2.1.2. Streamlining intra-organisational processes – inter-departmental and inter- 2.2. institutional collaboration Established interoperable digital environment (or platform) for inter- 2.2.1. institutional and inter-departmental collaboration Training and education opportunities for civil servants to develop 2.2.2. collaborative skills When considering organisational change within the public sector, it is essential to highlight the significance of collaboration among different departments and organisations within the sector. This form of collaboration, known as internal collaboration, involves coordinated and cooperative efforts between various departments within an organisation (inter-departmental collaboration) or between different PA organisations (inter-institutional collaboration) (Pereira et al., 2017). Similarly, Yahia et al. (2021) use the term "collaborative networks" to describe various forms of internal collaboration. The government (as executive authority) plays a crucial role in pursuing streamlining (or improving) these collaborative efforts. By establishing a digital environment or platforms, the government can facilitate and encourage mutual collaboration and the exchange of information and knowledge between PA organisations and their departments (Šiugzdiniene et al., 2017). As Bouckaert et al. (2010) pointed out over a decade ago, successful internal collaboration also hinges on adopting collaborative approaches and cohesive working methods among civil servants. Consequently, it is imperative to organise P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 813 regular education and training programs for civil servants, particularly developing collaborative competencies (skills), as highlighted in recent literature (Šiugzdiniene et al., 2017). 3.1.3 Changes in citizen-government interaction 'Streamlining of external processes – collaboration and participation' has been identified as an attribute (criteria) of the third category of 'changes in citizen- government interaction' (Šiugzdiniene et al., 2017). External collaboration involves the participation of governmental and non- governmental stakeholders, such as private stakeholders, academic institutions, and civil society (Ruhlandt, 2018; Pereira et al., 2018). External participation, therefore, refers to the participation of citizens – as individual forms of participation (citizens who are either individual y participating) or in more or less organised groups (associative forms) (Meerkerk, 2019). External participation is considered a (dynamic) process (Ruhlandt, 2018; Bolivar & Meijer, 2016; Cano, 2014) that has been shaped or created by technology for a decade. It has enabled new ways of consultation and dialogue (or reflection) between government and citizens and changed (or transformed) citizen participation in consultation and decision-making processes (Cunha et al., 2013). Collaboration between government and citizens can be initiated from both top- down (government-driven) and bottom-up (citizen-driven) approaches, as explained by Meerkerk (2019). Regarding top-down collaboration, the literature frequently discusses two primary directions, (i) direct involvement of citizens in the design, implementation, and decision-making processes of public policies, and (ii) co- creation of public services, where citizens take on roles as co-implementers, co- designers, or initiators (Voorberg et al., 2014; Bovaird et al., 2014). In the context of top-down collaboration, the critical factors are citizens' trust, willingness, and motivation to participate. Conversely, bottom-up collaboration centres on community-led initiatives to address common needs. While Meerkerk (2019) emphasises the importance of facilitative leadership in this context, Šiugzdiniene et al. (2017) frame this process more regarding internal collaboration. According to them, facilitative leadership 814 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION involves supporting and enabling public servants to develop and apply their skils and competencies. Leaders in this role act as "political entrepreneurs" by encouraging innovation, promoting new ideas, tolerating mistakes, facilitating dialogue, and fostering trust within their teams. Table 4: Model's attributes based on the third category, 'changes in citizen-government interaction' Attributes deriving from changes in citizen-government interaction Attribute number Streamlining of external processes – collaboration and participation 3.1. Established collaboration tools for participation with external stakeholders 3.1.1. Collaborative decision-making –taking into account citizens' opinions and 3.1.2. proposals Facilitate leadership – leaders act like policy entrepreneurs – they promote 2.2.3. new ideas, encourage innovations, and build trust in the team 3.1.4 Outcomes – creating public value The concept of public value (PV) was initiated by Moore (1995), or rather by his idea of how to guide public managers in creating public value. His thinking was that public organisations should be equivalent to private organizations, where private managers create private (economic) value for their customers (Hartley et al., 2019; Benington & Moore, 2011). The symbol of this idea became a “strategic triangle”, which helps public managers focus their attention on three complex issues they need to consider before committing themselves and their organizations to a particular course of action (Benington & Moore, 2011; Moore & Khagram, 2004): • first, what is the "public value" that the public organization wants to create (present to civil society)? • second, on which "sources of legitimacy and support" could they rely to empower the public organization to act and provide the resources needed to create this public value? • third, on which "operational capabilities" (including innovations and improvements) would the public organization rely to deliver the desired outcomes? P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 815 However, since Moore formulated his idea in the mid-1990s, it has understandably already been criticised. First, as Rhodes and Wanna (2007) note, it is not clear whether Moore is proposing a theoretical framework, a concept, or a strategic tool for public managers. Second, because Moore does not provide a definition of the term public value (but uses it anyway), some scholars argue that public value can, therefore, be considered as a paradigm (e.g. Stoker, 2006; Benington, 2005), concept (e.g. Kelly et al., 2002), model (O'Flynn, 2005), heuristic device or even a story (Smith, 2004). Third, Oakley et al. (2006) note that Moore sometimes appears to be talking about public goods and, at other times, about the public interest or even the public domain. Later, therefore, critically point out that public value is another "fuzzy" term which seems to be a messy hybrid of al three. In other words, where the term ‘public value’ has any meaning, it generally refers to public goods, the public interest or the public domain while offering nothing new to any of them. Regardless, Moore remains the so-cal ed father of public value creation (Meynhardt, 2022), although it remains unknown how to empirical y measure the extent to which public organisations actually create public value (Faulkner & Kaufman, 2017; Williams & Shearer, 2011; Talbot & Wiggan, 2010). This is partly due to the fact that the term public value remains conceptually unresolved, as there is no universally accepted definition among scholars (Pang et al., 2014). Nevertheless, the creation of public value should remain a goal of public organisations, as public organisations use public value to meet the needs and wishes of the public – citizens (Brown et al., 2021; Neumann et al., 2019; Jørgensen & Bozeman, 2007). For the purposes of this paper, we have defined public value creation as the outcomes that are the results of the previous three categories. On this basis, 'public service quality provision capability – improved efficiency and effectiveness of public service production' and 'citizen engagement capability – public administrations identify and respond more quickly to citizens' aspirations' were identified as attributes (criteria) of the fourth category 'outcomes – public value creation' (Twizeyimanaa & Andersson, 2019; Faulkner & Kaufman, 2017; Scott et al., 2016; Spano, 2014; Pang et al., 2014). In this paper, outcomes – public value creation are written as verbs - e.g. 'efficiency' is not a public value in itself, whereas 'doing or performing something in an efficient way' becomes one. While this way of defining public value may seem unusual, it proves extremely useful, especially in the context 816 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION of different views on what public value is and how to measure it (Twizeyimanaa & Andersson, 2019). The ability to deliver quality public services is key to creating public value. Delivering public services more efficiently and effectively, with minimal use of public resources and in a faster (more responsive) time, makes a significant contribution to improving quality (Scott et al., 2016; Pang et al., 2014). This was already highlighted three decades ago by Moore (1995), who pointed out that it is not enough for public managers to produce results that are supported by the public (citizens or civil society at large). Public managers need to demonstrate that these results justify public spending, as only then can it be argued that public value has been created. Alford and Hughes (2008) further state that if the desired results are achieved with minimal public expenditure and within a reasonable period, it can be concluded that PA is also efficient and effective. In addition to providing quality public services, citizens' ability (or capacity) to engage in the public policymaking process is also crucial for creating public value (Rasmussen & Rehe, 2023; Pang et al., 2014). Stoker (2006) points out that the paradigm of public value creation relies on citizens' trust in PA and the recognition of its legitimacy. Therefore, for public value creation to be supported and responsive to citizens' needs and desires, it is imperative that the government facilitates and supports active citizen participation. Without citizen participation and consent, public values cannot be created, no matter how good the government considers the quality of the outcomes created, as citizens may not consider them to be so (Wilson & Knighton, 2021; Faulkner & Kaufman, 2017; Pang et al., 2014; Talbot & Wiggan, 2010). In the digital age, where more and more tools for two-way communication are available, PAs are even more expected to foster citizen participation, which wil increase the trust and legitimacy of their actions. Accountable and transparent PA means better access to government information and improves the transparency of PA while reducing the risks of corruption (Castro & Lopes, 2022). This is considered in an increasingly dynamic environment where ICTs are transforming the design of public services, and it is becoming increasingly clear that PA alone can no longer respond effectively to the changing needs and demands of citizens (Pang et al., 2014). However, as ICTs enable two-way communication more than ever before, PA is also expected to work more closely with citizens in the co-creation of public services. This mutual collaboration enables public services to become truly citizen- P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 817 centred and tailored to citizens' needs (Li & Shang, 2023; Scott et al., 2016; Hellang & Flak, 2012; Jansen, 2012). Table 5: Model's attributes based on the fourth category, 'outcomes – creating public value' Attributes deriving from outcomes – creating public value Attribute number Public service quality provision capability – improved efficiency and 4.1. effectiveness of public service production Citizen engagement capability – public administrations identify and respond 4.2. more quickly to citizens' aspirations Citizen participation in policy-making and improved democracy 4.2.1. Increased transparency of public administration operations – citizens have 4.2.2. better access to government information Co-creation capability delivering more inclusive public services that are 4.2.3. citizen-centred and tailored to citizens' needs Trust and legitimacy – increase citizens' trust in public administration 4.2.4. operations and recognision of legitimacy 3.2 Hierarchical structure of the decision support model With the help of the content analysis, we identified 29 subordinate and 13 single- parent – see Table 1. These attributes represent the foundation for the development of the structure of the multi-attribute model, designed to measure the maturity of SPG in public administration institutions presented in this section. Figure 1 represents the tree-like structure of the multi-attribute model consisting of four level attributes (criteria), which support the SPG maturity measurement in public administration institutions. In the tree-like structure, each subordinate attribute affects a single parent attribute (Bohanec, 2012). In this structure, there are basic and derived attributes. An example of a basic (subordinate) attribute is 'the impact of artificial intelligence to automate routine processes' (1.2.1.1.) since it has no subordinated attributes. In the model, it represents the final node or "leaf". Such attributes are the model's input (operational attributes). Other attributes are derived (superior or single parent attribute); an example is 'the use of emerging technologies in PA' (1.2.), meaning they are aggregated nodes within the model (also cal ed aggregated attributes). The superior 818 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION attributes are calculated based on the values assigned to the basic attributes by the interview group. The highest attribute, i.e., the final maturity of SPG in public administration institutions, is the main output attribute – the root of the tree-like structure (Bohanec, 2012). It should be stressed, however, that the attributes within the multi-attribute model are treated as being non-redundant (avoiding unnecessary attributes that could impact the model's size and complexity), mutual y independent (one attribute for a specific decision factor), and operable (applicable in practice) (Bohanec, 2012). Figure 1: Multi-attribute model for measuring smart public governance maturity in public administration institutions P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 819 4 Future research development The paper makes a pioneering step aiming to stimulate a broader academic debate about which attributes (criteria) corresponding to the subject/area should be considered when measuring the maturity of SPG in public administration institutions. The multi-attribute model creation is based on several iterations, using the DSR approach (Dresch et al., 2015) and the DEX method, belonging to a multi-attribute utility theory method group. Prior research findings obtained through content analysis played a crucial role in identifying attributes (criteria) essential for measuring the maturity of SPG. Considering what we already learned, we can argue that maturity in SPG is not only about the use of technology but also about changes that public sector organisations face internally, as well as evolving dynamics of citizen interaction with public sector organisations in order to create public value (outcomes) for society. Further research should focus on the practical application and validation of the multi-attribute model introduced in this paper. Future studies are encouraged to test and refine the identified attributes (criteria), enhancing the model’s robustness and applicability. References Abuljadail, M., Khalil, A., Talwar, S., & Kaur, P. (2023). Big data analytics and e-governance: Actors, opportunities, tensions, and applications. Technological Forecasting and Social Change, 193:22612, 1-17. doi: 10.1016/j.techfore.2023.122612 Albino, V., Berardi, U., & Dangelico, M.R. (2015). Smart Cities: Definitions, Dimensions, Performance, and Initiatives . Journal of Urban Technology, 22(1), 3-21. doi: 10.1080/10630732.2014.942092 Alford, J. & Hughes, O. (2008). Public Value Pragmatism as the Next Phase of Public Management. The American Review of Public Administration, 38(2), 130–148. doi:10.1177/0275074008314203 Alves, H. (2013). Co-creation and innovation in public services. The Service Industries Journal, 33:7-8, 671-682, doi: 10.1080/02642069.2013.740468 Andersen, N.K., Henriksen, Z.H., Medalia, R., Danziger, N.J., Sannarsen, K.M., & Enemærke, M. (2010). Fads and Facts of E-Government: A Review of Impacts of E-government (2003– 2009). International Journal of Public Administration, 33(11), 564-579. doi: 10.1080/01900692.2010.517724 820 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Androutsopoulou, A., N. Karacapilidis, E. Loukis, & Y. Charalabidis. (2019). Transforming the Communication Between Citizens and Government Through AI-Guided Chatbots. Government Information Quarterly, 36(2), 358–367. doi: 10.1016/j.giq.2018.10.001 Angelopoulos, K., Diamantopoulou, V., Mouratidis, H., Pavlidis, M., Salnitri, M., Giorgini, P., & Ruiz, J.F. (2017). A Holistic Approach for Privacy Protection in E-Government. In Proceedings of the 12th International Conference on Availability, Reliability and Security (pp. 1-17). doi: 10.1145/3098954.3098960 Ansel , C & Gash, A. (2012). Stewards, Mediators, and Catalysts: Toward a Model of Col aborative Leadership. The Innovation Journal: The Public Sector Innovation Journal, 17(1), 1-20. Dostopno na: https://innovation.cc/wp-content/uploads/2012_17_1_7_ansel _gash_innovate-leadrship.pdf Ansell, C. & Gash, A. (2008). Collaborative Governance in Theory and Practice. Journal of Public Administration Research and Theory, 18(4), 543–571. doi: 10.1093/jopart/mum032 Anthopoulos, L., Reddick, C., & Sirakoulis, K. (2021). Conceptualizing Smart Government: Interrelations and Reciprocities with Smart City. Digital Government: Research and Practice, 2(4), 33:2-33:28. doi: 10.1145/3465061 Arntz, M., Gregory, T., & Zierahn, U. (2016). The Risk of Automation for Jobs in OECD Countries: A Comparative Analysis. OECD Publishing, Paris. Babić, A., Soklić, D., & Antonić, J.J. (2022). Development of Smart Governance in Croatian Cities - the Size of a City as a Determinant of Smart Governance. Ekonomski vjesnik, 35(2), 315-335. doi: 10.51680/ev.35.2.7 Bannister, F. (2001). Dismantling the silos: extracting new value from IT investments in public administration. Information Systems Journal, 11, 65–84. doi: 10.1046/j.1365-2575.2001.00094.x Bailey, E.D. & Barley, R.S. (2019). Beyond design and use: How scholars should study intel igent technologies. Information and Organization, 30(2), 100286. doi: 10.1016/j.infoandorg.2019.100286 Bartelt, L.V., Urbaczewski, A., Mueller, G.A., & Sarker, S. (2020) Enabling collaboration and innovation in Denver’s smart city through a living lab: a social capital perspective. European Journal of Information Systems, 29(4) 369-387. doi: 10.1080/0960085X.2020.1762127 Bason, C. (2010). Leading public sector innovation. Bristol: Policy Press. Batagan, L. (2011). Smart Cities and Sustainability Models. Informatica Economica, 15(0), 80-87. Available on IE Paper Template (ase.ro) Becker, T. (2016). Big Data Usage. In J.M. Cavanil as, E. Curry, & W. Wahlster (eds.), New Horizons for a Data-Driven Economy: A Roadmap for Usage and Exploitation of Big Data in Europe (pp. 143-165). doi: 10.1007/978-3-319-21569-3_8 Benington, J. & Moore, M.H. (2011). Public value in complex and changing times. In J. Benington & M.H. Moore (eds.), Public value theory & practice (str. 1–30). doi: 10.1007/978-0-230-36431-8_1 Benington, J. (2005). From Private Choice to Public Value. Paper presented to the Public Management and Policy Association. London: PMPA. doi: 10.1007/978-0-230-36431-8_2 Bertot, J.C. & Choi, H. (2013). Big data and e-government: Issues, policies, and recommendations. In Proceedings of the 14th Annual International Conference on Digital Government Research (pp. 1-10). doi: 10.1145/2479724.2479730 Bohanec, M. (2012). Odločanje in modeli. Ljubljana: DMFA založništvo. Bolivar, P.M.R. & Meijer, A. (2016). Smart Governance: Using a Literature Review and Empirical Analysis to Build a Research Model. Social Science Computer Review, 34(6), 1-20. doi: 10.1177/0894439315611088 Bouaziz, S., Nabli, A., & Gargouri, F. (2017). From Traditional Data Warehouse to Real-Time Data Warehouse. V J. Kacprzyk (ur.), Advances in Intel igent Systems and Computing (str. 467-477). doi: 10.1007/978-3-319-53480-0_46 Bouckaert, G., Peters, B.G., & Verhoest, K. (2010). The Coordination of Public Sector Organizations: Shifting Patterns of Public Management. London: Palgrave Macmillan. P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 821 Bovaird, T., Ryzin, V.G.G., Loeffler, E., & Parrado, S. (2014). Activating Citizens to Participate in Collective Co-Production of Public Services. Journal of Social Policy, 44(1), 1-23. doi: 10.1017/S0047279414000567 Bovens, M. & Zouridis, S. (2002). From Street-Level to System-Level Bureaucracies: How Information and Communication Technology Is Transforming Administrative Discretion and Constitutional Control. Public Administration Review, 62(2), 174-184. doi: 10.1111/0033-3352.00168 Brennan, M.N., Subramaniam, N., & Staden, J.C. (2019). Corporate governance implications of disruptive technology: An overview. The British Accounting Review, 51(6), 1-15. doi: 10.1016/j.bar.2019.100860 Campion, A., M. Gasco-Hernandez, M. Esteve, and S. Mikhaylov. (2022). Overcoming the Challenges of Collaboratively Adopting Artificial Intelligence in the Public Sector. Social Science Computer Review, 40(2), 462–477. doi: 10.1177/0894439320979953 Cano, J., Ros, S., & Hernandez, R. (2014). Distributed Framework for Electronic Democracy in Smart Cities. Computer, 47(10), 65-71. doi: 10.1109/MC.2014.280 Cardullo, P., & Kitchin, R. (2019). Being a ‘citizen’ in the smart city: Up and down the scaffold of smart citizen participation in Dublin, Ireland. GeoJournal, 84(1), 1–13. doi: 10.1007/s10708-018-9845-8 Castro, C. & Lopes, C.I. (2022). E-Government as a Tool in Control ing Corruption. International Journal of Public Administration, 46(1), 1-14. doi: 10.1080/01900692.2022.2076695 Chen, K., & Aitamurto, T. (2019). Barriers for crowd’s impact in Crowdsourced policymaking: Civic data overload and filter hierarchy. International Public Management Journal, 22(1), 99–126. doi: 10.1080/10967494.2018.1488780 Chen, C.Y. & Hsieh, C.T. (2014). Big Data for Digital Government: Opportunities, Chal enges, and Strategies. International Journal of Public Administration in the Digital Age, 1(1), 1-14. doi: 10.4018/ijpada.2014010101 Choi, Y., Gil-Garcia, R.J., Burke, G.B, Costel o, J., Werthmul er, A., & Aranay, O. (2021). Towards data-driven decision-making in government: Identifying opportunities and chal enges for data use and analytics. V Proceedings of the 54th Hawaii international conference on system sciences (str. 2183-2192). Dostopno na: http://hdl.handle.net/10125/70881 Chun, A.S., Luna-Reyes, F.L., & Sandoval-Almazán, R. (2012. Collaborative e-government. Transforming Government: People, Process and Policy, 6(1), 5 – 12. doi: 10.1108/17506161211214868 Cilliers, F., & Greyvenstein, H. (2012). The impact of silo mentality on team identity: An organisational case study. Journal of Industrial Psychology, 38(2), 1-9. doi:10.4102/sajip.v38i2.993 Cooper, A.L., Holderness Jr, K.D., Sorensen, L.T., & Wood, A.D. (2018). Robotic Process Automation in Public Accounting. Accounting Horizons, 33(4), 1-62. doi: 10.2308/acch-52466 Craglia, M., Annoni, A., Benczur, P., Bertoldi, P., Delipetrev, P., De Prato, G., … Vesnic Alujevic, L. (2018). In M. Craglia (eds.), Artificial Intel igence: A European perspective. European Commission Publications Office. Available on:https://op.europa.eu/en/publication-detail/- /publication/ed2148f3-0288-11e9-adde-01aa75ed71a1/language-en Criado, J.I. & Gil-Garcia, J.R. (2019). Creating public value through smart technologies and strategies: From digital services to artificial intelligence and beyond. International Journal of Public Sector Management, 32(5), 438-450. doi: 10.1108/IJPSM-07-2019-0178 Cunha, A.M., Coelho, R.T., & Pozzebon, M. (2013). The Use Of ICT In Public Decision-Making Participation. In Proceedings of the 21st European Conference on Information Systems (pp. 22-26). Available on http://aisel.aisnet.org/ecis2013/22 Danziger, N.J. & Andersen, V.K. (2002). The impacts of information technology on public administration: an analysis of empirical research from the “golden age” of transformation. International Journal of Public Administration, 25(5), 591-627. doi: 10.1081/PAD-120003292 Demirel, D., & Mülazımoglu, E. M. (2021). How the smart governance model shapes cities? Cases from Europe. Journal of Enterprising Communities: People and Places in the Global Economy, 16(1), 8-25. doi:10.1108/JEC-08-2021-0115 822 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Desouza, C.K. & Jacob, B. (2014). Big Data in the Public Sector: Lessons for Practitioners and Scholars. Administration & Society, 49(7) 1043–1064. doi: 10.1177/0095399714555751 Dibouliya, A. (2023). Review on: Modern Data Warehouse & how is it accelerating digital transformation. International journal of advance research, ideas, and innovations in technology, 9(2), V912-1224. Available on: https://www.ijariit.com Dresch, A., Lacerda, P. D., & Antunes, V. A. J. (2015). Design Science Research. London: Springer. Faulkner, N. & Kaufman, S. (2017). Avoiding Theoretical Stagnation: A Systematic Review and Framework for Measuring Public Value. Australian Journal of Public Administration, 77(2), 1-18. doi:10.1111/1467-8500.12251 Fredriksson, C., Mubarak, F., Tuohimaa, M., & Zhan, M. (2017). Big Data in the Public Sector: A Systematic Literature Review. Scandinavian Journal of Public Administration, 21(3), 39-61. doi: 10.58235/sjpa.v21i3.11563 Favoreu, C., Maurel, C., & Queyroi, X. (2024). Influence of public innovation laboratories on the development of public sector ambidexterity. Public Management Review, 88(2), 1-27. doi: 10.1080/14719037.2024.2322720 Giest, N.S. & Klievink, B. (2024) More than a digital system: how AI is changing the role of bureaucrats in different organizational contexts. Public Management Review, 26(2), 379-398. doi: 10.1080/14719037.2022.2095001 Gil-Garcia, J. R. (2012). Towards a smart State? Inter-agency col aboration, information integration, and beyond. Information Polity, 17(0), 269–280. doi: 10.5555/2656990.2656996 Giuliodori, A., Berrone, P., & Ricart, E.J. (2023). Where smart meets sustainability: The role of Smart Governance in achieving the Sustainable Development Goals in cities. Business Research Quarterly, 26(1), 27-44. doi: 10.1177/23409444221091281 Giulio, D.M. & Vecchi, G. (2021). Implementing digitalization in the public sector. Technologies, agency, and governance. Public Policy and Administration, (0)0, 1-15. doi:10.1177/09520767211023283 Guillaumie, L., Vezina-Im, L.A., Bourque, L., Boiral, O., Talbot, D., & Harb, E. (2024). Best Practices for Municipalities to Promote Online Citizen Participation and Engagement on Facebook: A Narrative Review of the Literature. Social Sciences, 13(127), 1-16. doi: 10.3390/socsci13030127 Hansen, A. (2014). Dismantling Silos – From Creative Col aboration to Col aborative Creativity. [Master thesis]. University of Washington, Science in Construction Management. Hartley, J., Sancino, A., Bennister, M., & Resodihardjo, S. (2019). Leadership for public value: Political astuteness as a conceptual link Symposium introduction. Public Administration, 0(0), 1-19. doi: 10.1111/padm.12597 Hel ang, Ø. & Flak, L. (2012). Assessing effects of e-government initiatives based on a public value framework. V International Conference on Electronic Government (str. 246-259). doi: 10.1007/978-3-642-33489-4_21 Höchtl, H., Parycek, P. & Schöllhammer, R. (2016). Big data in the policy cycle: Policy decision making in the digital era. Journal of Organizational Computing and Electronic Commerce, 26(1-2), 147-169. doi: 10.1080/10919392.2015.1125187 Interoperable Europe Academy. (b.d.). https://joinup.ec.europa.eu/interoperable-europe Janssen, M., Konopnicki, D., Snowdon, L.J., & Ojo, A. (2017). Driving public sector innovation using big and open linked data (BOLD). Information Systems Frontiers, 19(4), 189–195. doi: 10.1007/s10796-017-9746-2 Janssen, M. & Hoven, J. (2015). Big and Open Linked Data (BOLD) in government: A chal enge to transparency and privacy. Government Information Quarterly, 32(4), 363-368. doi: 10.1016/j.giq.2015.11.007 Jansen, A. (2012). The Understanding of ICTs in Public Sector and Its Impact on Governance. In H.J., Schol et al. (eds.), Electronic Government. Berling: Springer. Jiang, H., Geertman, S., & Witte, P. (2022). Smart urban governance: an alternative to technocratic “smartness.” GeoJournal, 87(0), 1639-1655. doi:10.1007/s10708-020-10326-w P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 823 Jiang, H., Geertman, S., & Witte, P. (2019). Smart urban governance: An urgent symbiosis? Information Polity, 24(9), 245-269. doi: 10.3233/IP-190130 Jørgensen, T. B. & Bozeman, B. (2007). Public values: An inventory. Administration & Society, 39(3), 354–381. doi: 10.1177/0095399707300703 Jucevicius, R. & Juceviciene, P. (2018). Knowledge dimension in smart development. In European Conference on Knowledge Management (pp. 369-376). Padua: University of Padua. Jukić, T. & Vrbek, S. (2023). Co-creation service readiness model: a decision support for the selection of public services suitable for improvement through co-creation. Transforming Government: People, Process and Policy, 67(1), 1750-6166. doi: 10.1108/TG-03-2023-0031 Jukić, T., Pluchinotta, I., Hržica, R., Vrbek, S. (2021). Organizational maturity for co-creation: Towards a multi-attribute decision support model for public organizations. Government Information Quarterly, 39(1):101623. doi:10.1016/j.giq.2021.101623 Kel y, G., Mulgan, G., & Muers, S. (2002). Creating Public Value. London: Cabinet Office. Kim, Y.H. & Cho, S.J. (2017). Data Governance Framework for Big Data Implementation with a Case of Korea. In the 6th International Congress on Big Data (pp. 384-391). doi: 10.1109/BigDataCongress.2017.56 Klievink, B., Romijn, J.B., Cunningham, S., & Bruijn, H. (2017). Big data in the public sector: Uncertainties and readiness. Information Systems Frontiers, 19(0), 267–283. doi: 10.1007/s10796-016-9686-2 Kolkman, D. (2020). The usefulness of algorithmic models in policy making. Government Information Quarterly, 37(3), 101488. doi: 10.1016/j.giq.2020.101488 Kostoff, N.R., Boylan, R., & Simons, R.G. (2004). Disruptive technology roadmap. Technological Forecasting and Social Change, 71(1-2), 141-159. doi: 10.1016/S0040-1625(03)00048-9 Kreuter, F., Ghani, R., & Lane, J. (2019). Change Through Data: A Data Analytics Training Program for Government Employees. Harvard Data Science Review, 1.2, 1-26. doi: 10.1162/99608f92.ed353ae3 Kumar, J.T.L. (2015). E-Governance for Smart Cities. London: Springer. Kuziemski, M., & Misuraca, G. (2020). AI governance in the public sector: Three tales from the frontiers of automated decision-making in democratic settings. Telecommunications Policy, 44(6), 101976. doi: 10.1016/j.telpol.2020.101976 Li, Y. & Shang, H. (2023). How does e-government use affect citizens’ trust in government? Empirical evidence from China. Information & Management, 60(7), 103844. doi:10.1016/j.im.2023.103844 Lin, Y. (2018). A comparison of selected Western and Chinese smart governance: The application of ICT in governmental management, participation and col aboration. Telecommunications Policy, 42(10), 800-809. doi: 10.1016/j.telpol.2018.07.003 Lindgren, I. & Veenstra, F.A. (2018). Digital government transformation: a case illustrating public e-service development as part of public sector transformation. In 19th Annual International Conference (pp. 2-6). doi: 10.1145/3209281.3209302 Lindsay, C., Pearson, S., Batty, E., Cul en, A. M., & Eadson, W. (2018). Co-production as a route to employability: Lessons from services with lone parents. Public Administration, 96(2), 318–332. doi:10.1111/padm.12408 Marche, S. & McNiven, D.J. (2003). E-government and E-governance: The future is not what it used to be. Canadian Journal of Administrative Science, 20(1), 74.86. doi: 10.1111/j.1936-440'.2003.tb00306.x Mergel, I., Dickinson, H., Stenvall, J., & Gasco, M. (2023). Implementing AI in the public sector. Public Management Review. doi: 10.1080/14719037.2023.2231950 Margetts, H., & Dorobantu, C. (2019). Rethink government with AI. Nature, 568(7751), 163–165. doi: 10.1038/d41586-019-01099-5 Margel, I., Rethemeyer, K.R., & Isett, K. (2016). Big Data in Public Affairs. Public Administration Review, 76(6), 1-19. doi: 10.1111/puar.12625 824 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Meijer, A. & Bolivar, M.P.R. (2015). Governing the smart city: a review of the literature on smart urban governance. International Review of Administrative Sciences, (0)0,1-17. doi:10.1177/0020852314564308 Meijer, A. (2014). Why don’t they listen to us? Reasserting the role of ICT in Public Administration. Information Polity, 12(4) 233–242. Meerkerk, I. (2019). Top-down versus bottom-up pathways to col aboration between governments and citizens: reflecting on different participation traps. In A. Kekez, M. Howlett & M. Ramesh (eds.), Col aboration and Public Service Delivery: Promise and Pitfal s, pp. 149-167. Eheltenham: Edward Elgar Publishing. Meynhardt, T. (2022). Public Value. V H.K., Anheier & S. Toepler (ur.), International Encyclopedia of Civil Society. New York, NY: Springer. Meynhardt, T. (2009). Public Value Inside: What is Public Value Creation? International Journal of Public Administration, 32(3-4), 192–219. doi: 10.1080/01900690902732632 Meuleman L. (2021). Public Administration and Governance for the SDGs: Navigating between Change and Stability. Sustainability,13(11):5914. doi: 10.3390/su13115914 Mikalef, P., Fjørtoft, S. O., & Torvatn, H. Y. (2019). Artificial Intelligence in the Public Sector: A Study of Chal enges and Opportunities for Norwegian Municipalities. In 18th IFIP WG 6.11 Conference on e-Business, e-Services, and e-Society (pp. 267–277). doi: 10.1007/978-3-030-29374-1 Milakovich, E.M. (2011). Digital Governance: New Technologies for Improving Public Service and Participation. New York (NY): Routledge. Miloslavskaya, N. & Tolstoy, A. (2016). Big Data, Fast Data and Data Lake Concepts. Procedia Computer Science, 88(0), 300–305. doi: 10.1016/j.procs.2016.07.439 Misuraca, G. & Van Noordt, C. (2020). Overview of the use and impact of AI in public services in the EU. Publications Office of the European Union: Luxembourg. doi:10.2760/039619, JRC120399 Misuraca, G. & Viscusi, G. (2015). Shaping public sector innovation theory: an interpretative framework for ICT-enabled governance innovation. Electronic Commerce Research, 15(0), 303– 322. doi: 10.1007/s10660-015-9184-5 Moore, M.H. & Khagram, S. (2004). On creating public value: What Business May Learn from Government about Strategic Management. Corporate Social Responsibility Initiative Working paper. Cambridge, MA: Harvard University Moore, M.H. (1995). Creating public value: Strategic management in government. Cambridge: Harvard University Press. Munne, R. (2016). Big Data in the Public Sector. In J.M. Cavanil as, E. Curry, & W. Wahlster (eds.), New Horizons for a Data-Driven Economy: A Roadmap for Usage and Exploitation of Big Data in Europe (pp. 195-208). doi: 10.1007/978-3-319-21569-3_11 Nam, T., & Pardo, T. A. (2014). The changing face of a city government: A case study of Phil y311. Government Information Quarterly, 31, S1–S9. doi: 10.1016/j.giq.2014.01.002 Navarra, D.D. & Cornford, T. (2005). ICT, Innovation and Public Management: Governance, Models and Alternatives for eGovernment Infrastructures. In ECIS 2005 Proceedings (pp. 121-133). Available on http://aisel.aisnet.org/ecis2005 Nesti, G. (2020). Defining and assessing the transformational nature of smart city governance: insights from four European cases. International Review of Administrative Sciences, 86(1) 20–37. doi: 10.1177/0020852318757063 Ndou, D.V. (2004). E-government for developing countries: opportunities and chal enges. The Electronic Journal of Information Systems in Developing Countries, 18(1), 1-24. doi: 10.1002/j.1681-4835.2004.tb00117.x Neumanna, O., Mattb, C., Hitz-Gamperb, S.B., Schmidthuberc, l., & Stürmer, M. (2019). Joining forces for public value creation? Exploring collaborative innovation in smart city initiatives. Government Information Quarterly, 36(4), 101411. doi: 10.1016/j.giq.2019.101411 Newman, J., Mintrom, M., O'Neill, D. (2022). Digital technologies, artificial intelligence, and bureaucratic transformation. Futures, 136, 102886. doi: 10.1016/j.futures.2021.102886 P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 825 Oakley, K., Naylor, R., & Lee, D. (2006). Giving them what they want: the construction of the public in 'public value' . London: BOP Consulting. Okuyucu, A. & Yavuz, N. (2020). Big data maturity models for the public sector: a review of state and organizational level models. Transforming Government: People, Process and Policy, 14(4), 681-699. doi: 10.1108/TG-09-2019-0085 O’Flynn, J. (2005). Adding Public Value: A New Era of Contractual Governance. In IPAA National Conference, (pp. /). Canberra, Australia Örsel i, E., Bayrakci, E., & Bilici, Z. (2022). Analysis of smart city projects in Turkey in the context of smart people and smart governance. Lex Humana, 14(1), 328-399. Page, M., Hajduk, M., Arriëns, L.N.E., Cecconi, G., & Brinkhuis, S. (2023). Open data maturity report 2023. Luxembourg: Publications Office of the European Union. Pang, M.S., Lee, G., & DeLone, W.H. (2014). IT resources, organizational capabilities, and value creation in public-sector organizations: a public-value management perspective. Journal of Information Technology, 29(3), 187–205. doi:10.1057/jit.2014.2 Pencheva, I., Esteve, M., & Mikhaylov, J.S. (2020). Big Data and AI – A transformational shift for government: So, what next for research. Public Policy and Administration, 35(1), 24–44. doi: 10.1177/0952076718780537 Pereira, G. V., Parycek, P., Falco, E., & Kleinhans, R. (2018). Smart governance in the context of smart cities: A literature review. Information Polity, 23(2), 143–162. doi: 10.3233/IP-170067 Pereira, V.G., Cunha, M.A., Lampoltshammer, T.J., Parycek, P., & Testa, M.G. (2017). Increasing col aboration and participation in smart city governance: A cross-case analysis of smart city initiatives. Information Technology for Development, 23(3), 526-553. doi: 10.1080/02681102.2017.1353946 Popova, Y. & Popovs, S. (2023). Effects and Externalities of Smart Governance. Smart Cities, 6(0),1109–1131. doi: 10.3390/smartcities6020053 Rackwitz, M., Breaugh, J., & Hammerschmid, G. (2024). Leadership for intergovernmental col aboration towards digital transformation. V K. Verhoest idr. (ur.), Col aborating for Digital Transformation. Cheltenham: Edward Elgar Publishing. Ramos, G., Fernandes, D., Coelho, J., & Aquino, A. (2023). Toward Data Lake Technologies for Intelligent Societies and Cities. V C.F.S., Portela (ur.), Sustainable, Innovative and Intel igent Societies and Cities (str. 3-29). doi: 10.1007/978-3-031-30514-6_1. Rasmussen, A. & Rehe, S. (2023). (Inequality in) Interest Group Involvement and the Legitimacy of Policy Making. British Journal of Political Science, 53(0), 45–64. doi:10.1017/S0007123422000242 Regal, B., Ferlie, E., Aagaard, P., & Vrbek, S. (2024). Governance modes, mayoral leadership and transitions to public sector co-creation across Europe. Public Policy and Administration, 0(0) 1– 25. doi: 10.1177/09520767241231598 Rhodes, R.A.W. & Wanna, J. (2007). The limits of public value or rescuing responsible government from the platonic guardians. Australian Journal of Public Administration, 66(4), 406-421. doi: 10.1111/j.1467-8500.2007.00553.x Riduan, A. (2024). Citizen participation in Policy Decision-Making. International Journal of Multidisciplinary Approach Sciences and Technologies, 1(1), 65-74. doi: 10.62207 Rogge, N., Agasisti, T., & Witte, K. (2017). Big data and the measurement of public organisations’ performance and efficiency: the state-of-the-art. Public Policy and Administration, 32(4), 263– 281. doi: 10.1177/0952076716687355 Romansky P.R. & Noninska, S.I. (2020). Challenges of the digital age for privacy and personal data protection. Mathematical Biosciences and Engineering, 17(5), 5288-5393. doi: 10.3934/mbe.2020286 Ronzhyn, A., Wimmer, A.M., Spitzer, V., Pereirs, V.G., & Alexopoulos, C. (2019). Using Disruptive Technologies in Government: Identification of Research and Training. In 18th International Conference on Electronic Government (pp. 276-287). doi: 10.1007/978-3-030-27325-5_21f Ruhlandt, S. W. R. (2018). The governance of smart cities: A systematic literature review. Cities, 81(0), 1-23. doi: 10.1016/j.cities.2018.02.014 826 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Ruijer, E., Twist, A., Haaker, T., Tartarin, T., Schuurman, N., Melenhorst, M., & Meijer, A. (2023). A Systematic Literature Review. Smart Cities, 6, 878–896. doi: 10.3390/smartcities6020042 Sarker, I.N.M., Wu, M., & Hossin, A.M. (2018). Smart Governance through Bigdata: Digital Transformation of Public Agencies. In International Conference on AI and big data (pp. 62-70). doi: 10.1109/ICAIBD.2018.8396168 Savignon, B.A., Zecchinelli, R., Costumato, L., & Scalabrini, F. (2024). Automation in public sector jobs and services: a framework to analyze public digital transformation’s impact in a data-constrained environment. Transforming Government: People, Process and Policy, 18(1), 49- 70. doi: 10.1108/TG-04-2023-0044 Schol , J.H. & Alawadhi, S. (2016). Creating Smart Governance: The key to radical ICT overhaul at the City of Munich. Information Polity, 21-42. doi: 10.3233/IP-150369 Schol ., H. J. & Schol , M. (2014). Smart Governance: A Roadmap for Research and Practice. In iConference 2014 Proceeding (pp. 163–176). doi:10.9776/14060 Scott, I. & Gong, T. (2021). Coordinating government silos: chal enges and opportunities. Global Public Policy and Governance, 1:20-23. doi: 10.1007/s43508-021-00004-z Scott, I. (2020). Governing by Silos. Oxford Research Encyclopedia of Politics, 1-23. doi: 10.1093/acrefore/9780190228637.013.1414 Scott, M., DeLone, W., & Golden, W. (2016). Measuring eGovernment success: a public value approach. European Journal of Information Systems, 25(3), 187–208. doi:10.1057/ejis.2015.11 Smith, R.F.I. (2004). Focusing on Public Value: Something New and Something Old. Australian Journal of Public Administration, 63(4), 68–79. doi: 10.1111/j.1467-8500.2004.00403.x Söderström, F., Melin, U., Lindgren, I., & Galzie, Z. (2018). Coordinating Public E-services – Investigating Mechanisms and Practices in a Government Agency. In P. Parycek (eds.), Electronic Government (pp. 85-97). Cham: Springer. Sørensen, E., Bryson, J., & Crosby, B. (2021). How public leaders can promote public value through co-creation. Policy and Politics, 49(2), 267-286. doi: 10.1332/030557321X16119271739728 Sørensen, E. & Torfing, J. (2019). Designing institutional platforms and arenas for interactive political leadership. Public Management Review, 21:10, 1443-1463. doi: 10.1080/14719037.2018.1559342 Stoker, G. (2006). Public value management - A new narrative for networked governance? American Review of Public Administration, 36 (1):41–57. doi: 10.1177/0275074005282583 Šiugždinienė, J., Gaule, E., & Rauleckas, R. (2017). In search of smart public governance: the case of Lithuania. International Review of Administrative Sciences, 85(0), 1-20. doi:1177/0020852317707814 Talbot, C. & Wiggan, C. (2010). The public value of the National Audit Office. International Journal of Public Sector Management, 23(1), 54-70. doi: 10.1108/09513551011012321 Tett, G. (2014). The Silo Ef ect: The Peril of Expertise and the Promise of Breaking Down Barriers. New York, NY: Simon & Schuster. Torfing, J., Ferlie, E., & Jukić, T. (2021). A theoretical framework for studying the co-creation of innovative solutions and public value. Policy & Politics, 49(2), 189–209. doi: 10.1332/030557321X16108172803520 Torfing, J., Sørensen, E., & Røiseland, A. (2019). Transforming the Public Sector into an Arena for Co-Creation: Barriers, Drivers, Benefits, and Ways Forward. Administration & Society, 51(5), 795–825. doi: 10.1177/0095399716680057 Torfing, J. & Ansell, C. (2016): Strengthening political leadership and policy innovation through the expansion of col aborative forms of governance. Public Management Review, 0(0), 1-18. doi: 10.1080/14719037.2016.1200662 Voorberg, H.W., Bekkers, M.J.J.V., & Tummers, G.L. (2014). A systematic review of co-creation and co-production: Embarking on the social innovation journey. Public Management Review, 1-39. doi: 10.1080/14719037.2014.930505 Tõnurist, P., Kattel, R., & Lember, V. (2017). Innovation labs in the public sector: what they are and what they do? Public Management Review, (0)0, 1–25. doi:10.1080/14719037.2017.128 Twizeyimanaa, D. J. & Andersson, A. (2019). The public value of E-Government – A literature review. Government Information Quarterly, 36(2), 167-178. doi: 10.1016/j.giq.2019.01.001 P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 827 Zhao, W. & Zou, Y. (2021). Smart Urban Governance in Epidemic Control: Practices and Implications of Hangzhou. CPAR, 12(1), 51-60. doi: 10.1177/153967542101200104 Zhu, Y. & Kou, G. (2019). Linking Smart Governance to Future Generations: A Study on the Use of Local E-Government Service among Undergraduate Students in a Chinese Municipality. Informatics, 6(4), 45. doi:10.3390/informatics6040045 Valle-Cruz, D. & Garcia-Contreras, R. (2023). Towards AI-driven transformation and smart data management: Emerging technological change in the public sector value chain. Public Policy and Administration, 0(0) 1–22. doi: 10.1177/09520767231188401 Veale, M., & Brass, I. (2019). Administration by algorithm? Public management meets public sector machine learning. Algorithmic Regulation, 1–30. doi: 10.31235/OSF.IO/MWHNB Van Noordt, C. & Misuraca, G. (2022). Artificial intelligence for the public sector: results of landscaping the use of AI in government across the European Union. Government Information Quarterly, 39(9), 101714. doi: 10.1016/j.giq.2022.101714 Veenstra, F.A., Gromme, F., & Djafari, S. (2019). The use of public sector data analytics in the Netherlands. Transforming Government: People, Process and Policy, 15(4), 396-419. doi: 10.1108/TG-09-2019-0095 Vrbek, S. & Jukić, T. (2023). Co-creation service readiness model: a decision support for the selection of public services suitable for improvement through co-creation. Transforming Government: People, Process and Policy, 18(1), 13-32. doi: 10.1108/TG-03-2023-0031 Vujković, P. & Jukić, T. (2023). Towards understanding smart public governance: a literature review. In 31st NISPAcee Annual Conference – The future of public administration enabled through emerging technologies (pp. 1-16). Vydra, S. & Klievink, B. (2019). Techno-optimism and policy-pessimism in the public sector big data debate. Government Information Quarterly, 36(4), 101383. doi: 10.1016/j.giq.2019.05.010 Zuiderwijk, A., Chen, Y., & Salem, F. (2021). Implications of the use of artificial intelligence in public governance: A systematic literature review and a research agenda. Government Information Quarterly, 38(3), 101577. doi: 10.1016/j.giq.2021.101577 Webster, C. W. R. & Leleux, C. (2018). Smart governance: Opportunities for technological y-mediated citizen co-production. Information Polity, 23(1), 95–110. doi: 10.3233/IP-170065 Wilson, C. & Knighton, D. (2021). Legitimacy, autonomy, and trust: a recipe for organizations to operate in the public interest. Corporate Communications: an International Journal, 26(4), 773-792. doi: 10.1108/CCIJ-03-2021-0029 Wil ems, J., Schmid, J.M., Vanderelst, D., Vogel, D., & Ebinger, F. (2023). AI-driven public services and the privacy paradox: do citizens real y care about their privacy? Public Management Review, 25(11), 2116-2134. doi: 10.1080/14719037.2022.2063934 Williams, I. & Shearer, H. (2011). Appraising public value: Past, present and futures. Public Administration, 89(4), 1367 – 1384. doi: 10.1111/j.1467-9299.2011.01942.x Willke, H. (2007). Smart governance: Governing the global knowledge society. New York, NY, USA. Campus Verlag. Wimmer, A.M., Pereira, V.G., Ronzhyn, A., & Spitzer, V. (2020). Transforming Government by Leveraging Disruptive Technologies: Identification of Research and Training Needs. Journal of e-Democracy, 12(1), 87-113. doi: 10.29379/jedem.v12i1.594 Xiao, X., Califf, B.C., Sarker, S., & Sarker, S. (2013). ICT innovation in emerging economics: a review of the existing literature and framework for future research. Journal of Information Technology, 1-15. doi: 10.1057/jit. 2013.20 Yahia, B.N., Eljaoued, W. Saoud, B.B.N., & Colomo-Palacios, R. (2021). Towards sustainable collaborative networks for smart cities co-governance, International Journal of Information Management, 56(0), 1-16. doi:10.1016/j.ijinfomgt.2019.11.005 Yolles, M. (2016). Governance through Political Bureaucracy: An Agency Approach. Kybernetes, 48(1), 1-29. doi: 10.1108/K-09-2017-0329 828 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Appendix Table 1: Selection of various SPG definitions Author SPG definition Pereira et al., 2018 (in Demirel & SPG “emphasizes participation in decision-making processes, and is closely related Mülazımoglu (2021, p. 8) to the transparency of administrative systems and the availability of public services” SPG is “a new model of administration, which includes changing and reshaping the roles of local government, central government, citizens and other social actors in the Örselli et al. (2022, p. 388) administrative mechanism, new communication structures and a new relational process” SPG is “a multidimensional and multilevel construct that includes aspects such as transparency, stakeholder collaboration, the ability to secure social infrastructure Giuliodori et al. (2022, p. 32) through public-private partnerships, a citizen-centric approach to solving problems, a long-term perspective, a proactive management style, sensible use of public resources and a strong wil ingness to innovate” SPG is “the adoption of a new approach based on experimentation, col aboration with al stakeholders and the reorganisation of existing government structures. Nesti (2020, p. 20) Public actors should drive this process and should be supported by appropriate tools to manage interactions, foster coordination, enhance democratic legitimacy and accountability, and ensure tangible results for citizens” SPG is “a way to take advantage of various ICTs, aimed at bringing changes in Jiang et al. (2019, p. 246) public policy and government institutions from a public administration perspective” SPG is “the importance of technology-based tools in transforming Jiang et al. (2019, p. 246) government institutions from a public administration perspective (i.e., technology interaction with the institution)” SPG is “applying ICTs in the processing of information and decision-making in Zhu & Kou (2019, p. 2) order to improve the capacity of governance” SPG is “the combining of digital technologies with innovative practices to improve government service delivery and citizen inclusion in developing and implementing IRI, 2015 (in Yol es, 2019, p. 1) policy. This enables responsive, transparent, and inclusive policy decisions that build citizen trust in government institutions at al levels, and create a dialogue between supply (government) and demand (citizen)” Giffinger et al., 2007 (in Ruhlandt, SPG “comprises aspects of political participation, services for citizens as wel as the 2018, p. 13) functioning of the administration” SPG is “not only about digitising existing processes and services but also about Andermatt & Göldi, 2018 (in Babić developing and establishing entirely new processes and public services in a et al., 2022, p. 317) participatory way for citizens” SPG is “a mode of governance that relies on rational y utilizing internal and external resources, making adequate progress, and making advanced decisions Šiugzdiniene et al. (2017, pp. 589- relevant to specific circumstances to create shared value to make a social system 590) (country, region or city) and its actors (government, citizens, communities, businesses and non-governmental organizations) operate ef ectively in a fast-changing and complex environment” SPG is “the capacity of employing intel igent and adaptive acts and activities of Schol & AlAwadhi (2016, p. 22) looking after and making decisions about something” Meijer & Bolivar, 2016 (in Babić et SPG is “make the right policy decisions and implement them ef ectively, and the al., 2022, p. 317) need for smart decision making that includes the processes and implementation of P. Vujković: Measuring Smart Public Governance Maturity in Public Administration Institutions: A Multi-Attribute Approach 829 Author SPG definition those decisions. The new technologies are used to strengthen the rationality of management through the use of more complete and better information in the decision- making process” Gil-Garcia, 2014 (in Orselli et al., SPG is “the foundation of smart, open, and participatory administration. It is 2022, p. 388) important to use ICT frequently to utilize these areas more ef ectively” SPG is “a new form of electronic governance that uses sophisticated information technologies to interconnect and integrate information, processes, institutions, and Gil-Garcia, 2012 (in Kumar, 2015, physical infrastructure to better serve citizens and communities. This type of smart p. 36) governance is at a higher level of transformation in administration since it requires the restructuring of the internal organization of government” SPG is “collaborating across departments and with communities, helping to Batagan, 2011 (in Kumar, 2015, p. promote economic growth and, at the most important level, making operations and 40; services truly citizen-centric. It may be noted that smart governance is the widespread Boliver & Meijer, 2016, p. 4) adoption of a more community-based model of governance with greater connectivity being facilitated by new technologies” SPG is “the ensemble of principles, factors and capacities that constitute a form of Wil ke (2007, p. 7) governance able to cope with the conditions and exigencies of the knowledge society” 830 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION DOCTORAL CONSORTIUM DEVELOPING PUBLIC VALUES BASED AI SYSTEMS USING VALUE SENSITIVE DESIGN ERIK SLINGERLAND,1 GUIDO ONGENA,1 MARLIES VAN STEENBERGEN2 1 HU University of Applied Sciences, Research Group Process Innovation & Information Systems, Utrecht, Netherland erik.slingerland@hu.nl, guido.ongena@hu.nl 2 HU University of Applied Sciences, Research Group Digital Ethics, Utrecht, Netherland marlies.vansteenbergen@hu.nl The growing prevalence of AI systems in society, has also prompted a growth of AI systems in the public sector. There are however ethical concerns over the impact of AI on society and how this technology can impact public values. Previous works do not connect public values and the development of AI. To Keywords: address this, a method is required to ensure that developers and public public servants can signal possible ethical implications of an AI values, value system and are assisted in creating systems that adhere to public sensitive values. Using the Research pathway model and Value Sensitive design, Design, we will develop a toolbox to assist in these challenges digital ethics, and gain insight into how public values can be embedded artificial throughout the development of AI systems. intelligence DOI https://doi.org/10.18690/um.fov.4.2024.50 ISBN 978-961-286-871-0 832 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Within the public sector, the growing reliance on digitalization has prompted the rise of e-government. A domain of research within public administration focussed on utilizing digital applications in various aspects of the public domain (Bannister & Connolly, 2014). The implementation of these digitalisation applications is not always successful and can have serious ethical implications. An example is the reveal of the NSA surveillance activities which sparked a global debate surrounding the balance between the values of privacy and (inter)national security (MacAskill et al., 2013). More recently in the Netherlands, the child benefit scandal surfaced, where the illegitimate use of personal information, led to parents incorrectly being classified as fraudulent by algorithms (Autoriteit Persoonsgegevens, 2020). The increasing potential of Artificial intelligence (AI) systems has prompted public servants to utilize this technology in the public domain but, as the example demonstrates, not always with positive outcomes for citizens. In the proposed AI act, the European Parliament defined several requirements for the use of AI in Europe. These also include public values like equality, sustainability, and transparency (European Parliament , 2023). The demarcation of what constitutes a public value and how these values relate is ambiguous, for example in the relation and distinction between transparency and openness (Meijer, 2013; Whittlestone et al., 2019). Various researchers state that the development and use of technology contain underlying values. Technology is increasingly viewed as a socio- technical system, which focuses on the reciprocal interaction between humans and technology. (Bannister & Connolly, 2014; Flanagan et al., 2008). Achieving values like fairness in these socio-technical systems, is only possible when examining both the social and technical aspects of a system (Selbst et al., 2019). Currently, there is a gap between the implementation of AI in the public domain and research into public values. To contribute to the implementation of AI systems that adhere to public values, this research aims to answer the following question: How can public values be implemented and validated in Artificial Intel igence systems in the public domain? E. Slingerland, G. Ongena, M. van Steenbergen: Developing Public Values Based AI Systems Using Value Sensitive Design 833 This research question will be inquired from two perspectives. Firstly, the process of identifying and operationalizing public values for AI systems and secondly developing tools to implement these public values in AI systems or validate their presence. 2 Related Work The concept of embodied values states that a digital application derives its ethical value from a combination of its designed properties and its usage (Flanagan et al., 2008; van de Poel, 2020). This is related is based on two ethical concepts. Firstly, normative ethics aims to judge morality and formulate recommendations about how to act or live. Secondly, value theory states that we can make evaluations of technical artefacts based on ethical values. These values are lasting convictions or matters that people feel ‘ought to be.’ (van de Poel & Royakkers, 2011). By using this as the basis of an ethical framework, a digital application can be examined on how it contributes to or disrupts the presence of a specific normative value. In the coming section, the concept of public values and a method for designing AI systems to adhere to values are explored. Within this research, an AI system is defined according to Article 3 of the proposed AI act. This definition is useable within the context of e-government as it has political support and international recognition. ‘An AI system is a machine-based system designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments’ (Artificial Intelligence Act, 2024). 2.1 Public Values Within the field of public administration, there has been a shift towards policy based on public values (Molina & McKeown, 2012). The previous economic approach to decision-making was criticised, as it did not account for the broader societal impact of policy. This slowly evolved into the concept of public interest and prompted policymakers to consider public values (Bozeman, 2002). A commonly cited definition of public values is: “…values providing normative consensus about (a) 834 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION the rights, benefits, and prerogatives to which citizens should (and should not) be entitled; (b) the obligations of citizens to society, the state, and one another; and (c) the principles on which governments and policies should be based” (Fukumoto & Bozeman, 2018). This paper looks at public values based on the third aspect of this definition. These public values also apply when we look at IT innovations in the public domain. Socio-technical systems have the potential influence and be influenced by values (Bannister & Connolly, 2014). Researchers therefore cal upon the public sector to recognise that technology is not neutral and has underlying values in its usage. This is reflected in European regulations like the European Data Protection Act and AI Act, which add various ethical obligations (Royakkers et al., 2018). Some researchers also reject the notion that there is a one-dimensional set of public values that can be defined. Values can overlap, have different meanings depending on the context and derive their importance from the social context. A practical approach to deal with these conflicts and overlap is to define concrete and measurable conceptualisations of public values and make a context-dependent decision on which values to include in a system (Wal & Van Hout, 2009). 2.2 Value Sensitive Design Value Sensitive Design (VSD) was developed as a theoretical approach for designing technology that accounts for human values in a principled and comprehensive manner through the design process. By investigating a design question from conceptual, empirical, and technical perspectives with various techniques, the developer can establish ethical requirements for an artefact and develop a plan on how to achieve them (Friedman et al., 2013). VSD contains various techniques like the stakeholder analysis and value source analysis that can be used in the conceptual and empirical investigation to establish stakeholders values and use them in the design of an artifact (Friedman et al., 2017). Applying VSD in the design of AI systems prompts unique chal enges. For example, the capability of some AI systems to adjust their behaviour over time, can cause them to disembody a value for which it was designed (Tsamados et al., 2022; Umbrello & Yampolskiy, 2022). To account for this, a few design methods have been proposed (Sadek et al., 2023). An example is Umbrel o & van de Poel who expand the scope of VSD to the entire lifecycle of an AI system. This method maps the investigations of VSD into four activities: E. Slingerland, G. Ongena, M. van Steenbergen: Developing Public Values Based AI Systems Using Value Sensitive Design 835 context analysis, value identification, design requirements and prototyping. These activities are a cyclical process that go through multiple iterations, as illustrated in Figure 1. By utilising these steps throughout the lifecycle of an AI system, the value- sensitive design process for AI technologies (VSD for AI) al ows users to determine whether the system stil adheres to normative values through its deployment and if necessary adjust the system (Umbrello & van de Poel, 2021). Figure 1: Values sensitive design process for AI technologies Source: adapted from Umbrello & van de Poel, 2021 There is a gap in linking public values and VSD. There are various papers related to the development of AI systems that adhere to values like fairness and transparency, but there is little research with a focus on public values. Approaching public values as context-dependent phenomenon, al ows for VSD to operationalise these public values with techniques like stakeholder and value source analysis. This could create a practical approach to the development of AI systems that adhere to public values. 836 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 3 Methodology To structure the research, the research pathway model is used. In this model, the trajectory is positioned in theoretical, conceptual and practical contexts. This ensures both scientific rigour and practical relevance. In these contexts; creation, exploration and delivery activities are employed (van Beest et al., 2021). Public values are approached as normative values that systems can be tested against. To guide developers and public servants through the actions in VSD for AI, a toolbox wil be developed. The toolbox wil also include an instrument to measure the degree to which an AI system embodies various public values. This will allow developers and public servants to examine the AI systems periodically and signal whether the system stil embodies the intended values. The toolbox wil consist of three main components: 1. A method for mapping relevant public values during the design of an AI system 2. A library of code chunks and design patterns to assist during development. 3. An instrument for testing and evaluating an AI system on public values. In the fol owing section, the structure from the Research Pathway Model wil be used to examine the development of the toolbox. This is also visualised in Figure 2. Creation phase As the problem has been identified, the project starts by investigating current state- of-the-art knowledge and assessing the needs of stakeholders. For this task, five activities have been identified. To gain insight into the theoretical context surrounding public values, a literature review on public values is conducted. This will be used as input for a Delphi study. This Delphi study with domain experts is used to create an initial prioritization of public values to include in the theoretical framework. Based on this prioritization, the values will be conceptualized so norms and measures can be identified for each value. In the conceptual context, the prototype of the toolbox wil be developed with a focus group using techniques and principles from VSD as inspiration. Lastly in the practice context, interviews will be conducted with AI developers and public servants in the public domain to gain a deeper understanding of the context in which the toolbox wil be deployed. E. Slingerland, G. Ongena, M. van Steenbergen: Developing Public Values Based AI Systems Using Value Sensitive Design 837 Exploration phase The exploration phase consists of an iterative process with three main activities. The phase starts by using the input from the creation phase to form the prototype of the toolbox. Secondly, the prototype is tested as an experiment with a test and control group (Mettler et al., 2014). This experiment wil be evaluated on two main measures. Firstly, the participants wil be interviewed to establish their awareness of the ethical implications before and after using the toolbox. Secondly, the final AI system wil be examined by a focus group of ethics and AI experts to examine whether the developed AI system embodies the values that were defined at the start of the project. By doing this for the test group who utilized the toolbox and the control group who did not, it is possible to establish the validity of the toolbox and examine whether participants have an increased ethical awareness surrounding AI systems. Lastly, the framework of public values is reevaluated and redefined where necessary. The new framework iteration and the outcome of the experiment are used to redesign the toolbox for a new iteration. This process is repeated until a final version is reached. The protocol for these experiments and evaluations is being developed. Delivery phase Figure 2: Visualisation of the methodology based on the research pathway model with the methods used. 838 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The delivery phase is concerned with communicating the research results to the public. Three main activities are defined. Firstly, the framework of public values in AI development wil be published as a paper to be used in further research. Secondly, the toolbox wil be released as an open-source application and be accompanied by a paper detailing the development process. Lastly, to ensure that the toolbox wil be properly maintained, a consortium of partners wil be realized from the actors involved during the research. This consortium will be tasked with maintaining the toolbox and organizing workshops to instruct new parties in how to use the toolbox once it has been released. 4 Future development and next steps The first step is to create an initial framework for public values. Currently, the literature review is completed, and a Delphi study is being prepared. In a Delphi Study, domain experts are asked about their opinion anonymously about an issue. This is done by seeking consensus between the experts over a series of rounds (von der Gracht, 2012). Our Delphi study will be conducted with experts in ethics, digital ethics, AI and e-government. It consists of two phases. In the first phase, the participants receive a list of public values distilled from the literature review. The participants are tasked with eliminating overlapping values. Here consensus will be defined per public value where the majority must agree on its inclusion in at least two consecutive rounds. In the second phase, the participants are tasked with ranking the remaining values on importance. This wil be done using a tournament ranking system, based on the Q methodology (Brown, 1996). Each participant sees sets of two public values. For every set, the participant specifies which value is more important. Here consensus is reached when after two consecutive rounds, the ranking does not shift. The resulting list of public values is used to create the initial framework. To operationalize these values, they need to be conceptualized. This involves specifying the values to a concrete norm and defining requirements that it can be tested against (Veluwenkamp & Van Den Hoven, 2023). To define the norms and requirements, a combination of literature, (inter)national laws, interviews with domain experts and industry standards wil be used. An example would be taking the value of privacy, using a norm from the European Data Protection Act and linking these to an ISO requirement. This will result in a structured approach to operationalizing public values into measurable requirements that AI systems can be evaluated against. E. Slingerland, G. Ongena, M. van Steenbergen: Developing Public Values Based AI Systems Using Value Sensitive Design 839 References Autoriteit Persoonsgegevens. (2020). De verwerking van de nationaliteit van aanvragers van kinderopvangtoeslag [Report]. Autoriteit Persoonsgegevens. https://www.rijksoverheid.nl/documenten/rapporten/2020/07/17/de-verwerking-van-de- nationaliteit-van-aanvragers-van-kinderopvangtoeslag Bannister, F., & Connol y, R. (2014). ICT, Public Values and Transformative Government: A Framework and Programme for Research. Government Information Quarterly, 31. https://doi.org/10.1016/j.giq.2013.06.002 Bozeman, B. (2002). Public-Value Failure: When Efficient Markets May Not Do. Public Administration Review, 62, 145–161. https://doi.org/10.1111/0033-3352.00165 Brown, S. R. (1996). Q Methodology and Qualitative Research. Qualitative Health Research - QUAL HEALTH RES, 6, 561–567. https://doi.org/10.1177/104973239600600408 European Parliament. (2023, June 8). EU AI Act: First regulation on artificial intelligence. EU AI Act: First Regulation on Artificial Intelligence. https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first- regulation-on-artificial-intel igence Flanagan, M., Howe, D. C., & Nissenbaum, H. (2008). Embodying Values in Technology: Theory and Practice. In J. Van Den Hoven & J. Weckert (Eds.), Information Technology and Moral Philosophy (1st ed., pp. 322–353). Cambridge University Press. https://doi.org/10.1017/CBO9780511498725.017 Friedman, B., Hendry, D. G., & Borning, A. (2017). A Survey of Value Sensitive Design Methods. Foundations and Trends® in Human–Computer Interaction, 11(2), 63–125. https://doi.org/10.1561/1100000015 Friedman, B., Kahn, P. H., Borning, A., & Huldtgren, A. (2013). Value Sensitive Design and Information Systems. In N. Doorn, D. Schuurbiers, I. van de Poel, & M. E. Gorman (Eds.), Early engagement and new technologies: Opening up the laboratory (pp. 55–95). Springer Netherlands. https://doi.org/10.1007/978-94-007-7844-3_4 Fukumoto, E., & Bozeman, B. (2018). Public Values Theory: What Is Missing? The American Review of Public Administration, 49, 027507401881424. https://doi.org/10.1177/0275074018814244 MacAskil , E., Dance, G., Cage, F., Chen, G., & Popovich, N. (2013, November 1). NSA files decoded: Edward Snowden’s surveil ance revelations explained. The Guardian. http://www.theguardian.com/world/interactive/2013/nov/01/snowden-nsa-files- surveillance-revelations-decoded Meijer, A. (2013). Understanding the Complex Dynamics of Transparency. Public Administration Review, 73(3), 429–439. https://doi.org/10.1111/puar.12032 Mettler, T., Eurich, M., & Winter, R. (2014). On the Use of Experiments in Design Science Research: A Proposition of an Evaluation Framework. Communications of the Association for Information Systems, 34, 223–240. https://doi.org/10.17705/1CAIS.03410 Molina, A., & McKeown, C. (2012). The Heart of the Profession: Understanding Public Service Values. Journal of Public Affairs Education, 18, 375–396. https://doi.org/10.2307/23208659 Proposal for a Regulation of the European Parliament and of the Council Laying down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts, Council of the European Union (2024). https://data.consilium.europa.eu/doc/document/ST-5662-2024-INIT/en/pdf Royakkers, L., Timmer, J., Kool, L., & Van Est, R. (2018). Societal and ethical issues of digitization. Ethics and Information Technology, 20(2), 127–142. https://doi.org/10.1007/s10676-018- 9452-x Sadek, M., Calvo, R. A., & Mougenot, C. (2023). Designing value-sensitive AI: A critical review and recommendations for socio-technical design processes. AI and Ethics. https://doi.org/10.1007/s43681-023-00373-7 840 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and Abstraction in Sociotechnical Systems. Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–68. https://doi.org/10.1145/3287560.3287598 Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022). The ethics of algorithms: Key problems and solutions. AI & SOCIETY, 37. https://doi.org/10.1007/s00146-021-01154-8 Umbrello, S., & van de Poel, I. (2021). Mapping value sensitive design onto AI for social good principles. AI and Ethics, 1, 3. https://doi.org/10.1007/s43681-021-00038-3 Umbrello, S., & Yampolskiy, R. V. (2022). Designing AI for Explainability and Verifiability: A Value Sensitive Design Approach to Avoid Artificial Stupidity in Autonomous Vehicles. International Journal of Social Robotics, 14(2), 313–322. https://doi.org/10.1007/s12369- 021-00790-w van Beest, W., Boon, W., Andriessen, D., Pol, H., van der Veen, G., & Moors, E. (2021). A Research Pathway Model for evaluating the implementation of practice-based research: The case of self-management health innovations. Research Evaluation, 31. https://doi.org/10.1093/reseval/rvab023 van de Poel, I. (2020). Embedding Values in Artificial Intelligence (AI) Systems. Minds and Machines, 30(3), 385–409. https://doi.org/10.1007/s11023-020-09537-4 van de Poel, I., & Royakkers, L. M. M. (2011). Ethics, technology, and engineering: An introduction. Wiley-Blackwell. Veluwenkamp, H., & Van Den Hoven, J. (2023). Design for values and conceptual engineering. Ethics and Information Technology, 25(1), 2. https://doi.org/10.1007/s10676-022-09675-6 von der Gracht, H. A. (2012). Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536. https://doi.org/10.1016/j.techfore.2012.04.013 Wal, Z., & Van Hout, E. (2009). Is Public Value Pluralism Paramount? The Intrinsic Multiplicity and Hybridity of Public Values. Intl Journal of Public Administration, 32, 220–231. https://doi.org/10.1080/01900690902732681 Whittlestone, J., Nyrup, R., Alexandrova, A., & Cave, S. (2019). The Role and Limits of Principles in AI Ethics: Towards a Focus on Tensions. 195–200. https://doi.org/10.1145/3306618.3314289 DOCTORAL CONSORTIUM HOW ARE COMPANIES STRATEGIZING WITH DIGITAL TRANSFORMATION TO INTEGRATE SUSTAINABILITY OBJECTIVES? FUAD KHAN University of Turku, Faculty of Information Systems Science, Turku, Finland fuad.khan@utu.fi Digitalization offers innovative solutions through which firms can address sustainable chal enges and be sustainable in the long run. The implementation of digitalization for sustainability sake can create both positive and negative impacts internal y and externally to a business. While digital technologies can optimize resource management and enhance social outcomes, they also pose risks of increased energy consumption and e-waste generation. This research aims to explore how firms can balance the benefits of digitalization with that of the associated risks to enhance sustainability performance. By investigating the dynamics between digital transformation strategies, business models, and sustainability, it seeks to create new knowledge on the method of how digital sustainability should work. The paper will employ a multiple case study approach with qualitative and quantitative data collection methods. Although the research is in Keywords: its initial stages, it anticipates that businesses will increasingly digitalization, adopt digital transformation to comply with environmental sustainability, regulations. The study expects to contribute in digital strategy, business sustainability strategy in response to evolving regulatory model, landscapes. technologies DOI https://doi.org/10.18690/um.fov.4.2024.51 ISBN 978-961-286-871-0 842 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Digitalization is essential for sustainable development. Digital technologies can address global chal enges like climate change and inequality with new business models and ways of working (Millet, 2020). As organizations continue to leverage the power of technology to improve operational efficiency and customer experience, they are also becoming increasingly aware of the impact of digitalization on sustainability. While some firms have sustainable practices, the number is not significant, and can vary by different industry types. Digital technologies can solve this disparity and henceforth increase firms’ sustainable conducts. The United Nations Global Compact (UNGC) has highlighted the critical role of digitalization in achieving the Sustainable Development Goals (SDGs) and cal ed on companies to integrate digitalization into their sustainability strategies ( EU Statement – UN Global Digital Compact: Deep Dive on Accelerating Progress on the SDGs, n.d.). 2 Problem definition The relationship between digitalization and sustainability is complex and multifaceted, with both positive and negative impacts on sustainability performance. On the positive side, digitalization has the potential to reduce resource consumption and waste generation by enabling more efficient and circular business models. For example, the use of digital technologies such as the Internet of Things (IoT) and artificial intel igence (AI) can optimize supply chain management, energy consumption, and waste management (Soori et al., 2023). Furthermore, digitalization can enhance social outcomes by increasing access to education, healthcare, and other services, as wel as promoting transparency and accountability through the use of digital platforms (Brenner & Hartl, 2021). On the negative side, digitalization can increase energy consumption and e-waste, as wel as create new privacy and security risks. For example, the growing use of data centers and cloud computing has led to significant increases in energy consumption, with data centers accounting for approximately 1% of global electricity consumption (Rong et al., 2016). Moreover, the proliferation of digital devices and platforms has resulted in a massive amount of electronic waste, with only a smal percentage being properly recycled or disposed of (Schaltegger et al., 2016). In addition, the collection and use of personal data by digital platforms raise concerns about privacy and security, as wel as the potential for discrimination and bias (World Economic Forum: Annual Report 2021-2022, F. Khan: How are Companies Strategizing with Digital Transformation to Integrate Sustainability Objectives? 843 2023). Therefore, the question arises that how can firm balance between maximizing the benefits of digitalization in the pursuit of sustainability performance and reporting enhancement and minimizing potential risks and costs related to their digital initiatives? Current body of research on digital transformation has an underlying focus that emphasizes on the economic feasibility of firms. By using new digital technologies, firms innovate new digital value propositions and offerings. Such improvements are necessary for a firm to remain relevant and competitive in the converging digital markets, where incumbent firms, digital startups and software firms compete and collaborate. A founding assumption in digital transformation research is that the pressure to improve competitiveness in order to remain relevant in converging digital markets drives digital transformation. (Baiyere et al., 2020) However, digital transformation and the development of digital business models and value offerings, can also have both positive and negative effects on environmental and social sustainability. Such acknowledgement in prior digital transformation research is not prevalent. Therefore, this research intends to gain knowledge and answer such gaps by investigating the dynamics between a firm’s digital transformation strategy, business model and sustainability. Such knowledge would pave the way for a better firm strategy for matured or newer firms that intend to create values with their sustainability performance. 3 Methodology For this research method, multiple case study has been chosen. Within a single study, multiple cases would be assessed due to the fact that multiple firms are being examined. This research will also follow multiple level of analysis (Yin, 1984) within each single case, for example investigating digital strategy that often is established in the corporate function and studying value proposition or business model which is seen in the operational level. Such multiple level analysis wil also be employed in firm and industry level, which is one of the elements in this research design. 844 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION In a case study, the data to be collected can be both qualitative and quantitative. Both primary and secondary data are good sources of data collection. For example, interviews and questionnaire are considered as qualitative primary data collection methods to find evidence of the phenomenon. However quantitative format such as questionnaire can also complement the qualitative method if necessary. Secondary data sources are annual and sustainability reports, 3rd party reports, archives, newsfeed etc. which publish quantifiable numbers that may complement the primary data collected or vice versa. Essentially, case study is suitable for this research as it can help a researcher to describe the phenomenon (Kidder, 1982), test a prebuilt or conceived theory on a specific case (Pinfield, 1986; Anderson,1983) or perhaps create a theoretical framework based on the case learning (Gersick, 1988; Harris & Sutton, 1986). (Eisenhardt, 1989) 4 Expected results The research is in its initial stage. We have not been able to collect enough data to forecast an expected result. 5 Future Development So far, we have observed continuous regulations and legislations imposed on businesses to limit their carbon emission and move towards a state of environmental y friendly operation. We expect that more and more businesses wil adapt to this changing and tighter environmental regulation through digital transformation or other means that wil help abide the regulation. We can also expect more regulation on the technology side. For example, as data-based service and AI i.e., generative AI becomes more accessible and applied by businesses in their operation, we can expect new regulations on the use of technologies. Inevitably, we will likely see new strategies and business models that effectively use digital technologies to achieve sustainability and other core organizational goals. References Anderson, P. (1983) Decision making by objection and the Cuban missile crisis. Administrative Science Quarterly, 28, 201-222. F. Khan: How are Companies Strategizing with Digital Transformation to Integrate Sustainability Objectives? 845 Brenner, B., & Hartl, B. (2021). The perceived relationship between digitalization and ecological, economic, and social sustainability. Journal of Cleaner Production, 315, 128128. https://doi.org/10.1016/j.jclepro.2021.128128 Eisenhardt, K. M. (1989). Building Theories from Case Study Research. Academy of Management Review, 14(4), 532. https://doi.org/10.2307/258557 EU Statement – UN Global Digital Compact: Deep Dive on Accelerating progress on the SDGs. (n.d.). EEAS. https://www.eeas.europa.eu/delegations/un-new-york/eu-statement- %E2%80%93-un-global-digital-compact-deep-dive-accelerating-progress-sdgs_en?s=63 Gersick, C. (1988) Time and transition in work teams: Toward a new model of group development. Academy of Man- agement Journal, 31, 9-41. Harris, S., & Sutton, R. (1986) Functions of parting ceremo- nies in dying organizations. Academy of Management Journal, 29, 5-30. Kidder, T. (1982) Soul of a new machine. New York: Avon. making. Administrative Science Quarterly, 31, 365-388. Mil et, T. (2020, January 1). How to weave ESG factors into your digital strategy. EY Canada. https://www.ey.com/en_ca/mining-metals/how-to-weave-esg-factors-into-your-digital- strategy Pinfield, L. (1986) A field evaluation of perspectives on orga- nizational decision Rong, H., Zhang, H., Xiao, S., Li, C., & Hu, C. (2016). Optimizing energy consumption for data centers. Renewable & Sustainable Energy Reviews, 58, 674–691. https://doi.org/10.1016/j.rser.2015.12.283 Schaltegger, S., Hansen, E. G., & Lüdeke-Freund, F. (2016). Business Models for Sustainability: Origins, Present Research, and Future Avenues. Organization & Environment, 29(1), 3–10. https://www.jstor.org/stable/26164751 Soori, M., Arezoo, B., & Dastres, R. (2023). Internet of things for smart factories in industry 4.0, a review. Internet of Things and Cyber-Physical Systems, 3, 192–204. https://doi.org/10.1016/j.iotcps.2023.04.006 World Economic Forum: Annual Report 2021-2022. (2023, October 9). World Economic Forum. https://www.weforum.org/publications/annual-report-2021-2022/ Yin, R. (1984) Case study research. Beverly Hills, CA: Sage Publications. 846 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION DOCTORAL CONSORTIUM ENHANCING PROCESS MODEL VISUALISATION TO FACILITATE THE UNDERSTANDING OF STAKEHOLDERS IRIS MULDER University of Applied Sciences Utrecht, Utrecht, Netherland iris.mulder@hu.nl Over the past forty years, the use of process models in practice has grown extensively. Until twenty years ago, remarkably little was known about the factors that contribute to the human understandability of process models in practice. Since then, research has, indeed, been conducted on this important topic, by e.g. creating guidelines. Unfortunately, the suggested modelling guidelines often fail to achieve the desired effects, because they are not tied to actual experimental findings. The need arises for knowledge on what kind of visualisation of process models is perceived as understandable, in order to improve the understanding of different stakeholders. Therefore the objective of this study is to answer the question: How can process models be visually enhanced so that they facilitate a common understanding by different stakeholders? Consequently, five sub- Keywords: understanding, research questions (SRQ) will be discussed, covering three process studies. By combining social psychology and process models we model, can work towards a more human-centred and empirical-based stakeholder, designer, solution to enhance the understanding of process models by the visualisation different stakeholders with visualisation. psychology DOI https://doi.org/10.18690/um.fov.4.2024.52 ISBN 978-961-286-871-0 848 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Over the past forty years, the use of process models in practice has grown extensively. As the creation and use of process models involves a broad range of stakeholders, it is crucial that these models are intuitive and easy to understand (Dumas et al., 2018; Mendling et al., 2007; Reijers and Mendling, 2011). The notion of ‘process model’ is a broad one, also involving a wide range of notations and visualisations. We regard process models as simplified and abstract representations of systems, and their interactions, that are essential for a particular purpose (Geissdoerfer et al., 2018; Kerim, 2023; Smirnov et al., 2012). Until twenty years ago, remarkably little was known about the factors that contribute to the human understandability of process models in practice (Mendling et al., 2007; Petrusel and Mendling, 2013). Since then, research has, indeed, been conducted on this important topic, by e.g. creating guidelines (Slagter et al., 2017; Mendling et al., 2010). Most guidelines focus on the visualisation of process models and are intended for the designer to use when model ing processes. When choosing a guideline, designers consider multiple factors, including the audience and purpose of the model. Unfortunately, the suggested modelling guidelines often fail to achieve the desired effects, because they are not tied to actual experimental findings (Mendling, 2012). The practical need for more transparent and understandable processes grows more important in light of the transition to a more circular economy. This increasing need is illustrated and underlined by the VMRG (the Dutch industry organisation for metal facades). The VMRG closes the leakage flows of materials from supply chains and moves towards zero carbon in the construction sector. This VMRG achieves with a process-based approach by using the Business Process Model ing Notation (BPMN). Fledderman (2023) from VMRG states, “Formulating and model ing processes [.. ] is done from the user perspective. This, knowing that on the one hand, we have to support processes with digitalisation, [.. ], and on the other hand, that we have 1000+ companies as users, with a very diverse level of knowledge, to include in the communication. We are already experiencing user-(director)-designer communication issues in daily practice. How do we ensure that we do this bet er and more ef ectively?” In the Dutch construction sector, in which VMRG operates, VISI software is the mandatory standard for the exchange of construction process information (Bakker & Spees, 2024). The basis of this mandatory standard is the process model notation Design and Engineering Methodology for Organisations (DEMO), stressing the importance of understandable process notation. I. Mulder: Enhancing Process Model Visualisation to Facilitate the Understanding of Stakeholders 849 In practice, there are currently several drawbacks and chal enges concerning process models when presented to the intended readers, to stakeholders. Based on Freeman (2020), we define a stakeholder as a group or an individual that affects or is affected by process models (e.g. executives, customers, employees, management, investors, media). Researchers are taking steps to map and improve said drawbacks and challenges. Mulder (2019) highlights this by showing five cases that lack communicability with different stakeholders using the same process model, based on DEMO. Simply because process models are not usual y designed with different stakeholders in mind. Different stakeholders have different factors contributing to their understanding. Experiences from practice show that the human element is often overlooked (Jans, 2023), especial y for the stakeholders who do not (want to) see process models regularly. As Van Gils (2023) stated when interviewed: “[designers] need to create one process model with multiple visualisations for multiple stakeholders at multiple levels of abstraction” . For example, a process model as shown in figure 1, used in the construction sector, might not be perceived as transparent and understandable to other stakeholders and could be interpreted differently. The need arises for knowledge on what kind of visualisation of process models is perceived as understandable, in order to improve the understanding of different stakeholders. Figure 1: Example of a BPMN process model Source: Own Stakeholders might need to make decisions based on the process model from the previous example, shown in Figure 1. If these stakeholders base their decisions on the (wrong) assumption of understanding, this could lead to negative effects, e.g. planning risks or negative financial impact. Thus, the designers must not make the reading, understanding and visualisation of the process model harder than it needs to be. Based on Mandelburger (2021), we define a designer as the person who makes a process model, e.g. architects, process analysts, and others. 850 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The general population familiar with complex visualisations is growing, although the number of people able to deduce the relevant information from these visualisations is difficult to estimate (Boy et al., 2014), this is called visual literacy. In the proposed study we define visual literacy as the ability to effectively, efficiently, and confidently understand, use, create, and extract information from wel -established data visualisations, inferred from Aisami (2015) and Lee (2016). In today’s society, with more individuals using and encountering visualisation, the need grows for individuals to become more visually literate regarding complex visualisations (Börner et al., 2019; Kiper et al., 2012). Current research into the understanding of process models mainly comes from the enterprise engineering and process model ing communities. Interdisciplinary research has much potential to address some of the larger problems facing organisations (Waldman, 2013). So far, the number of interdisciplinary studies is limited, an overview of several relevant studies can be found in Table 1. For example, Rosenthal et al. (2022) combine process model ing and cognitive psychology to look into the understanding of process models. Cognitive psychology focuses on the inner workings of the brain. However, because we focus on the stakeholder’s understanding of visual process models the domain of social psychology may be more relevant. The study of social psychology scientifically investigates how individuals think, feel, and behave in a social context (Kassin, 2023). Social psychology has a diverse set of topics, everything the stakeholder thinks and therefore acts on fal s within this scope. It examines both the ‘social’ and the ‘non- social’ factors that affect people. Accordingly, how designers visual y influence the stakeholders through process models fal s wholly within social psychology. By combining social psychology and process models we can work towards a more human-centred and empirical-based solution to enhance the understanding of process models by the different stakeholders with visualisation. What are process models? Every notation al ows for an “informational payload” needed for the various kinds of decisions stakeholders need to make. Therefore, with its (informational) payload, each notation serves a specific purpose and might not be suitable for use in every situation. There are many process model ing notations available (e.g. BPMN, DEMO, VISI, Flowchart, ArchiMate, EPC, DFD, IDEF0, Petrinet) (B). Within this PhD research, we wil focus on two notations that have significance for practice: BPMN and DEMO. I. Mulder: Enhancing Process Model Visualisation to Facilitate the Understanding of Stakeholders 851 Table 1: Overview of interdisciplinary studies Study Area of interest PM CP SP ET V U Other BPMN DEMO Abbad et al. (2023) X X Bera et al. (2019) X X X Figl (2017) X X Hipp et al. (2014) X X Malinova Mandelburger and X X Mendling (2021) Mendling et al. (2007) X X Petrusel and Mendling X X (2013) Reijers and Mendling (2011) X X Rosenthal et al. (2022) X X X Zimoch et al. (2017) X X Note. PM: Process modelling. CP: Cognitive Psychology. SP: Social Psychology. ET: Eye-tracking. V: Visualisation. U: Understanding. BPMN is a functional notation used to model the activities and decisions of an organisation (OMG, 2013). This notation focuses on the “how” of the organisation. BPMN is used frequently because people focus more on the “how” of the organisation. BPMN allows for a payload of e.g. functions, activities, decisions, and sequences (OMG, 2013). DEMO is an abstract method, containing a notation to model the construction of an organisation (Dietz and Mulder, 2020). This method abstracts the organisation from implementation and realisation. It gives a view of the “what” of the organisation. DEMO is also often used but less frequently than BPMN because people focus less on the “what” of their organisation. DEMO al ows for a payload of e.g. responsibilities, products, roles, functions, data, rules, and dependencies (Dietz and Mulder, 2020). DEMO and BPMN are important to study due to their significance for practice and are complementary to each other as they al ow for different payloads serving different stakeholders. Why is visualisation so important to understand? So far, there is no standardised terminology, typology, or classification system for core visualisation concepts anywhere (Börner et al., 2019); visualisation can mean various things to different people (Csinger, 1992). Visual literacy is becoming as important as the ability to read 852 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION and comprehend text (Lee et al., 2016). Despite the importance of visual literacy and visualisation, researchers have paid little attention to the application and development of visual literacy (Kiper, 2012). As visualisation plays a crucial and essential role in communicating models (Hoppenbrouwers et al., 2012), being able to measure visual literacy is increasingly important. To our knowledge, only a few visual literacy scales exist (e.g. Aisami, 2015; Boy et al. 2014; Kiper et al., 2012), and they are not widely accessible. When using process models for communication, it is essential to be aware of the different views people involved have (Hoppenbrouwers et al., 2012); all meaning is relative to culture (Ware, 2004). Thus, stakeholders from different cultures and with different knowledge could have a different understanding of one specific process model visualisation (A, B). 2 Problem Definition As mentioned, different stakeholders have different visual needs that need to be met to communicate process models effectively (Börner et al., 2019; Hoppenbrouwers et al., 2012). The designer does not always keep the different stakeholders and their level of understanding in mind. This requires a balance between human-oriented communication and rational engineering, which can be described as “a chal enge and often a bit of a struggle” as you cannot assume that al people are familiar with process models (Hoppenbrouwers et al., 2012). Additionally, designers often make the mistake of too easily considering a process model to be effectively communicated without thorough validation. Similarly, stakeholders tend to assume they possess a sufficient understanding of process models and their visualisations. This proposed PhD study aims to enhance the understanding of process models of the different stakeholders in practice. 2.1 Research question As discussed above, the biggest chal enge to increasing the added value of process model ing in practice, is to enhance the understanding of process models by stakeholders. This research aims to develop an artefact that helps to improve the visualisation of process models to enhance common understanding and thereby increase the value of using process models in practice. We work towards solving this I. Mulder: Enhancing Process Model Visualisation to Facilitate the Understanding of Stakeholders 853 chal enge by combining the enterprise engineering and process model ing domains with the social psychology research domain. Based on the previous sections, the following main research question is formulated: How can process models be visual y enhanced so that they facilitate a common understanding by dif erent stakeholders? To answer the main research question the following sub-research questions (SRQ) are formulated: SRQ 1: What is known about the occurrence of misunderstanding when reading process models? SRQ 2: What are the visually oriented factors that facilitate the understanding of process models? With SRQ one and two, we establish the current state-of-the-art regarding this topic. This al ows us to more accurately define research gaps and future research that we may have missed before. SRQ 3: What is the difference between different types of stakeholders when reading and interpreting process models? SRQ 4: What aspects related to visualisation will improve the understanding of process models by different types of stakeholders? With SRQ three and four, we wil measure understanding of process models among different types of stakeholders (e.g. experts and non-experts) to develop a baseline of understanding without interventions. With this baseline, we have a foundation to experimental y compare potential improvements. SRQ 5: What artefact can be designed to be useful in facilitating the understanding of process models in relation to their visualisation? 854 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION With SRQ five, we want to improve the visualisation of process models and research what kind of artefact would be best for that. With this question, we can see what direction can work to help the practice 3 Methodology To answer the main research question and the sub-research questions discussed in the previous section, the proposed research project consists of three studies. The first study wil be a systematic literature review to establish the current state-of-the- art regarding this topic. The second study wil be an eye-tracking study combined with in-depth interviews to develop a baseline of understanding without interventions. The third study wil be the designing and testing of an artefact fol owing Design Science Research (DSR) to develop the artefact while using eye- tracking as a key instrument in evaluating the artefact. 3.1 Study 1: Literature Review To answer SRQs 1 and 2, study 1 will be a systematic literature review. The study wil use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach (Page et al., 2021). This will help provide the knowledge foundation regarding process model understanding and visual factors that facilitate understanding. Furthermore, the aim is to identify knowledge gaps in the literature from the stakeholder perspective. This review wil be the foundation for the entire PhD project. However, as it is important to be up to date with recent literature, updating this foundation is something that wil be done during the entire PhD period. 3.2 Study 2: Eye-tracking To answer SRQs 3 and 4, study 2 wil consist of two parts. The first part wil be an eye-tracking experiment. When reading and interpreting process models specific links between attention distribution and task performance have not yet been established (Bera et al., 2019). The advantage of eye-tracking is its ability to measure physiological responses to visual stimuli and record these responses in real-time (Hassan and Bialowas, 2017). Employing eye-tracking to compare various process model ing notations provides valuable insights into how stakeholders understand I. Mulder: Enhancing Process Model Visualisation to Facilitate the Understanding of Stakeholders 855 process models. (Zimoch et al., 2017). With eye-tracking, we aim to find differences in how types of stakeholders (e.g. experts and non-experts) read and interpret process models. By mapping these differences, a baseline of understanding can be made for these groups. The experiment wil consist of a questionnaire about the two process model notations, Design and Engineering Methodology for Organisations (DEMO) and Business Process Model ing Notation (BPMN). The participant wil answer multiple-choice questions, minimising additional work requirements and the risk of data entry errors (Hassan and Bialowas, 2017). The second part wil be an in-depth interview. The participants of the eye-tracking experiment wil be invited for an interview to further enrich and detail their answers to the eye-tracking questionnaire. With these interviews, we aim to get a deeper understanding of what the participants' thoughts were during the questionnaire. Through this retrospective thinking, we can hear from the various groups about their thoughts and opinions. With this information, we want to learn about what might be a cause of misunderstanding and what the various stakeholders think might facilitate their understanding. 3.3 Study 3: Artefact To answer SRQ 5, study 3 wil be based on Design Science Research (DSR). With the DSR approach an artefact (e.g. a guideline) can be created based on experimental findings. The results from study two wil form the basis from which we develop an intervention targeting practice. The aim is to create an artefact or guideline that facilitates understanding significantly compared to the baseline established in study two. Through the same method used to create the baseline of understanding in study two, eye-tracking, we will evaluate whether there are significant improvements in understanding. 4 Expected Results By combining the domains of process management (model ing) and social psychology, this PhD has the potential to address some of the problems facing organisations concerning process models. Because there is limited interdisciplinary work in combining these fields, this study is one of the building blocks for more studies like these in the future and creates more direction for future research. By 856 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION focusing on the stakeholders, we can get more focus on the human element, social psychology and visualisation. We aim to explore what visual adaptations can help facilitate the understanding of process models by stakeholders. The practice has shown that there are currently several drawbacks and chal enges concerning process models when presenting these to stakeholders. It is essential to be aware of the different kinds of stakeholders because the interpretation of models is contextual (e.g. relative to international/organisational culture as all meaning is relative to culture). The results of the three studies will help facilitate the understanding of the stakeholders and thus help practice make wel -informed decisions based on process models. 5 Future Development This research explores the visualisation and subsequent understanding of process models. This is not only an essential topic for practice but also for education. Using visuals in various learning environments is a vital learning enhancer (Aisami, 2015), making it a valuable tool for educational applications. Visualisation will improve how education is taught and help the students prepare for practice. References Abbad-Andaloussi, A., Lübke, D., & Weber, B. (2023). Conducting eye-tracking studies on large and interactive process models using EyeMind. SoftwareX, 24, 101564. https://doi.org/10.1016/j.softx.2023.101564 Aisami, R. S. (2015). Learning styles and visual literacy for learning and performance. Procedia-Social and Behavioral Sciences, 176, 538–545. https://doi.org/10.1016/j.sbspro.2015.01.508 Bakker & Spees. (2024). VISI software | Slimmer samenwerken. Retrieved 1 February 2024, from ttps://www.bakkerspees.nl/visi/?gclid=EAIaIQobChMIyrHfpZuKhAMVfopoCR0TJwSfE AAYASAAEgJuPvD_BwE Bera, P., Soffer, P., & Parsons, J. (2019). Using eye tracking to expose cognitive processes in understanding conceptual models. MIS quarterly, 43(4), 1105-1126. Börner , K., Bueckle, A., & Ginda, M. (2019). Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences, 116(6), 1857–1864. https://doi.org/10.1073/pnas.1807180116 Boy, J., Rensink, R. A., Bertini, E., & Fekete, J.-D. (2014). A principled way of assessing visualization literacy. IEEE Transactions on Visualization and Computer Graphics, 20(12), 1963–1972. 10.1109/TVCG.2014.2346984 Csinger, A. (1992). The psychology of visualization. University of British Columbia, Department of Computer Science. Dietz, J. L., & Mulder, H. B. (2020). Enterprise ontology: A human-centric approach to understanding the essence of organisation. Springer Nature. I. Mulder: Enhancing Process Model Visualisation to Facilitate the Understanding of Stakeholders 857 Dumas, M., Rosa, L. M., Mendling, J., & Reijers, A. H. (2018). Fundamentals of business process management. Springer-Verlag. Figl, K. (2017). Comprehension of procedural visual business process models: a literature review. Business & Information Systems Engineering, 59, 41–67. Freeman, R. E. (2020). The Stakeholder Approach Revisited. In T. Beschorner, A. Brink, B. Hol stein, M. C. Hübscher, & O. Schumann (Eds.), Wirtschafts- und Unternehmensethik (pp. 657–671). doi:10.1007/978-3-658-16205-4_55 Geissdoerfer, M., Vladimirova, D., & Evans, S. (2018). Sustainable business model innovation: A review. Journal of Cleaner Production, 198, 401–416. https://doi.org/10.1016/j.jclepro.2018.06.240 Hassan, M., & Białowąs, S. (2017). Research design in eye-tracking experiments. Yayımlanmamış ders notu. Poznań University of Economics and Business, Poznan, 12, 2019. Hipp, M., Strauss, A., Michelberger, B., Mutschler, B., & Reichert, M. (2014, September). Enabling a user-friendly visualization of business process models. In International Conference on Business Process Management (pp. 395-407). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-15895-2_33 Hoppenbrouwers, S. J. B. A., van Stokkum, W., Iacob, M. E., Wilmont, I., Van der Linden, D. J. T., Amrit, C., & Joosen, M. (2012). Stakeholder communication. Agile Service Development: Combining Adaptive Methods and Flexible Solutions, 141-176. https://doi.org/10.1007/978-3-642-28188-4_7 Jans, C. (2023, September 18). S02E10: The BPM conference 2023. Retrieved from https://www.linkedin.com/feed/update/urn:li:activity:7108475538767253505/ on September 22, 2023 Kassin, S., Fein, S., & Markus, H. R. (2023). Social psychology. SAGE Publications. Kerim, Y. (2023). A Structured Literature Analysis on Understandability in Human-Model Interaction. Kiper, A., Arslan, S., Kıyıcı, M., & Akgün, Ö. E. (2012). Visual literacy scale: the study of validity and reliability. The Online Journal of New Horizons in Education, 2(2), 73–83. Lee, S., Kim, S. H., & Kwon, B. C. (2016). Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics, 23(1), 551-560. Malinova Mandelburger, M., & Mendling, J. (2021). Cognitive diagram understanding and task performance in systems analysis and design. MIS Quarterly, 45(4), 2101–2157. Mendling, J. (2012, June). Managing structural and textual quality of business process models. In International Symposium on Data-Driven Process Discovery and Analysis (pp. 100-111). Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-40919- 6_6 Mendling, J., Reijers, H. A., & Cardoso, J. (2007). What makes process models understandable?. In Business Process Management: 5th International Conference, BPM 2007, Brisbane, Australia, September 24-28, 2007. Proceedings 5 (pp. 48-63). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-75183-0_4 Mendling, J., Reijers, H. A., & van der Aalst, W. M. (2010). Seven process modeling guidelines (7PMG). Information and software technology, 52(2), 127-136. https://doi.org/10.1016/j.infsof.2009.08.004 Mulder, M. A. (2020). A Design Evaluation of an Extension to the DEMO Methodology: Five Model and Process Instantiations. In Advances in Enterprise Engineering XIII: 9th Enterprise Engineering Working Conference, EEWC 2019, Lisbon, Portugal, May 20–24, 2019, Revised Papers 9 (pp. 55-65). Springer International Publishing. https://doi.org/10.1007/978-3-030- 37933-9_4 OMG, Object Management Group, Business process model and notation (bpmn), version 2.0.2, 2013. URL: https://www.omg.org/spec/BPMN/2.0.2/About-BPMN. 858 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., .. & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Bmj, 372. https://doi.org/10.1136/bmj.n71 Petrusel, R., & Mendling, J. (2013). Eye-tracking the factors of process model comprehension tasks. In Advanced Information Systems Engineering: 25th International Conference, CAiSE 2013, Valencia, Spain, June 17-21, 2013. Proceedings 25 (pp. 224-239). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-38709-8_15 Reijers, H. A., & Mendling, J. (2010). A study into the factors that influence the understandability of business process models. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 41(3), 449-462. Rosenthal, K., Strecker, S., & Snoeck, M. (2023). Modeling difficulties in creating conceptual data models: Multimodal studies on individual modeling processes. Software and Systems Modeling, 22(3), 1005-1030. https://doi.org/10.1007/s10270-022-01051-8 Slagter, R.J., Hoppenbrouwers, S. J. B. A., Lankhorst, M. M., Campschroer, J. (2017). Guidelines for Model ing. In: Enterprise Architecture at Work. The Enterprise Engineering Series. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-53933-0_7 Smirnov, S., Reijers, H. A., Weske, M., & Nugteren, T. (2012). Business process model abstraction: a definition, catalog, and survey. Distributed and Parallel Databases, 30, 63–99. https://doi.org/10.1007/s10619-011-7088-5 Waldman, D. A. (2013). Interdisciplinary research is the key. Frontiers in Human Neuroscience, 7, 562. https://doi.org/10.3389/fnhum.2013.00562 Ware, C. (2004). Information visualization: perception for design. San Diego, CA, USA: Academic Press. Zimoch, M., Pryss, R., Schobel, J., & Reichert, M. (2017). Eye tracking experiments on process model comprehension: lessons learned. In Enterprise, Business-Process and Information Systems Modeling: 18th International Conference, BPMDS 2017, 22nd International Conference, EMMSAD 2017, Held at CAiSE 2017, Essen, Germany, June 12-13, 2017, Proceedings 18 (pp. 153-168). Springer International Publishing. https://doi.org/10.1007/978-3-319-59466- 8_10 DOCTORAL CONSORTIUM METHODOLOGY DEVELOPMENT FOR OPEN DATA MATURITY ASSESSMENT IN SMALL AND MEDIUM-SIZED ENTERPRISES – A LITERATURE REVIEW STAŠA BLATNIK University of Maribor, Faculty of Organizational Sciences, Kranj, Slovenia stasa.batnik@student.um.si Over the past two decades, the global movement towards open government gained momentum, aiming to leverage vast amounts of data generated by government institutions to increase citizen participation in governing processes, increase the transparency of public resource al ocation, and increase organizations’ economic value. Despite legislative initiatives promoting the use of OGD little is known about its actual use and the impact it Keywords: generates. The study aims to determine whether a model to open measure and distinguish between different levels of OD maturity data, can be made. The scope of this research includes a review of the open government existing literature on OD and OGD, and models that measure data, the OD maturity level. We analyzed the research findings of the OD maturity model, identified literature and models used to measure the OD maturity preparedness of organizations to adopt OGD in their everyday assessment, processes. Nine models that measure the maturity level for OGD maturity multi-criteria adoption have been identified. We discovered that no existing decision model is ful y comprehensive in assessing the maturity level of model, SMEs to adopt and use OGD. A model that wil explain the small and medium-sized current OD maturity level of an SME and propose individualized enterprises, actions to increase it yet needs to be developed. DEX DOI https://doi.org/10.18690/um.fov.4.2024.53 ISBN 978-961-286-871-0 860 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction Over two decades have passed since the first initiatives to open public sector information appeared. Countries and companies have established processes that generate numerous data that could present a wealth of information for the economy. In the year 2003, Slovenia passed its first Public Sector Information Access Act (Republika Slovenija, 2003) and the European Union its Directive 2003/98/EC of the European Parliament and of the Council of 17 November 2003 on the re-use of public sector information (European Commission, 2003) that initialized the collecting and publishing of open government data (OGD) that governmental institutions produce in the course of their daily processes. The initiatives expand beyond the public sector, aiming to incentivize private organizations to make their data publicly available as open data (OD). Openly accessible online portals now provide a centralized platform for users to deploy, access, and use OD and OGD as an integral part of it. Despite countries' efforts to open their data to the public, very little is known about the actual use of OD. Initial y foreseen benefits of OD were to increase transparency, enhance public engagement, and enable organizations to create added value (Attard et al., 2016). However, to this day, it remains unclear whether these benefits are being realized. In this research, the focus wil be on the possibility of exploiting OD in small and medium-sized enterprises (SMEs) to increase organizations' resilience through better-informed decision-making and expedite its response to disruptive changes in their environment. To achieve this, enterprises need the organizational and technological capacities to adopt and use the vast amounts of OD and to transform them into useful information. To facilitate OD adoption, we initially need to establish an organization’s OD maturity level. With this assessment, further actions can be identified to aid organizations' progression to higher levels of OD maturity. With design science research (DSR) methodology, we intend to create an IT artifact - a model, that wil explain the organization’s OD maturity factors and propose unique actions to advance to the next level of maturity. In the scope of this research, we wil focus on the rigor phase of the DSR, reviewing existing literature and models, and identifying their objectives, proposed dimensions, sub-dimensions, and requirements to achieve a certain level. With this analysis, we aim to identify the S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 861 existing measured aspects of OD maturity and the aspects that have previously been overlooked or inadequately explained in current models. 2 Methodology For the research, we intend to follow the Design Science Research (DSR) approach (Hevner et al., 2004) where the output is an IT artifact designed on a real-world problem. The DSR consists of three main research cycles: scientific rigor, relevance to practice, and the central cycle of design and development. The research result of DSR is a developed artifact in our case a multi-decision model. We initialize our research with a review of currently existing theories and models. In the central research cycle of design and development, we intend to employ a multi- parameter decision method the Decision Expert - DEX methodology (Bohanec & Rajkovič, 1990). A multi-criteria model implemented in a web-based solution wil enable organizations to make a self-assessment, and based on the entered values provide individualized propositions to increase its maturity level. The problem is that we do not know what factors comprehensively describe the maturity of an organization to successfully adopt, use, and advance the existing use of OD. Based on that we defined a research question: RQ1: How can we develop a multicriteria model that would distinguish between different levels of OD maturity level? RQ2: Which dimensions can be used to measure the OD maturity level of SMEs in Slovenia? For this study, we will focus on the scientific rigor phase of the DSR with a systematic literature review. We wil focus on what past research has addressed, what insights have been gained, and specifical y what models have been proposed, their objectives, focus, and dimensions measured. 862 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION After a preliminary study, based on the research question, the best keywords to answer our research question were defined: • Open data • Open government data • Open data maturity model • Open data maturity assessment • Maturity multi-criteria decision model • Small and medium-sized enterprises • DEX We reviewed the following bibliographic databases, using specific combinations of keywords: Web of Science which yielded 88 resulting publications, Scopus with 154 results, ProQuest with 32 results and Google Scholar with 30 results. To obtain results that corresponded with our research questions we excluded publications that: • Did not focus on open data, it was only mentioned, • Focuses on other keywords but does not relate to open data, • The model did not assess OD or OGD maturity level, • The maturity model did not in any way assess the implementation of OD or OGD concepts, • The language was not English. After applying the exclusion criteria, we used the “snowbal ing” method and further identified 21 publications from the references of previously gained publications. The literature review research in the end resulted in 71 publications and 9 models, that have been included in this phase of our research. S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 863 3 Results We divided the research into two areas: 1. the overal research on the open data, to establish the base for our work, to identify areas and aspects that have been addressed and areas where more research is needed; and 2. research on models that have been used, created, or adapted to measure the OD aspects. 3.1 The overall literature review Academic society has, since 2009 devoted significant attention to the issue of opening the data to the public. Since then, research has been done investigating different aspects of OD, ranging from user perspective, through quality perspective, adopted policies, research categorization, drivers and barriers to impact, and maturity views of OD. User perspectives have been investigated from various standpoints. Numerous authors have surveyed user intent to use OD by utilizing various existing models and theories e.g. Unified Theory of Acceptance and Use of Technology (Saxena & Janssen, 2017; Shao, 2023; Talukder et al., 2019; Zuiderwijk et al., 2015), Technology Acceptance Model (Weerakkody et al., 2017), Relevant Social Groups (Lassinantti et al., 2019), Information Systems Theory (Khurshid et al., 2022), and Social Cognitive Theory (H.J. Wang, 2020). The main findings suggest that the easiness of use, perceived usefulness, and social approval are the main motivations that indicate users' intention to use OD. Other research investigated the motivation that drives OD use in organizations, public institutions, and entrepreneurs (Alawadhi et al., 2021; Mustapa et al., 2022; H. J. Wang & Lo, 2020; Zhou, Wang, Jiang, et al., 2023). How user perspective influence the publication of OD, how to use OD platforms to resolve public problems, and what is the IT professional’s responsibility in OD publication (Ruijer et al., 2020; Shepherd et al., 2019). The opening of data has various anticipated benefits, as evidenced by case studies of actual OD use (Apanasevic, 2021; Coutinho & Freitas, 2021; Cruz & Lee, 2015; Jetzek et al., 2014; McBride et al., 2019; Ruijer & Meijer, 2020; Shao, 2023). The impact of open data on democratic processes has been acknowledged (Ruijer & Martinius, 2017). How the impact of OD can be measured through added value, and along what processes, during the lifecycle of OD, added value can be created has 864 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION been investigated (Attard et al., 2016; Magalhaes & Roseira, 2020). The impact of OD can be observed in the organization’s innovations (Gottfried et al., 2021; Huber et al., 2022) and in public administration operation (Apanasevic, 2021; Coutinho & Freitas, 2021; Maccani, 2016; McBride et al., 2019; Wilson & Cong, 2021). The aspect of OD policies has also been researched notably often in previous literature. A proposition of the maturity level of OD policies and their classification has been reviewed (Attard et al., 2015) and an analytical framework for the studying of OD policies developed (Ruijer & Meijer, 2020). In the initial phases of OD implementation countries adopted various OD policies resulting in different levels of implementation and usage. A benchmark for comparison of these various policies, levels of their implementation, and their impact has been developed (Zuiderwijk & Janssen, 2014) that has later been revised, updated, and implemented in comparison to OGD policies (Zuiderwijk et al., 2021). Based on conducted research new policies have been proposed (Lee, 2021; Van Loenen et al., 2020) to increase OD use and re-use. Additionally, other various research has been conducted on other individual aspects of OD policies e.g. factors influencing the performance of OGD policies (Hossain et al., 2021) and the impact of OGD policies on organizational performance (Zhou, Wang, Huang, et al., 2023). Aspects frequently addressed in previous literature are the drivers and barriers of OD. The first OD drivers were government directives, such as Directive 2003/98/EC (European Commission, 2003), European Union (Granell et al., 2022), Open government initiative (Transparency and Open Government, 2009) and ZDIJZ (Republika Slovenija, 2003). After that, it was upon government institutions, organizations, entrepreneurs, civil society organizations, individuals, and academic society to advocate for OD publication and re-use. However past research has predominantly focused on the barriers hindering the widespread adoption of OD. The literature provides insights into organizational, technical, and legal obstacles to the implementation and use of OD (Çaldağ & Gökalp, 2023; Crusoe & Melin, 2018). Additional y, studies have concentrated on identifying barriers encountered during implementation and offered recommendations to successfully overcome these barriers (Huber et al., 2020; Sugg, 2022; F. Wang et al., 2019; Wieczorkowski, 2019). S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 865 The quality of the provided OD has been recognized as an important aspect, which is reasonable since any meaningful use of OD depends on the quality of OD, its correct value as wel as the metadata giving it context. Previous research has focused on the quality of OD (Ham et al., 2019; Krasikov et al., 2020) and on the themes of its inclusion, maintenance and governance (Bachtiar et al., 2020; Schultz & Kempton, 2022). Low quality of available OD has been observed resulting in research focusing on the increase of OD quality to enhance its use (Moradi et al., 2022; Zuiderwijk et al., 2014). Given the growing volume of research on OD, which has been increasing since the concept of OD appeared, required its categorization. In an analysis of 101 academic studies about OGD (Safarov et al., 2017) divided the research based on its main focus and suggested future directions for research. Based on a socio-technical model a framework for future categorization was proposed (Cruz & Lee, 2016) and academic research about OD tools for visualization has been categorized (Ansari et al., 2022). A more quantitative approach implementing hierarchical clustering (Ferencek et al., 2022) exhibited that the authors are general y focusing on one of two directions: one that summarizes government policies, initiatives, and portals for OGD sharing; or the other that summarizes regional use cases, adoption of OGD, platforms and barriers for OGD implementation. A review of empirical research by (Wirtz et al., 2022) resulted in the development of a framework, that showed that general y past research can be categorized into one of six groups identified. Research investigating the maturity level of either companies, countries, or policies to adopt and use OD and to stimulate its utilization is another concept that has been addressed in past literature. A literature review by (Çaldağ & Gökalp, 2022) presented that existing maturity models do not cover al the OD aspects. Presented in the literature were models for evaluation of OD maturity for publication and re- use (Dodds & Newman, 2015; Solar et al., 2012) and benchmarks for the evaluation of the progress of OD adoption (Susha et al., 2015; Zuiderwijk et al., 2021). These models are theoretical and mostly assess the maturity of public sector institutions or open data portals for OD publication, not the maturity of organizations for OD adoption and infusion into its processes to add value to its services and/or products. 866 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION This examination of the literature did not identify a model that would comprehensively measure the OD maturity level. While various organizations measure OD maturity levels, these assessments are mostly based on questionnaires that do not provide insight into the state of an individual company. The number of OGD research has been increasing since its appearance. User perspective for the use of OD has been investigated from many viewpoints, impact and OD policies have been addressed, the question of quality was emphasized, maturity of various aspects of OD has been investigated, and the categorization of the OD literature conducted. Nevertheless, until this day, still very little is known about the actual use of OD, what OD sets are used in enterprises to add value to its services and products, and the main drivers and barriers that promote or hinder OD use in organizations' processes. 3.2 Review of OD models To measure organizational maturity level for the adoption of open data a theoretical y supported model must first be defined. We surveyed existing literature to identify what models have so far been created, what were the main dimensions that they have been measuring, and what levels of maturity they have proposed. In the literature, we identified 9 models that meet our criteria. 3.2.1 E-government Openness Index – eGovOI Veljković et al. (2014) introduced an index that enables an assessment of government performance in publishing OD as an essential element of open government. The main dimensions proposed in this benchmark are: 1. The existence of Basic Data Sets, which are the data sets recognized as the big value datasets from previous literature. 2. Data Openness, which is evaluated on 8 open government data principles (Open Government Working Group, 2007). 3. Transparency, which is a dimension calculated from the Government Transparency and Data Transparency sub-dimensions. 4. Participation index dimension, which evaluates the possibility of citizen engagement; and 5. Collaboration index dimension, which evaluates the enablement for cooperation across different levels of government with private institutions or citizens. The eGovOI proposes a five-level openness scale, based on S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 867 a calculation derived from the values gained in the dimension evaluation. The levels of maturity can be seen in Table 1 below. The index can be used to benchmark open governments to qualitatively evaluate whether e-government goals have been satisfied, and to assess the maturity of e- government to change and embrace open concepts. 3.2.2 Open government maturity model – OGMM OGMM model by Lee & Kwak (2012) was designed specifical y for the assessment of open government initiatives to enable public engagement. The main dimensions of the model are the Data dimension, which evaluates whether data is available in a way that its potential can be achieved; and the Participation/Collaboration dimension, which evaluates the easiness of public engagement. The government entity can progress through five levels of maturity, based on their effort to engage the public in various government activities. A higher level can only be achieved if the requirements of the previous are fulfilled. The model is designed to help government agencies implement their open government initiatives effectively through building technological and organizational capabilities. 3.2.3 Open data maturity model – OD-MM The model by Solar et al. (2012) is designed to assess the commitment and capabilities of public agencies to implement open data practices for publication. A three-level structure is proposed: domains, sub-domains, and critical variables. The main domains are 1. Establishment and Legal Perspective, which evaluates the organization's IT strategy alignment with its business strategy that implements a decision and a vision to be incorporated into open government. 2. Technological perspective, which establishes the technological capacities needed to incorporate open government; and 3. Citizen and Entrepreneurial Perspective establishes the organizations' ability to involve citizens to develop applications that improve transparency. Each domain contains three subdomains, and each sub-domain 868 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION contains three to four critical variables. To each critical variable, a capacity level is asserted ranging from 1. Inexistent capabilities to 4. Advanced capabilities. The capacity level of each critical variable is then weighted according to their importance and the result presents the capacity level of a sub-domain. The OD-MM assessment yields the overal maturity level of the public agency to publish OD. Additional y, a roadmap for implementation is provided, outlining the steps necessary to progress towards a higher maturity level. 3.2.4 Open data maturity model – ODMM How wel an organization publishes and consumes open data is determined with the use of a model by Dodds & Newman (2015) who presented the model as a result of an Open Data Institute’s effort to help organizations with the assessment of their operational and strategic activities regarding OD. The model consists of five main dimensions: 1. Data management process, which identifies the key business processes that ground data management and publication. 2. Knowledge and skil s dimension, which is focused on creating a culture of OD within an organization. 3. Customer support and engagement, which addresses the importance of engagement with both the data supplier as wel as the data consumer. 4. Investment and financial performance, which highlights the need to have an insight into the cost and the value of data for publication and consumption. And 5. Strategic oversight describes the need for an organization to have a clear strategy and leadership with the responsibility and capacity to deliver that strategy. Each dimension is measured on a five-level scale from 1. Initial level to 5. Optimizing level. Another considerable benefit of this model is the beneficial effect for each of the activity of dimensions so that decision-makers have a clear view of the advantages of the implementation. The model intends to provide guidance on potential areas of improvement and identify their strengths and weaknesses, adopt best practices, and improve their processes. S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 869 3.2.5 A stage model A model designed to provide a roadmap for OGD use and to enable the evaluation of relevant initiatives’ sophistication by Kalampokis et al. (2011) focuses on socio- technical issues related to organizational and technological chal enges for the publication of data. The model evaluates two dimensions: Added value and Organizational and Technological complexity. It proposes 4 levels of maturity: 1. Aggregation of government data, meaning simply gathering data from various sources and publishing it online; 2. Integration of government data in which a unified view is provided for the aggregated data; 3. Integration of government data with non- government formal data; to the final stage 4. Integration of government data with non-government formal and social data that enables public administration insight into real-world public opinion. With the organizational and technological complexity rising so is the possibility for higher added value. With this model, a step-by-step model is proposed to implement OGD into an organization's everyday processes to exploit the available OD to create new or innovative added-value services and products. 3.2.6 Metric for evaluating Brazilian OGD – DGABr A metric by Silva & Pinheiro (2018) is focused on the evaluation of the publishing of data sets by governmental institutions in Brazil. The metric could be, with appropriate modifications, applied for global use. The model focuses on five dimensions of OD and gives each dimension a respective weight based on its importance. The first dimension is the Open Data perspective, which is evaluated based on the 8 open government data principles (Open Government Working Group, 2007) and the final two stages of the five-star linked data principles (Berners-Lee, 2012). The second dimension is the Legal perspective that evaluates compliance with legislation. The third is the Technical perspective, which evaluates the use of technical standards for publication of OGD. Next is the Managerial perspective, which involves the management related to the planning, monitoring, and control of published OGD. The last dimension is the Reuse 870 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION perspective which evaluates whether OGD is being reused in other applications. The metric proposes six levels of implementation from 0 to 5, of which the first three portray non-existence or non-fulfil ment of the dimension, level 3 is the minimum for OGD to be reused. The maturity level value is obtained by the sum of 28 sub- dimensions, describing basic dimensions, multiplied by their respective weight. The score is calculated and the degree of maturity of the organization to publish OD is reflected by the result. The GDABr metric provides a base for the evaluation of OGD publication. It represents an instrument to measure results obtained from the efforts and investments in the publication of governmental OD. 3.2.7 A trust-based conceptual framework on OGD A model by Zainal et al. (2018) attempts to introduce a framework to identify the determinant factors that influence the user's behavioral intent to use OD by integrating the UTAUT and trust factors. The model presents acceptance factors of the UTAUT, namely Performance expectancy, Ef ort expectancy, Social influence, and Facilitating conditions; and extends the theory by adding trust factors that influence users' trust in OD websites: Trust to government and Trust to technology. This work presents a highly theoretical model that focuses on evaluating whether the proposed factors impact users’ intention to use OD. No levels of implementation are assumed in this model. The model describes and determines the factors that influence behavioral intention to use ODG from a user’s viewpoint. 3.2.8 A model for post-adoption of OGD The model focuses on the users’ needs after accepting OGD. The study by Mustapa et al. (2022) attempts to propose a research model for OGD implementation in the post-adoption phase. The Technology-Organization-Environment (TOE) framework and the innovation adoption process present the theoretical foundation for the model. The technological context represents the availability and characteristics of technology. In S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 871 the organizational context, the structure of an organization through its formal and informal systems of connections and hierarchy are considered. The environmental context represents the business features, the market structure, and legal regulations. The innovation adoption theory explains the dynamics of influences and adoption patterns in an organization. The model evaluates the TOE and innovation subdimensions considering their positive or negative influence on OGD acceptance in the public sector in the phases of Acceptance, Routinization, and Infusion of OD into the organizations’ processes. The proposed model observes the OGD adoption as an ongoing process rather than a one-time decision. It is anticipated that the model will assist policymakers develop such strategies that will enable long-term OGD implementation. 3.2.9 Digital maturity model The model by Kljajić Borštnar & Pucihar (2021) does not directly address the OGD publication or use, it does however investigate and propose a model for organizations, specifically small and medium-sized organizations, to implement technological solutions to enhance their digital maturity level. From a viewpoint that OGD is one of the digital technologies that contributes to adding value in an organization the choice to include this model in our review seems natural. The proposed model addresses the problem of assessing digital maturity for SMEs, with a design science research approach, and presents a multi-attribute model as an IT artifact. The main dimensions of the model are Organizational capabilities and Technological capabilities, each divided into meaningful sub-dimensions further divided until a basic level is reached and additional division would no longer contribute to the easiness of understanding for the assessor. Each dimension is assessed on a four- level scale and the result presents digital maturity of a SME. The levels of digital maturity range from 1. Lagging behind; 2. Initial; 3. Advanced; to 4. Digital winner. The model serves as a valuable tool for SMEs, indicating their current digital maturity level and identifying their strengths and weaknesses in this domain. Furthermore, it proposes improvement activities, guiding SMEs to digitally evolve and achieve higher maturity levels. 872 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION The models, their objectives, focus, basic dimensions, and proposed levels by the authors are presented in Table 1: Table 1: Maturity model review Focus Name Authors Objective (Publication Dimensions Levels / Re-Use) Basic Data Set 0 - 5% Cradle eGovernment 6 - 25% Basic openness Data Openness Openness Evaluation of level Index Veljković et eGovernment Publication Transparency 25 - 65% Average al. (2014) through an OD openness level Participation eGovOI perspective 66 - 82 % Openness level Index > 83% High openness Collaboration level Index Open Assessment of Data Government open 1. Initial conditions Maturity Lee & government 2. Data transparency Participation Model Kwak initiatives for Publication 3. Open participation and (2012) enabling public 4. Open Collaboration collaboration OGMM engagement 5. Ubiquitous engagement Establishment offers an in- & Legal depth insight Technological Open Data into the 1. Inexistent Capacities Maturity maturity and 2. Emerging Capacities Model Solar et al. existing Publication (informal) (2012) capabilities of 3. Existent Capacities OD-MM public Citizen and organizations 4. Advanced Capacities Entrepreneurial regarding OD initiatives. Publication Data & re-use Management Processes Knowledge and Open Data skil s 1. Initial Maturity Doods & Customer 2. Repeatable Model Newman support & 3. Defined (2015) engagement 4. Managed ODMM 5. Optimizing Investment & financial performance Strategic oversight A Stage Model Publication Added value S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 873 Focus Name Authors Objective (Publication Dimensions Levels / Re-Use) To provide a roadmap for 1. Aggregation of GD OGD re-use Organizational 2. Integration of GD Kalampokis and to enable and 3. Integration of GD with et al. evaluation of Technological Non-Gov Formal data relevant complexity 4. Integration of GD with initiatives' Non-Gov Formal and sophistication Social Data To evaluate the Open data Metric for OGD in 0. Nonexistent Legal evaluating federal Public 1. Under construction Brazilian OGD Silva & Administration Publication 2. Not executed Technical Pinheiro of Brazil based 3. Partially performed DGABr on metrics and 4. Existing results Managerial international 5. Advanced results indicators Reuse An attempt to Trust to OD propose a new websites A trust-based model for conceptual Zainal et al. measuring the framework on (2018) level of use of Re-use Acceptance OGD OGD by integrating Factors UTAUT and trust factors. Technological To propose a research model A research for OGD Post-adoption phase: Organizational model for Mustapa et implementation 4. Acceptance Post-adoption al. (2020) in the post- Re-use 5. Routinization of OGD adoption phase 6. Infusion in (Malaysia's) public sector. Environmental Innovation Develop a multi-attribute Digital Kljajić model for the 1. Lagging behind Maturity of Borštnar & assessment of Digital 2. Initial SMEs Pucihar capabilities 3. Advanced (2021) the digital maturity of 4. Digital winner SMEs 874 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 4 Discussions The research findings wil make contributions in various domains including scientific, economic, and social. In the scientific domain, the contribution wil be a developed artifact – a comprehensive methodology for assessing organizations' readiness for OD use. The model will present an innovative solution in the field of decision-making methods within organization and management studies. Based on this research – a systematic literature review – it was found that a model enabling companies to comprehensively assess their OD maturity level does not exist. The development of such a tool wil introduce new knowledge and enable further research in the field. In the economic domain, the contribution wil present an insight into the state’s economy OD maturity level to enhance its added value. The developed artifact wil not only enable the assessment of individual companies’ readiness, it will also provide individualized suggestions of activities to enhance their capabilities for OD use. This will enable organizations to improve their decision-making based on quality information. At a country level, the data gathered from a large number of enterprises will provide insight into the state of the population. This can enable further policy adoption to increase awareness, encourage adoption, and boost economic activities. Key aspects representing significant progress at the societal level include raising awareness among citizens and organizations to advocate increased transparency and active citizen participation. The developed artifact designed for assessing OD maturity level along with essential recommendations for its enhancement wil enable social organizations and individuals to become acquainted with OD and increase the possibility of OD utilization. Through their engagement, social organizations and individuals will be able to contribute to the formulation of legislation as well as monitor the transparency of public expenditure. There are various models and tools for assessing the maturity of OD. The problem is that these models are theoretical, are not comprehensive, and mostly assess the maturity of public sector institutions or open data portals. We have not yet found such a model in the literature or in practice that comprehensively assesses the maturity level for the use of OD in companies and presents a transparent S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 875 interpretation of the results along with recommendations for future activities to improve the maturity level. 5 Conclusions The comprehensive methodology for the assessment of OD use will provide feedback for any individual enterprise, but the analysis of a large number of assessed enterprises, will offer valuable information to the policymakers about the efficiency of their actions and support, and thus support further support actions planning. In conclusion of this study, we can give initial answers to our research questions: RQ1: With a DSR methodology, we could create a model that would measure the maturity level of an SME and distinguish between different levels of maturity. RQ2: Based on the literature review we identified dimensions that could be used to measure OD maturity level of SMEs in Slovenia. Some dimensions have previously not been identified; however, based on the review, we can see that they could significantly influence the model. Both must be further analyzed, and based on that analysis, included, or excluded from the model that we are developing. A comprehensive tool, the result of this research wil contribute to the three main objectives of opening data: Increase transparency through public engagement, increase collaboration between government and its citizens, and increase economic activity through better-informed decision-making. Achieving these three key elements OD and a maturity model that offers guidance for enhancement of OD readiness level, can lead to enhanced societal wel -being as a whole. Acknowledgments This research was supported by the Slovenian Research Agency: P5-0018 - Decision Support Systems in Digital Business, and Slovenian Research Agency and Ministry of Digital Transformation of Republic of Slovenia: V5-2356 - Open Data Maturity Assessment and Encouragement in Slovenian Enterprises: Methodology Development and Web-Based Solution. 876 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION References Alawadhi, N., Al Shaikhli, I., Alkandari, A., & Kalaie Chab, S. (2021). Business Owners’ Feedback toward Adoption of Open Data: A Case Study in Kuwait. Journal of Electrical and Computer Engineering, 2021, 1–9. https://doi.org/10.1155/2021/6692410 Ansari, B., Barati, M., & Martin, E. G. (2022). Enhancing the usability and usefulness of open government data: A comprehensive review of the state of open government data visualization research. Government Information Quarterly, 39(1), 101657. https://doi.org/10.1016/j.giq.2021.101657 Apanasevic, T. (2021). Socio-economic effects and the value of open data: A case from Sweden. 23rd ITS Biennial Conference, 2021. http://hdl.handle.net/10419/238004 Attard, J., Orlandi, F., & Auer, S. (2016). Value Creation on Open Government Data. 2016 49th Hawai International Conference on System Sciences (HICSS), 2605–2614. https://doi.org/10.1109/HICSS.2016.326 Attard, J., Orlandi, F., Scerri, S., & Auer, S. (2015). A systematic review of open government data initiatives. Government Information Quarterly, 32(4), 399–418. https://doi.org/10.1016/j.giq.2015.07.006 Bachtiar, A., Suhardi, & Muhamad, W. (2020). Literature Review of Open Government Data. 2020 International Conference on Information Technology Systems and Innovation (ICITSI), 329–334. https://doi.org/10.1109/ICITSI50517.2020.9264960 Berners-Lee, T. (2012, January 22). Linked data. https://5stardata.info/en/ Bohanec, M., & Rajkovič, V. (1990). DEX: An Expert System Shel for Decision Support. Sistemica, 1990(1), 145–157. Çaldağ, M. T., & Gökalp, E. (2022). The maturity of open government data maturity: A multivocal literature review. Aslib Journal of Information Management, 74(6), 1007–1030. https://doi.org/10.1108/AJIM-11-2021-0354 Çaldağ, M. T., & Gökalp, E. (2023). Understanding barriers affecting the adoption and usage of open access data in the context of organizations. Data and Information Management, 100049. https://doi.org/10.1016/j.dim.2023.100049 Coutinho, E. D., & Freitas, A. S. (2021). PUBLIC VALUE THROUGH TECHNOLOGIES DEVELOPED WITH OPEN GOVERNMENT DATA: THE LOVE SERENADE OPERATION CASE. RAM. Revista de Administração Mackenzie, 22(6), eRAMD210079. https://doi.org/10.1590/1678-6971/eramd210079 Crusoe, J., & Melin, U. (2018). Investigating Open Government Data Barriers: A Literature Review and Conceptualization. In P. Parycek, O. Glassey, M. Janssen, H. J. Schol , E. Tambouris, E. Kalampokis, & S. Virkar (Eds.), Electronic Government (Vol. 11020, pp. 169–183). Springer International Publishing. https://doi.org/10.1007/978-3-319-98690-6_15 Cruz, R. A. B., & Lee, H. J. (2015). Opening the Nation: Leveraging Open Data to Create New Business and Provide Services. Knowledge Management Review, 16(4), 157–168. https://doi.org/10.15813/KMR.2015.16.4.009 Cruz, R. A. B., & Lee, H. J. (2016). A Socio-Technical Model for Open Government Data Research. Asia Pacific Journal of Information Systems, 26(3), 339–366. https://doi.org/10.14329/apjis.2016.26.3.339 Dodds, L., & Newman, A. (2015). Open Data Maturity Model [Technical Report]. Open Data Institute. https://theodi.org/insights/reports/open-data-maturity-model/ European Commission. (2003, November 17). Directive 2003/98/EC of the European Parliament and of the Council of 17 November 2003 on the re-use of public sector information. https://eur-lex.europa.eu/eli/dir/2003/98/oj Ferencek, A., Borštnar, M. K., & Žagar, A. P. (2022). Categorisation of Open Government Data Literature. Business Systems Research Journal, 13(1), 66–83. https://doi.org/10.2478/bsrj-2022-0005 S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 877 Gottfried, A., Hartmann, C., & Yates, D. (2021). Mining Open Government Data for Business Intelligence Using Data Visualization: A Two-Industry Case Study. Journal of Theoretical and Applied Electronic Commerce Research, 16(4), 1042–1065. https://doi.org/10.3390/jtaer16040059 Granel , C., Mooney, P., Jirka, S., Rieke, M., van den Broecke, J., Sarretta, A., Verhulst, S., Dencik, L., Oost, H., Micheli, M., Minghini, M., Kotsev, A., & Schade, S. (2022). Emerging approaches for data-driven innovation in Europe: Sandbox experiments on the governance of data and technology. Publications Office of the European Union. Ham, J., Koo, Y., & Lee, J.-N. (2019). Provision and usage of open government data: Strategic transformation paths. Industrial Management & Data Systems, 119(8), 1841–1858. https://doi.org/10.1108/IMDS-04-2019-0218 Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design Science in Information Systems Research. MIS Quarterly, 28(1), 75–105. https://doi.org/25148625 Hossain, M. A., Rahman, S., Quaddus, M., Hooi, E., & Olanrewaju, A.-S. (2021). Factors Affecting Performance of Open Government Data Initiatives: A Multi-Method Approach Using Sem and FSQCA. Journal of Organizational Computing and Electronic Commerce, 31(4), 300– 319. https://doi.org/10.1080/10919392.2021.2018258 Huber, F., Ponce, A., Rentocchini, F., & Wainwright, T. (2022). The wealth of (Open Data) nations? Open government data, country-level institutions and entrepreneurial activity. Industry and Innovation, 29(8), 992–1023. https://doi.org/10.1080/13662716.2022.2109455 Huber, F., Wainwright, T., & Rentocchini, F. (2020). Open data for open innovation: Managing absorptive capacity in SMEs. R&D Management, 50(1), 31–46. https://doi.org/10.1111/radm.12347 Jetzek, T., Avital, M., & Bjorn-Andersen, N. (2014). Data-Driven Innovation through Open Government Data. Journal of Theoretical and Applied Electronic Commerce Research, 9(2), 15–16. https://doi.org/10.4067/S0718-18762014000200008 Kalampokis, E., Tambouris, E., & Tarabanis, K. (2011). Open Government Data: A Stage Model. In M. Janssen, H. J. Schol , M. A. Wimmer, & Y. Tan (Eds.), Electronic Government (Vol. 6846, pp. 235–246). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-22878- 0_20 Khurshid, M. M., Zakaria, N. H., Arfeen, M. I., Rashid, A., Nasir, S. U., & Shehzad, H. M. F. (2022). Factors Influencing Citizens’ Intention to Use Open Government Data—A Case Study of Pakistan. Big Data and Cognitive Computing, 6(1), 31. https://doi.org/10.3390/bdcc6010031 Kljajić Borštnar, M., & Pucihar, A. (2021). Multi-Attribute Assessment of Digital Maturity of SMEs. Electronics, 10(8), 885. https://doi.org/10.3390/electronics10080885 Krasikov, P., Obrecht, T., Legner, C., & Eurich, M. (2020). Is Open Data Ready for Use by Enterprises? Learnings from Corporate Registers: Proceedings of the 9th International Conference on Data Science, Technology and Applications, 109–120. https://doi.org/10.5220/0009875801090120 Lassinantti, J., Ståhlbröst, A., & Runardotter, M. (2019). Relevant social groups for open data use and engagement. Government Information Quarterly, 36(1), 98–111. https://doi.org/10.1016/j.giq.2018.11.001 Lee, G., & Kwak, Y. H. (2012). An Open Government Maturity Model for social media-based public engagement. Government Information Quarterly, 29(4), 492–503. https://doi.org/10.1016/j.giq.2012.06.001 Lee, J. W. (2021). The Data Sharing Economy and Open Governance of Big Data as Public Good. The Journal of Asian Finance, Economics and Business, 8(11), 87–96. https://doi.org/10.13106/JAFEB.2021.VOL8.NO11.0087 Maccani, G. (2016). Exploring the factors that influence adoption of open government data for commercial service innovation in cities. PhD, 2016. 878 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Magalhaes, G., & Roseira, C. (2020). Open government data and the private sector: An empirical view on business models and value creation. Government Information Quarterly, 37(3), 101248. https://doi.org/10.1016/j.giq.2017.08.004 McBride, K., Aavik, G., Toots, M., Kalvet, T., & Krimmer, R. (2019). How does open government data driven co-creation occur? Six factors and a ‘perfect storm’; insights from Chicago’s food inspection forecasting model. Government Information Quarterly, 36(1), 88–97. https://doi.org/10.1016/j.giq.2018.11.006 Moradi, M., Mazoochi, M., & Ahmadi, M. (2022). A Comprehensive Method for Improving the Quality of Open Government Data and Increasing Citizens’ Wil ingness to Use Data by Analyzing the Complex System of Citizens and Organizations. Complexity, 2022, 1–14. https://doi.org/10.1155/2022/5876035 Mustapa, M. N., Hamid, S., & Md Nasaruddin, F. H. (2022). Factors influencing open government data post-adoption in the public sector: The perspective of data providers. PLOS ONE, 17(11), e0276860. https://doi.org/10.1371/journal.pone.0276860 Open Government Working Group. (2007, December 8). Open Government Data Principles. Open Government Data Principles. https://public.resource.org/8_principles.html Republika Slovenija. (2003, March 7). Zakon o dostopu do informacij javnega značaja. Ruijer, E., Grimmelikhuijsen, S., Van Den Berg, J., & Meijer, A. (2020). Open data work: Understanding open data usage from a practice lens. International Review of Administrative Sciences, 86(1), 3–19. https://doi.org/10.1177/0020852317753068 Ruijer, E., & Martinius, E. (2017). Researching the democratic impact of open government data: A systematic literature review. Information Polity, 22(4), 233–250. https://doi.org/10.3233/IP-170413 Ruijer, E., & Meijer, A. (2020). Open Government Data as an Innovation Process: Lessons from a Living Lab Experiment. Public Performance & Management Review, 43(3), 613–635. https://doi.org/10.1080/15309576.2019.1568884 Safarov, I., Meijer, A., & Grimmelikhuijsen, S. (2017). Utilization of open government data: A systematic literature review of types, conditions, effects and users. Information Polity, 22(1), 1–24. https://doi.org/10.3233/IP-160012 Saxena, S., & Janssen, M. (2017). Examining open government data (OGD) usage in India through UTAUT framework. Foresight, 19(4), 421–436. https://doi.org/10.1108/FS-02-2017-0003 Schultz, C., & Kempton, A. (2022). Governance Chal enges in Open Government Data Ecosystems: A Case Study from the Financial Sector in Norway. Hawai International Conference on System Sciences. https://doi.org/10.24251/HICSS.2022.321 Shao, D. (2023). Empirical analysis of open government data usage in Tanzania. Information Discovery and Delivery, 52(1), 73–84. https://doi.org/10.1108/IDD-10-2022-0098 Shepherd, E., Bunn, J., Flinn, A., Lomas, E., Sexton, A., Brimble, S., Chorley, K., Harrison, E., Lowry, J., & Page, J. (2019). Open government data: Critical information management perspectives. Records Management Journal, 29(1/2), 152–167. https://doi.org/10.1108/RMJ-08-2018-0023 Silva, P. N., & Pinheiro, M. M. K. (2018). DGABr: Metric for evaluating Brazilian open government data. Informação & Sociedade: Estudos, 28(3). https://doi.org/10.22478/ufpb.1809-4783.2018v28n3.42183 Solar, M., Concha, G., & Meijueiro, L. (2012). A Model to Assess Open Government Data in Public Agencies. In H. J. Schol , M. Janssen, M. A. Wimmer, C. E. Moe, & L. S. Flak (Eds.), Electronic Government (Vol. 7443, pp. 210–221). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-33489-4_18 Sugg, Z. (2022). Social barriers to open (water) data. WIREs Water, 9(1), e1564. https://doi.org/10.1002/wat2.1564 Susha, I., Zuiderwijk, A., Janssen, M., & Grönlund, Å. (2015). Benchmarks for Evaluating the Progress of Open Data Adoption: Usage, Limitations, and Lessons Learned. Social Science Computer Review, 33(5), 613–630. https://doi.org/10.1177/0894439314560852 S. Blatnik: Methodology Development for Open Data Maturity Assessment in Smal and Medium-Sized Enterprises – A Literature Review 879 Talukder, M. S., Shen, L., Hossain Talukder, M. F., & Bao, Y. (2019). Determinants of user acceptance and use of open government data (OGD): An empirical investigation in Bangladesh. Technology in Society, 56, 147–156. https://doi.org/10.1016/j.techsoc.2018.09.013 Transparency and Open Government (2009). Van Loenen, B., Wel e Donker, F., Zuiderwijk-van, E., Alexopoulos, C., & Tutic, D. (2020). Towards an Open Data Research Ecosystem in Croatia. Proceedings of Ongoing Research, Practitioners, Workshops, Posters, and Projects of the International Conference EGOV- CeDEM-ePart 2020, 2020(2797), 59–70. Veljković, N., Bogdanović-Dinić, S., & Stoimenov, L. (2014). Benchmarking open government: An open data perspective. Government Information Quarterly, 31(2), 278–290. https://doi.org/10.1016/j.giq.2013.10.011 Wang, F., Zhao, A., Zhao, H., & Chu, J. (2019). Building a Holistic Taxonomy Model for OGD-Related Risks: Based on a Lifecycle Analysis. Data Intel igence, 1(4), 309–332. https://doi.org/10.1162/dint_a_00018 Wang, H. J., & Lo, J. (2020). Factors Influencing the Adoption of Open Government Data at the Firm Level. IEEE Transactions on Engineering Management, 67(3), 670–682. https://doi.org/10.1109/TEM.2019.2898107 Wang, H.-J. (2020). Adoption of Open government data: Perspectives of user innovators. Information Research, 2020(25 (1)). https://web.archive.org/web/20200224193543/http://informationr.net/ir/25- 1/paper849.html Weerakkody, V., Kapoor, K., Balta, M. E., Irani, Z., & Dwivedi, Y. K. (2017). Factors influencing user acceptance of public sector big open data. Production Planning & Control, 28(11–12), 891–905. https://doi.org/10.1080/09537287.2017.1336802 Wieczorkowski, J. (2019). Barriers to Using Open Government Data. Proceedings of the 2019 3rd International Conference on E-Commerce, E-Business and E-Government - ICEEG 2019, 15–20. https://doi.org/10.1145/3340017.3340022 Wilson, B., & Cong, C. (2021). Beyond the supply side: Use and impact of municipal open data in the U.S. Telematics and Informatics, 58, 101526. https://doi.org/10.1016/j.tele.2020.101526 Wirtz, B. W., Weyerer, J. C., Becker, M., & Mül er, W. M. (2022). Open government data: A systematic literature review of empirical research. Electronic Markets, 32(4), 2381–2404. https://doi.org/10.1007/s12525-022-00582-8 Zainal, N. Z., Hussin, H., & Miro Nazri, M. N. (2018). A Trust-Based Conceptual Framework on Open Government Data Potential Use. 2018 International Conference on Information and Communication Technology for the Muslim World (ICT4M), 156–161. https://doi.org/10.1109/ICT4M.2018.00037 Zhou, M., Wang, Y., Huang, X., & Li, G. (2023). Can open government data policy improve firm performance? Evidence from listed firms in China. Managerial and Decision Economics, 44(5), 2593–2603. https://doi.org/10.1002/mde.3835 Zhou, M., Wang, Y., Jiang, H., Li, M., & Li, G. (2023). How Leadership Influences Open Government Data (OGD)-Driven Innovation: The Mediating Role of Organizational Commitment. Sustainability, 15(2), 1219. https://doi.org/10.3390/su15021219 Zuiderwijk, A., & Janssen, M. (2014). Open data policies, their implementation and impact: A framework for comparison. Government Information Quarterly, 31(1), 17–29. https://doi.org/10.1016/j.giq.2013.04.003 Zuiderwijk, A., Janssen, M., & Davis, C. (2014). Innovation with open data: Essential elements of open data ecosystems. Information Polity, 19(1,2), 17–33. https://doi.org/10.3233/IP-140329 Zuiderwijk, A., Janssen, M., & Dwivedi, Y. K. (2015). Acceptance and use predictors of open data technologies: Drawing upon the unified theory of acceptance and use of technology. 880 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Government Information Quarterly, 32(4), 429–440. https://doi.org/10.1016/j.giq.2015.09.005 Zuiderwijk, A., Pirannejad, A., & Susha, I. (2021). Comparing open data benchmarks: Which metrics and methodologies determine countries’ positions in the ranking lists? Telematics and Informatics, 62, 101634. https://doi.org/10.1016/j.tele.2021.101634 DOCTORAL CONSORTIUM COMPARATIVE ANALYSIS OF THE PERSONAL BANKRUPTCY MODEL AND THEIR EFFECTS IN THE LEGAL ORDERS IN SLOVENIA AND CROATIA REA ŠAINA University of Ljubljana, Faculty of Law, Ljubljana, Slovenia rea.gashi@gmail.com Personal bankruptcy is a procedure in which a natural person who is over-indebted or insolvent is helped by paying off the debts with his or her own assets. It is an institution that is changing and is increasingly being used by people in situations of over-indebtedness. In practice, it has become popular because debtors seek to obtain a discharge of their liabilities by concealing Keywords: their assets, thereby failing to achieve the essential objective of personal personal insolvency - the equal and simultaneous repayment of bankruptcy, comparative creditors. The following analysis focuses on the differences and analysis of the similarities between the Slovenian and the Croatian legal systems Slovenian and Croatian in the area of personal bankruptcy and the legal solutions offered legal by both legal systems. Particular attention is paid to a system, comparative legal analysis of statistical data showing the equal treatment of importance of personal bankruptcy over selected years. The creditors, analysis also addresses a peculiarity of the Croatian legal system remission of - "simple" consumer bankruptcy, which is not known in the liabilities, »simple« Slovenian legal order, and critical y assesses the obstacles and consumer dilemmas that arise in practice. bankruptcy DOI https://doi.org/10.18690/um.fov.4.2024.54 ISBN 978-961-286-871-0 882 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 1 Introduction A comparative legal perspective of the two systems, Slovenian and Croatian, in terms of the possibility of individual regulation certainly brings similarities, but it is also crucial to take into account the differences, which may not be expected due to the coherence and similarity of the legal rules in the different branches of law. It should be mentioned at the outset that both systems, the Croatian system in 2022 and the Slovenian system in 2023, bring changes in the area of personal bankruptcy. The changes were necessitated by the adoption of the new European Directive (EU) 2019/1023 on a framework for preventive restructuring, for the discharge of liabilities and prohibition of the performance of duties, and on measures to increase the effectiveness of restructuring, insolvency and debt relief proceedings and amending Directive (EU) 2017/1132 (the "Restructuring and Insolvency Directive"). 1 The umbrella law of the Slovenian legal system in the field of insolvency law is Zakon o finančnem poslovanju, postopkih zaradi insolventnosti in prisilnem prenehanju (hereinafter: ZFPPIPP)2, which dedicates a specific chapter to personal bankruptcy. 3 On the other hand, the Croatian legal system regulates personal bankruptcy in two different acts, in Stečajni zakon (hereinafter: SZ)4 and Zakon o stečaju potrošača (hereinafter: ZSP)5. The latter deals with only one type of debtor, namely consumers. Would we conclude from the above that such a distinction also entails certain pitfalls? Of course, the yellow light would immediately go on and we would ask ourselves why this is so. The global financial crisis of 2008-2009 and the euro area crisis of 2011-2013 have had an additional impact on Croatia, increasing the number of blocked citizens. 6 On January 1, 2016 Croatia introduced the SPZ into its legal system to provide legal protection to citizens in financial difficulties and over-1 Directive (EU) 2019/1023 of the European Parliament and of the Council of 20 June 2019 on preventive restructuring frameworks, on discharge of debt and disqualifications, and on measures to increase the efficiency of procedures concerning restructuring, insolvency and discharge of debt, and amending Directive (EU) 2017/1132 (Directive on restructuring and insolvency), accesible on https://eur-lex.europa.eu/eli/dir/2019/1023/oj (May 17, 2024). 2 Zakon o finančnem poslovanju, postopkih zaradi insolventnosti in prisilnem prenehanju (ZFPPIPP), Uradni list RS, št. 176/21, 178/21 – popr., 196/21 – odl. US, 157/22 – odl. US, 35/23 – odl. US, 57/23 – odl. US in 102/23. 3 Section 5.11 of the ZFPPIPP. 4 Stečajni zakon (SZ), Narodne novine, br. 71/15, 104/17, 36/22. 5 Zakon o stečaju potrošača (ZSP), Narodne novine, br. 100/15, 67/18, 36/22. 6 https://arhivanalitika.hr/blog/blokirani-kako-uravnoteziti-financijsku-disciplinu-i-solidarnost/ (May 17, 2024). R. Šaina: Comparative Analysis of the Personal Bankruptcy Model and Their Ef ects in the Legal Orders in Slovenia and Croatia 883 indebted citizens who have been unable to meet their financial obligations due to blocked accounts. 7 In contrast, the Slovenian system only knows one form of personal bankruptcy and does not distinguish between different types of personal bankruptcy. 8 Details of the systems wil be explained below. 2 Methodology Initial y, a descriptive approach was used to explore the chosen topic. The focus was on describing general concepts, definitions, facts, institutes and processes in the study of regulation for both the Slovenian and Croatian systems. The description referred to the main differences of personal bankruptcy, which broadened our view of the current situation and development. We paid a lot of attention to the institutes that have been drastical y charged after the changes in the law in both systems. In addition, we have analysed the comparison of personal bankruptcy statistics over the years, which wil be presented below. The study of the selected topic was based on domestic and foreign laws, professional books and scientific articles obtained from databases. By reviewing them, we have examined the shortcomings of personal bankruptcy, the pitfal s brought about by the changes, given a critical opinion and the possibilities for improvement. We also proposed mechanisms to shorten the duration of the personal bankruptcy procedure, better repay creditors and prevent abuse of this important institution. 3 Opening of personal insolvency proceedings 3.1 Personal bankruptcy participants In the Slovenian system, personal bankruptcy is addressed to three groups of persons, as follows: • all natural persons - consumers (employees, unemployed, pensioners, students,. .); • sole traders; 7 Akšamović D., Šimunović L. (2023). Zakon o stečaju potrošača u službi zaštite građana: postignuća i izazovi. Godišnjak Akademije pravnih znanosti Hrvatske, 88-89. 8 Višje sodišče v Ljubljani, sklep Cst 178/2018. 884 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION • private individuals. 9 The Croatian system considers the same persons as defined in the Slovenian system as debtors, with the exception of consumers, as they are not explicitly mentioned in the SZ. As a result, natural persons as insolvent debtors are not conceptualise clearly and precisely on the basis of a single law, which may require a clearer definition of their meaning in the future. 10 Consumers, in the Croatian system, are subject of a separate law, the SPZ, where are specifical y defined and differ from those in the Slovenian system. In the SPZ, consumers would be divided into two types, according to the procedure that can be proposed, i.e. according to the consumer's »regular« or »simple« procedure. 11 The simple consumer procedure stands out as it alien to the Slovenian legal order and aims to reduce the number of blocked citizens and enforcement proceedings. 12 3.2 Procedure - opening of personal insolvency proceedings The procedure starts with the filing of a written application by the debtor or the creditor. 13 The latest amendment to the Croatian law 2022 also includes the possibility for a creditor to file for personal bankruptcy, which has not been the case so far. 14 In Slovenia, more than 90% of personal insolvency proceedings are initiated by the debtors themselves. 15 After the motion is filed, the court checks whether the motion is complete. The proposal must be accompanied by a report on the debtor’s financial situation. 16 The Croatian legal system does not require the latter, except at the request of the court. 17 The debtor may file a proposal to open insolvency proceedings together with the proposal for remission of liabilities, but it must be filed no later than the end of the 9 Article 381 of the ZFPPIPP. 10 Article 3 of the SZ. 11 Article 4 of the SPZ; Article 79.a of the SPZ. 12 Marković N., (2022). Izmjene Zakona o stečaju potrošača, Novi Informator, 2-3. 13 Article 231 ZFPPIPP. 14 Article 109 SZ; Marković N., Čuveljak J., (2022). Izmjene i dopune stečajnog zakona, priručnik za polaznike, Pravosudna akademija, accesible on https://www.pak.hr/wp-content/uploads/2021/12/Stecajni-zakon.pdf (May 17, 2024). 15 https://nasodiscu.si/osebni-stecaj#kako-se-zacne (May 17, 2024). 16 Article 384(3)(1) of the ZFPPIPP. 17 Article 117(2) of the SZ. R. Šaina: Comparative Analysis of the Personal Bankruptcy Model and Their Ef ects in the Legal Orders in Slovenia and Croatia 885 insolvency proceedings. 18 On the other hand, the “Croatian” debtor must file the proposal for remission of liabilities with the proposal for personal insolvency or on the basis of an invitation from the court. 19 3.3 Commencement of personal insolvency proceedings The insolvency proceedings are opened by the court's order commencing the insolvency proceedings, appointing the insolvency administrator and cal ing on creditors to lodge their claims and divestment and strike-off rights in the debtor's bankruptcy estate in due time. The deadline for filing claims is three months under the ZFPPIPP20 and sixty days under the Croatian SZ. 21 The insolvency estate in personal insolvency proceedings comprises al the assets and income of the debtor held at the commencement of the personal insolvency proceedings, as wel as assets that the individual acquires (inherits, receives, etc.) in the course of the personal insolvency proceedings. 22 3.4 Legal consequences of the commencing of insolvency proceedings The debtor's capacity to act and dispose of his assets is restricted. The debtor's rights to administer and dispose of the debtor's assets are transferred to the insolvency administrator on the commencement of personal insolvency. 23 An important duty of the debtor is to keep the insolvency administrator and the court informed and to cooperate with them. 24 3.5 Conclusion of personal insolvency proceedings The personal insolvency formal y ends when the order closing the personal insolvency proceedings is made, so once al the assets have been realised and the distribution to creditors has been made. However, it should be noted that the end of personal insolvency does not necessarily mean the end of the personal insolvency 18 Article 398 of the ZFPPIPP. 19 Josipović T., (2016). Razlučni vjerovnici u novom hrvatskom insolvencijskom pravu, v: Liber amicorum Gašo Knežević, 65; Article 373 in conjuction with Article 131 of the SZ. 20 Article 59(2) of the ZFPPIPP. 21 Article 129(1)(4) of the SZ. 22 Article 224 ZFPPIPP; Article 134 SZ. 23 Article 386 of the ZFPPIPP: Articles 159 and 161 of the SZ. 24 Article 384 of the ZFPPIPP; Article 177 of the SPZ. 886 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION for the debtor, but rather for the creditors. It should be borne in mind that personal insolvency proceedings are primarily conducted to protect the interests of creditors. 25 4 Personal bankruptcy statistics in Slovenian and Croatian legal order The personal bankruptcy data to be presented below relates to the period from 2016 to 2023 (this period of study has been taken into account in order to avoid excessive divergences in the two legal systems) for Slovenia and Croatia, which are publicly available or provided by the Ministry of Justice. It should be pointed out, however, that while these data may indicate and guide the effectiveness of an institution, they cannot ultimately indicate and measure the ultimate success of the institution itself, even in terms of the number of personal insolvency proceedings opened and pending. It is possible to ascertain that the number of cases is decreasing. In 2016, the number of cases received, resolved, settled and pending was higher than in the previous year. Most personal insolvency proceedings are proposed by debtors. In 2016, almost more than 70% of personal insolvency proceedings ended without a distribution of the insolvency estate, but this figure has decreased over the years, reaching a percentage of around 50% last year. This figure is stil quite high, as the purpose of personal bankruptcy – to proportional y repay creditors – remains unrealized. The duration of personal insolvency depends on various factors, one of which is discharge procedure together with the probationary period, which has been shortened (from six months to three years)26 by the Slovenian amendment to the ZFPPIPP. In the Croatian legal order, this period is three years. 27 25 Article 286 of the SZ; Article 396 of the ZFPPIPP. 26 Article 400(5) of the ZFPPIPP. 27 Article 373(2) of the SZ. R. Šaina: Comparative Analysis of the Personal Bankruptcy Model and Their Ef ects in the Legal Orders in Slovenia and Croatia 887 Table 1: Slovenian il ustration of personal insolvency proceedings Year 2016 2017 2018 2019 2020 2021 2022 2023 Cases Received cases 3.883 2.628 2.331 2.121 1.615 1.318 1.223 1.052 Settled cases 2.847 3.991 4.464 3.529 2.574 2.124 1.780 1.440 Pending cases at the end 10.561 9.203 7.072 5.666 4.707 3.900 3.341 2.946 Applicant - Creditor 266 205 234 239 189 148 145 118 - Debtor 3.616 2.423 2.096 1.881 1.425 1.170 1.078 932 SALE TRADE/ PRIVATE 109 135 170 216 146 150 128 114 INDIVIDUAL CONSUMER 2.552 3.851 4.135 3.304 2.422 2.030 1.644 1.342 Finally disposed 2.661 3.986 4.485 3.520 2.568 2.180 1.772 1.456 REMISSION OF 1.862 3.292 3.865 2.932 2.001 1.652 1.331 1.055 LIABILITIES Sale trade/private 68 82 116 134 95 76 73 60 individual Consumer 1.794 3.210 3.749 2.798 1.906 1.576 1.258 995 INSOLVENCY ESTATE (in ≈ 5,3 ≈ 37,3 ≈ 14,8 ≈ 26,9 ≈ 18,3 ≈ 24,8 ≈ 14,5 ≈ 19,7 millions) Sale trade/private ≈ 1,2 ≈ 2,9 ≈ 1,4 ≈ 10,3 ≈ 1,2 ≈ 10,3 ≈ 3,5 ≈ 4,6 individual Consumer ≈ 4,0 ≈ 34,4 ≈ 13,4 ≈ 16,6 ≈ 17,1 ≈ 14,5 ≈ 11,0 ≈ 15,2 SOLUTION METHOD WITHOUT distribution 75,38% 80,03% 76,12% 67,72% 63,33% 60,97% 58,15% 55,14% By distribution 8,47% 13,25% 18,64% 26,13% 28,55% 29,90% 31,52% 32,71% 888 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION Table 2: Croatian il ustration of personal consumer’s insolvency proceedings Country courts TYPE OF DISPUTE Simple consumer bankruptcy procedure Consumer bankruptcy At the YEAR Start Received Settled At the end Start Received Settled end 2016 0 319 52 267 2017 267 372 281 358 2018 365 178 248 295 2019 0 96.043 43.969 52.074 295 152 226 221 2020 52.074 18.460 55.885 14.649 221 66 150 137 2021 14.649 17.842 22.295 10.196 137 123 113 147 2022 10.196 21.393 18.811 12.775 146 128 122 152 2023 12.775 37.982 28.416 22.341 153 155 154 154 In the Croatian legal system, two types of courts decide on the type of personal insolvency proceedings. Country courts rule on consumer bankruptcy and simple consumer bankruptcy proceedings. District courts rule on the bankruptcy of individual debtors, specifically sole traders and private individuals (Table 3). The Table 2 above shows information on the bankruptcy of the consumer. With the entry into force of the SPZ in 2016, consumers gained the possibility to be part of the personal insolvency proceedings. In 2019, the SPZ introduces a new procedure, mentioned above, simple consumer bankruptcy procedure, which is being introduced ex officio by the Financial Agency (hereinafter: FINA) for consumers with smal debts and a continuous lockage of accounts lasting more than three years. At the beginning, we ascertain a high percentage of proposal field, despite that this percentage has decreased over the years. According to FINA, the number of consumers in blockade is 211 thousand R. Šaina: Comparative Analysis of the Personal Bankruptcy Model and Their Ef ects in the Legal Orders in Slovenia and Croatia 889 and the debt is EUR 0,76 billion. 28 What should be worked on is the implementation and enforcement of a col ective strategy to tackle consumer insolvency, not just constant adoption of new rules that fail. Table 3: Croatian il ustration of personal bankruptcy proceedings of an individual debtor29 District TYPE OF DISPUTE courts Bankruptcy of a natural person Bankruptcy over the assets of an individual debtor At the At the YEAR Start Received Settled Start Received Settled end end 2016 13 4 1 16 85 9 23 71 2017 16 1 7 11 71 5 16 61 2018 11 0 5 7 61 3 16 48 2019 7 0 1 6 46 1 15 32 2020 6 0 3 3 32 3 7 28 2021 3 1 2 2 28 2 7 23 2022 2 0 1 1 23 0 10 13 2023 1 2 2 1 13 0 1 12 In Table 3 above, we can determine that very few personal insolvency petitions were filed in 2016 compared to personal consumer’s insolvency proceedings. Also, the trend of settled cases remains very low, and this type of personal bankruptcy is not often proposed. 28https://www.fina.hr/novosti/broj-ovrha-na-novcanim-sredstvima-poslovnih-subjekata-i-potrosaca-31.- prosinca-2023 (May, 17 2024). 29 Please note that these two types of debtor (natural person and individual debtor are identical objects) do not include consumers according to the clarification of the Ministry of Justice and Administration in Croatia. These are business owners. 890 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 5 »Simple« consumer bankruptcy procedure 5.1 Procedure In 2019, amendments and additions to the SPZ introduce the new institution of a simple consumer bankruptcy procedure, which aims to settle creditors out of eligible assets and release consumers from residual debt. 30 The procedure is intended only for a specific category of consumers whose accounts have been blocked continuously for more than three years and whose principal debt does not exceed €2,654.46. 31 It shal be initiated ex officio by FINA on the basis of its records, after the consumer has been given the opportunity to state whether he or she agrees to the procedure. 32 If the debtor consents to the procedure, he must submit a list of his assets within the time limit specified. The declaration and the list of assets shall be submitted on the prescribed form. If the consumer fails to make a declaration within the time limit, or submits a list of assets without a declaration, or consents to the proceedings, he shal be deemed to have consented to the simple consumer insolvency proceedings being conducted over his assets. Even if it is stated that he agrees to the simple insolvency proceedings of the consumer over his assets but does not list the assets, the consumer wil be deemed to have declared that there are no assets from which his creditors can be recovered settled. 33 The application for the simple bankruptcy of the consumer shal be submitted by FINA to the competent local court on the prescribed form in electronic form if the consumer has consented to the simple bankruptcy of the consumer over his assets. 34 The court is obliged to establish the value of the consumer's assets and the disposals of assets made by the consumer in the three years preceding the opening of the consumer's simple insolvency proceedings. 35 30 Marković N., (2022). Izmjene Zakona o stečaju potrošača, Novi Informator, 2-3. 31 Article 79.a(2) of the SPZ. 32 Article 79.b(1) of the SPZ. 33 Ibid. 34 Article 79.d of the SPZ. 35 Ibid. R. Šaina: Comparative Analysis of the Personal Bankruptcy Model and Their Ef ects in the Legal Orders in Slovenia and Croatia 891 5.2 The value of the assets is equal to or less than EUR 1.327,23 If the courts finds that the value of the consumer's assets that could be realised in the bankruptcy estate is equal to or less than EUR 1,327.23, the court shall, of its own motion, issue an order opening and closing the consumer's simple bankruptcy proceedings. In this case, the court wil not appoint a insolvency administrator, nor will it set a time limit for verifying behaviour, but will release the consumer from the remaining obligations. 36 5.3 The value of the assets is greater than EUR 1.327,23 If the value of the consumer's assets that could be realised in the bankruptcy estate is greater than EUR 1,327.23, the court will, of its own motion, issue an order opening the consumer's simple bankruptcy proceedings and appoint an insolvency administrator. 37 Real estate owned by consumers is not considered a realisable asset and cannot be used to pay creditors. The assets consist of movable property and the consumer's rights. 38 5.4 Conclusion of a simple consumer insolvency procedure After the goods and rights have been realised, i.e. after the consumer has fulfilled the obligations imposed by the court's order, the court shal , by order, close the consumer's simple insolvency proceedings without setting a behavioural review period. In the decision closing the consumer's simple insolvency proceedings, the court wil release the consumer from the remaining obligations. 39 6 Concluding thoughts Based on the legislative changes in the Slovenian and Croatian legal order, the duration of personal insolvency proceedings could result in their shortening, because the purpose of insolvency proceedings is the principle of speed and economy. 36 Article 79.g of the SPZ. 37 Article 79.h of the SPZ. 38 Article 79.i(2) of the SPZ. 39 Article 79.n of the SPZ. 892 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION However, in actual practice, it is not the case, they could at least last several years (an average of 3 years). 40 We expected the two systems to be very similar, but there are differences. Consumers in the Croatian system "own" their own law, which specifies the specific conditions for the possibility of personal bankruptcy. In the case of receiving a loan or any other borrowing, it is only logical to return the received. It would be the best if we live in a society where debts are paid promptly and on time, and the institute of personal bankruptcy would not be necessary, but that is too naive to seek. We live in a time when many things are unpredictable, such as bank interest rates, the economy in general, social and societal relations, labour relations, unemployment, reckless disposal of money, high cost of living. Overnight the matter can change and we become unable to repay our debts or over-indebted. In practice, many people are uninformed and do not even know about the institute of personal bankruptcy or the possibility of their introduction. This is why the principle of informing citizens is important and crucial in such unpredictable situations in which we find ourselves. Basically, the main purpose of personal bankruptcy is equal y repay creditors, to forgive honest debtors' their debts and offer them a fresh start. Changes from the amendements of law are visible in both legal systems. One of them concerns one of the most important principles of personal bankruptcy - the speed of the process. In Croatian insolvency proceedings, which should last within the legal term of one and a half years, the legislator has complicated matters and extended their duration in the event that a third party comes forward and claims to own a part of the bankruptcy estate, the court decides to send the bankruptcy administrator to litigation. Are such provisions which complicate and hinder the procedure real y necessary? Of course not, which is why it is crucial to address these problems wisely and to define the articles of the law appropriately in the light of judicial practice and by adapting to the economic situation of society. 40 https://nasodiscu.si/osebni-stecaj#trajanje-in-stroski (May, 17 2024). R. Šaina: Comparative Analysis of the Personal Bankruptcy Model and Their Ef ects in the Legal Orders in Slovenia and Croatia 893 References Akšamović D., Šimunović L. (2023). Zakon o stečaju potrošača u službi zaštite građana: postignuća i izazovi. Godišnjak Akademije pravnih znanosti Hrvatske, Vol. XIV, No1., 87-110. Garašić J. (2017). Najznačajnine novine stečajnog zakona iz 2015. godine. Zbornik Pravnog fakulteta Sveučilišta u Rijeci, 38, 131-184. Josipović T. (2016) Razlučni vjerovnici u novom hrvatskom insolvencijskom pravu, v: Liber amicorum Gašo Knežević, 65. Directive (EU) 2019/1023 of the European Parliament and of the Council of 20 June 2019 on preventive restructuring frameworks, on discharge of debt and disqualifications, and on measures to increase the efficiency of procedures concerning restructuring, insolvency and discharge of debt, and amending Directive (EU) 2017/1132 (Directive on restructuring and insolvency), accesible on https://eur-lex.europa.eu/eli/dir/2019/1023/oj (May 17, 2024). Marković N. (2022) Izmjene Zakona o stečaju potrošača. Novi Informator, 2-3. Marković N., Čuveljak J., (2022). Izmjene i dopune stečajnog zakona, priručnik za polaznike, Pravosudna akademija, accessible on https://www.pak.hr/wp- content/uploads/2021/12/Stecajni-zakon.pdf (May 17, 2024). Stečajni zakon (SZ), Narodne novine, br. 71/15, 104/17, 36/22. Zakon o finančnem poslovanju, postopkih zaradi insolventnosti in prisilnem prenehanju (ZFPPIPP), Uradni list št. 176/21, 178/21 – popr., 196/21 – odl. US, 157/22 – odl. US, 35/23 – odl. US, 57/23 – odl. US in 102/23. Zakon o stečaju potrošača (ZSP), Narodne novine, br. 100/15, 67/18, 36/22. Višje sodišče v Ljubljani, sklep Cst 178/2018. https://arhivanalitika.hr/blog/blokirani-kako-uravnoteziti-financijsku-disciplinu-i-solidarnost/ (May, 17 2024). https://www.fina.hr/novosti/broj-ovrha-na-novcanim-sredstvima-poslovnih-subjekata-i-potrosaca- 31.-prosinca-2023 (May, 17 2024). https://nasodiscu.si/osebni-stecaj#kako-se-zacne (May 17, 2024). https://nasodiscu.si/osebni-stecaj#trajanje-in-stroski (May 17, 2024). 894 37TH BLED ECONFERENCE: RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION 37TH BLED ECONFERENCE RESILIENCE THROUGH DIGITAL INNOVATION: ENABLING THE TWIN TRANSITION ANDREJA PUCIHAR (ET AL.) University of Maribor, Faculty of Organizational Sciences, Kranj, Slovenia andreja.pucihar@um.si The Bled eConference, organised by the University of Maribor, Faculty of Organizational Sciences, has been shaping electronic interactions since 1988. The theme of the 37th conference is "Resilience through digital innovation: enabling the twin transition". The theme addresses the critical convergence of digital transformation and sustainability, in line with the European Commission’s top priorities and policy initiatives. At the conference, we aim to highlight the opportunities of digital technologies that can contribute to building organisational and Keywords: resilience, societal resilience, with a focus on social and environmental digital goals. The papers in this conference proceedings explore a range transformation, of topics including the opportunities and chal enges of the twin digital innovation, transition, emerging technologies, artificial intel igence and data twin science, decision analytics for business and societal changes, transition, digital innovation and business models, restructured work and sustainability, digital the future workplace, digital health, digital ethics, digital transition, education, smart sustainable cities, digital consumers, and the green digital transformation of the public sector. transition DOI https://doi.org/10.18690/um.fov.4.2024 ISBN 978-961-286-871-0 Document Outline 1 Introduction 2 Academic Integrity 3 Generative Artificial Intelligence (GenAI) 3.1 The Adoption of ChatGPT 3.2 Enhancing Educational Praxis with GenAI 3.3 Challenges Posed by GenAI for Academic Integrity 4 Research Method 5 Findings 5.1 Insights from One to One Interviews with University Academics 6 Limitations 7 Conclusions and Future Studies 1 Introduction 2 Related Literature 2.1 Digital Assets and Digital Strategic Initiatives1F 3 Methodology and Case Description 4 Discussion 4.1 Programmatic Bitstring Encapsulation 4.2 Hardware Abstraction 4.3 Physical extensibility and Decoupling 4 Theoretical Implications and Conclusions 1 Introduction 1.1 Smart Governance 1.2 Challenges to digital transformation 2 Method 3 Result 3.1 Indonesian Digital Transformation Policy Design 3.2 Smart Governance Transformation in Jambi City 4 Discussion Challenges On Smart Governance Implementation 5 Conclusion 1 Introduction 2 Methodology 3 Results 3.1 Definition of Metaverse 3.2 Types of Paper 3.3 Advantages of Metaverse 3.4 Challenges of Metaverse 3.5 Types of Metaverse 3.6 Technologies 4 Discussion of Research Gaps 5 Conclusion 1 Introduction 2 Methods and analysis 2.1 Data collection and participants 2.2 Measures 3 Results 4 Discussion and conclusions 1 Introduction 2 Background and Related Work 3 Methodology 4 Synthsesis of Results 4.1 Publication Trend Analysis 4.2 Qualitative Content Analysis 4.3 SWOT Analysis of 5G in Healthcare 4.4 Diabetes Care Challenges and Thematic Analysis 4.5 Emerging Trends and Gap Analysis 4.6 Feature Analysis for the Development of an Effective 5G-Enabled Dashboard for Personalized Diabetes Self-Management 5 Discussion 6 Limitations and Future Research 7 Conclusion Appendix1: PRISMA Flow Diagram Appendix 2: Publications Per Year Appendix 3: Detailed Methodology Components Appendix 4: Publication trend analysis Appendix 5: Tables and Figures with Example Raw Data Qualitative Analysis- Summary of Articles Included in the Review 1 Introduction 2 Related Work 2.1 Data-driven Business Models 2.2 Tools for Data-driven Business Model Innovation 3 Research Approach 4 Designing Tools for Data-driven Business Models 4.1 Initial Artifact 4.2 Data-driven Design Features 5 Evaluation 5.1 Focus Group 5.2 Experiment 6 Concluding Remarks Appendix 1 Introduction 2 Background 2.1 Researcher bias 2.2 Adoption factors hinging on business functions 3 Methodology 3.1 Participants 3.2 Procedure 4 Results 4.1 Researcher bias in AI adoption 4.2 AI adoption in different business functions 5 Discussion 6 Contribution 6.1 Theoretical contributions 6.2 Practical contribution 6.3 Methodological contribution 7 Limitations and future research 8 Conclusion Appendix A Appendix B Appendix C 1 Introduction 2 Literature 2.1 Technology acceptance 2.2 eHealth adoption 2.3 Design principles for PID instrument 3 Method 3.1 Data gathering methods 3.2 Selection of participants 3.3 Data handling and analysis 4 Results 4.1 Suitable design principles for a research instrument for PID 4.2 Performance of the instrument 5 Discussion and limitations 6 Conclusion Appendix A Appendix B Appendix C Appendix D Appendix E Appendix F 1 Introduction 2 Review of the Literature 2.1 Resource Based View and Dynamic Capability Theory 2.2 Data Analytic Capability and Data Driven Culture 2.3 Employee Empowerment 2.4 Transformational Leadership 3 Methods 3.1 Sampling and Data Collection 3.2 Survey Design 4 Results 5 Discussion 5.1 Limitations and Further Research 5.2 Implications for Practitioners 1 Introduction 2 Data-driven Sustainability Evaluation 3 Methodology 4 The Data-Driven Sustainability Management Maturity Model 4.1 Data Handling and Data Sensors 4.2 Inventory Background Data 4.3 Process Descriptions 4.4 Analytics 4.5 Internal Communication 4.6 External Communication 4.7 Management KPIs 5 Demonstrating the DDSM Maturity Model 5.1 Data Handling and Data Sensors 5.4 Analytics 5.5 Management KPIs 6 Discussion and conclusions 1 Introduction 2 Theoretical background 3 Research Method 4 Results 4.1. Design principles 4.2 Operationalisation 4.3 Evaluation 5 Discussion and Conclusion 1 Introduction 2 Literature 2.1 (Personal) Data Ecosystems 2.2 Balancing Data Sovereignty in Data Ecosystems 2.3 Data Intermediary Models 2.4 Research Gap 3 Methdology 4 Taxonomy of Data Control Mechanisms for Data Intermediaries 4.1 Data Access Control 4.2 Power Dynamics 4.3 Revenue Sharing Model 5 Discussion 6 Conclusion 7 Annex: overview of data intermediaires and mapping of data control mechanisms 1 Introduction 2 Methodology Analysis of the functionality and use of the ePrescription solution was carried out in December 2023 and January 2024. Statistical data were obtained from the business and administrative modules in January 2024. 3 Results 3.1 Description of the solution and analysis 3.2 Usage data 4 Discussion 5 Conclusions 1 Introduction 2 Interoperability in the European Union 3 Research Design and Methodology 4 Research Findings 5 Proposed Solution for Identity Matching and Recommendations 6 Limitations and Future Work 7 Conclusion 1 Introduction 2 Related work 3 Methods and Materials 4 Results from the User Evaluation on the Health AI tool concept 5 Discussion & AI Tooling implications 1 Introduction 2 Fuzzy-set qualitative comparative analysis (fsQCA) 3 Methodology 3.1 Search strategy 3.2 Inclusion & exclusion criteria and study selection 3.3 Data extraction and data synthesis 4 Findings 4.1 An overview of the reviewed articles 4.2 Consumer behavior towards technology adoption 4.3 Organizational technology decision-making 4.4 Other methods used with fsQCA 4.5 Novel insights and unique properties offered by fsQCA 5 Discussion and future implications 6 Conclusions and limitations 1 Introduction 4.1 Secondary data collection 4.2 Semi-structured interviews 4.3 Data analysis 5.5 Potential use cases (and technologies leveraging the implementation and use) of DTs at Railorg 1 Introduction 2 Methodology 2.1 Systematic literature review 2.2 Focus groups & survey 2.3 Interviews & Q-Methodology 2.4 Data Analysis 3 Results 3.1 Survey 3.2 Interviews 3.2.1 Q-methodology 4 Discussion, Limitations & Outlook 5 Conclusion 1 Introduction 2 Background and Related Work 3 Research Method 4 Data Analysis and Results 6 Discussion and Future Research 7 Conclusion 1 Introduction 2 Twin transformation 3 Conceptualizing digital sustainability 1 Introduction 2 Research Background 3 Research Methods Industry questionnaire Job vacancy analysis Focus groups with experts 4 Results Analysis of Job Vacancies 5 Discussion, Conclusions, and Implications 1 Introduction 2 Environmental scan 3 Research design 4 Mandatory IMO and UN regulations on data sharing 4.1 Maritime Single Window 5 The ePIcenter project recent results 5.2 MJC2: Synchromodality optimization 5.3 Smart containers 6 Conclusion 1 Introduction 2 Review of Relevant Works 3 Relevant Theories: Design Science Research Methodology (DSRM) 4 Methodology 4.1 Problem Identification 4.2 Defining Objectives 4.3 Designing 4.4 Development 4.5 The Artifact 4.6 Demonstration 4.7 Feedback to be Taken Forward 5 Results 6 Discussion and Conclusions 1 Introduction 2 Decision Analytics for HEPA Programs – First Explorations 3 HEPA Programs – User Reactions and Comments 4 Decision Analytics for HEPA Programs – Digital Ecosystems and Smart Systems 5 Summary and Conclusions 1 Introduction 2 Theoretical approach 3 Results and discussion 3.1 Nudging towards saving goals 3.2 Privacy 4 Towards design recommendations 1 Introduction 2 Background 2.1 Process mining 2.2 From decision management and modeling to decision mining Emerging from this complex decision-making landscape is decision mining, a discipline that extends the traditional focus of process mining by exploring the impact of data attributes on decision-making within processes (Beerepoot et al., 2023). Decisio... 3 Research method 4 The initial methodological framework: PM2xDM 5.1 Case study 5.2 Context 6 Discussion 1 Introduction 1.1 Urban Energy Transitions with Citizen Insights 1.2 Digital Twins for Urban Energy Transitions 1.3 Research Objective 1.4 Paper Outline 2 Methodology 2.1 Case Study Background 2.2 Research Approach 3 Social Perspectives for Digital Twins 4 Current Advancements in Digital Twins 4.1 Urban Digital Twins 4.2 Energy Digital Twins 4.3 Digital Twins for Energy Transitions 5 Towards Social Digital Twins 5.1 Social Perspective Inclusion Strategies 5.2 Conceptual Implementation Approach 6 Discussion & Conclusion 1 Introduction 2 Literature Review 3 Research Methods 4 Results 5 Conclusions Appendix A – References used in table 1 1 Introduction 2 Theoretical background 2.1 Smart City in a context of Port City 3 Methodology 3.1 Data collection 3.2 Data analysis 4 Results 4.1 Reasons for visiting or using the area of the port city 5 Discussion References Appendix 1 Introduction 2.1 Health benefits of experiencing nature 2.2 Wearable measurement of emotional arousal 3 Methods 3.1 Study sites 3.1.1 Fort de Roovere 3.1.2 Nationaal Park Sallandse Heuvelrug 3.2 Sample 3.3 Measures 3.4 Data analysis 4 Findings 4.1 Descriptive findings 4.2 Spatial analysis of emotional arousal 4.3 Self-reported health change as moderator 4 Discussion 4.1 Limitations 1 Introduction 2 Background and Hypotheses 3 Method and Procedure 4 Results 5 Discussion and Outlook 1 Introduction 4 Results References 1 Introduction 2 Methodological Approach 3 Immersion and Related Concepts 3.1 Immersion 3.2 Technological Immersion 3.3 Immersive Technologies 3.4 Psychological Immersion 3.5 Concepts Related to Immersion 4 Summary of the Literature Review 5 Discussion 6 Limitations and Future Research Suggestions Appendix A: Summary of Technological Immersion and Senses Appendix B: Summary of Immersive Technologies Appendix C: Summary of Psychological Immersion and Dimensions 1 Introduction 2 Showrooming behaviour 3 Methodology 4 Results 5 Discussion and Conclusion Appendix 1: Selected samples and constructs for the meta-analysis 1 Introduction 2 Literature review 2.1 Sustainability in the logistics industry 2.2 Digital technologies as means to achieve logistics sustainability 2.3 Value proposition in sustainable business model 2.4 Conceptual Framework 3 Methodology 4 Results 4.1 Small, local & single mode case descriptions 4.2 Large, global & multi-modal case descriptions 4.3 Findings from the cross-case analysis of value propositions in logistics services 4.3.1 Financial value proposition 4.3.2 Functional value proposition 4.3.3 Emotional value proposition 4.3.4 Digital technologies supporting sustainable value propositions 5 Discussion, limitations, and conclusions References 1 Introduction 2 Data-based Business models 3 Methodology 4 Findings 4.1 Overall Insights from the European Company Dataset 5.2 Analysis by Company Size Categories 5.3 Analysis of specific industries 6 Discussion and Conclusions Appendix 1 Introduction 2 In-store identification of customers 3 Methodology 4 Findings 5 Discussion References Appendix 1: Participant’s background information 1 Introduction 2 Methodology 3 Preliminary Results 4 Summary and next steps References 1 Introduction 2 Background and Related Work 3 Research Method 4 Data Analysis and Results 5 Discussion and Future Research 6 Conclusion References 1 Introduction 2 Theoretical Foundation 3 Methodology 4 Results 4.1 Construct Reliability and Validity 4.2 Latent Profile Analysis 4.3 Post-Hoc Analyses 5 Discussion and Conclusion 6 Limitations and Future Research 1 Introduction 2 Methodology 2.1 Data Collection 2.2 Bibliometric analysis 2.3 Co-occurrence analysis 2.4 Data Curing 2.5 Thematic analysis 2.6 Trend analysis 2.7 Network analysis 3 Findings 4 Discussion and Conclusions References 1 Introduction 2.1 Data 2.1 References 1 Introduction 2 Related literature 3 Methodology 4 Preliminary results 4.2 Refining the frontend processes 4.3 Improving the backend systems 5 Conclusion and further research 1 Introduction 2 Systematic Literature Review Methodology 3 Literature Analysis 3.1 Data and Tools 3.2 Challenges and Enablers 4 Conclusions and Future Work 1 Introduction 2 Literature Review 3 Data Collection 4 Data Analysis 5 Results 5.1 Unique Applications 5.2 Types of Applications 5.3 SWOT and TOWS Results 6 Discussion 7 Conclusion and Future Work 1 Introduction 2 Literature review 3 Methodology 4 Findings and conclusions 4.1 Preliminary findings and work to be completed 4.2 Conclusions 1 Introduction 2 Problem definition 2.1 Thesis 2.2 Expected results 3 Methodology 3.1 Identification of the problem and motivation 3.2 Definition of objectives and proposed solution 3.3 Design and development 3.4 Demonstration 3.5 Evaluation 3.6 Communication 4 Preliminary/expected results 4.1 Identification of the problem and motivation 4.2 Definition of objectives and proposed solution 4.3 Design and development 4.5 Evaluation 5 Future development 1 Introduction and Problem Definition 2 Methodology 2.1 SPG attributes (criteria) 2.1.1 Content analysis of the Web of Science (WoS) and Scopus records 3 Preliminary Results and Discussion 3.1 Summary of attributes based on four categories 3.1.1 (ICT)-enabled governance innovation in the public sector 3.1.2 Inter-organisational changes 3.1.3 Changes in citizen-government interaction 3.1.4 Outcomes – creating public value 3.2 Hierarchical structure of the decision support model 4 Future research development References Appendix 1 Introduction 2 Related Work 2.1 Public Values 2.2 Value Sensitive Design 3 Methodology Creation phase Exploration phase Delivery phase 4 Future development and next steps 1 Introduction 2 Problem definition Current body of research on digital transformation has an underlying focus that emphasizes on the economic feasibility of firms. By using new digital technologies, firms innovate new digital value propositions and offerings. Such improvements are nece... Therefore, this research intends to gain knowledge and answer such gaps by investigating the dynamics between a firm’s digital transformation strategy, business model and sustainability. Such knowledge would pave the way for a better firm strategy for... 3 Methodology 4 Expected results 5 Future Development References 1 Introduction 2 Problem Definition 2.1 Research question 3 Methodology 3.1 Study 1: Literature Review 3.2 Study 2: Eye-tracking 3.3 Study 3: Artefact 4 Expected Results 5 Future Development 1 Introduction 2 Methodology 3 Results 3.1 The overall literature review 3.2 Review of OD models 3.2.1 E-government Openness Index – eGovOI 3.2.2 Open government maturity model – OGMM 3.2.3 Open data maturity model – OD-MM 3.2.4 Open data maturity model – ODMM 3.2.5 A stage model 3.2.6 Metric for evaluating Brazilian OGD – DGABr 3.2.7 A trust-based conceptual framework on OGD 3.2.8 A model for post-adoption of OGD 3.2.9 Digital maturity model 4 Discussions 5 Conclusions 1 Introduction 2 Methodology 3 Opening of personal insolvency proceedings 3.1 Personal bankruptcy participants 3.2 Procedure - opening of personal insolvency proceedings 3.5 Conclusion of personal insolvency proceedings 4 Personal bankruptcy statistics in Slovenian and Croatian legal order 5 »Simple« consumer bankruptcy procedure 5.1 Procedure 5.2 The value of the assets is equal to or less than EUR 1.327,23 5.3 The value of the assets is greater than EUR 1.327,23 5.4 Conclusion of a simple consumer insolvency procedure 6 Concluding thoughts