ERK'2020, Portorož, 397-400 397 Quality management of Erasmus+ BENEFIT project Urban Burnik 1 , Francesco Marcuzzi 2 , Dušanka Bošković 3 , Drago Žagar 4 , Nataša P. Maleš-Ilić 5 , Darko Huljenić 6 , Matej Zajc 1 1 University of Ljubljana, FE, Tržaška 25, SI-1000 Ljubljana 2 University of Klagenfurt, NES, Lake Side Park B02 – A-9020 Klagenfurt, Austria 3 University of Sarajevo, FER, Zmaja od Bosne bb, 71000 Sarajevo, BiH 4 University of Osijek, FERIT, Kneza Trpimira 2b, 31000, Osijek, Croatia 5 University of Niš, FEE, Aleksandra Medvedeva 14, 18106 Niš, Serbia 6 Ericsson Nikola Tesla d.d., Krapinska 45, HR-10 002 Zagreb, Croatia E-mail: urban.burnik@fe.uni-lj.si Abstract. The paper presents the quality management system of the BENEFIT Erasmus+ project. Quality management is an important part of planning Erasmus+ projects. After a brief project introduction, we present general quality management aspects related to projects of this nature. We then present the project management structure followed by a detailed description of quality assurance plan and its development. 1 Introduction The members of the consortium have initiated project BENEFIT to modernize study programmes in the area of telecommunication engineering towards ICT engineering, intending to meet the demands of regional industry [1]. The project provides a framework supporting regional collaboration among the stakeholders in order to increase the quality and attractiveness of telecommunication studies and to harmonize the study programmes and methodologies with EU anticipations. The consortium consists of several universities, of which three are located in Bosnia and Hercegovina, and three in Serbia, while the action is supported by three universities from EU member states. Within the project, the consortium has introduced modernized study programmes, several new teaching mechanisms, combined with laboratory and industrial training. Industrial and educational experts evaluated the concept of BENEFIT and its new teaching and training methods. The main motivation of the participating Universities was to regain the student interest in study programs in telecommunication engineering, as the enrolment numbers recently decreased despite the dominant role of ICT industry in the modern economy. The labour market struggles for a new profile of telecom engineer (TE): a professional who is trained in a broad ICT domain, possessing both technical and market- oriented skills and consequently can contribute to the techno-economic society challenges. Accordingly, BENEFIT shapes a modern profile of graduates in telecom/ICT engineering, bridging the gap with the job market needs, specifically demanded by Western Balkan (WB) countries, which tend to be integrated in the EU. The EU character of the project supports modernization of the telecommunication engineers’ profile by bringing the experience of EU High Education Institutions (HEI) to enhance the quality of study programmes in WB. More skilled and competent graduates will reflect in better-qualified engineers who will contribute to the innovation process of the regional industry. Study programme improvements, innovative teaching and training methodologies, new labs, and internships will prepare students for a flexible international job market and enhance their mobility opportunities. The projects’ idea success depends on the partners’ active engagement in exchanging of good practices, enhancement of curricula and their contents, facilitation of mutual studies and degrees recognition as well as on active cooperation with industry. In order to persistently monitor the project actions and to foster continuing BENEFIT activities throughout and after the project duration, a formal Quality Assurance Plan was established, which will be presented in the following. 2 Quality management in Erasmus+ projects Relevant quality control measures are among the required elements in Erasmus+ assessment criteria [2]. The existence of relevant quality control measures was already part of the project proposal. However, creation of a detailed Quality Assurance Plan was one of the project tasks that underwent within the first months of the project lifetime and were regularly updated to reflect the identified requirements as the project was in effect. The project consortium comprises 15 members: 9 universities and 6 industry partners. Project duration was originally foreseen for 36 months (October 2017 – October 2020), but due to the COVID-19 pandemic introducing some delays in implementation it is extended for additional 4 months (until February 2021). It is important to note the scope of the project and significance of an efficient and effective quality management system for the overall success of the project. We have resourced to formal project quality control requirements with a reference to former successful Erasmus+ projects in the region. J. Kontio et al represent a systematic approach of quality assurance procedures where eight European universities identified a need for enhancement of self- evaluation and creation of processes with peers in order to reduce the level of inertia due to evaluation procedures in modernisation of their own study programmes [3]. T Hjeltnes et al present a comprehensive overview of quality assurance methodology in form of a case study, based on project “9-conversations”, where they study internal, external and national evaluation of the 398 Erasmus+ projects based on interviews and empirical data from the project in order to advise on evaluation and quality assurance processes in general [4]. We have identified a highly relevant source of knowledge, including templates and sample documents in quality assurance documentation related to project “Re@WBC”, which addressed management of human research potential at WB universities in line with national and EU strategies for researchers [5]. Another valuable resource is a well elaborated Quality Control and Monitoring manual of Erasmus+ project “PH-ELIM”, available online at [6]. Apart from these resources, the following formal requirements and guidelines served as a resource in creation of the Quality Assessment Plan: • EACEA – BENEFIT project Grant Agreement • BENEFIT project Dissemination and Exploitation Plan • BENEFIT project budget and task assignment • Erasmus + Guidelines for the Use of Grants • Erasmus + Frequently Asked Questions 3 BENEFIT Project Management and Quality Board The BENEFIT Project Management (PM) is implemented via the following structure: • Management Board (MB) • Development Board (DB) • Quality Control and Management Board (QB) The MB consists of one representative per project partner country and is responsible to monitor the project progress, the achievement of milestones and the delivery of planned results as well as monitor the financial aspects and the use of resources. The DB consists of WP leaders and reports the progress of activities, achievements and possible difficulties. Quality Control and Management Board (QB) comprises of one representative of the participating universities per country, of three external experts outside the consortium and of two student representatives. It is responsible for quality control and monitoring throughout the project. In case of identified issues, MB analyses the situation and eventually proposes a contingency plan. Quality Control and Monitoring (QCM) is being performed continuously during the project duration and includes: • Internal control of project progress and outcomes. • Monitoring and evaluation of deliverables. • Monitoring and evaluation of the events, (management meetings, students and teacher trainings, Webinars). • Suggestion of improvement strategies, including evaluation of offered modified and new courses, improved teaching and lab facilities as well as teaching staff and student training. • Monitoring the achievement of objectives such as impact of the project at each university, profile of graduates, improvements in skills, cooperation with industry, response to industry needs, impact of the project at regional/national level, enrolment of students in new and updated courses and employability of the graduates resulting from the project actions. • Collecting feedback from students and industry representatives via questionnaires and surveys, taking advantage of direct communication and social networks. • Monitoring students’ ability to enter the job market in the industry and through entrepreneurial initiatives as start-ups. The QCM activities follow the requirements specified in the projects’ Logical Framework Matrix and are coordinated by QB, which organizes monthly conference calls complemented by annual board meetings and assessment visits at selected participating institutions. The Board by cooperation and approval of each project partner prepared projects’ Quality Assessment Plan, which details the required procedures regarding QCM. 4 BENEFIT Quality Assessment Plan Following the state-of-the art findings and good practices in Erasmus+ project management, we have analysed the requirements and specifics of the BENEFIT project. We have identified the main domains of quality assurance for the specific objectives and actions identified within BENEFIT. The result was a Quality Assessment Plan, published as one of the key deliverables within the project. The core quality assessment requirements are categorised and described as follows. 4.1 Quality of project implementation Most important project requirement is to meet the general objectives of the proposal, which is to render the existing telecommunications study programmes more attractive and to boost the TE profile in the region. In an operational setting, the goal depends on the three BENEFIT pillars (1) cooperation between HEIs and industry to modernize the study program in telecommunication engineering, (2) adoption of modern teaching methodologies and tools, the upgrade of the lab infrastructure, and the creation of joint university-industry labs and (3) implementation of training of both teachers and students. 4.2 Progress indicators The effectiveness and impact are the main focus of BENEFIT quality control. In order to adequately track the qualitative and quantitative status of achievements on the project, a series of progress indicators have been defined. Scope of the project activities and outreach can be illustrated by providing current values for some of the key indicators: • Number of modernized courses at 1st cycle of study: 45 /at 2nd cycle of study: 20 • Number of study programs involved in modernization: 12 • Number of university-industry labs ruled according to an agreement: 6 • Number of events: 18 /project consortium meetings: 4 399 • Number of direct beneficiaries in the Partner countries per year: academic staff 101 / students 1207 • Number of partner country academic staff trained: average of 40 participants p 4 trainings face-to-face, average 50 participants at 3 online trainings A complete list of project indicators is listed in Table 1. Table 1.Project Indicators 1. Indicators of progress 1.1 Number of milestones reached 1.2 Number of meetings 1.3 Number of webinars 1.4 Number of deliverables 2. Indicators of dissemination 2.1 Project web page 2.2 Number of visits to project web 2.3 Central web platform 2.4 Number of visits to Central web platform 2.4 Designed, printed and disseminated project promotion material. 2.5 Newsletters, e-bulletins, newspapers. 2.6 Reports from presentation meetings, presentations for media and interested groups. 3. Indicators of quality 3.1 QCB established 3.2 Quality plan 3.3 Quality reports 3.4 Feedback, questionnaire results from clusters and other industrial partners. 3.5 External audits 3.6 Field visits 3.7 Minutes from QCB meetings. 4. Indicators of direct impact 4.1 Number of direct beneficiaries in the Partner country(ies) per year: academic staff from HEIs 4.2 Number of direct beneficiaries in the PCs (/year): administrative staff from HEIs 4.3 Number of direct beneficiaries in the PCs (/year): HE students 4.4 Number of direct beneficiaries in the PCs (/year): non HE individuals 5. Indicators of training activities 5.1 Number of partner country "HEIs' students trained 5.2 Number of partner country "HEIs' academic staff" trained 5.3 Number of partner country "HEI's administrative staff" trained 5.4 Number of partner country "non-HEI individuals" trained (priv. sector, NGOs, civil servants, etc.) 5.5 Number of trainings 6. Indicators of modernized curricula 6.1 Number of modernized courses at 1 st cycle of study. 6.2 Number of modernized courses at 2 nd cycle of study. 6.3 Number of novel specific-knowledge courses at 1 st cycle of study. 6.4 Number of novel specific-knowledge courses at 2 nd cycle of study. 6.5 Number of study programs involved in modernization 6.6 Number of knowledge areas 6.7 Number of knowledge areas with modernized courses 6.8 Number of study programs included in the web portal (central platform) 6.9 Number of courses on the web portal (BSc and MSc) 6.10 Number of textbooks published including guides for new equipment. 6.11 Number of new and modernized courses included into audio-library. 6.12 Number of study programs having implemented modernization 6.13 Number of delivered modernized courses (BSc and MSc ) 6.14 Number of accredited study programs 6.15 Number of students enrolled to modernized study programs. 6.16 Satisfaction survey of new learning material quality reviewers. 6.17 Students’ satisfaction survey reports. 7. Indicators of modernized teaching methodologies 7.1 Number of new methods 7.2 Number of courses implementing new methods 7.3 Number of hackatons and team competitions 8. Indicators of industry co-operation 8.1 Number of university-industry labs ruled according to an agreement. 8.2 Number of purchased equipment units (list in the inventory books) and material. 8.3 Number of students included into internships. 8.4 Number of theses co-supervised with industry. 8.5 Number of companies listed in the web catalogue 8.6 Number of collaborating enterprises 9. Indicators of management 9.1 Partnership agreements 9.2 Internal semi-annual financial reports 9.3 PM board meetings and minutes 9.4 PM board reports 10. Indicators of sustainability 10.1 Number of university-industry labs ruled according to an agreement. 10.2 Number of signed agreement about long-term cooperation between HEIs and non-academic partners. 4.3 Quality of project activities and documentation The transparency of project results is best demonstrated through consistent reporting of activities and well- organized documentation. The Quality Board has imposed several good practices in order to ensure consistency and appearance of project activities and documentation. These practices are systematically described as follows. Document deliverables The tangible project deliverables are organised in a form of reports, publications, manuals, methodology, plans, printed and electronically available promotional material and follow the common project appearance guidelines. All the deliverables are being analysed and evaluated in terms of relevance (is the deliverables coherent with the planned content), effectiveness (did the deliverable contribute to the achievement of the general and specific objectives) and timing (was the deliverable delivered according to the planned timeline). Being among the most representative of the project results, the quality of deliverables is being maintained at four quality verification levels. These levels include (1) deliverable approval by responsible WP leader, (2) internal review of deliverable provided by project partner not involved in deliverable authoring, (3) quality evaluation of deliverable by a QB board member and (4) acknowledgment of deliverable by each projects partner. 400 Events Organised events (training, conferences, info days, etc.) are considered intangible deliverables of the project and are therefore an important project result. The organizers of all the project events (working meetings, training abroad, etc.) should provide a full information package to the participants including the draft agenda, letter of invitation and a note on the logistics in due time. The organizers record the minutes of the meetings, which are distributed to all the project participants and made accessible via project portal. The quality of the project events is ensured by collecting a variety of information using visits, interviews, questionnaires, consultation, and other forms of activities. These will bring awareness of the satisfaction of beneficiaries and other target groups. Posters and other promotional materials are set-up during the event in order to increase visibility of the events. Each event should be documented using project website, Intranet site (Confluence), news, agenda, list of participants, list of trainees, report, gallery, presentations, and/or video materials. Reports include summary review of statistical data to help in final reporting. BENEFIT websites and other electronic tools The BENEFIT project foresees an immediate set up of a dedicated project website for dissemination purposes. A dedicated intranet portal is set up and maintained by project coordinator. It can be accessed by all partners depending on their assigned tasks and roles and represents a single point of reference for the project documentation and communication among partners. All partners are asked to promote BENEFIT project on their websites and other electronic tools (such as: Facebook, Twitter and LinkedIn profiles/groups, newsletters, etc.) by providing short description of the project, logo and link to BENEFIT website. Publicity control The project coordinator (UNI-KLU) is responsible for the design of the promotional material. The draft version will be sent to all partners for comments and suggestions, before printing, publishing and distribution. The materials will be disseminated by all the partners at appropriate events in order to reach the project’s target group. Document templates Quality board has prepared and maintained a repository of document templates, which ensure common project appearance of public and internal project documents. The templates were made available via an intranet project repository and included templates of (1) minutes of the meetings (2) project deliverables, (3) PowerPoint presentations, (4) event attendance sheets, (5) meeting evaluation forms and (6) event reports. In addition, templates for internal milestone verification and deliverable approval at several levels were provided. The common user of provided templates promotes the graphical identity of the project and ensures the publicity guidelines of EU are strictly followed. This project logo serves to increase the visibility of the project and has to be used for all the project deliverables and its official project documents. 4.4 Risk Management Regular risk assessment is to be carried out, which shall lead to corrective actions and potential adaptations of the workplan. This assessment will take care of issues that could endanger the project achievements. In case of serious risks, MB should suggest alternatives, workarounds and the proposed corrective actions that will make the risk consequences acceptable for the consortium. The identification and assessment of new risks is a joint responsibility of all project partners and of external evaluation. Partners may propose preventive actions (avoiding that the risk occurs) and corrective actions (decreasing the severity and impact), specifying also the resources that would be needed. 5 Comments and conclusions The paper presents a Quality Assurance plan of the BENEFIT project as it has been accepted and enhanced throughout the first two years of project duration. There were five plan revisions, based on the comments of internal and external reviewers and midterm project revision comments of the EC. The quality management activities of the project were commenced as planned. We plan to publish the actual results of project quality evaluation after the project is finished. Acknowledgements “This work was carried out within the project BENEFIT - Boosting the telecommunications engineer profile to meet modern society and industry needs - 2017-2021 - cofounded by the ERASMUS+ KA2 program – grant agreement n. 585716-EPP-1-2017-1-AT-EPPKA2- CBHE-JP”. References [1] “Project-Benefit,” [Online]. Available: https://www.project-benefit.eu/. [Accessed 22 7 2020]. [2] “Capacity Building in the field of higher education - Erasmus+ - European Commission,” [Online]. Available: https://ec.europa.eu/programmes/erasmus- plus/programme-guide/part-b/three-key-actions/key- action-2/capacity-building-field-higher-education_en. [Accessed 22 7 2020]. [3] J. Kontio, K. Heikkinen, F. Georgsson, J. Bennedsen and R. Clark et. al., “QA and Enhancement Marketplace for HEIs: An Erasmus+ project,” in CDIO 2015 : 11th international CDIO conference Conceive Design Implement Operate: Collaboration and Extension, Chengdu, China, 2015. [4] T. Hjeltnes, A. Fox, T. Hjeltnes, K. Strand and M. Storvik, “Quality assurance and evaluation in Erasmus+ projects, a case study,” INTED2019 Proceedings, pp. 8758-8764, 2019. [5] “Re@WBC,” [Online]. Available: http://rewbc.ni.ac.rs/. [Accessed 22 7 2020]. [6] “QUALITY CONTROL AND MONITORING,” [Online]. Available: http://ph-elim.net/activities/qcm/. [Accessed 22 7 2020].