Slovenian Quality Assurance Agency for Higher Education av. azpr . Zbornik r 20 VIS 20 očilo NAK GUIDELINES FOR DISTANT etno por vo. L EVALUATION acijo na dalja valv a e e z ) - Smernic ANNUAL REPORT VIS tvu (NAK NAKVIS 2020 em šols t v visok ovosak a k CONFERENCE encija RS z PROCEEDINGS Nacionalna ag Guidelines for Distant Evaluation. NAKVIS Annual Report 2020. Conference Proceedings. Published by: Slovenian Quality Assurance Agency for Higher Education Editors: Tatjana Horvat and Tatjana Debevec Authors: Tatjana Debevec, Klemen Šubic, Nataša Kramar, Olgun Çiçek, Jernej Širok, Matjaž Štuhec Authors of discussions: Peter Purg, Peter Verovšek, Nejc Šarabon, Goran Đakovic, Anna Gover Translated by: Polona Glavan Design: RB Grafika 1st e-Edition https://www.nakvis.si/wp-content/uploads/2021/10/Guidelines-for-Distant-Evaluation.-NAKVIS-Annual-Report-2020.-Conference-Proceedings.pdf For NAKVIS: dr. Franci Demšar Ljubljana 2021 This publication is free of charge. Kataložni zapis o publikaciji (CIP) pripravili v Narodni in univerzitetni knjižnici v Ljubljani COBISS.SI-ID 80974083 ISBN 978-961-93476-8-3 (PDF) CONTENTS ABOUT THE AGENCY 1 PROCEEDINGS OF THE CONFERENCE 57 Tatjana Debevec Prof. dr. Peter Purg BRIEF OVERVIEW OF THE AGENCY'S WORK IN 2020 8 NEW ROLE AND PARADOXES OF QUALITY IN THE NATIONAL PROGRAMME FOR HIGHER EDUCATION 2021–2030 59 Tatjana Debevec, Jernej Širok, Tatjana Horvat, Matjaž Štuhec Prof. dr. Peter Verovšek TENTH ANNIVERSARY OF THE AGENCY 16 WORK OF THE NAKVIS COUNCIL DURING THE PANDEMIC 62 I. OVERVIEW OF SIGNIFICANT EVENTS AT THE AGENCY AND THE HIGHER EDUCATION COUNCIL AFFECTING THEIR ACTIVITIES AND COOPERATION WITH STAKEHOLDERS 20 Prof. dr. Nejc Šarabon II. DEVELOPMENT OF THE ACCREDITATION AND EVALUATION CRITERIA OR CRITERIA FOR THE EXPERIENCE OF IMPLEMENTING INSTITUTIONAL RE-ACCREDITATION REMOTELY, EXTERNAL QUALITY ASSESSMENT 21 THROUGH THE EYES OF THE PRESIDENT OF THE GROUP OF EXPERTS IN THE PROCEDURES OF THE TWO MAJOR SLOVENIAN UNIVERSITIES 65 III. DEVELOPMENT OF IN-DEPTH SELF-EVALUATION AND COOPERATION OF THE AGENCY WITH STAKEHOLDERS 25 Goran Đaković in Anna Gover IV. INTERNATIONAL VISIBILITY OF THE AGENCY 28 DIVERSIFICATION OF EXTERNAL QUALITY ASSURANCE IN THE EHEA - REFLECTION FOLLOWING THE ANALYSIS OF ENQA AGENCY REVIEWS CONDUCTED IN 2020¬2021 69 V. ANALYTICAL AND DEVELOPMENT ACTIVITY OF THE AGENCY WITH A VIEW TO FUTURE DEVELOPMENT 29 Tatjana Debevec STRATEGIC DEVELOPMENT OF THE SLOVENIAN QUALITY ASSURANCE AGENCY FOR HIGHER EDUCATION FOR THE 2021¬2025 PERIOD 32 I. INTRODUCTION 32 II. OVERVIEW OF STRATEGIC GOALS ACHIEVED IN THE 2017¬2020 PERIOD 32 III. STRATEGIC OBJECTIVES FOR THE 2021¬2025 PERIOD 36 IV. CONCLUSION 42 GUIDELINES FOR DISTANCE EVALUATION, CEENQA 45 Klemen Šubic and Nataša Kramar BACKGROUND FOR WRITING GUIDELINES 47 Klemen Šubic and Olgun Çiçek GUIDELINES FOR DISTANCE EVALUATION, GUIDELINES BY THE CEENQA NETWORK 50 ACRONYMS AND ABBREVIATIONS CEENQA Central and Eastern European Network of Quality Assurance Agencies in Higher Education HEI Higher Education Institution EHEA European Higher Education Area ENQA European Association for Quality Assurance in Higher Education EQAR European Quality Assurance Register for Higher Education ESG or ESG Standards Standards and Guidelines for Quality Assurance in the European Higher Education Area MIZŠ Ministry of Education, Science and Sport, Republic of Slovenia NAKVIS or Agency Slovenian Quality Assurance Agency for Higher Education ZViS Higher Education Act ABOUT THE AGENCY Since quality higher education is the foundation of every modern society, the signatories of the Bologna Declaration committed themselves in the 1990s to make their efforts to improve and ensure the quality of higher education institutions, higher vocational colleges and study programmes their ongoing task. We are talking about the European dimension of the quality assurance system in higher and higher vocational education, which means: self-evaluation and accreditation or external evaluation of higher education institutions, their study programmes and higher vocational colleges with a mandatory visit of a group of independent experts to assess compliance with the prescribed quality standards. External evaluations and accreditations are the responsibility of independent institutions (agencies). Slovenian Quality Assurance Agency for Higher Education (hereinafter: the Agency) started operating in 2010, when it assumed the tasks previously carried out by the Council of the Republic of Slovenia for Higher Education (hereinafter: the Council for Higher Education). Accreditations in higher and higher vocational education, of which an independent external quality assessment is a mandatory part, have been, namely, performed in Slovenia in one form or another since 1994. The establishment of the Agency was a decisive step towards recognising and understanding the importance of independent quality assessment in higher and higher vocational education and deciding on accreditations and opinions on the achievement of quality standards. After a few years of good work, it has established itself well in the international arena and achieved full international recognition by being registered on the European Quality Assurance Register for Higher Education (EQAR) in 2013 and joining the European Association of Quality Assurance Agencies in Higher Education – ENQA in 2015. External assessment of the quality of higher and higher vocational education has constantly evolved and partly changed, both in Slovenia and internationally. The Standards and Guidelines for Quality Assurance in the European Higher Education Area – ESG as well as the Higher Education Act have been amended; especially important amendment to the latter is the ZViS-K adopted at the end of 2016, which increases the autonomy of higher education institutions in assessing the quality of study programmes, as it eliminates their reaccreditation, which was within the competence of the Agency. On the other hand, it emphasises the development and consulting work of the Agency, which is also made possible by the evaluations of samples of study programmes focused on such work. From 2017 to 2021, the Agency focused on improving professional and development work, strengthening the culture of quality, its own independence and authority, renewal of regulations, computerisation and reorganisation. In cooperation with external stakeholders, it adopted new criteria and various regulations, taking into account both new laws and revised European standards. A thorough revision of the Accreditation Criteria in 2017 was followed by amendments in 2019, 2020 and 2021. During this period, the Agency also revised the Criteria for External Evaluation of Higher Vocational Colleges, the Criteria for Agency Experts, the Criteria for Transferring Between Study Programmes and the Criteria for the Allocation of Credits to Study Programmes. In order to adapt to the circumstances of the COVID-19 pandemic, which began in the spring of 2020, it also adjusted the Minimum Standards for Appointment to Titles of Higher Education Teachers, Scientific Staff and Higher Education Staff at Higher Education Institutions for a limited period. It supported the changes to the regulations through training, consultations, interpretation of accreditation criteria and guidelines for assessment according to quality standards, revision of the Quality Manual, guidelines for distance evaluation and recommendations for work in emergency situations during the pandemic. The Agency also established information system and reorganised its work. 2 3 ANALYTICS DEPARTMENT • plans and reports (annual work and financial plan and report on the work and operation of the Agency, strategy); AGENCY APPEALS • analyses, documents, publications; COUNCIL COMMITTEE • self-evaluation of the Agency; • manuals, guides; • organisation and cooperation in different events (conferences, consultations, training courses, workshops); • establishment of databases; • translation. DEPARTMENT OF INTERNATIONAL COOPERATION AND INFORMATION TECHNOLOGY • international cooperation; The Agency's departments perform all tasks in the basic areas of its operation, which means that they draft criteria and other regulations, conduct accreditation and evaluation procedures, update • organisation of annual international conference; the Agency's information system, prepare analyses and guidelines to support the systems and • eNakvis information system and links with databases (SICRIS, IZUM); quality assessment, engage in international networking, oversee the transparency of work and communication with stakeholders, and prepare materials for decision-making of the Agency Council • internal information system iNakvis; and for the Appeals Committee. One of the basic activities is constant international cooperation with related agencies and associations of agencies, as well as development and consulting work. Detailed • keeping and updating records (on accreditations, evaluations, modifications, overview of tasks related to the external quality assessment of higher and higher vocational education transnational higher education ...); by departments: • communication. QUALITY ASSURANCE DEPARTMENT • criteria and other provisions from the field of quality; GENERAL AFFAIRS DEPARTMENT • accreditations and evaluations; • legislation, preparation of internal Agency acts; • modifications of study programmes; • assistance in the implementation of activities from the field of quality assurance; • cooperation with stakeholders (institutions/colleges, experts); • work for the Appeals Committee; • keeping and updating records (on accreditations, evaluations, modifications, • financial and human resource affairs; transnational higher education ...); • access to public information; • archiving applications and other documents. • business and administrative tasks; • keeping and updating human resource and other records within its powers. 4 5 The AGENCY COUNCIL is the highest decision-making body. Its composition enables decision-making The Agency constantly strives to ensure and improve qualitative, professional and objective quality independent of the politics and ensures the participation of all important stakeholders: assessment in accreditation and evaluation procedures. To this end, it appoints independent groups of • 3 members are appointed by the Rectors’ Conference, experts – which must include a foreign expert and a student – who carry out an in-depth assessment of • 1 member is appointed by the representative association of independent higher education institutions, compliance with quality standards in all areas of assessment. It regularly trains experts and candidates • 1 member is appointed by the representative association of higher vocational colleges, at special meetings and workshops, acquaints them with novelties in laws, Agency regulations and • 2 members are appointed by the representative organisation of students in cooperation with student international standards for quality assessment, the need for impartial, in-depth substantive quality councils, assessment, information on the characteristics of external assessments and satisfaction with them, • 1 member is appointed by representative employers’ associations by agreement, the protocol of visits to institutions and colleges and the needs to meet other, equally important • 1 member is appointed by representative trade unions in the field of higher education by agreement, conditions for the Agency expert, such as the ability to work in a group and a respectful attitude • 2 members are appointed by the Government of the Republic of Slovenia based on public invitation (1 is an towards interlocutors. The findings of the group of experts are evident from the accreditation and expert in the field of higher education or its quality assurance + 1 is an expert in the field of higher education evaluation reports and are the basis for decisions on accreditations, external evaluations or important or its quality assurance who studies or works abroad). recommendations for quality improvement. The Agency Council: Importance of cooperation • determines and adopts criteria for accreditation and external evaluation of higher education institutions, study programmes, higher vocational colleges and other regulations related to it; all are published on the In Slovenia, external assessment of the quality of higher and higher vocational education is inextricably Agency's website; linked to the active participation of students, teachers and researchers. The Agency, as well as its • decides on the following: predecessor, the Council for Higher Education, was one of the first in Europe and beyond to start working • initial accreditation of higher education institutions and their reaccreditation, which may be granted for with students at all levels: students are compulsory members of the Agency Council and all groups a maximum of 5 years, of experts, and their organisations participate in drafting legislation, criteria and other regulations. • accreditation of study programmes, including international joint study programmes, for an indefinite External quality assessment primarily addresses them, as well as teachers and researchers. The task of period of time, the Agency is to contribute to quality education according to state-approved study programmes. The • accreditation of transformations of higher education institutions, state-approved status or validity of diplomas is, namely, guaranteed by a granted accreditation, which is • compliance with conditions for the entry of a transnational higher education in the public records, within the competence of the Agency. • notifications of international study programmes accredited abroad; • adopts opinions about compliance with quality standards of higher vocational colleges; • issues recommendations to higher education institutions and higher vocational colleges to improve the quality of all their activities, and especially self-evaluation, updating and implementation of study programmes. Decisions in accreditation and evaluation procedures – on granting accreditation or rejecting an application for it, reaccreditation or possible withdrawal – are decisively influenced by the compliance with quality standards by the areas of quality assessment determined by the criteria: AREAS OF QUALITY ASSESSMENT ACCORDING TO THE TYPES OF ACCREDITATIONS OR EXTERNAL EVALUATIONS INITIAL ACCREDITATION REACCREDITATION OF A HIGHER EDUCATION INSTITUTION OF A HIGHER EDUCATION AND EXTERNAL EVALUATION OF A HIGHER PROFESSIONAL INSTITUTION COLLEGE 1. Operation of the higher 1. Operation of the higher education institution or higher education institution vocational college 2. Human resources 2. Human resources 3. Material conditions 3. Students 4. Material conditions 5. Internal quality assurance and improvement, modification, updating and implementation of study programmes ACCREDITATION OF A STUDY EVALUATION OF A STUDY PROGRAMME OR A SAMPLE OF PROGRAMME STUDY PROGRAMME 1. Structure and content of a study 1. Internal quality assurance and improvement of the quality of programme a study programme 2. Study programme 2. Modification and updating of a study programme implementation concept 3. Study programme implementation 6 7 BRIEF OVERVIEW OF THE AGENCY'S WORK IN 2020 The tasks in the Agency's Annual Work Plan were defined by taking into account the following: strategic objectives of the Agency from 2017 to 2020: • improving the professional work of the Agency, • strengthening the independence and authority of the Agency, • strengthening the culture of quality, In 2020, the Agency completed all important tasks from the action or work plan for the year, despite • renewal of criteria and other acts of the Agency, the COVID-19 pandemic, which completely changed the working conditions; in some cases, the • establishing an efficient and user-friendly information system of the Agency, Government even temporarily suspended accreditation and evaluation procedures. In addition to the • extending membership in ENQA and EQAR and strengthening the international cooperation of the planned tasks, it had to perform a number of others to adapt all or part of its activities to the situation. Agency, It had to examine the changed way of working of the Agency Council and the Appeals Committee • reorganisation of the Agency work, (videoconference sessions), the method of implementing procedures (what suspension of deadlines • development work; in accreditation and evaluation procedures means in practice), and promptly notify its stakeholders via the website and in individual documents. It also had to adjust and legally regulate the possibility of areas of assessment according to the Quality Manual: performing work from home for all Agency employees, and oversee prompt provision of information • accreditations and external evaluations, and regulation of working conditions. As the employees started working from home practically • internal quality assurance system of the Agency, overnight, they had to be trained for teleworking, provided with adequate material conditions and • external quality assurance system of the Agency, a stable infrastructure. A system of remote communication using a web platform (Zoom) has been • criteria and other provisions of the Agency, established. • information system and provision of information; The pandemic had the greatest impact on the basic activity of the Agency, i.e. accreditations and values of the Agency: external evaluations of higher education institutions, study programmes and vocational colleges. • independence, accountability, professionalism, efficiency, These are very diversified and complex tasks, which include many different actions in the accreditation • public and transparent operation, and evaluation procedures, the most important and essential for quality assessment being site visits • commitment to progress. to institutions and colleges. To make work possible, different orientations for a different way of working had to be adopted, such as distance visits, which required considerable coordination with A brief overview of the Agency work in 2020 begins with the core activity, accreditations and evaluations, all stakeholders involved, adjustments, meetings or consultations, adaptation of the information and goes on to present work in important areas: system, including training on the use of online tools, and the like. The Agency drafted and published recommendations to higher education institutions for work in pandemic-related emergencies, and, Accreditations and evaluations in higher education and higher vocational education in cooperation with the ministry responsible for higher education (MIZŠ) and the Institute of Public Health, guidelines for the implementation of study activities in the winter semester. Despite the COVID-19 pandemic and the two-month suspension of accreditation and evaluation procedures by the intervention act, the Agency sought to carry out work in this area as planned. It At the end of 2020, the strategic period of the Agency's development from 2017 to 2020 came to an developed guidelines for conducting a distance site visit (adopted by the Agency Council in May end, so – while planning new strategic orientations from 2021 to 2025 – its achievements in the past 2020), which include guidelines for preparing for a site visit, conducting a distance visit and examples period were thoroughly assessed. We can establish that the development goals have been achieved. of the visit schedule. Distance site visits were initially carried out in the procedures of evaluations of samples and evaluations of higher vocational colleges. In addition to accreditations and evaluations of higher education institutions, higher vocational colleges and study programmes, the main tasks of the Agency in 2020 were the following: After conducting the first set of distance site visits, the participating stakeholders participated in • drafting a new medium-term strategy from 2021 to 2025; a special survey, stating their opinions mainly about the applicability of distance procedures and • drafting an independent act on the Agency; satisfaction with this method of work. The results were used to supplement the guidelines or special • implementation of the second specialised annual international conference and preparation of a technical instructions for the organisation and implementation of distance site visits at the end of publication with proceedings of discussions; August 2020. • preparation of a draft guide to external assessments – accreditations and evaluations of higher education institutions and study programmes with guidelines for assessment according to quality Compared to 2019, the number of positive decisions adopted by the Agency Council in accreditation standards; and evaluation procedures was somewhat smaller, namely: • training of Agency experts and consultations with experts and chairs of groups of experts on the guidelines in the guide; Higher education • consultation with higher education and higher vocational education teachers on the guidelines in • accreditations of study programmes: 29 in 2019, 13 in 2020; the guide; • initial accreditations of higher education institutions: 0 in 2019, 0 in 2020; • development of iNakvis and further development of eNakvis; • reaccreditations of higher education institutions: 2 in 2019, 7 in 2020; • adapting the work of the Agency to the conditions caused by the COVID-19 pandemic and adopting • transformations of higher education institutions: 2 in 2019, 2 in 2020; regulations or guidelines for a changed way of work; • evaluations of samples of study programmes: 12 in 2019, 18 in 2020; • renewal of some criteria and internal acts of the Agency. Total: 45 positive decisions in 2019, 40 positive decisions in 2020. 8 9 In 2020, 37 visits took place in procedures of reaccreditation of higher education institutions, transformations of higher education institutions, accreditation of study programmes and sample evaluation of study programmes (35 in 2019). 2020 The Agency examined and recorded 400 modifications of study programmes (656 in 2019). No. of % of ISCED-F 2013 / KLASIUS-P-16 accredited accredited Higher vocational education programmes programmes • site visits: 13 in 2019, 4 in 2020; • opinions on the achievement of standards prescribed by the law: 15 in 2019, 8 in 2020. (01) Education 2 15 In 2020, the Agency carried out more procedures or reaccreditation of higher education institutions than in (02) Arts and humanities 1 8 2019; these procedures are the most complex and often include two site visits. (03) Social sciences, journalism and information 2 15 In 2020, the Agency Council did not grant accreditation to one higher education institution, one study programme (04) Business, administration and law 1 8 and one transformation of a higher education institution. One decision is still not final. The number of negative decisions in 2020 is therefore the same as in 2019. (05) Natural sciences, mathematics and statistics 2 15 The Council determined that the sample of study programmes for external evaluation in 2021 will contain (06) Information and communication technologies (ICTs) 1 8 programmes in the field of teacher training, and coordinated its decision with higher education institutions. (07) Engineering, manufacturing and construction 1 8 (08) Agriculture, forestry, fisheries and veterinary 0 0 NUMBER OF ACCREDITED STUDY PROGRAMMES ACCORDING TO KLASIUS-P-16 – (09) Health and welfare 3 23 COMPARISON BETWEEN 2019 AND 2020 (10) Services 0 0 Interdisciplinary 0 0 2019 Total 13 100 ISCED-F 2013 / KLASIUS-P-16 No. of % of NOTE: KLASIUS-P-16 is the implementation of classification ISCED-F 2013 that replaced ISCED 1997 and entered into force by the accredited accredited amended Decree on the introduction and use of the education and training classification system (Official Gazette of the Republic of programmes programmes Slovenia [Uradni list RS], No 8/17). Klasius-P-16 is intended for national and international statistical data collection and reporting. (01) Education 1 3 Tenth anniversary of the Agency (02) Arts and humanities 4 14 On the occasion of the tenth anniversary of the Agency's operation, an annual publication issued in autumn published a paper on the development of quality assessment in higher education since 1994, which was, (03) Social sciences, journalism and information 2 7 until the Agency’s establishment in 2010, carried out by the National Commission for Quality and the (04) Business, administration and law 3 10 Council for Higher Education with its senates. In addition to the annual brief review of its work, the Agency provided an insight into its operation and development through thematic discussions on the development (05) Natural sciences, mathematics and statistics 0 0 of criteria for external quality assessment, in-depth substantive assessment of quality standards and the strengthening of analytical activity. On the occasion of the tenth anniversary of the Agency, a (06) Information and communication technologies (ICTs) 3 10 videoconference consultation was organised in June 2020 featuring representatives of former and current bodies and institutions that helped Slovenia to establish a recognised external quality assurance system (07) Engineering, manufacturing and construction 0 0 in higher education comparable at the European level. Brief statements by former chairs of commissions, Council and senates, Agency Council presidents and Agency Directors are summarised in the publication. (08) Agriculture, forestry, fisheries and veterinary 0 0 (09) Health and welfare 12 41 New Agency development strategy until 2025 (10) Services 0 0 In order to formulate a new strategy and strategic goals of the Agency after 2020, when the previous strategic period expired, it was necessary to thoroughly analyse the Agency's work in the past four years, Interdisciplinary 4 14 the results of various analyses, self-evaluation reports, stakeholder responses to the Agency's work as well as findings of groups of foreign experts that assessed the Agency with the aim of renewal of ENQA Total 29 100 membership and registration in EQAR. 10 11 In the autumn of 2020, the Agency management formulated strategic objectives until 2025. These were International cooperation the basis for the document: Strategic Development of the Slovenian Quality Assurance Agency for Higher Education for the 2021¬2025 Period (strategy). The Agency Council adopted it at the December session In the autumn of 2020, the Agency director was elected president of CEENQA. The presidency has in 2020. The strategic goals for the new period are focused primarily on the adoption or enforcement changed the method of work of the network. The Agency introduced regular meetings of working groups of an independent Agency act – the act on quality assurance in higher education, the improvement of of agencies, members of the network, to identify common challenges, exchange examples of good practice quality assessment, further development of the information system, as well as sustainable development. and develop appropriate solutions to improve the quality of distance site visits or strengthen the new work The full text of the Strategy is published in the next chapter. method. Legislation and other provisions The Agency played an important role in the aforementioned EUniQ project, where partner organisations first established a reference assessment framework for the external evaluation of European university alliances. Although the draft Agency act has not been submitted to the legislative procedure, its full text has In accordance with predetermined assessment standards and developed methodology, it participated in been prepared. In January, the text of amendments to the current ZViS was prepared for discussion the first external evaluation of the EUTOPIA alliance coordinated by the University of Ljubljana. The aim at the Ministry of Science, Education and Sports; these amendments would, even without a new was to assess the adequacy of the prepared reference framework and to learn more about the new form independent act, increase the Agency’s autonomy, simplify accreditation procedures and expand the and ways of connecting European universities into alliances. The creation of these alliances also promotes Agency’s competences in relation to international cooperation. The proposed amendments to the ZViS the tendency to create alliances between quality assurance agencies, so participation in such projects is have not been discussed, so a reasoning was prepared for the draft Agency act entitled Act on Quality key to creating new directions for the development of all national and sectoral quality assurance agencies. Assurance in Higher Education containing an assessment of the situation and reasons for it as well as an The findings of the EuniQ project were presented at the dissemination conference in early March 2021. international comparison. The document was presented to the MIZŠ, but the talks were stopped mainly due to the situation related to COVID-19. According to the MIZŠ, they are expected to continue at the Cooperation with foreign agencies has been strengthened, including cooperation with the Ukrainian beginning of 2021. agency NAQA, with which a respective agreement has been signed. Agency employees and Director have attended numerous international meetings, conferences and events and regularly participated in Some criteria or standards have been amended as well. The Agency Council adopted an amendment to working groups of associations of which the Agency is a member, or in various projects. For example, they the minimum standards, which extends the time required for the appointment to the title of associate presented positive experiences and examples of good practice in the transition to the implementation professor, senior research associate and language instructor during the COVID-19 pandemic. It concerns and organisation of distance evaluations at the ECA webinar organised in May 2020; they participated a temporary suspension of their work abroad. The amended Criteria for Accreditation and External in the preparation of proposals in the projects Erasmus + (KA2) – Professional higher education Learning Evaluation of Higher Education Institutions and Study Programmes enables the accreditation of study Outcomes Development (PELOD), for ensuring accessibility for students with fewer opportunities (ESF programmes of the International Alliance of Universities EMUNI or the assessment of study programmes project) and building the institutional capacity of agencies (INQAAHE project). of this alliance taking into account its status specifics. The Agency informed the public about its work through various communication channels, mainly on the The Agency had to amend a number of internal acts, including issuing new rules on work from home, websites of foreign agencies (NAQA - Ukrainian National Agency for Higher Education Quality Assurance, which take into account the changed circumstances for such work during the epidemic, and new rules CEENQA) and in various newsletters (ECA, CEENQA, INQAAHE). on working hours. The project of researching the independence of agencies for higher education quality assurance continued, In 2020, the Agency obtained the basic Family-Friendly Enterprise certificate, which was awarded for the focusing on the impact of different stakeholders on the operation of agencies and the responsibilities of key period of three years by the non-governmental organisation Ekvilib Institute. Therefore, it adopted special actors at different levels of decision-making. rules on the implementation of measures to obtain a full certificate – these contain 16 measures for the reconciliation of work and family life of employees. The Agency will have to monitor the implementation ENQA visit to assess the Agency's progress since the last visit of these measures, and after the end of the three-year period, a special audit board will assess whether the conditions for granting a full certificate are met. The Agency seized an opportunity for a special visit by ENQA representatives to assess the compliance of its operations with ESG standards. The purpose of the meeting was to present to international experts International Conference: Shaping the Universities of the Future its achievements and progress since the last external assessment by ENQA in 2018, in connection with the recommendations made at the time. The visit was organised in the form of an online meeting; it was The international conference was held in November 2020 in the form of an online event presenting the attended by the assessment coordinator of ENQA and two members of the assessment team, the Agency results of the EUniQ project – European Approach for Comprehensive Quality Assurance of European Director, the president and vice-president of the Agency Council and employees of the Agency. An open University Networks, in which the Agency participates as one of 17 partners coordinated by the conversation between the representatives of the Agency and the ENQA association also indicated the Accreditation Organisation of the Netherlands and Flanders (NVAO). The first part of the conference direction of simplification of individual actions in the Agency's self-evaluation, so measures were taken to presented the preliminary findings of participation in the EUniQ project and the results of the external update the Agency's Quality Manual, a key document for its internal quality assurance system. evaluation of EUTOPIA, as well as the challenges for the external quality assurance system in assessing new forms of European Universities Alliances. The second part was reserved for the discussion, where Analytical work and self-evaluation participants in various networks of associations from Slovenia, Croatia and the European Students' Union (ESU) presented their experiences, expectations and views. The conference featured more than The tasks related to the self-evaluation of the Agency continued in 2020. The self-evaluation report for 100 participants. 2018 and 2019 was finalised at the end of February. 12 13 The analytical work of the Agency in 2020 was focused on a thorough review of contents and analysis The consultations were well received, attended by about 250 stakeholders. The starting points for the of the final reports of groups of experts for the accreditation and evaluation of study programmes and assessment guide presented at consultations were then adopted by the Agency Council at the December reaccreditation of higher education institutions. The Agency determined whether the quality standards session. prescribed for an individual type of accreditation or evaluation were properly assessed, especially in terms of in-depth content, in accordance with all the provisions of the criteria. The result was an extensive analysis Information system of the Agency (eNakvis and iNakvis) of the characteristics of assessments, deviations from high-quality assessment or analysis according to quality standards with the presentation of inadequate or deficient assessments and findings of experts; the The greatest progress has been made in connecting the external information system eNakvis with other results were used in the draft guide to external assessments. external systems. The structure of data on compulsory components of study programmes has been finally harmonised, and a web interface has been set up through which external systems can obtain data An analysis of research data on higher education institutions and course holders of study programmes from or transmit it to eNakvis. Electronic supplementation of applications and internal communication was also prepared, although it was not foreseen in the annual plan. The result was a diversified and useful within eNakvis between members and the Rectorate are enabled, the interdependence of individual database, which helps in more accurate and even better substantiated external quality assessments in the parts of electronic application forms is regulated, the content of forms is harmonised with changes in field of scientific, professional, research or artistic work of higher education institutions and staff, especially criteria, and the entire information system is stabilised. higher education teachers. The data prepared by the Agency as additional material for experts in quality assessment in the procedures of reaccreditation of higher education institutions, evaluations of samples of In 2020, the Agency began to develop the internal information system iNakvis in order to improve the study programmes and extraordinary evaluations are data on scientific achievements in the last five years management and administration of tasks and records, automate certain processes and facilitate the for holders of study programmes or their courses, higher education institutions and as well as faculties, communication between employees. Record keeping in the web application also makes it easier to members of universities. These are data on publications from the SICRIS database for articles covered manage complex data structures, and the latest updated version of individual records is always available by WOS, for monographs and for publications published in journals that are in the top quarter of WOS in to all employees. Easier and more efficient communication is enabled with automatic e-mails, a shared terms of impact factor. The data are standardised according to the sciences of research. calendar of events and the possibility of user communication in real time. Guide to external assessments At the end of the short report on the work of the Agency in 2020, we should mention the evaluations of universities, including the largest Slovenian university, which, despite very complex and diversified Based on a thorough analysis of the reports of groups of experts, the development of the central, most procedures that, in addition to the Agency, domestic and foreign experts, involve many groups of different important part of the guide to external assessments could begin. Its purpose is to raise the level of stakeholders at the university and outside it, have adapted to the new situation caused by the COVID-19 assessments, progress to even more in-depth determination of facts, and produce justified assessments. pandemic by introducing distance evaluations. The Agency continuously obtained assessments from The chapter discusses the types of accreditations and evaluations with all areas of assessment, quality universities and other higher education institutions or study programmes assessed in 2020 on the new standards and provisions for their assessment according to the criteria. It points out inconsistencies or method of evaluations, thus improving its work, where necessary, and maintaining the quality of external shortcomings and provides instructions for proper assessment. With the guide, the Agency wants to assessments. A thorough evaluation of the distance work in 2020 will be made in 2021. We can already set the basic starting points for further raising the level of assessments, deepening the understanding see, however, that we have lain good foundations for work in the changed situation in the future. of assessments and their effects, deepening the understanding of quality for responsible management of various ideals in higher education or balanced consideration of internal specifics of higher education. Tatjana Debevec, NAKVIS The guide is intended not only for the Agency experts, but also for higher education institutions, agency employees and other interested stakeholders. In early 2021, it was published on the website as an open document for discussion and possible comments. The finalisation of its content is planned for 2022, when it is also expected to be printed. Experts – training courses and consultations In March 2020, before the proclamation of the COVID-19 pandemic, two training courses for candidates for Agency experts were organised. The participants were presented the accreditation and evaluation criteria, specifics of assessments by types of accreditations and evaluations with corresponding quality standards and provisions for their assessment and criteria for Agency experts. The training is compulsory, followed by practical training of candidates – participation in the procedure of accreditation or evaluation of a specific institution, college or study programme. In November and December 2020, the Agency organised three consultations: with Agency experts, higher education teachers and chairmen of groups of experts on key substantive issues in assessments. It presented the structural features and peculiarities of external assessments relating to higher education teaching, scientific, professional, research or artistic work, cycles and types of studies, disciplinary integration of study programmes, institutional characteristics of higher education institutions, (self-)evaluation of their work and especially study programmes they implement, and a draft of the central part of the guide to external assessments. 14 15 TENTH ANNIVERSARY OF THE AGENCY1 Presidents of the Agency Council Professor Peter Purg, Ph.D. 2019 ¬ Professor Franc Čuš, Ph.D. 2018 ¬ 2019 Professor Andreja Kocijančič, Ph.D. 2013 ¬ 2018 Since its establishment, the Agency has strived for reliable quality assessment and high-quality counselling, Professor Maja Makovec Brenčič, Ph. D. 2010 ¬ 2013 which is supported by competent experts, employees and Council members. It complements the efforts for external evaluations that lead to quality improvement, as well as autonomous and fair decisions on accreditations and evaluations, with a focus on supporting internal quality assurance systems in higher Composition of the Agency Council, as of 2020: education institutions and higher vocational colleges. On the one hand, it relies on its development work, which is based on the preparation of relevant manuals, interpretations of regulations and instructions, and Appointed by which further includes analytical activities, project participation, monitoring and collection of practical solutions, development of guidelines for self-evaluation, maintenance and continuous updating of various Professor Peter Purg, Ph.D., President Rectors’ Conference computer databases and public records and the establishment of a team of consultants. On the other hand, it Boris Dular, Ph.D., Deputy President Representative employers’ associations assigns an important role to the transfer and deepening of expertise and the strengthening of competencies. Professor Marjan Mernik, Ph.D. Rectors’ Conference It regularly organises trainings, consultations, professional conferences or symposia and exchanges. Professor Ivan Svetlik, Ph. D. Rectors’ Conference Professor Sebastjan Kristovič, Ph.D. Representative association of independent higher education In my opinion, the most important thing is that the Agency has maintained and built institutions independence, autonomy, professionalism and international competitiveness. All this is the Branko Škafar, Ph.D. Representative association of higher vocational colleges achievement of all of us, you and all those stakeholders who have supported us in improving standards, regulations, procedures and internal documents – everything that made it possible Klemen Peran Representative organisation of students in cooperation with student councils for both the Agency's staff and external evaluators and all persons carrying out the procedures to credibly assess what was of sufficient and what was of insufficient quality for our higher Andrej Pirjevec Representative organisation of students in cooperation with student councils education area. At that time, we also prepared the first and very critical self-evaluation report, Professor Bruno Završnik, Ph.D. Representative unions in the higher education area where we wrote down especially what and who the Agency is. We also received very good responses from all international institutions that assessed the Agency. This was a great Professor Mirko Pečarič, Ph.D. Government of the Republic of Slovenia achievement for all of us and especially for the Agency's staff, and a great international Professor Peter Verovšek, Ph.D. Government of the Republic of Slovenia recognition. [...] Even as a minister, I have always been committed to creating the basis for an independent Agency act that the Agency deserves. All the relevant articles in the ZViS are a good basis for considering how to establish even greater autonomy as well as credibility of the Presidents of the Council for Higher Education, Director of the Secretariat of the Council and Agency – also through independent legislation. Presidents of the Agency Council at the meeting at the 10th anniversary, 10 June 2020: Professor Maja Makovec Brenčič, Ph. D. All together, from staff, members of bodies and students to higher education institutions, participated in the development of the field of quality assurance in Slovenian higher education. It was an extremely difficult period. As is customary in Slovenia, the prevalent opinion was that it was not possible. Our stubbornness, however, helped establish a system that I am proud of. This means that it is comparable at the European level, and our higher education institutions have reached a level where they are constantly updating their study programmes and institutionally developing through accreditation and evaluation procedures, which of course means great progress and success. Professor Franc Čuš, Ph.D. 1 The text was first published in Slovene in the publication Deseta obletnica Nacionalne agencije Republike Slovenije za kakovost v visokem šolstvu: Zbornik razprav o zvezah evropskih univerz – univerzah prihodnosti, in 2020. 16 17 Directors of the Agency: • establishing an efficient and user-friendly information system of the Agency, • extending membership in the ENQA and EQAR and strengthening the international cooperation Professor Franci Demšar, Ph.D. 2018 ¬ of the Agency, • reorganisation of the Agency work and its Professor Ivan Leban, Ph.D. 2013 ¬ 2018 • development work. Tatjana Debevec (acting director) 2012 ¬ 2013 Professor Mojca Novak, Ph. D. 2010 ¬ 2012 In 2011, based on the materials and practices of the Higher Education Council, the Agency, in cooperation with stakeholders, developed a set of guidelines that, in addition to regulations, guided external quality assessment. The set consisted of the Agency's quality manual, Rules of Procedure Present heads of departments of the Agency: of the Agency Council, a handbook for experts and templates for writing reports – expert opinions. These materials were regularly updated, adapted to the changes in regulations, supplemented and Jožica Kramar Quality Assurance Department improved. Two years later, the Agency supplemented the guidelines by a comprehensive overhaul of the handbook for experts and a protocol of the evaluation visit, which sets out in more detail the course Klemen Šubic Department of International Cooperation and Information Technology of the visit together with the tasks of the group of experts. By amending the Accreditation Criteria in Tatjana Debevec Analytics Department 2017, it also offered stakeholders an explanation of the criteria or individual quality standards with Barbara Zupančič Kočar General Affairs Department guidelines for assessment. The Agency organised five trainings for members of the Agency Council and, in order to help with Agency Directors and the first secretary of the Council for Higher Education at the meeting at the consistent decision-making, made an analysis of the Agency's selected decision-making practices 10th anniversary, 10 June 2020 : in accreditation and evaluation procedures and all practices of the Appeals Committee and the Administrative Court of the Republic of Slovenia, particularly focusing on the objectivity and impartiality and sectoral regulations and reasoning of decisions. The Agency staff attended numerous professional training courses and seminars, most frequently in the field of ethics, integrity, conflict of interest and general administrative procedure. In 2018 and 2019 alone, they participated in 41 education and training courses, workshops and consultations in Slovenia and abroad. They also participated in the exchanges with the Austrian, Cyprian and Latvian quality assurance agencies. In October 2019, three staff members participated in the international training for professionals according to the European approach. 21 training courses were organised for the Agency experts; in addition, candidates for entry in the register of experts participated in a number of evaluation visits as observers before they were entered in the register. They learned about sectoral regulations, methods of assessment, the role of experts in assessments, the Agency's information system, the characteristics of evaluation practices and the related results of systems analyses. Before amending the criteria for experts in 2018, the Agency also participated in the training of students – candidates for experts organised by the Student Organisation of Slovenia. It also offered its experience and expertise in the projects of some higher education institutions aimed at the establishment of internal evaluation systems. Thus, it participated several times in the training of internal evaluators of universities and independent higher education institutions. The work and development of the Agency were based on the strategy or strategic goals that it had set Its training programmes were adapted to the proposals and needs of the participants, changes in at the beginning of its operation and revised at the end of 2016. These goals concerned especially the regulations and its experience with training on the one hand and external evaluations on the other. One following: of the key changes in the training of candidates for entry in the register of experts was the introduction of mandatory participation of candidates in evaluation visits, so that they no longer got acquainted • development and functioning of the quality assurance system, with the role of the assessor through simulation, but on the basis of observing actual assessments and • monitoring of progress and strengthening of higher education quality culture, trial participation. Special recognition was given to the training system by an international group of • positioning and recognition of the role, significance and quality of the Agency's operation in the public, assessors when the Agency was in the process of obtaining the membership in ENQA and EQAR. • co-creation and development of higher education policy in the area of quality assurance, • membership of the Agency in international associations (ENQA and EQAR); The Agency and its predecessor, the Evaluation Senate, organised 11 consultations both for their experts • while in the period 2017¬2020, they concerned the following: and for other stakeholders. The topics of the consultations included consideration of changes or proposals • improving the professional work of the Agency, for changes in international and national regulations and guidelines, organisation and implementation • strengthening the independence and authority of the Agency, of evaluation visits, writing reports, activities of other agencies and organisations in the field of quality • strengthening the culture of quality, assurance and improvement, results of systems and thematic analyses, development trends and higher • renewal of criteria and other acts of the Agency, education policies. 18 19 The consultations also covered topics such as the issue of blind and partially sighted students and students Year Amendments to Guidelines Training courses Consultations and International exchanges Analyses with disabilities in higher education, or the Slovenian higher education area – on the way to excellence. the legislation and conferences and projects adoption of criteria The anniversary consultations, the symposium on the occasion of the second and the conference on 2016 ZViS-K Training course for Consultation for the occasion of the fifth anniversary of the Agency also featured a reflection on the operation of the members of the experts Agency Council Agency and a discussion on its future. The Agency exchanged views, experiences and open issues with (Commission for the Prevention of participants at each consultation. In 2018 and 2019, it organised two consultations for commissions for Corruption) quality at higher education institutions and individuals involved in self-evaluation at higher education 2017 New Accreditation Interpretation of 4 training courses Opinion analysis of the Criteria the Accreditation for experts concept of quality and institutions and higher vocational colleges. Criteria as a strategic orientations of New Criteria for constituent part of the Agency International the regulation The professional development of the Agency was significantly enhanced by international activities in Cooperation projects, namely in the ECA projects and the Erasmus + EIQAS project. 2018 New Evaluation Criteria Consultation for Quality assurance experts Consultation in Slovenian higher of the Commission and higher vocational for Quality education from 2014 to OVERVIEW OF SIGNIFICANT EVENTS AT THE AGENCY AND THE HIGHER EDUCATION COUNCIL 2017 AFFECTING THEIR ACTIVITIES AND COOPERATION WITH STAKEHOLDERS Quality of staff in Slovenian higher education from 2014 to 2017 Year Amendments to Guidelines Training courses Consultations and International exchanges Analyses 2019 Amended Accreditation Revision of the 6 training courses Consultation for Exchange – visit of a Proceedings of discussions Criteria Agency’s Quality for experts experts representative of the on quality assurance in the legislation and conferences and projects Manual Cyprus Agency of Quality higher education: from adoption of criteria Consultation of the Assurance to Slovenia minimum standards to excellence, and SQAA 2008 Instruments for 2 training courses Qrossroads, ECA, 2007 and 2008 Commission for Quality Visit of a delegation of annual report 2018. In external evaluations for experts database of reports of metareport Amended Criteria for cooperation with the Transfers representatives of the accredited programmes Slovenian Academy Conference Latvian Agency of Quality of Science and Arts and institutions, today Assurance to Slovenia Amended ECTS criteria Stakeholder opinion on DEQAR Conference external quality assurance Slovenian higher Visit of representative in 2019 2009 ZViS-G Consultation for 2009 meta-report education area of the Croatian Agency experts – on the path to of Quality Assurance excellence? (AZVO) 2010 Accreditation Criteria Manual for Experts Training course for MULTRA, ECA, agreement Minimum standards members of the on mutual recognition 2020 Amended Accreditation Guidelines for Training course for Historical overview Criteria for Transferring Agency Council of JOQAR accreditation Criteria – regulation distance evaluations members of the of the Agency and of provisions for an Agency Council its predecessors ECTS Criteria decisions, ECA, quality international alliance of Recommendations (Commission for assurance and recognition universities for work in the Prevention of Symposium on the of degrees awarded for emergency situations Corruption) occasion of the 10th joint study programmes Amended Minimum anniversary of the Standards for the Training course for Agency 2011 Criteria for External Quality Manual of 5 training courses Consultation for Exchange of staff members Appointment to a Title members of the Evaluation of Higher the Agency for experts and experts – visit to the Austrian Council (decision- Vocational Colleges Manual for Experts Agency employees Quality Assurance Agency making) 2012 Participation in 2 consultations for CeQuInt, ECA, certificate Training of training for student experts for the quality of candidates for experts – SOS internationalisation experts Agency symposium 2013 Revision of the 2 training courses 2 consultations for Quality in the Slovenian Manual for Experts for experts experts higher education and The following chapters bring a more detailed presentation of the Agency's work in terms of subject areas that higher vocational have influenced its development and visibility at home and abroad. The basis for performing the basic activity Visit protocol Participation in education area from 2010 training for student to 2013 of the Agency, accreditation and external evaluation of higher education institutions and study programmes experts – SOS and higher vocational colleges, are the Accreditation and Evaluation Criteria harmonised with stakeholders, so their development is described first. This is followed by a presentation of the number of accreditations and 2014 Amended Accreditation 2 training courses EEP, ECA, exchange Analysis of selected evaluations by year, cooperation with stakeholders and provision of information, international visibility of the Criteria for experts of experts for external decision-making practices assessment of quality of the Agency Council and Agency, self-evaluation, and finally analytical and development activities with a view to future development. all practices of the Appeals Committee and the Administrative Court DEVELOPMENT OF THE ACCREDITATION AND EVALUATION CRITERIA OR CRITERIA FOR 2015 Revision of the 4 training courses Consultation on the Erasmus + EIQAS – Agency’s Quality for experts fifth anniversary training, development of EXTERNAL QUALITY ASSESSMENT Manual of the Agency guidelines, examples of Participation in operation good practice and analysis training for student of the assessment of experts – SOS internal quality assurance Immediately after its establishment, the Agency started preparing new criteria for accreditation and external systems evaluation in accordance with the ZViS 2009, which had to consider that external evaluation of higher Training course for internal evaluators education institutions and study programmes is a condition for their reaccreditation at least every seven of UP ECAPedia, ECA, years. It was decided that both – accreditations and external evaluations – would therefore be governed Refreshes seminar information on higher a single regulation rather than two, as was the case before the establishment of the Agency. On the other on the General education Administrative hand, the Agency no longer applied the “higher education” criteria mutatis mutandis to external evaluation Procedure Act for Hosts of a CEENQA employees workshop of higher vocational colleges; instead, the latter got their own, as college evaluations do not have the same role or consequences as they do in higher education, where they are part of the reaccreditation procedure. Accreditations in higher vocational education are decided by the competent minister. 20 21 The new Criteria were created on the basis of the old Accreditation Criteria from 2004 and the Evaluation While institutions are still granted initial accreditation and reaccreditation – as a rule, every five years Criteria from 2006, as they enabled the assessment of all important areas of a higher education and not seven, as was the case until the ZViS 2016 – this is granted indefinitely to higher education institution, study programme and higher vocational college, also taking into account the ESG. The first study programmes, provided that the legal provisions and quality standards from the Criteria are Agency criteria entitled Criteria for the Accreditation and External Evaluation of Higher Education met. What has been preserved is the instrument of extraordinary evaluation, which allows for the Institutions and Study Programmes were adopted in November 2010 and entered into force in early withdrawal of accreditation of study programmes. December 2010. In June 2014, the Agency Council adopted new criteria with the same title, which was repeated in 2017 – therefore the names of the accreditation criteria referred to below include the year in The ZViS 2016 introduced a novelty to strengthen the advisory and development role of the Agency, which they were adopted. i.e. evaluations of samples of study programmes that are performed every year. The 2010 Accreditation Criteria were a good basis for the start of the Agency work or for assessing The 2017 Accreditation Criteria adopted on the basis of the ZViS 2016 constitute a shift from meeting institutions and study programmes. They brought a significant change in the assessment of the the minimum standards for accreditation or reaccreditation to a qualitative assessment of the quality fulfilment of the criteria for the reaccreditation of both institutions and study programmes. Due to a of institutions and study programmes they implement. Therefore, they specified for the first time the high number of procedures, they set the so-called minimum standards that the institutions had to meet. quality standards that must be met in accreditation and evaluation procedures, major deficiencies or Their biggest weakness was that they did not (sufficiently) distinguish between the criteria for the initial inconsistencies that affect the (non) reaccreditation, reaccreditation for a period shorter than 5 years, accreditation and the criteria for reaccreditation, that they were too detailed, sometimes unclear, and that as well as withdrawal of accreditation of study programmes, and follow-up procedures for institutions. the application form was uniform for all types of accreditations (evaluations). The 2010 Accreditation Criteria were amended four times, and the fifth amendment in 2014 was so thorough that new ones The areas of assessment of higher education institutions and study programmes differ by the type were drafted. The Agency decided on a thorough revision based on self-evaluation findings, comments of accreditation and evaluation. Quality standards or their assessment differ in the same areas of from higher education stakeholders, recommendations of an international group of evaluators for the assessment depending on whether it is the initial accreditation or reaccreditation of an institution, entry of the Agency in EQAR and renewal of its membership in ECA in 2013 and difficulties due to or the accreditation or evaluation of a study programme. They also differ between accreditations of unclear provisions. individual types of study programmes. Compliance with the ESG can mostly be linked to standards by the areas of assessment for reaccreditation or evaluation. It is with regret that I look at the process of transformation of professional colleges into In order to make the criteria as friendly as possible for the participants in the procedures, saving faculties, which took place in the past when I was the Agency Director. One of the important them work and time, the assessment areas and quality standards with provisions on their assessment points of a faculty is, namely, that its core operation is scientific research work, while some are repeated in the electronic application forms. The provisions are also intended for a uniform professional colleges skipped this condition in the appeal procedure with a decision of the interpretation of the regulation. Administrative Court. Different regulations first regulated the accreditation of joint study programmes implemented only Professor Mojca Novak, Ph. D. by Slovenian institutions (in the 2017 Accreditation Criteria) and the accreditation of joint study programmes implemented by Slovenian institutions together with foreign ones – the so-called international joint study programmes: these were the Criteria for International Cooperation in Higher Education (Criteria for International Cooperation). The latter also regulate the notification of study The 2014 Accreditation Criteria did not change much in terms of content. They were, however, thoroughly programmes and higher transnational education, which are not part of accreditation and evaluation consolidated, and the number of assessment provisions was significantly reduced, especially for the procedures. The criteria governing international cooperation of Slovenian institutions in one place initial accreditation of study programmes. The accreditation of the transformation of higher education provide a better insight into the specifics of these procedures. institutions and their branches was regulated in more detail, and the procedures on which the Agency Council based its decisions were formulated more clearly. Superfluous appendices for accreditation of Following the example of the Accreditation Criteria and taking into account the characteristics of study programmes were also deleted. higher vocational education governed by the Higher Vocational Education Act, the new Criteria for External Evaluation of Higher Vocational Colleges were adopted, first in 2011 and then in 2018. The consolidated Accreditation Criteria made work easier for everyone: for higher education institutions Evaluation is an independent procedure. Colleges are assessed in the same areas as higher education and experts as well as for the Agency staff and Council. They were used in their unchanged form for institutions, and the Agency Council adopts an opinion on meeting the prescribed quality standards, applications submitted to the Agency until the amendment of the ZViS in 2016. as accreditations are not within its competence. A large number of procedures where the Agency had to regularly reaccredit all – more than 900 – study Other criteria and standards programmes prevented an in-depth substantive assessment of quality. Dealing with more than 200 accreditation and evaluation cases a year, the Agency ran out of time for qualitative assessments as well Besides the Accreditation Criteria, the Agency determined upon its establishment the Minimum as for other urgent basic tasks, which include assistance to institutions in establishing internal quality Standards for Appointment to Titles of Higher Education Teachers, Scientific Staff and Higher assurance systems and in self-evaluation, education of stakeholders or cooperation with them, consulting, Education Staff at Higher Education Institutions (Minimum Standards for Appointment to Titles), systematic and transparent communication, facilitation of procedures through the information system which still apply. In 2010, it also determined the Criteria for the Allocation of Credits to Study (eNakvis), and much more. The turning point was the ZViS 2016, whose implementation brought the so-Programmes under ECTS (ECTS criteria), which did not differ from the criteria used by the Council called transition to institutional reaccreditation, which in practice meant a significant reduction in the number for Higher Education, and the Criteria for Transferring Between Study Programmes (Criteria for of procedures. Transferring). Both regulations were updated in 2019. 22 23 NUMBER OF EVALUATIONS BY YEAR, SITUATION ON 31 DECEMBER 2019 Overview of fields of study programmes (Classification Klasius-P-16) When the Agency was established, there were four universities and 38 independent higher education EDUCATION 57 130 01 institutions. While two universities were formed over a 10-year period, which now makes six in total, the number of independent higher education institutions fluctuated considerably, also due to the fact that they ARTS AND HUMANITIES 132 168 11 23 later merged into two universities. Six universities and 40 independent higher education institutions are SOCIAL SCIENCES 87 104 10 17 accredited in 2020. BUSINESS, ADMINISTRATION AND LAW 54 61 48 73 NATURAL SCIENCES, MATHEMATICS AND STATISTICS 56 59 8 10 INFORMATION AND COMMUNICATION TECHNOLOGIES 21 28 4 4 ENGINEERING, MANUFACTURING AND CONSTRUCTION 80 91 18 24 AGRICULTURE, FORESTRY, FISHERIES AND VETERINARY 27 31 1 2 HEALTH AND WELFARE 32 39 8 27 TRANSPORT, SECURITY, CATERING AND TOURISM, PERSONAL SERVICES 26 34 4 9 INTERDISCIPLINARY 8 15 0 1 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% PUBLIC (2010) PUBLIC (2019) PRIVATE (2010) PRIVATE (2019) Independent higher education institutions Universities DEVELOPMENT OF IN-DEPTH SELF-EVALUATION AND COOPERATION OF THE AGENCY WITH Number of study programmes of higher education institutions in Slovenia STAKEHOLDERS The principal aim of self-evaluation of the Agency is to assess the quality of its operation as a whole, the implementation of tasks from the work or action plan by evaluating the accomplishment of strategic objectives of the Agency, especially in comparison with the previous self-evaluation period, to improve the Agency's operation according to the second and third sections of ESG standards in compliance with the self-evaluation findings and recommendations of groups of experts, and to define the opportunities for the improvement of work. The basic areas of quality are accreditations and external evaluations, the external quality assurance system of the Agency, the internal quality assurance system of the Agency, criteria and other regulations, as well as the information system and provision of information. The self-evaluation of the Agency is prepared by a self-evaluation group consisting of at least three employees and one Council member. External stakeholders (representatives of students, the Rectors' Conference, the Association of Independent Higher Education Institutions, the Association of Higher Vocational Colleges) also cooperate with 17 the group. 17 UL - University of Ljubljana UM – University of Maribor UP – University of Primorska UNG – University of Nova Gorica The Agency has developed well and improved its work throughout its existence. It has developed UNM – University of Novo mesto NU – New University analytical and critical self-evaluation and its own information system with strong databases; it SVZ – Independent higher education institutions provides updated information about its activities to stakeholders and places a particular focus on students, helping them in choosing their studies by providing information on accredited study programmes, higher education institutions and evaluated higher vocational colleges. The following figure shows the number of accredited study programmes according to the Klasius-P-16 In recent years, it has been working to improve quality assessment; it is a transition from classification in 2010 and 2019, which are also distributed by public or private higher education institutions. quantitative assessment to qualitative assessment with more in-depth content. The Agency The increase in the number of programmes in individual fields (e.g. educational sciences and also provides advice to institutions and colleges; its international and project cooperation is arts and humanities – public institutions) is also a result of the transition from non-Bologna to very diverse and vibrant, and it boasts strong analytical and development activity, which will Bologna programmes. In private institutions, programmes in business and administrative sciences be further diversified in the future. and law strongly predominate, with programmes in health care and social security, engineering, production technologies and construction, and the humanities constituting approximately the Tatjana Debevec same share. 24 25 In its self-evaluation reports, the Agency deals in depth with topics concerning its basic activity, the The aim of the survey was to gain insight into the average perceptions or expectations of stakeholders broader context of higher education and the relationship between internal and external stakeholders. by interest groups regarding ensuring or improving quality, and especially to identify their expectations It strives for the preservation and consolidation of independence, stable and orderly operation, active regarding the work of the Agency and its strategic development. The results of the survey, without any international operation, and oversees the balance between internal stakeholders. The Agency's individual significant deviations, showed a tendency for the Agency to focus on strengthening its advisory role self-evaluation reports show that the self-evaluation group critically monitors developments that could and assistance to higher education institutions and higher vocational colleges in developing quality, jeopardize the Agency's operations, both at the decision-making level and at the level of legislation and on performing development tasks or preparing professional analyses. Some stakeholder groups and higher education policy, and proposes necessary measures. Annual work plans and work reports also emphasised the strengthening of the professionalism in the implementation of procedures and of the Agency demonstrate the observance of these measures and therefore the strengthening of its assessments, especially the Agency experts and Council members. The strengthening of professionalism development and importance. was also highly ranked by the representatives of the management of institutions and colleges. The Agency conducted its first self-evaluation for the period 2010-2012, assessing all internal processes The Agency conducted a comprehensive survey on the opinion of stakeholders on external quality of its operation and evaluating the achievement of the set priority strategic goals by 2013 and ESG assessment in 2019. Thus various stakeholders communicated to the Agency their views on the standards. During this period, it set key milestones for its development, such as the constitution of Agency's regulations or criteria, the work of the Agency staff, expert assessments, the composition its bodies, the adoption of regulations for external quality assurance, the initial assessments of study of groups of experts, their reports, decisions of the Agency Council and appeal procedures, as well programmes and higher education institutions, the start of international activities in cooperation with as the effects of accreditation and evaluation procedures on the quality of institutions and colleges agencies and other associations for quality in higher education. In 2013, it successfully completed external and the extent to which the Agency ensures that quality standards are met. Good scores were given evaluation which enabled the extension of its ECA membership, accession to the MULTRA project and mainly to the Agency staff and experts with their assessments and reports. While the decisions of entry in EQAR; in 2015, it became a member of ENQA. The development flourished, which was also a the Agency Council and its appeal procedures were rated below average, the worst score was given to result of the self-evaluation: the Agency revised the strategy for the period 2017 to 2020, updated the sub-questions about the Agency's contribution to the quality of tertiary education, not so much about quality manual and formalised the system of continuous monitoring of quality improvement of higher the contribution to the success of internal quality assurance systems in institutions or colleges than education institutions and study programmes, arranged long-term and stable funding of the Agency, to the quality of implementation and content of study programmes, and especially to the quality of adopted new, better criteria, made progress in the development of the eNakvis information system, in material and human resource conditions for study and research at institutions or colleges. The analysis development and analytical work, international cooperation and communication with stakeholders, for of the survey emphasises that the relatively positive self-image shared by internal stakeholders and which the Agency's self-evaluation group determined from year to year that it needed an urgent and stakeholders who are better acquainted with the work of the Agency must be verified by a more critical thorough improvement, and reorganised its work. attitude of stakeholder groups on the other hand. The communication plan for strengthening communication with internal and external stakeholders was The latter will have to be taken into account in the self-evaluation period 2020-2021, during which thoroughly revised in 2018. The aim of the revised communication plan was optimise the cooperation of the Agency will prepare a new medium-term strategy until 2025. It currently performs, monitors and higher education institutions in the procedures for amending regulations and to increase the common evaluates the implementation of procedures for reaccreditation of universities and independent higher understanding and expectations in quality assessment in higher education among different stakeholders. education institutions and evaluations of a sample of study programmes, implementation of the eNakvis Regular meetings were introduced to resolve open issues regarding accreditation procedures, information information system and development of tools for systems and thematic analyses and self-evaluation, system, legislation and other important issues with the Ministry of Education, Science and Sport, monitors and updates its regulations, when necessary, and prepares an independent Agency act or the Student Organisation of Slovenia, the Employment Service of Slovenia, the Rectors' Conference, supplement or amendment to the ZViS in the part relating to the Agency. universities and other higher education institutions. The Agency's self-evaluation is also based on obtaining the opinions of various internal and external Recently, the Agency has made the greatest progress in the field of transparency. We stakeholders on its work. One of the methods is periodic surveying. Let us mention the findings of the keep stakeholders informed about all important developments related to the Agency, last three surveys. investors have a detailed insight into the procedures that are being conducted, and the implementation of the procedures is upgraded by the new information system eNakvis; at In 2018, the Agency conducted an opinion survey to check the satisfaction of stakeholders from various the same time, our website offering a variety of information has also become an important aspects of its operations. The results showed that stakeholders were satisfied with the content offered entry point for future students. We are also working on contents. We have developed by the Agency at events such as training courses, consultations and conferences. They believe that the various guidelines to help experts and other stakeholders understand the evaluation Agency takes into account the expectations, initiatives and rights of stakeholders, but there remains procedures. Of course, the Agency has adapted to the current situation, so we launched an opportunity to strengthen cooperation with representatives of the wider intellectual or cultural distance evaluations. community. The latter refers mainly to the strengthening of cooperation with higher education institution libraries and the academic community, as well as quality assurance committees at institutions and Professor Franci Demšar, Ph.D. colleges. According to the written comments of the respondents, they want an agency that would listen to the real problems of higher education and higher vocational education, enable a better understanding of the situation in various scientific disciplines and be better recognised especially among students. The publicly available records of the Agency are an accurate, transparent and up-to-date public database The Agency also conducted an opinion survey before devising its strategy for the period 2017-2020. The of accredited higher education institutions, study programmes and evaluated higher vocational colleges. survey focused on strategic priorities and basic content-based dilemmas of the Agency important for its The website also contains a constantly updated list of higher education institutions, study programmes further strategic development. and higher vocational colleges in accreditation and evaluation procedures with all relevant information. 26 27 The Agency encourages students to check, before enrolling in tertiary education, whether the internationalisation, establishing a register of expert reports on higher education institutions and selected study programme or higher education institution or higher vocational college is state-study programmes, establishing an encyclopaedic overview of European higher education, and cross-approved and to make sure that their studies lead to a state-approved graduation certificate. It border exchanges of experts. In the Erasmus + EIQAS project, it exchanged views and experiences provides students and all interested public high-quality and diverse information on Slovenian higher on the operation and assessment of internal quality assurance systems among various universities, and higher vocational education. Since 2014, the website has regularly published expert reports on quality assurance agencies, experts and students. A students’ guide to European ESG standards and the assessments of study programmes, higher education institutions and higher vocational colleges, a comparative analysis of methods of external assessment of internal quality assurance systems have which are an important source of information on the state of quality and development, which can been prepared. Another important field of international cooperation is strategic networking with help the general public to create an impression of the strengths and weaknesses of each institution, agencies in the area of former Yugoslavia, which takes place through participation in international study programme or college. projects and associations and frequent visits to agencies. Particular attention is paid to the exchange of thematic positions and good practices in the field of independence of the operation of agencies for The Agency's website was completely renovated in 2018, both graphically and in terms of content, quality assurance in higher education. now providing access to all groups of users, especially the blind and partially sighted, as well as people with reading disabilities, for which the Agency was the first organisation in the public sector awarded The Agency also encourages internationalisation in the context of evaluations of study programmes the A3C certificate – accessible to all. Through its own example and actions, it encourages and raises that are more closely related to the international environment. According to the Criteria for International the awareness of stakeholders in higher education about the importance of ensuring accessibility, Cooperation, it assesses the compliance with the conditions for the accreditation of joint study while following the guidelines for accessibility of the website also helps to improve the user experience programmes of Slovenian and foreign higher education institutions, for transnational higher education for all users, including those without disabilities. and for the notification of study programmes accredited abroad. The purpose of the Criteria is to assure appropriate quality of international study programmes according to the standards of quality Since 2019, the Agency has published a monthly newsletter Novičnik for active provision of information determined by the Accreditation Criteria, ESG, and the quality of transnational higher education. The to its stakeholders. It is also active on social networks. Its website also contains information on Agency also recognises accreditation decisions adopted by EQAR-listed agencies. individual higher education institutions – their research activities, other accreditations, rankings on international scales – enrolment information, other useful links and presentation videos of higher The Agency has developed a strategy for international cooperation, the areas of which are described education institutions, thus enabling them to present and point out their achievements, situation and in the chapter on the Agency work in 2019. plans themselves. The Agency's website provides higher education institutions access to the eNakvis information I will turn my gaze to the future. I would really like to see the Agency get ing its own Agency system, where electronic applications for most procedures can be currently submitted. In addition act. I would not like the Agency to continue to be part of the Higher Education Act in Articles to providing electronic support for procedures, eNakvis contains a wide digital database of higher 51a to 51ž. Of course, I am also in favour of a transparent act on higher education, which education institutions, higher vocational colleges and study programmes, which is the basis for should also feature a student ombudsman, and, given all the scandals in the past, the connecting to databases of higher education institutions, colleges and other state institutions, which establishment of an inspectorate for higher education. This inspectorate should oversee the will help increase efficiency, eliminate duplication of work and enable the creation of accurate records implementation of Article 79 of the ZViS. It is the ministry that has to see to the legality of as well as support for analytical processes at the Agency. the operation of higher education, instead of the Agency taking this burden. [...] I believe that high quality of education is very important for our country. It is an important part of Activities for the establishment, testing and implementation of an independent information system culture. Education must remain a public good just like public health care, as we now could of the Agency have been taking place since mid-2014, during which time the entire system has been see. Together with citizens, public health care is what has defied the epidemic we have thoroughly considered and supplemented. During the establishment of eNakvis, there was an ongoing experienced. [.] Quality is very difficult to achieve and maintain. But it can be lost very communication with stakeholders, including presentations or explanations of technical specifications. quickly. Stakeholders’ comments on the operation of eNakvis were closely monitored and largely taken into account in the further development of the system. Professor Ivan Leban, Ph.D. INTERNATIONAL VISIBILITY OF THE AGENCY The Agency is a full member of EQAR, ENQA, ECA, CEENQA and INQAAHE. Its membership in ANALYTICAL AND DEVELOPMENT ACTIVITY OF THE AGENCY WITH A VIEW TO FUTURE international associations for quality in higher education proves the compliance of its operation with DEVELOPMENT European standards and guidelines. Active participation in various European and international projects strengthens the Agency’s international visibility and the visibility and international comparability of the The analytical work of the Agency and its predecessor is marked by four extensive systems analyses. Slovenian higher education area; the Agency thus contributes to the development of high-quality, open These describe the cross-sectional state of quality of higher vocational colleges, higher education and accessible higher education, monitors and co-creates guidelines for quality assessment in higher institutions and their study programmes by all areas of assessment or related quality standards. They education and the values of the European higher education area. The Agency has thus established also address the quality of their self-evaluations and, since 2013, shed more light on the characteristics and formalised cooperation with several related agencies and participates in international projects, of evaluation practices of experts – what the experts particularly emphasised and what they criticised. in executive committees, general assemblies and working groups of these international associations. Similarly, they shed light on the properties of self-evaluation reports of institutions and colleges. They In cooperation with ECA, the Agency developed solutions in the field of mutual recognition provide an in-depth insight into the operation of the quality assurance system in Slovenian higher of accreditation decisions regarding joint study programmes, assessment and quality of education. 28 29 With five thematic analyses, the Agency focused on selected topics, including the characteristics of 3. It will strengthen its international activities to get a good overview of examples of good practice, decision-making practices of the Council and the Appeals Committee, quality of staff in Slovenian higher policies, analyses and comparisons on a European and global scale. It will adapt their application education, and the opinion of stakeholders on the work and reputation of the Agency, its contribution to in the Slovenian higher education area subject to systematic conditions and, if necessary, limit it by quality, as well as on the concept of quality and quality-related expectations among different groups of critical reflection and professional consultation with stakeholders. stakeholders and their views on the strategic orientations of the Agency. One of the thematic analyses addresses the transition from minimum standards to excellence. The analytical work of the Agency is 4. It will involve scientists in higher education didactics and research in important phases of the first developing towards more quantitative and statistically supported research. It is based on the publicly three points, and strengthen consultations, conferences, symposia and training sessions. available methodology and procedure for the preparation and dissemination of analyses. In the future, the Agency intends to develop analytical work through substantively more in-depth analyses in selected areas. In can see increasingly clearly how the Agency, including the Agency Council, is becoming a relevant civil society pendant, a kind of critical partner to the Ministry, on the one hand, while Analytical and development activities are of key importance for the further development of the Agency. on the other, the wider social value of the Agency is increasingly visible. It is revealing itself This article tried to show that the Agency has successfully implemented its current strategic goals in as a relevant interlocutor to the profession and getting a clear role in the higher education the field of professional work, independence, quality culture, regulations, computerisation, as well as ecosystem. The latter, namely, sometimes gives the impression that it exists in a kind of autarkic its international activities and development work. With the exception of the reorganisation of work distance from social reality, social conditions and economic frameworks, as well as culture. By and the establishment of the information system, its strategy until 2020 was directed towards the assuring quality, the Agency is firmly attaching the higher education area to society and thus future, towards the continuation of the set course of development. This orientation will take it to the gaining the civil-social and political role that belongs to such an agency or a good system of next strategic period, in which the goals from different areas will be linked by further efforts aimed at academic quality. professional external assessments at high level, which will be sensitive to the needs of students and the external environment, as well as better take into account the internal specifics of higher education Professor Peter Purg, Ph.D. related to different cycles and types of studies, disciplines to which study programmes are linked, and differences in the status and organisation of higher education institutions. Aiming at balancing the expectations of the external environment coming from the economic, social The Agency will continue its efforts to maintain a high level of its institutional autonomy and the and cultural spheres with the expectations of the academic community, which is best acquainted with autonomy of Slovenian higher education. It will also seek to strengthen autonomy at the transnational the specifics of studies and knowledge, we will try to cultivate a constructive dialogue at consultations, level and freedom in co-shaping national higher education policy. It will try to consolidate its regulations conferences, training courses, and harmonisation and development of external quality assurance system to ensure their long-term stability and consequently the stability of external assessments. The Agency between representatives from different environments. This dialogue should lead to synergies between believes that such a development will convince Slovenian higher education and Europe that its decisions autonomous higher education institutions and the Agency, which must continue to perform not only in accreditation procedures are excellently supported, fair and right for all. advisory but also supervisory tasks, and which at the same time influences the development of higher education and internal quality systems through its assessments and policies. Tatjana Debevec, Jernej Širok, Tatjana Horvat, Matjaž Štuhec, NAKVIS In the future, the Agency will focus on improving the already reliable and useful external audits. Our plan is to make the evaluation reports establish the actual situation even better, taking into account the specifics of the assessed subject. In their assessments and report writing, experts will rely on more sources, researches and comparisons and thus make better, more knowledgeable, understandable and useful recommendations. At the same time, the Agency will ensure that the content of the reports will be more accessible and transparent to a wider readership, making decisions easier in particular for future students and employers. It will try to make this shift by four key orientations: 1. It will strengthen analytical and research work in order to offer experts more useful references, reviews, averages and deviations in terms of quality in Slovenia through systems and thematic, in-depth analyses and scientific research, and at the same time provide better information about the characteristics, methods and consequences of their work. Its long-term efforts will be directed towards establishing a research group. 2. It will provide additional guidelines for external assessments, which will place greater emphasis on substantive dilemmas and starting points for assessments and on a more in-depth interpretation of regulations instead of operations and protocols already developed. These guidelines will transcend the generic level of rules. They will also address the particularities of cycles, types and disciplinary integration of a study, and differences in its institutional structure. They will try to get closer to meeting different academic standards. The Agency is preparing a special guide to accreditations and evaluations. 30 31 STRATEGIC DEVELOPMENT OF THE SLOVENIAN It has laid the foundations for improving quality assessment; for the transition from quantitative assessment to qualitative assessment with more in-depth content, which is reflected in the new criteria and other acts it has QUALITY ASSURANCE AGENCY FOR HIGHER adopted. It has strengthened the counseling of institutions and colleges and cooperation with them, switched EDUCATION FOR THE 2021–2025 PERIOD to the so-called institutional accreditation, and begun to perform sample evaluations of study programmes intended to help institutions improve and self-evaluate study programmes. The international and project-based cooperation of the Agency has been diverse: it has its representatives in the bodies of various international institutions and in 2020, the Agency director was elected president of CEENQA. Analytical and development activities have also developed and will be further diversified in the future. There has been a great improvement in communication with stakeholders, which the Agency's self-evaluation group found deficient from year to year. In order to implement the above as soon as possible and as easily as possible, the Agency first thoroughly I. INTRODUCTION reorganised its work in 2018 and then in 2019 by adding two more departments to the Quality Assurance Department and the General Affairs Department, namely the Analytics Department and the Department The period in which the Slovenian Quality Assurance Agency for Higher Education (the Agency) has set of International Cooperation and Information Technology – the latter two were created by the division of the medium-term strategic goals from 2017 to 2020 is coming to an end. Before describing the achievement of Quality Assurance Department. It also regulated the long-term and stable funding of the Agency.In 2018 and strategic goals in the period and defining new ones until 2025, let us emphasise that since the beginning of 2019, it extended its membership in ENQA and registration on EQAR for a full period of five years. its operation in 2010 – this year marks the tenth anniversary of its founding and the development in all these years is summarised in a special annual publication – the Agency has strived for continuous improvement Overview of events related to the professional and analytical work of the Agency and cooperation in all areas of its work. We can see that it has been quite successful, following the values it has set for itself: with stakeholders from 2017 to 2020: professionalism, transparency and development. The establishment of the Agency in 2010 was a decisive step towards recognising and understanding the importance of independent quality assessment in higher Year Amendments to Guides/guidelines/ Training Consultations and International exchanges Analyses and higher vocational education and deciding on accreditations, evaluations and opinions on the achievement the legislation and interpretations conferences and projects of quality standards. After a few years, the Agency has established itself well in the international arena and adoption of criteria 2017 New Accreditation Interpretation of 4 training courses Opinion analysis of the achieved full international recognition by being registered on the European Quality Assurance Register for Criteria the Accreditation for experts concept of quality and Higher Education (EQAR) in 2013 and joining the European Association of Quality Assurance Agencies in Criteria as a strategic orientations of Higher Education – ENQA in 2015. In the years from 2017 to 2020, it focused mainly on the improvement of constituent part of the Agency the regulation professional, consulting and development work, renewal of criteria, better communication with stakeholders 2018 Consultation for Quality assurance and computerisation. It reorganised its work into four sectors and reinforced its human resources in the field experts in Slovenian higher of information technology. Consultation for and higher vocational the Commissions education from 2014 to for Quality at 2017 The Agency connects the strategic orientations for the period from 2021 to 2025 mainly with the further higher education Quality of staff in institutions Slovenian higher education development of enhanced professional, analytical and development work and with more focused training from 2014 to 2017 (and cooperation) of stakeholders. The basic strategic goal of the agency until 2025 is the adoption of an 2019 Amended Accreditation Revision of the 6 training courses Consultation for Exchange – visit of a Proceedings of discussions independent act on the Agency – the act on quality assurance in higher education. Criteria Agency's Quality for experts experts representative of the on quality assurance in Manual Consultation for Cyprus Agency of Quality higher education: from Amended Criteria for the Commissions Assurance to Slovenia minimum standards to Transfers for Quality at Visit of a delegation of excellence, and SQAA higher education representatives of the annual report 2018. In II. OVERVIEW OF STRATEGIC GOALS ACHIEVED IN THE 2017¬2020 PERIOD Amended ECTS criteria institutions Lithuanian Agency of cooperation with the Quality Assurance to Slovenian Academy of Conference Slovenia Science and Arts The strategic goals adopted by the Agency Council at its 107th session on 15 December 2016: Slovenian higher Stakeholder opinion on • improving the professional work of the Agency, education area external quality assurance – on the path to in 2019 • strengthening the independence and authority of the Agency, excellence? • strengthening the culture of quality, 2020 Amended Accreditation Guidelines for Training course for Historical overview Proceedings of discussions • renewal of criteria and other acts of the Agency, Criteria – regulation distance evaluations members of the of the Agency and on alliances of European of provisions for an Agency Council its predecessors universities – universities • establishing an efficient and user-friendly information system of the Agency, international alliance of Recommendations (Commission for of the future, Agency • extending membership in the ENQA and EQAR and strengthening the international cooperation of the universities for work in the Prevention of Symposium on the annual report 2019 and emergency situations Corruption) occasion of the 10th tenth anniversary of the Agency, Amended Minimum Training course for anniversary of the Agency • reorganisation of the Agency work and its Standards for the Online guide to members of the Agency Appointment to a Title assessment of Council (decision- • development work. quality standards for making) International accreditation and Training of conference on evaluation candidates for federations In-depth and analytical self-evaluation, annual reports on the work of the Agency and other documents show experts of European that the Agency has made great progress in recent years and achieved the set goals. It has put in place a universities Two consultations new information system eNakvis with strong databases, it provides updated information about its activities for Agency experts to stakeholders and helps students in choosing their studies by providing information on accredited study Consultation for programmes, higher education institutions and evaluated higher vocational colleges. higher education teachers 32 33 The Agency has organised several trainings for the Agency Council and the Appeals Committee The database is the basis for connecting with the databases of higher education institutions, higher members and prepared various analyses to help with consistent decision-making. The analyses focused vocational colleges and other state institutions, which will eliminate duplication of work and enable on objectivity and impartiality, as well as to sector-specific regulations and reasoning of decisions. More real-time creation of accurate records. The database will also help in further developing the analytical than ten training courses and consultations have been also organised for the Agency experts in these work at the Agency. three years. In particular, they learned about the new criteria for accreditation and external evaluation, methods of assessment, the Agency's information system, the characteristics of evaluation practices The Agency has comprehensively redesigned its website, both in terms of graphics and contents, which and the related results of systems analyses. Special training courses were organised for the introduction made it the first organisation in the public sector to receive the A3C Certificate, accessible to all. It has of sample evaluations of study programmes. In 2018 and 2019, the Agency organised first two broad published eNovičnik (eNewsletter) since 2019. It regularly publishes adopted regulations, and informs the consultations for commissions for quality and individuals involved in self-evaluation at higher education stakeholders about the content of the emerging regulations and constructively addresses their proposals. institutions and higher vocational colleges. In these two years, employees participated in 41 education and training courses, workshops and consultations in Slovenia and abroad. They participated in exchanges Its website also contains information on individual higher education institutions – their research activities, with foreign agencies, and three of them attended international training for experts according to the data on other accreditations, rankings on international scales, enrolment information –, other useful links European approach. and presentation videos of higher education institutions, thus enabling them to present their achievements, situation and plans themselves. The communication plan of the Agency was thoroughly revised in 2018. The aim was to maximise the cooperation of higher education institutions in changing regulations and to increase the common The publicly available records of the Agency are an accurate, transparent and up-to-date public database understanding of quality assessment in higher education among different stakeholders. Regular of accredited higher education institutions, study programmes and evaluated higher vocational colleges, meetings were introduced to resolve open issues regarding accreditation procedures, information together with decisions of the Agency Council and reports of groups or experts. The website also contains system, legislation and other important issues with the Ministry of Education, Science and Sport, a constantly updated list of higher education institutions, study programmes and higher vocational colleges the Student Organisation of Slovenia, the Employment Service of Slovenia, the Rectors' Conference, in accreditation and evaluation procedures with all relevant information. universities and other higher education institutions. The goal of “extending membership in ENQA and EQAR and strengthening the international cooperation of The Agency conducted the last comprehensive survey on the opinion of stakeholders on external the Agency” has been achieved. The Agency participates in international projects, in executive committees, quality assessment in the expiring strategic period in 2019. The results showed that stakeholders want general assemblies and working groups of international associations of agencies. In 2020, the Agency a greater Agency’s contribution to the quality of implementation and content of study programmes, director was elected president of CEENQA. and especially to the quality of material and human resource conditions for study and research at institutions or colleges. A step towards it is the Criteria for the Accreditation and External Evaluation Another important field of international cooperation is strategic networking with agencies in the area of of Higher Education Institutions and Study Programmes, which place a great focus on the assessment former Yugoslavia, which takes place through participation in international projects and associations and of these issues. They set quality standards whose assessment differs according to the type of frequent visits to agencies. Particular attention is paid to the exchange of thematic positions and good accreditation and evaluation and the type of higher education institution or the type and cycle of a practices in the field of independence of the operation of quality assurance agencies in higher education. study programme. They also differ from the old Criteria in the detailed provisions on assessment, which are intended for a uniform interpretation of the regulation. The impact of the new Accreditation The Agency also promotes internationalisation through the accreditation of international joint study Criteria on the experts’ assessment and decision-making of the Agency Council began to show at the programmes of Slovenian and foreign higher education institutions, the notification of study programmes end of 2019. The Agency is currently examining and analysing it and preparing the basic documents accredited abroad and transnational higher education in accordance with the new Criteria for International for the consultations that started at the end of November and continued in the first half of December Cooperation in Higher Education. It recognises accreditation decisions adopted by EQAR-listed agencies. 2020. Following the example of the Accreditation Criteria and taking into account the characteristics of higher education, new Criteria for External Evaluation of Higher Vocational Colleges were adopted in In the last two years it organised two regular annual international conferences on current topics in 2018. The Criteria for International Cooperation in Higher Education and the new Criteria for Agency Experts higher education, published two annual publications of the Agency with proceedings of discussions on were adopted in the same year. selected topics at the conference, a short presentation of its work in the past year and selected thematic contribution. Also updated in 2019 and 2020 were the Criteria for the Allocation of Credits to Study Programmes under the European Credit Transfer System, Criteria for Transferring Between Study Programmes and Minimum The analytical and development work of the Agency is most evident from the extensive systems and Standards for the Appointment to the Title of Higher Education Teacher, Researcher and Faculty Assistant thematic analyses and in-depth self-evaluation of the Agency, for which it also received recognition from at Higher Education Institutions. external auditors of ENQA. The analyses describe the cross-sectional state of quality of higher vocational colleges, higher education institutions and their study programmes by all areas of assessment or related The Agency has finally developed a new eNakvis information system, which enabled the submission of quality standards, address the quality of their self-evaluations or reports and shed more light on the applications for accreditation and evaluation in electronic form in 2018 and improved the cooperation characteristics of evaluation practices of experts. They provide an in-depth insight into the operation of with institutions and other users in 2019. During the establishment and development of eNakvis, the the quality assurance system in Slovenian higher education. Agency actively cooperated with stakeholders. In the expiring strategic period, the Agency has prepared and published an analysis of quality in the The greatest progress has been made in establishing a reliable, up-to-date and supplemented Slovenian higher education area from 2014 to 2017, an analysis of the quality of staff in Slovenian higher database on higher education institutions and study programmes; at the end of 2019, it contained education, a contribution on the transition from minimum standards to excellence, and the findings of three-quarters of all data required by the Agency from higher education institutions. stakeholder opinions on external quality assurance and on distance education. 34 35 It has updated and published the methodology and procedure for the production and dissemination of 1. Adoption of the act on quality assurance in higher education systems and thematic analyses. By providing solutions different from those determined for the work of the Agency by the ZViS, the quality In order to improve self-evaluation and its work, the Agency has also amended its Quality Manual, Rules assurance in higher education act (quality assurance act) will enable high-quality and full implementation of Procedure of the Agency Council and other internal acts. It also responded quickly to the COVID-19 of the tasks related to strategic orientations. The quality assurance act will regulate procedures by assigning epidemic by adapting accreditations and evaluations to the situation. It has developed guidelines for the essential role to accreditation and evaluation decisions made on the basis of expert substantive distance site visits and published recommendations during the COVID-19 related emergency. assessments or findings; it will establish a basis for accreditations and evaluations abroad, regulate inconsistencies in applicable regulations and gather their provisions in one place. In addition to the eNakvis information system and the established system of distance external evaluations, the Agency set up in the second half of 2020 an internal information system intended for managing The reasons for the adoption of an autonomous quality assurance act are therefore the following: employees' time and tasks, called “iNakvis”. The latter enables both monitoring and updating of tasks a) consolidation of the independence of the Agency; from the action plan, creation of events and online meetings by automatically generating web links for b) regulation of the appeal procedure, which would ensure the importance of substantive them, as well as fast communication between co-workers. assessment and decisions in accreditation and evaluation procedures; c) provision of a legal basis for the work of the Agency abroad – international accreditations; III. STRATEGIC OBJECTIVES FOR THE 2021¬2025 PERIOD d) determination of the Agency's procedures in an act and not in an implementing regulation – as also recommended by the decisions of administrative inspectors; Mission of the Agency e) elimination of a significant number of inconsistencies in the current legislation; The Agency provides for development and operation of the quality assurance system in the f) merging the Agency-related provisions of the ZViS, Memorandum of Association of the Agency, Slovenian higher education area. It operates with substantive and formal responsibility and Accreditation and Evaluation Criteria, Criteria for International Cooperation and other relevant provides counselling for all stakeholders and participants in tertiary education in accordance with regulations in a single document, which would also contribute to de-bureaucratisation. the European and global development orientations. The Agency will strive to consolidate and enhance its autonomy as well as the autonomy of Slovenian Vision of the Agency higher education. Independence is closely linked to other reasons for the adoption of the quality The Agency shall, with its system of quality assurance development, contribute to higher education assurance act; besides those listed above, these are the consolidation of the provisions governing the in Slovenia being of high quality in terms of education and research, internationally recognisable, principal activity of the Agency, or the criteria that will help external quality assessments develop in competitive and equally integrated in the global higher education area. the right direction and stabilise, which will boost the Agency’s reputation and trust of higher education stakeholders. Values of the Agency: • professionalism (P) The achievement of this goal is in accordance with the value of the Agency that commits it to development. • transparency (T) • development (R) 2. Improvement of assessment according to quality standards and accreditation and evaluation procedures Strategic objectives for the 2021¬2025 period: The assessment guide outlined under the next strategic goal will contribute to the improvement of 1. Adoption of the act on quality assurance in higher education (D) assessment according to quality standards in all prescribed areas and to the correct observance of 2. Improvement of assessment according to quality standards and accreditation and evaluation the provisions on their assessment. In the strategic period from 2021 to 2025, the Agency will strive procedures (P) primarily for professional external assessments at high level, which will be sensitive to the needs of 3. In-depth substantive analytic work in selected areas (P) students and the external environment, as well as better take into account the specifics of higher 4. Establishment of up-to-date databases on selected activities at higher education institutions education related to different cycles and types of studies, disciplines to which study programmes (T) are linked, and differences in the status and organisation of higher education institutions. It will 5. Pursuing sustainable development goals (D) therefore focus on improving the assessments of experts and, on their basis, the decisions of the 6. In-depth international cooperation (P) Agency Council. The Agency plans to organise many education and training courses, meetings with 7. Proactive communication (T) all important participants in accreditation and evaluation procedures, and together with them: The Agency connects the strategic orientations for the 2021¬2015 period mainly with the further a) address the inconsistencies in assessments made so far; development of analytical and development work; preparation of a special guide to accreditations and b) unify the understanding of individual quality standards and provisions on their assessment; evaluations with an emphasis on in-depth substantive assessment according to quality standards; c) shift the assessment of quality standards from formalisms to a thorough substantive with continuous and more focused training of participants in accreditation and evaluation procedures; assessment under all its provisions; with strengthening international activity in current areas; and with proactive communication. To d) improve, in particular, the assessment of pedagogical, professional, scientific, research and facilitate the work of the Agency experts, it wants to establish up-to-date databases on research artistic work of higher education institutions and teachers; and library activities at higher education institutions, their premises and equipment. The basic e) make better use of individual quality standards and provisions on their assessment, taking strategic goal of the Agency until 2025 is an independent act on the Agency – the act on quality into account the type of higher education institution and the type or nature and cycle of the assurance in higher education. study programme; f) improve and unify the accreditation and evaluation reports of experts, in which the factual They provide institutions and colleges information with which they can compare the quality of their situation will be even better established and the specifics of the subject of assessment will be educational and research activities and self-evaluation practices. The purpose of systems and thematic taken into account; analyses is to provide assistance in reflections on the quality of tertiary education, its assessment and g) strengthen awareness of the importance and effects of both external audits and the quality that improvement. Thematic analyses may be the basis for guidelines, recommendations and guides related to these assessment seek (It will encourage a prudent use of conceptual approaches to quality.). evaluation and accreditation procedures. Such documents are adopted by the Director (recommendations) or the Agency Council (guidelines and guides) and published on the website among the official acts of the A thorough and in-depth substantive assessment will be the basis for decisions on accreditations and Agency. They must be observed by institutions, colleges and experts in their assessments. evaluations. Therefore, in the draft quality assurance act, the Agency establishes special commissions by fields of study to assist the Agency Council; these commissions will, if necessary, discuss the findings In this strategic period, as in the past, a systems analysis of quality in the Slovenian higher education area of experts on study programmes. from 2018 to 2023 and extensive analyses of the opinions of internal and external stakeholders on the operation of the Agency will be made, especially in quality assessment, which will be one of the important The stages of accreditation and evaluation procedures also have some room for improvement: bases for Agency reports for 2020 and 2021 and for 2022 and 2023. a) filing applications for accreditation or evaluation; b) experts and their appointment: The Agency, in cooperation with external associates, will prepare analyses on individual issues related • removal of non-responsive or inappropriate experts from the register, to higher education, publish them and present them at domestic and international consultations. The • complaint of an institution, analyses will be the basis for guidelines to be used by experts in assessing quality standards, and will also • site visit of experts (work method, uniformity, impartiality, objectivity and professionalism influence amendments to regulations, such as guidelines for the assessment of higher education libraries. – in-depth substantive assessment), They will also be the basis for the Agency's thematic contributions in annual publications and for the • regular training and consultation sessions, including training courses for foreign experts; Agency's international conferences. c) findings or “interim” reports of experts; d) final reports of experts; In-depth analyses of the reports of groups of experts on accreditations and evaluations will provide additional e) consideration of applications and decision-making at the Agency Council; guidelines for external assessments, which will place greater emphasis on substantive dilemmas and f) appeal. starting points for assessments and on a more in-depth interpretation of regulations instead of operations and protocols already developed. They will try to get closer to meeting different academic standards. The To improve the above, the Agency plans to increase the number of employees for conducting accreditation Analytics Department is preparing a special guide to accreditations and evaluations or assessments. Based and evaluation procedures or to reinforce the human resources in the Quality Assurance Department. At on the results of the analysis of previous expert reports, quality assessment consultations were held at the all stages, their work with higher education institutions (applicants) as well as with the Agency experts end of November and the beginning of December 2020, to which higher education teachers and scientists – including in monitoring the production of accreditation and evaluation reports – and members of the from higher education didactics and science research were invited in addition to the Agency's experts and Agency Council needs to be further improved and standardised. The Agency will try to achieve a unified other internal stakeholders. Consultations, trainings, conferences, etc. will continue in this strategic period, understanding of the fulfilment of quality standards and legal provisions, and thus the correct guidance and the guide will be continuously updated and improved. The Agency plans to first publish it in full in of experts in audits, primarily through continuous and even more intensive training. electronic form at the end of 2021, and later to print it and distribute it to all stakeholders. The new quality assurance act provides for changes in the objections and appeals of applicants, with greater The achievement of this goal is in accordance with the value of the Agency that commits it to professionalism. emphasis on substantiated and in-depth substantive findings. Related to the latter are the consideration of applications and decision-making at the Agency Council. In addition to care and commitment to 4. Establishment of up-to-date databases on selected activities at higher education institutions responsible and fair conduct, independent decision-making, equal treatment of applicants, avoidance of conflicts of interest etc., attention will be focused on preparing Council members or its rapporteurs for The Agency has developed and put in place the eNakvis information system, which already enables much substantive findings regarding each application and related report of the group of experts. more than the old system. During this strategic period, it will develop it so as to make it maximally useful or easy to use. In addition to submitting and completing electronic applications for accreditations, evaluations, The achievement of this goal is in accordance with the value of the Agency that commits it to professionalism. transformations of institutions and modifications of study programmes or the possibility of communicating their transformations or modifications, creating various databases and updating them on an ongoing basis, 3. In-depth substantive analytic work in selected areas it will further assist the work of the Agency Council and the Appeals Committee, experts and Agency staff. It will support the Agency's self-evaluation and its analytical activity, including by obtaining data on In the future, the Agency intends to develop analytical work through substantively more in-depth analyses accreditations and evaluations according to the old criteria set before the establishment of eNakvis. in selected areas. It has written in the renewed methodology for the production and dissemination of systems and thematic analyses that the latter provide information by individual quality variables, namely By enabling the connection of various information systems and databases, the Agency will promote the for study programmes, institutions and colleges; general and specific information on the accreditation connection between the systems of higher education institutions, the Ministry of Education, Science and evaluation practices of the Agency, i.e. on the characteristics of the accreditation and evaluation and Sport, the Institute of the Republic of Slovenia for Vocational Education and Training (CPI) and other reports of experts and on the internal evaluation practices at institutions and colleges, as shown in their institutions. self-evaluation reports. In addition to informing the public, analyses can be the basis for the formulation of higher education and higher vocational education policies, implementing regulations and other New databases are planned: the Agency, together with IZUM, will create a database on the research regulations on the quality of higher education and for the planning of external evaluations. They provide activities of higher education institutions or their higher education teachers and associates and libraries, reference support in assessments in evaluation and accreditation procedures. as well as a database of compulsory components of study programmes. 38 39 A database on spatial capacities and equipment of higher education institutions will also be established. to training experts and other stakeholders for a more in-depth quality assessment in line with the These bases can be of great help to experts and the Agency Council in assessing the quality of higher emerging guide to accreditations and evaluations. education institutions and study programmes. In cooperation with the Ministry of Education, Science and Sport, a database on the employability of students by higher education institutions and study The Agency has obtained the “family-friendly company” certificate, which proves that it strives for programmes is also planned, taking into account the cycle, type or nature of the latter. the health and well-being of its employees. The latter is taken into account in the process of choosing new Agency premises by observing the opinion of employees and allowing them to tour the premises The Agency will provide access to a variety of data to the general public, especially students and at various locations. The relocation of the Agency is planned for the end of 2021. prospective students. The achievement of this goal is in accordance with the value of the Agency that commits it to development. The achievement of this goal is in accordance with the value of the Agency that commits it to transparency. 6. In-depth international cooperation 5. Pursuing sustainable development goals As can be seen from the above explanations, the Agency has established extensive international Taking into account the Strategic Sustainable Development Goals (SDG 2030) adopted at the United cooperation in the expiring strategic period. In the future, in line with the adopted plan, it will further Nations General Assembly in 2015 and the changes caused by the epidemic situation related to the strengthen its international activities to get a good overview of examples of good practice, policies, COVID-19 pandemic, the Agency will strive to achieve the sustainable development goals in the next analyses and comparisons on a European and global scale, and adapt their application in the Slovenian strategic period. In line with development trends in the European and international environment, it will higher education area to its specifics. take these objectives into account in its operation and management, as well as in the broader context of supporting and assisting the education area in reflecting and implementing good and transferable In addition to maintaining or renewing ENQA membership and registration on EQAR, which, if the practices aimed at improving higher education. This will demonstrate support for society on the path Agency works well, can be done every five years, the Agency will: of change towards a more sustainable lifestyle. This field is opening a new spectrum of knowledge and potential for research and development also in the field of higher education. It includes inclusive and a) cooperate even more actively with other agencies and associations of quality assurance integrative approaches to learning, teaching and cooperation, and paves the way for new or innovative agencies; during the presidency of CEENQA, it will introduce innovations according to which principles and priorities in education. all members of this association will prepare contributions in delineated subject fields and The Agency’s activities will therefore be oriented towards: present them publicly; • improvement of its internal quality assurance system and activities related to sustainable b) participate in international project groups, international networks and activities, in education development goals; and training courses, in the exchange of experts and employees; • active cooperation in the convergence of the elements of the external quality assurance system c) make international comparisons in the context of analyses; with the sustainable development goals in the Slovenian higher education area. d) after the adoption of the quality assurance act, also perform accreditations and evaluations abroad. Taking into account the sustainable development goals and the epidemiological situation due to COVID-19, the Agency will continue to: A particular challenge in this strategic period will be the Agency's work on the establishment and a) rationally plan travels abroad; operation of federations of European universities (European universities), the so-called universities b) implement distance: of the future. Their present participants include two Slovenia’s biggest universities. This integration • accreditation and evaluation procedures or visits of higher education institutions and will require the adjustment of quality assessment in all participating countries, agreements and higher vocational colleges; creation of criteria or assessment frameworks tailored to such universities, mutual recognition of • education and training courses of experts and organise consultations with them and accreditation and evaluation decisions and, of course, full and mutual trust and great professionalism other participants in distance accreditations and evaluations; of all participants related to the above. c) organise several meetings with representatives of domestic institutions and foreign agencies, associations, etc. through video conferencing tools; The achievement of this goal is in accordance with the value of the Agency that commits it to professionalism. d) enable the acceptance of applications in electronic form; e) take care of health and well-being of its employees; 7. Proactive communication f) find new premises for the Agency: it will strive to make them less costly than at present and better tailored to the needs of the Agency's activities and its employees. The Agency has made a great step forward in the communication with stakeholders. It intends to develop and deepen it further in the coming years. It is a commitment to transparent and open two-way Already in 2020, the Agency had to adapt its activities to the changed situation caused by the outbreak communication, inclusion of external stakeholders in self-evaluation procedures and other activities of the of the COVID-19 pandemic. According to experts, the pandemic will not completely slow down, so it Agency, and prompt notification of interested parties (website, monthly e-newsletter, social networks, will continue to shape the work of the Agency in the future. With the listed adjustments, it will try to etc.). do it as well as possible while maintaining its quality. Distance accreditations, evaluations, trainings, consultations and meetings must therefore be carefully planned and managed. Some advantages may The Agency wants to provide its stakeholders with the most up-to-date, accurate and diverse information also be identified in this method of work: these certainly include a reduction in the cost of transport, on Slovenian higher and higher vocational education. The aim is to increase the visibility of the website, accommodation or overnight stays, lecture halls, etc.; as well as a greater chance of more events or which will be an information centre for raising awareness of the importance of assessing the quality of their repetition where necessary – the latter being particularly important in the Agency's commitment higher education. 40 41 Therefore the Agency will prepare web content following the example of presentation films, which provide in an accessible and popular way information on the operation of the Agency, higher education institutions and the structure of the Slovenian higher education area while guiding visitors towards important contents (e.g. reports of groups of experts) more inherently related to the basic activity of the Agency, i.e. the assessment of the quality of higher education institutions, higher vocational colleges and study programmes. In the coming years, the Agency's website will become the most important information point for students and future students, gathering all information relevant to them and links to other websites in one place. In order to popularise the website, the Agency will further strengthen its cooperation with higher education institutions, higher vocational colleges, secondary school and university student associations, while increasing its presence on social networks. The achievement of this goal is in accordance with the value of the Agency that commits it to transparency. IV. CONCLUSION In the expiring strategic period, the Agency has established a foundation or orientation for further development in 2021 to 2025. Although it concerns different areas of its operation, the biggest challenge among the strategic goals it has set is to improve the assessment of quality standards in accreditations and evaluations. This will help to make meaningful the role played by the Agency under the ZViS, i.e. to provide higher education institutions and higher vocational colleges a professionally substantiated, realistic, true insight into their activity, help them in its development, and strengthen the Agency's reputation as an indispensable partner in improving the quality of higher and higher vocational education. It will also improve and facilitate the decisions of the Agency Council and Appeals Committee. An independent act on the Agency – the act on quality assurance in higher education – will play an important role in achieving strategic goals. The Agency will strive to enable even more in-depth professional assessment of quality standards and decision-making of the Agency Council by introducing special commissions by fields of study, reorganising appeal procedures and enabling the work of its staff abroad. It will work to harmonise the quality assurance act with all relevant higher education stakeholders, and, moreover, make them identify it as a necessary step to improve as well as facilitate work in the field of quality assurance in higher and higher vocational education. The text »Strategic development of the Slovenian Quality Assurance Agency for Higher Education from 2021 to 2025« (Strategy) was written by Tatjana Debevec, Analytics Department. The Strategy was discussed in detail by the Agency's Council at its 156th session on 19 November 2020 and adopted at its 157th session on 17 December 2020. 42 43 GUIDELINES FOR DISTANCE EVALUATION, CEENQA BACKGROUND FOR WRITING GUIDELINES In this article we present the proposal of the first CEENQA document: Guidelines for distance evaluation. The COVID-19 pandemic has caused an enormous wave of change in the higher education area worldwide. Despite various new technologies and technological advancements, teaching and learning was mostly still a face-to-face process. However, in 2020, this changed for all stakeholders in the higher education area, including quality assurance agencies in higher education. We faced an enormous challenge in adapting our QA procedures, especially our evaluation site visits. At international events, the CEENQA agencies regularly exchanged their issues and solutions, which helped us compile numerous recommendations of experiences and practices that work. However, although agencies belong to the international higher education area and mostly adhere to the same standards (namely, European Standards and Guidelines), they conduct their work in many different ways due to national specifics, legislation, etc. Therefore, the CEENQA guidelines focus solely on site visits as a common part of the QA process and can thus be used by any agency for quality assurance in higher education. The document does not intend to restrict agencies in any capacity but to suggest possible solutions and offer advice if needed. These guidelines are based on the existing NAKVIS guidelines, with contributions and comments from other CEENQA agencies (YÖDAK, ACQUIN, NAB, EKKA, IAAR, AIKA, HEPDAK, NAQA, AQ). The process of creating these guidelines was coordinated by the CEENQA Board Member Olgun Çiçek. The guidelines consist of two parts: guidelines to be followed (1) before and (2) during a site visit. In the appendices to the guidelines, there are examples of good practices and challenges, which were collected from a survey that was conducted among CEENQA agencies in May 2021 (1), and (2) example of a site visit schedule. The first two parts advise on preparation, IT solutions, and clear definitions of the roles and distribution of work between stakeholders (e.g., higher education institution (HEI) coordinator, group of experts, agency’s employees, and interviewees). All these individuals must be well prepared and have sufficient digital and informational skills. The agency’s employee and HEI coordinator both play crucial roles, as they must explain and deliver the necessary information as well as ensure that the participants are familiar with the virtual conference tool in use. It is also important to test potential additional features (e.g., translation) and have an alternative tool ready and tested should any technical difficulties occur. The chair of the group must assign the roles and content very clearly already during preliminary meetings of the group. We advise that the interviewees check into online waiting rooms approx. 5 minutes early to avoid delays or possible technical problems and to enable the agency’s employee to check the attendance. Our guidelines strongly emphasise the importance of appropriate communication between stakeholders and adequate etiquette in the online environment. We recommend using additional channels/methods, such as mobile apps, for quick and efficient coordination between HEI coordinators and agency employees and within the group of experts. Writing the joint report via video conferencing may present some obstacles at first but has proved beneficial to the whole process later on. Although the guidelines focus on site visits, some advice and solutions provided can be incorporated into other procedures and day-to-day operations of any agency. The first appendix includes some good practices or challenging situations from the aforementioned survey, which could not be included in the guidelines. 46 47 The second appendix presents an example of a schedule that can be modified to serve the specifics Stakeholders should identify elements that ensure credible evaluation but do not diminish mutual trust. of a certain procedure. We emphasise the importance of longer breaks between sessions to allow the Achieving better credibility at the expense of a loss of trust is definitely not the right way forward. Conversely, experts some additional rest as well as to guarantee a sufficient amount of time to exchange the gathered paying attention to ethical principles and encouraging open dialogue and cooperation are essential. As information, make assessments, ask any follow-up questions, and provide conclusions. with face-to-face processes, maintaining the integrity of all stakeholders remains crucial. At the end of the introduction of these CEENQA guidelines, we will try to look into the future. Which In conclusion, we at SQAA and CEENQA hope that this document will benefit everyone who organises and novelties will we keep and how will we return to pre-COVID-19 times? What will the inevitable hybrid conducts site visits. As we are planning to implement a new hybrid system, we hope that our community system look like and where will we adapt our QA systems? There are many possible improvements to in the higher education area will continue to exchange solutions, good and bad practices, and any thoughts the previous face-to-face system that could be part of the future hybrid model, and we will discuss some on the dilemmas listed above. of them: Klemen Šubic and Nataša Kramar, NAKVIS, including contributions from the agencies YÖDAK, ACQUIN, NAB, 1. Availability and accessibility: Online site visits offer a much broader choice of experts from all around EKKA, IAAR, AIKA, HEPDAK, NAQA, and AQ. the world. As no time-consuming travel is involved, experts are more willing to participate in a procedure that is geographically further away. Additionally, it is also easier for the institutions to recruit interviewees, who may have busy schedules or are located elsewhere, even abroad, at the time of evaluation. This is especially relevant when an institution or study programme has many international students, alumni, or teachers. While this can cause some issues due to potentially different time zones, it is still a major advantage in many cases. 2. Flexibility: Since there are no additional travel costs, an additional day off can be included in the schedule for longer visits. This gives the experts extra time to work on a report or conclusions as well as much-needed rest after long hours behind the computer screen. Institutions thus have more time to prepare additional materials, if needed, and to focus on their daily tasks. If the group decides that there are some material conditions that cannot be fully presented online, the local part of the panel can use the day off to review the material conditions and practical training in person. 3. Cost reduction and easier logistics: Different national legislations have different price policies for (national) agencies and higher education institutions. However, online site visits substantially decrease costs, which benefits both the institutions and agencies. Logistic organisation proves much easier in a virtual environment. It certainly has its challenges; in general, however, everyone included saves considerable time. 4. Sustainable future: As we inevitably face global climate change and its consequences, shifting our work online, which results in much less travel, makes a small but nevertheless important contribution to achieving environmentally sustainable work habits and reducing our carbon footprint. Concern for the environment is a global responsibility, and designing a new model for conducting our agency’s procedures provides us with an opportunity to play our part. However, this different format of site visits, whether fully or partially online, also creates new issues. 1. One of the most important challenges is to ensure the necessary credibility. Various ethical dilemmas, e.g., the attendance of participants’ superiors behind the camera may cause self-censorship during interviews. 2. Although non-consensual recording was possible with voice recorders before, it is now even easier in virtual environments, creating additional pressure on participants. 3. Spontaneous on-site interviews are not possible, which makes it much more difficult to reveal any potential hidden challenges. Such instances are rare during site visits; however, they present potential threats and must be taken seriously. The best way to avoid such issues is to build strong mutual trust between institutions and agencies, as this enables a respectful exchange of information and fair assessment. 48 49 Klemen Šubic and Nataša Kramar GUIDELINES FOR DISTANCE EVALUATION, Guidelines for conducting a distance evaluation GUIDELINES BY THE CEENQA NETWORK We are sure this will be a very practical and valuable tool for the member agencies in their virtual evaluations for the post pandemic era. The purpose of this guideline is to help and direct member agencies to find the practical information and tools for online evaluation. This will be a reference point for agencies during the time of force majeure. CEENQA will be ready and willing to provide training/workshop on the technical issues and practices of The presented guidelines are a document of the CEENQA association, the purpose of which is to offer all this guideline for member agencies whenever needed. members an additional or sometimes even alternative aspect on what is already in use in a different national context, taking into account the specifics of the legislative and procedural frameworks already in place. The In individual cases the evaluation of a higher education institution can be conducted exclusively remotely guidelines could serve as an alternative view and an opportunity for CEENQA members to improve the with the use of online tools (for example, Zoom, Skype, Google Meet, Jitsi or another platform). organisation and conduct of online site visits also by exchanging some adequate and transferrable good practices (where relevant). In order to conduct a remote evaluation, the agency provides a virtual conferencing platform. These guidelines are not intended to unify procedural steps between all CEENQA member agencies, as they 1. Prior to the evaluation do not constitute an obligation for them, but rather to provide additional input in how the site visit could be organised, what kind of circumstances should be taken into account and in what timeframe. They are a The planning of remote evaluation requires mainly the following: step towards sharing different practices and solutions to the challenges faced by agencies in organising and conducting online site visits, during and after the pandemic restrictions. 1. In order to participate in evaluation discussions, the group of experts, the institution and all participants are required to have functioning equipment (computer with microphone and camera or The basis for this document was taken from the national NAKVIS guidelines and supplemented with other smart device – e.g., smartphone, tablet) and a good internet connection. Use of headphones suggestions and specifics that CEENQA members highlighted in the answers to the questionnaire on and a headset microphone is recommended. organisational specifics of online site visits. The substantive aspect of the assessments is deliberately omitted from the guidelines, as differences between countries are substantial and therefore not relevant for such a 2. It is recommended that few days prior to the evaluation an agency employee and the coordinator at document. the higher education institution (HEI) test the online platform (e.g. Zoom, MS Teams, GoToMeeting, Google Meet, Jitsi, Skype, etc.). Several potential participants could participate in a test meeting. The aim is to provide guidance where needed, and to ask the right questions about what the important The purpose of such meeting could be for the agency employees to present the course and dynamics organisational challenges of online assessments are, as well as to indicate possible ways to address them. of the remote evaluation to the representatives of the institutions. The purpose of the test meeting is mainly to check the operation of online platforms in use and discuss the issues related to the Klemen Šubic, NAKVIS technical part of the evaluation and to prepare the participants for it. Not only teaching and learning have been deeply affected by the COVID-19 pandemic; similarly, all traditional 3. In case of technical issues with the online platform in use an alternative online platform (Zoom, forms of external quality assurance also needed to undergo a serious modification. Since March 2020, Google Meets, Skype, Jitsi, etc.) should be verified and provided. All the participants should be most agencies have adopted virtual approaches to evaluations, reviews, and site visits as a consequence of informed about this prior to the actual evaluation. COVID-19. 4. Schedule preparation: Considering the dynamics and the way a remote evaluation is conducted, Therefore, CEENQA Board has decided to prepare and publish a set of guidelines for its member agencies on careful and prudent planning is needed. One of the possibilities is that the discussion with individual online evaluation for the quality assurance of their virtual assessment. This guideline also intends to promote group of interlocutors is expected to last from 45 to 90 minutes, followed by a 15¬ 25 minutes break the consistency of the evaluation among the member agencies. in order to summarise the findings and prepare for the next discussion. Depending on the procedure (as well as the availability of participants, their number and position etc.), the time slots dedicated This paper is an outcome of extensive work performed by the participation and contribution of the member for each discussion session may vary. It is recommended that the agency and the applicant HEI agencies through a detailed survey, as well as workshop organised by CEENQA. agree on the maximum number of participants attending each discussion session/interview in order to guarantee an adequate time slot for constructive debate and discussions to each group I especially thank the NAKVIS team for the initiation of the guideline and the Director, Professor Franci of interviewees. A discussion should be held with all relevant stakeholders in order to evaluate the Demšar, Ph. D., for his commitment and leadership. I also thank all agencies who responded to the survey compliance with the quality standards and provisions of assessment. The chair of the group agrees with valuable input to finalise the guideline. The presentations of the agencies about the implementation, with an agency employee on the agenda. It is recommended that the initial and final discussions are challenges and best practices of the online evaluations became the backbone of this guideline. Special thanks held with the management of the HEI. also to the CEENQA Board for their initiative and commitment at every stage of this valuable guideline. I sincerely hope that the guidelines will be useful and valuable for our members on their online evaluations. 5. Site visit agenda may include active online links to the meetings and sessions scheduled, but also a list of participants with their titles and roles in the evaluation procedure. It is recommended that all Prof. Dr. Olgun Çiçek, YÖDAK participants and experts are informed about the site visit agenda in time (including active meeting links and list of participants), for example at least one week before the site visit takes place. 50 51 6. In order to agree on the content and solve any technical issues, it is recommended that an agency employee 8. On the final day of the evaluation, the discussions are held only in the morning. The afternoon is organises a preparatory meeting with the group of experts at least a week prior to the evaluation. intended for finalising the joint report. It is recommended that the Google Docs application is used for drafting and updating the report. 7. t is recommended that before the evaluation the group of experts prepares a comprehensive draft report based on documentation and discusses the remaining issues. The group can prepare a list of Appendix 1 additional documentation that the HEI provides on the first day of the evaluation. Together with an agency employee the group revises the draft report, questions and potential additional documentation. Good practices and challenges identified by CEENQA members For the technical part, cameras and microphones of all participants need to be tested, as well as the functioning of an alternative online platform. The survey that was done in May 2021 helped the authors a lot with preparing the guidelines. However, the agencies also highlighted some good practices or challenging situations, which could not be included in 2. During the evaluation the document. These are shared here and we hope that we will continue with exchanging of good practices and with sharing experiences. 1. Depending on the site visit form (fully online or hybrid), the experts join the meeting room (online or on site). The agency staff shall be responsible for the procedure and also for providing technical Evaluation of material conditions support when possible and needed. It is recommended that foreign experts participate in the evaluation procedure online/via video conference. This issue has been mentioned the most and remains a challenge. Agencies opted for live virtual tours with a camera or photos and videos prepared before the site visit. However, experiences with such solutions 2. In order to ensure smooth notification and prompt response (in case of technical difficulties or vary and many predict that they will opt for a physical visit of the premises when the situation allows. non-attendance for exceptional circumstances), an agency employee and the coordinator at the institution maintain contact by phone if needed, or use adequate communication channels such as Confidentiality of online meetings WhatsApp, Telegram etc. Site visits in a virtual environment raise certain issues and one of them that was mentioned by agencies is 3. A moderator is selected; this is the chair of the group of experts or a person temporary appointed by ensuring the confidentiality of the meetings with stakeholders. The meetings could easily be recorded with the chair. The moderator allocates and monitors time, assigns the floor to individual participants and various computer tools without anyone knowing. Besides the interviewee, other people can be present in ensures compliance with the principles of professional and respectful communication. a room who were not invited and can influence the answers. There are more solutions to these problems, from signing a confidentiality agreement to focusing on establishing mutual trust. 4. Every participant is required to join the meeting 5 minutes prior to its beginning. The use of the “waiting room” function in the video-conferencing platform is recommended to enable the participants to Interim day join the meeting in order and according to schedule. The group of experts and an agency employee are present in the online conference room throughout the conversations. Participants involved in Management of the breaks between interviews and so-called “Zoom fatigue” present a problem, especially discussions join them individually. For the purpose of discussions, participants should use full and when visits are longer. A potential solution to be considered is an interim day, where no interviews take proper names as their user names. Several people under one username/computer in a meeting room, place. The experts can focus on additional documentation, available databases, fact-checking, updating two people on the same computer /username on the desk should be avoided. the draft report, etc. In addition, the group of experts can also prepare concluding remarks and formulate thematic questions for the following day(s). While this extends evaluation for one day, it can help with 5. All participants in the conversation follow the rules of respectful conduct and online communication restoring the focus and mental preparedness of the group of experts. (raising hand before speaking, turning off the microphone when not speaking, being tolerant and respectful in communication). All participants as well as the members of the group of experts are Preparing the questions required to have their cameras on throughout the discussion. Preliminary meetings and compiling the questions have generally become easier with the help of online tools. 6. Assessment of material conditions: according to instructions of an agency employee and the group Therefore, the group of experts can gather the written questions before the site visit, which consequently of experts, the higher education institution records a video of premises used for the implementation enables a better exchange and extensive discussion between the group of experts and the HEI. of the study programme, premises of the whole institution/college (laboratory, library, the premises for students and equipment for the implementation of the study programme) when assessing Hybrid model material conditions. The video should not be longer than 15 minutes. During the remote evaluation in the presence of the representative of the institution/college, the group of experts reviews the As we look into the future and try to combine good practices from face-to-face and virtual site visits into a video and resolves the open issues. The whole or part of the group of experts can physically visit the new model, one option is to include HEIs in this decision. Especially when it is not the first re-accreditation premises if necessary. The visit of the premises and equipment is conducted for study programmes cycle, HEIs could be presented with an option of the site visit form: online site visit, physical site visit, or and institutions/colleges when appropriate and necessary. blended form. There are possible legislative restrictions; however, it is worth considering how to include opinions of HEIs in the potentially different future models. 7. At the end of each evaluation day, the group of experts shall meet in closed session to discuss the findings, conclusions, remaining outstanding issues and additional evidence necessary for a Training of experts comprehensive evaluation, if needed. This follow-up session is also intended for formulating the highlighted points (findings, conclusions) that are to be included in the evaluation report. In various trainings of experts, the online format can now be easily used. 52 53 Instead of traditional one-day training sessions, several shorter sessions with further independent and The second evaluation day* group-work are possible. The experience show that such format is more efficient, but requires clear goal, focus and structure and clear communication concerning expectations on the preparatory work. Meeting Participants Hour Appendix 2 Preliminary meeting of the group of experts The group of experts Sample schedule in the procedure of evaluation of a study programme (two-day site visit): day one Participants of the fifth discussion join the meeting 8:25 *The information provided in the sample schedule is solely for the purpose of illustrating how to organise a two-day online site visit. The schedule can differ significantly in the case of several daily visits and in Discussion with the specialist services Discussion with the representatives of support 8:30–10:00 services different procedures (initial accreditation, institutional procedures, etc.). Break 10:00–10:25 Meeting Participants Hour Participants of the sixth discussion join the 10:25 meeting Preliminary meeting of the group of experts/ The group of experts 7:40–8:00 Discussion with the institutional management, Discussion with the institutional management 10:30–12:00 contact with institutional management and review heads of programmes +if needed also heads of departments, chair of of potential difficulties the evaluation team... A short virtual visit of the premises and equipment Representatives of the institution 8:00–8:25 Lunch break 12:00–13:25 at the institution/in relation to the study (management, coordinator) programme evaluation Drafting a joint report 13:30–15:30 Participants of the first discussion join the meeting 8:25 The final presentation to representatives of the 15:30–16:30 higher education institution Discussion with higher education teachers Discussion with higher education teachers 8:30–10:00 (employed and contractual - all titles +faculty assistants) * Upon the decision of the group of experts, a site visit of particular premises and equipment might be needed and scheduled Break 10:00–10:25 for the second evaluation day. It is conducted to a minor extent (an agency employee and the chair of the group) whereby instructions and recommendations to prevent spread of infection are considered. After the evaluation the group of experts Participants of the second discussion join the 10:25 continues with video call discussions (in a safe location of the institution or in the agency premises). meeting Discussion with students and alumni Discussion with students of all grades + 10:30–12:00 Recommendation to experts after the evaluation alumni It is recommended that after an evaluation the group of experts meets once again (on the same or the Lunch break 12:00–13:25 following day while impressions are still fresh) and completes the substantive part of the evaluation Participants of the third discussion join the 13:25 report. The group of experts has enough time for editing the style of the report until the submission meeting deadline. Discussion with the evaluation team Discussion with the evaluation team + head of 13:30–15:00 the study programme Break 15:00–15:25 Participants of the fourth discussion join the 15:25 meeting Discussion with representatives of the external Discussion with representatives of external 15:30–17:00 environment environment, mentors of practical training Final meeting of the group and preparation for the 17:00–17:30 second evaluation day * When seeking and choosing the participants (students) who are not student representatives, the institution/college should cooperate with the student council. 54 55 PROCEEDINGS OF THE CONFERENCE Prof. dr. Peter Purg NEW ROLE AND PARADOXES OF QUALITY IN THE NATIONAL PROGRAMME FOR HIGHER EDUCATION 2021–2030 I present this paper in the role of a member of the group of experts that, in a complex process lasting more than a year, drafted the new Slovenian National Programme for Higher Education (NPVŠ), which will be valid until 2030. At the time of writing, the document is in the final steps of approval at the highest levels (the Council for Higher Education; it will be later adopted by the National Assembly). At the time of the development of the document, I was the President of the SQAA Council, thus covering the field of quality assurance in addition to the fields of arts and humanities. As one of the authors who, throughout the process, tried not only to draw attention to the importance of quality in strategic planning of higher education development, but also consistently asserted ways of assessing programme priorities and measures that are critical on the one hand and inclusive on the other, I will try to highlight especially those aspects of the NPVŠ that concern the field of quality. This is also defined as one of the five chapters of the document (the others being Social Development and Higher Education System, Legislation and Financing, Internationalisation, and Digitalisation), which indicates an important – in comparison to the previous document, significantly stronger – role of quality as a perspective for the development of higher education on the one hand, and perhaps even strategic support on the other. Although it may be somewhat paradoxical, but on the other hand also expected, the quality of higher education in the upcoming NPVŠ is related to some strategic development areas imbued with both ideological and ethical issues, as well as very concrete, significant and sensitive measures. Under the motto “Green and creative development of higher education based on excellence and openness for the benefit of the wider society” , the National Programme, in addition to all expected academic excellence, emphasises already in the preamble “accountable academic, institutional and economic autonomy” , which is related to “producing competitive and quality human resources for the needs of the labour market primarily in Slovenia, as well as in the wider EU” (all quotes in this paper are from the currently discussed draft NPVŠ, which was submitted for the second consideration at the Council for Higher Education on 17 August 2021)**. In this decade, Slovenian higher education is to be distinguished by “innovation, pedagogical, research and artistic excellence” and, as a kind of novelty, also “growing quality on an international scale.” For the latter, the Programme also highlights other areas of quality such as “stimulating working and study conditions, high standards of academic culture and ethics, responsiveness to current and future needs of the economic and public sectors, respect for institutional autonomy” and general ethical principles such as “social responsibility, integrity and trust, equality and non-discrimination, dignity, justice and inclusiveness” . Another distinctive novelty is the focus on the “ecosystem diversity through in-depth pluralisation of disciplines” , which will be harmonised with smart specialisation strategies, the focus on the green transition to a circular economy and open innovation, with Slovenian higher education also “co-shaping the trends of digital transformation in teaching, learning, research and innovation.” Specifically the field of quality is expected to “achieve an internationally comparable high quality and excellence above the European average in all standard indicators.” In addition to its wide accessibility, it is expected to show a growing international integration supported by “instruments of internationalisation”, which also include the implementation of study programmes in a foreign language. Among the six strategic goals in the field of quality, the following, which indicate the concrete priorities in the field of quality in this strategic document, should be highlighted in the present context: Sustained quality assurance in Slovenian higher education should be based on the system of “promoting continuous improvement and (self) evaluation in all areas of quality assurance in higher education.” 58 59 This goal envisages the concrete introduction of “appropriate models and principles (articulation of The measure of Creating modern and flexible educational programmes in accordance with international goals, measures, follow-up and, above all, continuous improvement through a combination of internal development trends and the needs of the environment envisages “in-depth, structured links with non-and external assessment or critical reflection both at the programme level and at the level of higher formal education and other aspects of lifelong learning.” Another measure interesting from the point education institutions).” This goal also unequivocally states that “quality in higher education is checked of view of SQAA's activities and competences is that of Preparing a strategy for integrating lifelong by the national, independent agency for quality assurance in higher education (SQAA).” Student-learning in higher education area, focusing on the development of a system of accreditations and centred teaching based on professional, research or artistic work, which envisages the establishment recognition of micro-credentials and a system of recognition and evaluation of previously acquired of a student-centred “appropriately updated method of teaching based on professional, research or knowledge.” artistic work.” Highlighted is the effect of “flexible adaptation of curricula to ensure the appropriate competences and knowledge of graduates”, which is to be supported by an appropriate “combination of A comprehensive package of measures is envisaged in the field of human resources, from providing virtual and in-person presentation of learning content”. suitably qualified human resources for the needs of the economy and wider society based on “faster and more efficient diversification of study programmes (and institutions) by individual professional The design of study programmes for professions and challenges of the future, with a particular fields with the aim of providing the most appropriate, interdisciplinarily trained human resources for emphasis on “green transition to a circular economy”, brings a unique paradox that will later, however, the economic and non-economic sectors” to a more flexible definition of workload. The latter measure prove to make great sense: in addition to “competitiveness in the domestic and international employment envisages “systemic solutions for the permanent education of higher education teachers and faculty market”, students will be encouraged towards “active citizenship, which means that students and assistants that will also include a more flexible and appropriately balanced definition of workload. At the graduates will be involved in social developments and decision-making, and strive to strengthen same time, appropriate reconciliation of work and family life, actions against burnout, introduction democracy and civic and professional solidarity.” At the same time, the programmes are expected to of green solutions (e.g. work from home), etc. will be ensured,” which shows the paradoxical tension, “support self-initiative in creating employment opportunities (e.g. start-up entrepreneurship, open similar to those described above, between the demand for effective if not downright servile utilitarianism innovation, cross-sectoral initiatives) with special focus on the non-profit and non-governmental of higher education on the one hand and care for the well-being of all stakeholders, especially employees sectors.” In a similar spirit, the promotion of entrepreneurship “provided (by appropriate legislation) and students, on the other. through the entities of the entrepreneurial environment of start-ups (spin-off, spin-out and start-up)” should be based (not on bare materialistic pragmatism but) on “scientific and artistic achievements The update of the minimum standards for appointment to the title through the implementation of and results of development, creative and research projects”. The establishment of a system of lifelong pedagogical and professional performance should be based on both qualitative and quantitative indicators learning at the higher education level aims at “empowering wider circles of the active population with of pedagogical performance. The latter are supposed to “have a comparable weight in the procedures new findings of science, art and profession”, and the “necessity of acquiring diverse but quality higher of appointment to the title and promotions in terms of indicators of scientific or artistic excellence; education knowledge and skills and creating a positive, education-based orientation for the future” is to consideration will also be given to success in the implementation of projects (including programmes be clearly and widely communicated in the context of quality. and networks) and other aspects of professional success, which will also include reputation in the professional and general public and public representation of professional or academic views, impact Measures to achieve strategic goals in the field of quality show even more concretely the continuation or recognised achievements in the field in which one’s title is awarded, and field of academic as well of the paradoxical tension between the social and economic roles of higher education, and its autonomy as social activity.” Greater involvement of experts from the economic and public sectors in the study and commitment to academic ethics; highlighted here are just a few of the eleven in total: process and external evaluation is supposed to take place through their time-limited appointment to such a title and their “inclusion in the study process and external evaluation. Successful and recognised The promotion of cooperation with the wider environment and the creation of joint study programmes experts who demonstrate high ethical principles will be encouraged in particular.” in the European area should be based on “mechanisms to promote international (transnational) cooperation, development of third-cycle study programmes and cooperation with the economy and Given the limited scope, the present summary of most goals and measures of the National Programme wider society (integration of higher education into the environment)”, which directly relates to one of for Higher Education in the field of quality confirms that quality in Slovenia is undoubtedly established the key areas of quality assessment of programmes as well as institutions. In particular, the “topics of as a fundamental concept and important tool, as one of the key areas of higher education development. solidarity, quality of work, social security, social cohesion, inclusion of vulnerable and marginalised On the one hand, this field, full of paradoxes and not yet fully populated, contains many value-based, groups, non-discrimination, etc.” are to be asserted in this process. perhaps even ideological definitions or political priorities – those that point to the unstoppable (neo) liberalisation, commodification or even economic instrumentalisation of higher education. On the other The expansion of SQAA's activities refers to the Agency's advisory role, which is supposed to “help hand, it is through quality that bold ethical orientations and priorities enter the strategic planning at the higher education institutions to continuously improve their quality on the basis of a combination of national level; these orientations and priorities bring back to the academic space an important humanistic external evaluations and self-evaluation”, which envisages “preparation of guides, analyses and breadth and values of creativity, diversity, equality and interdisciplinary and intercultural cooperation. comparative data on the Slovenian higher education system”. As SQAA has successfully established itself in recent years with self-initiated activities in this field, it should from now also “oversee the **NOTE: As an appointed member of the group of experts, the author of the text is entrusted with the appropriate evaluation and analysis of relevant international scales (with complex and field-balanced communication and verification of the content of the draft NPVŠ (in its final coordination phase) with various indicators) and the ranking of higher education institutions, as well as an objective public presentation stakeholders. of higher education institutions and programmes from the point of view of quality.” Professor Peter Purg, Ph.D., University of Nova Gorica, President of the NAKVIS Council Critical self-reflection of the higher education system is to be carried out at the “state level with the involvement of all key stakeholders in order to change higher education in accordance with the needs of society and with optimal exploitation of new technologies and scientific findings and intense cooperation with the cultural and creative sectors.” 60 61 Prof. dr. Peter Verovšek WORK OF THE NAKVIS COUNCIL DURING THE Since it enables all participants to access and share both audio and video (and slides), as well as use the chat function, this system was much more user friendly than the previous ad hoc approach to PANDEMIC remote working, as it meant that I was in the same position as all other members of the Council. Virtual meetings have other advantages as well. In addition to the fact that members no longer had to commute to attend sessions in person, such a format also facilitated the process of recusal for members whose home institutions were under evaluation. When meeting in person, this process required the members in question to leave the room while the discussion proceeded before being invited back once it was completed and the relevant action had been taken. While fairly straightforward, such recusals could be The Covid-19 pandemic has posed great and novel challenges across both the private and the public quite time-consuming, with members frequently filing in and out while the rest of the Council was forced sectors. In particular, it has forced all of us – whether or not we were prepared to do so – to move many to wait for them. By contrast, on Zoom the Agency employee acting as the virtual meeting host can activities that would normally take place in person into the digital realm. While this was necessary to recuse a member by temporarily removing them from the meeting at the click of a button, thus speeding ensure public health and enabled many individuals, particularly knowledge and other office workers, up and smoothing out the workflow of our meetings by a great deal. to take advantage of working from home, organisations across all sectors of the economy and government had to adapt rapidly to the new reality of a remote workplace, including all technical and While there are some minor advantages to virtual meetings, the challenges are obviously much greater. other difficulties it poses, with little or no advance preparation. I will not go into these in great detail, as all of us are aware of how difficult it is to deal with “Zoom fatigue” or to have to speak to a panel of empty squares when everyone’s camera is turned off. In order The pandemic has also had a great effect on the accreditation and evaluation procedures carried to ensure that we could continue our work in the best way possible given the circumstances, the Council out by SQAA as an organisation that was used to carrying out most of its work in person. Since has adopted a number of different norms in line with the best practices that have emerged over the our teams of expert evaluators normally carry out their visits in person, where they can determine past 18 months. For example, while we used to take a quick break during longer in-person meetings, the whether or not the higher education institution in question meets all necessary statutory provisions, President of the Agency Council, Peter Purg, now makes sure to schedule a comfort break every hour quality standards and other regulations adopted by the Agency by examining its infrastructure and and a half, so that we can stretch our legs and look away from our computer screens for 15 minutes. speaking with key stakeholders, moving these processes online required significant modification to our methods of working. While this resulted in some delays early in the pandemic, when it was not More substantively, we have also sought to ensure engagement by asking all voting members of the yet clear how long this extraordinary situation would last, the Agency was able to pivot quickly by Council to keep their cameras turned on at all times whenever possible. This has helped, as it has adapting to the circumstances. guaranteed that whoever was presenting the case before us was able to see whom they were addressing and to gauge their reactions. Most notably, it allows the chairperson to see if everyone is nodding I am a member of the Agency Council, whose task is to make decision-based written reports provided or if a member is frozen or simply having other difficulties with their Internet connection during our to us by our teams of experts. I will therefore leave it to my colleagues who are more directly involved deliberations. in the process of conducting these external evaluations remotely to address the concrete challenges this posed for them and their work. I will focus instead on how our work as the Agency’s highest Of course, the fact that many of us have been forced to work from home also raises important issues decision-making body was affected by the Coronavirus pandemic. Much like the Agency’s other regarding data privacy and security, as our sessions are held in confidence and are not open to members and bodies, the Council was also used to meeting and conducting its business in person outsiders. While we deal with these issues by operating behind closed doors when in person, special before March 2020. accommodations are necessary in a virtual environment, especially as many of us were forced to work from home for long periods during the pandemic when our offices were closed. In order to prevent any Since I teach at the University of Sheffield in the United Kingdom, I was nominated and appointed problems, members are asked to either work in a room alone or to use headphones to ensure that no one to the Council by the Government of the Republic of Slovenia in the autumn of 2019. By taking this else can hear or follow our proceedings. The President will usually remind us of this rule at the beginning position, I filled the seat reserved for a member who works in higher education outside Slovenia. of every meeting. In light of the fact that I do not work or live in the country, I attended many sessions of the Council virtually even before the onset of the pandemic. This does not, however, mean that I was unaffected Overall, the Agency and the Council have done a good job of adapting to the unfortunate and difficult by the changes brought by the SARS-CoV-2. circumstances brought on by the novel Coronavirus. However, many issues and questions still remain. In particular, it will be interesting to see how many of the changes we have adopted over the course of Before the whole Council was forced to move online, I was often its only member attending the sessions the last 18 months will remain part of our operations once we are able to return to in-person meetings remotely. As a result, special accommodations had to be made for me to dial into our sessions via and business as usual. Skype. While the employees of the Agency did their best to facilitate this process by setting up a camera and a microphone, it was difficult for me to follow the discussion as I often could not hear For example, as government restrictions within Slovenia started to ease in the spring, the Council started or see who was speaking. While I was still able to record my votes and voice my opinions when to hold its sessions in something approaching a hybrid format. This meant that the Council President and necessary, such engagement required an Agency employee to monitor my Skype connection at all the Agency Director, Franci Demšar, who is present in a non-voting capacity, attended in person from times. Overall, despite the best efforts of the Agency, this system was awkward at best. the SQAA headquarters, while the rest of the Council and the Agency’s other non-voting employees logged into our Zoom meeting remotely. Such a format works well when a limited number of members All of this changed when the whole Council was forced to move online, starting with its 149th session are attending in person, as they can all be captured by a single camera that makes them visible within on 19 March 2020. Suddenly, instead of making special accommodations for single member dialling one square for those working remotely. This will become a problem, however, once a majority begins to in via the Internet, all of us were working remotely. The Agency and its employees were able to adapt attend in person again, as I already highlighted when speaking about my own pre-pandemic experience to the circumstances quickly by adopting the digital conference platform Zoom for our activities. of logging into our sessions from abroad. 62 63 Prof. dr. Nejc Šarabon Dealing with such issues and seeing what pandemic-enforced changes we want to keep or carry over THE EXPERIENCE OF IMPLEMENTING into the post-COVID world to which we will hopefully return in the near future is a key question. While this is clearly an important issue for me (and will be for the successor to my seat reserved for someone INSTITUTIONAL RE-ACCREDITATION REMOTELY, working outside of the Republic of Slovenia), it has implications for the Agency and other Council THROUGH THE EYES OF THE PRESIDENT OF THE members as well. In particular, we will need to consider whether the advantages of meeting in person mean that we will go back to the status quo ante of the beforetimes, or whether the Council will proceed GROUP OF EXPERTS IN THE PROCEDURES OF THE with some sort of hybrid approach. TWO MAJOR SLOVENIAN UNIVERSITIES This decision has important cost and time implications, as the Agency covers travel expenses for members to attend meetings. While it is true that Slovenia is a small country with relatively short commute times (it is hard to drive for more than two hours without crossing a border), once life is back to normal, we will need to consider whether it makes sense to have all of our members attend in person again. While there are clear advantages to doing so, particularly in terms of the culture of the Council, which is easier to maintain if members occasionally meet in person and are able to develop a rapport, there are also In the time of the COVID-19 epidemic and the consequent restrictive measures for virus spread clear disadvantages. These refer to finance (for the Agency) and time and work-life balance (for Council prevention, I had the privilege and responsibility of leading a group of experts in the assessment members). However, they are also important in terms of congestion and fighting climate change, as procedure for the re-accreditation of the University of Ljubljana and University of Maribor. As every car trip saved means that less CO2 is emitted into the atmosphere. regards the former, our duties, namely the duties of the group of experts, have already been fulfilled, while the latter is still awaiting the second part of assessment, the one concerned with In light of my brief reflections, it is clear that while the Coronavirus has posed important challenges study programmes, in the autumn of this year. Institutional evaluation, especially of Slovenia’s two for both the Council and the Agency as a whole, it has also presented us with important opportunities major universities, presents a great challenge by itself. Implementing the evaluation procedure for growth by forcing us to adapt and think about our procedures rather than simply proceeding with remotely is a still greater one. In this contribution, I would like to discuss some activities, solutions “business as usual.” So far, SQAA has managed to deal with all of this with minimal disruption to its work. and practices that have been deployed in the given situation in an attempt to make the results of However, many questions still remain, especially given the uncertainties of a world which is moving our work useful and to maintain as high a level of quality as possible. back towards “normal” despite the fact that Covid-19 has become endemic and will be with us for the foreseeable future. Let me begin by noting that I enjoyed collaborating with my colleagues, the members of the group of experts – I would like to express great thanks to each and everyone for your cooperative There are a number of aspects of the accommodations made due to the existing epidemiological situation spirit and committed work! Although the mentioned re-accreditation procedures have been a over the past year and a half that we might consider keeping even after we are once again able to meet in considerable workload for me, I must admit that I took on this challenge in full consciousness of person. For example, given its ease of use and the time and energy we all save when meeting online – an its magnitude, complexity and the responsibility it brings. I would make the same decision again! issue that is increasingly important given the growing emphasis on climate change – we might consider This experience has enriched me with new knowledge and skills, and put my expert, organisational conducting most of our sessions remotely even when this is no longer necessary to comply with public and leadership competencies to the test. I believe it has improved the quality of my work as a health regulations. Since occasional face-to-face meetings are important as well, we could then consider pedagogue, researcher, innovator and dean. Therefore, I would like to thank all colleagues I have scheduling two or three meetings where we would all do our best to attend in person. These “special” had the opportunity to work with – special thanks to Professor Terence Clifford Amos, Ph.D., an meetings could focus on key issues and challenges facing the Agency and could also be combined with exceptionally experienced evaluator who occasionally worked, at my request, as my true unofficial team-building activities and/or a reception to encourage members of the Agency Council to mingle and mentor and selflessly shared with me his experience, suggestions and guidance. get to know each other in a more informal setting. From the appointment of the group of experts to the implementation of the e-visit, we held In light of our experiences over the past year and a half, I am confident that we will be able to deal with approximately five preparatory one-to-three-hour meetings in both cases (both universities, the whatever challenges await us in the future whatever measures we end up adopting once this situation visit related to the institutional part and the one related to the evaluation of study programmes). is behind us. Like it or not, the absence of in-person contact means less authenticity and warmth in conversations, Professor Peter Verovšek, Ph.D., The University of Sheffield, United Kingdom of Great Britain and Northern meetings or interviews. All tasks seem to be reduced to their technical aspects. At the same time, Ireland; Member of the NAKVIS Council thorough preparation and how well the work is structured seem to have played an even greater part than usual in the successful implementation of the evaluation procedure and e-interviews. Additional information and materials seem to be less readily attainable spontaneously during the interviews. In all the work stages – from the appointment of the group of experts to the submission of the final report to the Slovenian Quality Assurance Agency for Higher Education (SQAA) – we have organised our work in the Office365 environment and enabled interactive work on the files by using OneDrive structured folders. What seems to have been crucial is that we have all worked on the online version and not on the locally synchronised folders. Namely, working locally on personal computers and automatic synchronisations initially led to the replication of files with non-unitary trackable changes to documents. Since then, we have consistently worked exclusively online (Office365) and have not encountered a single problem. 64 65 When all the members of SQAA and the members of the group of experts were granted access to They had been informed beforehand about the function of the latter by their university. At the the OneDrive folder, some minor complication would appear almost as a rule (the causes varied: conclusion of our introductory speech, they were also asked to give short and concise answers and MAC computer, user profiles on office computers, the network, complexity of the previous use of to stick to the question at hand, as only very limited time was at our disposal. the MS online environment and conflicting user rights, an invitation to edit a particular file that has been miscategorised as junk mail …). A few small interventions aside, we have managed to When we conduct interviews online, interrupting the speaker and referring them back to the establish a functioning system and from then on our work with our special-purpose folders/files question – after they have digressed from the key focus of the question – can quickly create a sense continued unhindered. A particular advantage of working on an online file is that it is updated in of tactless communication. For this reason, we warned our interlocutors in advance at the start of virtually real time, which, in effect, allows us to edit it (and identify the individual at work). This each interview that this was something to be expected, specified the reasons behind this and asked has been of great help in all the stages of our work – the preparatory notes, the preparation of them for their understanding. Upon any actual interruption, in an act of conversation management, interview questions, the prompt recording of the answers during interviews and the drafting of the the interviewer would again tactfully alert the interviewee to stick to the point – e.g. »Please let consolidated and final versions of the report. me interrupt you, as I would like us to return to the original question. Only limited time is available to us and we would really like to collect some of the missing information with your help. This is The mentioned online folders/files have made interactive work possible in a way that enabled each why I would like to ask you: ….?«. On average, it has turned out that we could manage to address group member to see who modified which part of the text, comment on each other’s work and approximately ten questions of medium complexity per hour, including minor digressions from the thus exchange our views and opinions. The members agreed not to send each other the working main question. It should be emphasised that the questions prepared in advance served as our basic documents through e-mail, and to work exclusively directly in the mentioned e-environment, while framework, yet the interviewer could still decide on the spot to pose a different question based on e-mail was used solely for the purpose of informing everyone about the completed tasks and for where the discussion was going and considering the information already obtained. This form of requests by individuals about what their colleagues should check, comment on or further develop. organising our work (questions, answers and notes) has proven to be crucial, as the specifics of In this way, the report form matured, too, through the three stages which have been reflected working remotely could have led to considerable time losses, while the information contained in in the organisation of our subfolders: (1) the preparation stage – the draft of proceedings, with the responses could have easily been lost or forgotten. notes and highlights created in the stage of carefully reading through the received materials and reviewing target websites; (2) the consolidation stage – updating and upgrading of the draft from Unquestionably, the remote implementation has had, among other things, the effect of curtailing the previous stage during the evaluation visit, interviews and the additional information gained in the possibility of a reliable assessment of material conditions and more informal conversations this way; and (3) the stage of completion – the final modifications and corrections to the report, with students. If not much could be done as regards the latter, material conditions (premises, with a special attention to following the evaluation guide, the re-readings done by the colleagues equipment, disabled facilities, etc.) have been presented in either written form or in the form of from SQAA and refinements up to the final version submitted to the Agency Council for further photo/video materials. The presentations received have not provided a better insight; in case of reading. the future remote evaluations, it would be sensible to determine in advance the exact form and the essential components of such presentations. In addition to these versions of the evaluation report, a similar process unfolded as regards preparing for interviews, conducting them and using the obtained answers to provide the missing parts of the In conclusion, I would like to share a few observations, which are probably not specifically related substrate for our qualitative analysis by particular assessment standards as the key component to the remote implementation of the evaluation procedure, but are of a more general nature, yet of the final report. The Excel document was structured to reflect the interview timetable and I see them as an opportunity to improve future implementations of evaluation. Firstly, it would contained the proposed questions, the question posed by the members of the group of experts, the make sense to standardise the mandatory components of the evaluation application form, as primary addressee of the question from among the interviewees, assessment standard/s to which well as the mandatory elements of all the key documents. This significant step forward could be the question referred, and space for the prompt jotting down of responses or basic information performed side by side to a greater computerisation of procedures. Secondly and relating to the inferable from them. For the prompt recording of responses, we selected a member of the group same point, SQAA should, in my opinion, assume a more active role in the introductory phases of of experts in advance (not the same person as the one posing the question). As we all worked in the process, including the double-checks of all the received documents, calls for submitting the the environment Office365, all group members could simultaneously read the taken notes and mandatory documents known in advance, etc. Thirdly, again referring to my previous observation, complement and modify them as they saw fit immediately after the entry had been finished by the the procedure timeline should be subject to and based on at least two phases of verifying the person primarily responsible for it. adequacy of the submitted documents, which would enable a more organised and better work of the group of experts. For example, in concrete terms this would mean that the group of experts is As probably many of you have experienced in your own work in the time of COVID-19-related not appointed until the newly defined mandatory documents, including translations into English restrictions, some regular work procedures such as the meetings of the bodies of higher education when proposed members of the group of experts are foreigners, are obtained in their entirety. As institutions, other meetings, etc., usually last longer than in person. This is why the optimum use regards my final remark, the applying institution should be obliged to submit all the documents of the available time and managing the conversation (by steering it in the right direction) during in the English language, too. If such a request cannot be made in accordance with the current evaluation interviews poses a special challenge – with the goal of eliciting the responses and legislation, SQAA should be the one to take care of adequately translated documents. Fourthly, in information that the group of experts aims for or wants to clarify. At the start of each interview, the case when the group of experts includes foreigners and thus the work of the whole group is I had a short 1-to-2-minute introductory speech to greet everyone present and explain that we conducted in English, the finalised version of the final report is expected to be submitted in English, had, through the university, already sent everyone our personal presentations and received (in the too. Since the whole report needs to be submitted in the Slovenian language, the whole burden majority of cases) their names and professional functions, with the purpose of not wasting precious of translation is left with the Slovenian-speaking members. In the case of both accreditations time on personal presentations. During this short introduction, the interviewees were also asked to (University of Ljubljana and University of Maribor), the group members unanimously agreed that use the English language if possible, while being informed of their legal right to use the Slovenian translations should be either provided by SQAA or the members of the group of experts who take language and that we could provide simultaneous interpretation if this was their wish. on this extra task should be additionally paid for their services. 66 67 Goran Dakovic and Anna Gover Finally, in my opinion, the work of the group should be organised so that the days/week of interviews DIVERSIFICATION OF EXTERNAL QUALITY would be immediately followed by the group effort of drafting the report. To be more specific, in spite of how extremely mentally demanding the interviews may be (especially when conducted ASSURANCE IN THE EHEA - REFLECTION online, remotely), all the information and impressions are still fresh immediately afterwards. Due to the experienced strain, the majority of the group of experts seem to vanish into thin air for at FOLLOWING THE ANALYSIS OF ENQA AGENCY least a few days and usually several weeks immediately after the evaluation visit. The freshness of memory fades and I believe that if we had all known that a week of interviews would be followed REVIEWS CONDUCTED IN 2020–2021 by a day of writing the report together, the latter would have been better, more homogeneous, and drafted more quickly. I hope that the thoughts I have shared may serve as a starting point for our future work together – with the ultimate aim of improving the quality of the work done in the field of evaluation procedures. External quality assurance (QA) of higher education has its roots in the need for accountability and enhancement of higher education institutions, particularly with regards to accountability of these institutions towards their Professor Nejc Šarabon, Ph.D., University of Primorska, Faculty of Health Sciences, Slovenia stakeholders1. Agencies conduct external QA activities in accordance with the needs of the higher education system(s) they serve (and their various stakeholders), and taking into account the specificities of the context. In the last three decades, developments at national level across Europe moved at varying paces, but sped up considerably with the incorporation of quality assurance as one of the three key commitments of the Bologna Process (along with recognition and the three cycle degree structure2). Since then, a number of trends and developments have been observed, including varying shifts in focus between the accountability and enhancement functions of external QA3, and in recent years a general tendency towards external QA at institutional level rather than programme level4, although this appears to have stabilised in the past couple of years5 and it is also clear that many agencies perform a combination of the two approaches. While some challenges remain, the implementation of quality assurance in accordance with the Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) has become an established and accepted part of higher education systems in the European Higher Education Areas (EHEA)6. A further commitment made in the context of the Bologna Process, which is particularly relevant as background to this paper, is the assertion that higher education institutions in the EHEA should be permitted to choose any EQAR registered agency to conduct their external QA7. Full implementation of this commitment is far from being a reality but many systems are nonetheless opening up8. While this in theory creates more choice for institutions to select an agency that best fits their needs, it also creates a situation of ‘marketisation’ of external QA, where agencies that previously operated as a lone actor in their respective higher education systems now face the need to compete with other agencies9. As external QA in higher education continues to mature, discussions have recently emerged about the future and position of this activity in the higher education arena10. 1 See for instance the introduction chapters of Jeliazkova, M & Westerheijden, DF 2002, ‘Systemic adaptation to a changing environment: Towards a next generation of quality assurance models’, Higher education, vol. 44, no. 3/4, pp. 433¬448. 2 See the Bologna Declaration, http://www.magna-charta.org/resources/files/BOLOGNA_DECLARATION.pdf. 3 See for instance Vroejenstijn AI., 1995, ‘Improvement and accountability: navigating between Scylla and Charybdis: guide for external quality assessment in higher education’, London: Jessica Kingsley. 4 EQAR, 2020, ‘Policy Brief: External Quality Assurance Activities within and beyond the EHEA’, chart 1: Institutional, programme and joint programme reviews by year (2014¬2019), p. 4, https://www.eqar.eu/assets/uploads/2020/07/PolicyBrief_EQA_WithinandBeyondEHEA.pdf 5 Ibid, p. 4 6 Bologna Implementation Report, 2020, p. 73. 7 First mentioned in the Bucharest Commiqué in 2012 (EHEA, 2012, ‘Bucharest Communique: Making the Most of Our Potential: Consolidating the European Higher Education Area’, p. 2. https://ehea.info/Upload/document/ministerial_declarations/Bucharest_Communique_2012_610673.pdf), and reiterated in the Yerevan Communiqué in 2015, (EHEA, 2015, ‘Yerevan Communiqué’, p. 5 https://ehea.info/Upload/document/ministerial_declarations/YerevanCommuniqueFinal_613707.pdf). 8 21 systems in the EHEA fully realise this commitment, over twice as many as did so in 2013/14. Bologna Implementation Report, 2020, pp. 77¬78. 9 This said, it should be noted that some agencies have long operated in a competitive manner, particularly those providing external QA that forms a voluntary process for higher education institutions. For marketisation of the higher education market as such see for instance Jongbloed, B 2003, ‘Marketisation in Higher Education, Clark’s Triangle and the Essential Ingredients of Markets’, Higher Education Quarterly, vol. 57, issue 2, pp. 110¬135. For the developments in marketisation of external quality assurance in higher education see for instance Westerheijden, DF 2001, ‘Ex oriente lux? National and multiple accreditation in Europe after the fall of the wall and after Bologna’, Quality in Higher Education, vol. 7, issue 1, pp. 65¬75. 10 Hopbach, A & Flierman, A 2020, ‘Higher education: a rapidly changing world and a next step for the Standards and Guidelines for Quality Assurance in the European Higher Education Area’, in: ENQA, 2020, ‘Advancing quality in higher education: celebrating 20 years of ENQA’, ENQA: Brussels, pp. 29¬36, https://www.enqa.eu/wp-content/ uploads/Advancing-quality-in-European-higher-education-celebrating-20-years-of-ENQA.pdf. 68 69 In their need to successfully deliver both functions of external QA (i.e., provision of accountability and support For each of the analysed agency reviews, the agency’s self-assessment report19 was screened to obtain the for enhancement) and respond to the most recent changes in higher education (e.g., the Covid-19 pandemic, the research data (specifically the chapter describing the agency’s portfolio of external QA activities and the chapter rise of micro-credentials, the launch of European University Alliances), agencies are not only further developing ‘Current challenges and areas for future development’). More specifically, two sets of data were sought, in their existing external QA activities, but are also reconsidering their overall portfolio of activities as a whole. accordance with the twofold purpose of the paper: 1. The agency’s reflection on the activities and plans related Making a call for ‘innovation in external QA’ is a further manner in which agencies are highlighting their desire to the future diversification of external QA activities, including the content and characteristics of such changes; to experiment with the provision of services to higher education and this topic is gaining prominence in the field and 2. Motives and causes (i.e., internal and external drivers) of future diversification of external QA activities. of QA11. The purpose of this paper is thus twofold: Firstly, it aims to identify to what extent there is a discussion ongoing in QA agencies regarding the diversification of their external QA activities, and secondly, to explore the The paper does not include the agencies’ reflections on their diversification of external QA through the provision motives for, and characteristics of, such diversification. of external QA of research, since this type of provision does not fall within the scope of the ESG20, nor does it cover activities related to the external QA of other levels of education, outside higher education. Importantly, the paper does not seek to analyse or compare the existing portfolio of external QA activities of the agencies considered. Equally relevant, the paper does not observe the (planned) development of agencies’ Research Results existing external QA activities, since this aspect of their work rather falls within the remit of the ongoing enhancement of QA activities, following their own internal quality assurance processes. Instead, the research Actions and/or plans for future diversification of external QA activities wishes to understand whether the examined agencies are considering the possible future diversification of their external QA activities, and why they are considering expanding or changing their portfolio. This research shows that a large majority of the agencies in the sample reflected on the need to diversify their external QA activities in the future. In fact, the topic of diversification of external QA activities is strongly present, Finally, the findings of the paper should be considered within the overall context of an agency’s positioning within especially in the sections of the self-assessment reports where the agencies were asked to reflect on current its higher education system, funding, and mission to be fulfilled. For instance, a national quality assurance agency challenges and areas for future development. Out of 17 examined self-assessment reports, 13 agencies mention that is by large state funded and a sole provider of services to higher education institutions, will reflect on the at least once the aspiration and/or the need to further diversity their activities. diversification of external QA activities from a position that is profoundly different from the one of a field-specific agency competing for work across the EHEA. The analysed agencies state several areas to be considered when further diversifying the portfolio of external QA activities. For instance, they mention the need to develop new processes to foster the response of learning Research Methodology and teaching to society’s current challenges, e.g., “internationalisation, growing heterogeneity of students, different educational biographies and its consequences for higher education and training, the consequences of To empirically analyse agencies’ recent and planned diversification of their activities, the paper builds on the demographic developments for HEIs and the demand for lifelong and digital learning in the future” (SAR 1). data collected from external reviews of QA agencies that have been conducted by ENQA in 2020 and 2021. These reviews have followed the evaluation methodology as set out in the Guidelines for ENQA Agency Furthermore, the analysis reveals not only agencies’ changing approaches to external QA, but also the changing Reviews12. Altogether, 17 agencies13 from 11 countries14 have been included in this study. For the purposes of nature of agencies as organisations. In this regard, agencies reported to being faced with several new expectations data comparability, the analysis covers only full reviews15 against the ESG. No reviews of agencies that were by their governmental bodies (notably in the cases of national QA agencies), causing them to gain several new conducted outside the EHEA have been included in the study16. tasks and expand their scope of activities, such as “monitoring the student employability and compliance of study programmes with the needs of the labour market, the guarantee of equal access for all to higher education, The research period 2020¬2021 has been selected so as to include only the latest agency reviews as coordinated the internationalisation of higher education” (SAR 4). In another example, an agency operating in a competitive by ENQA, and to provide an opportunity to examine those agencies that have already reflected on the potential market of external QA noted a need to address the diverging needs of higher education institutions, and to impact of the Covid-19 pandemic on their external QA activities17. Moreover, during this period ENQA has profile itself as a ‘centre of competence for external quality assurance’ (SAR 3), which could potentially give, in observed intensified discussions in the quality assurance arena on the future of external quality assurance18. close cooperation with all stakeholders in higher education, “rise to new projects and approaches that would also help enhance the accreditation system” (SAR 3). These developments give rise to questions around what The research was conducted using the qualitative method of case study research, thus no statistical probability expectations can reasonably be placed on QA (and particularly external QA), as QA cannot be a universal should be concluded following the research results. panacea for managing higher education. Motives and/or causes of future diversification of external QA activities 11 See for instance Elken, M & Stensaker, B 2020, ‘Innovative practices in higher education quality assurance. A study of new activities, tasks and roles in six quality assurance agencies in Europe’, Nordic Institute for Studies in Innovation, Research and Education, working paper no. 7. Also Jacob, AK 2013, ‘Quality Assurance and Quality Enhancement in Higher Education and Innovation’, in: Carayannis, EG (eds), ‘Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship’, Springer: New York, NY. Probably the least surprising finding of this research is that agencies are discussing the need for diversification 12 Available at: https://www.enqa.eu/wp-content/uploads/ENQA-Guidelines-2021.pdf. of their external QA activities due to so-called ‘evaluation fatigue’ among higher education institutions and/or 13 External reviews of the following quality assurance agencies have been considered for this paper: ACQUIN, Germany, AI, Denmark, AQAS, Germany, ASHE, Croatia, ECAQA, Kazakhstan, FIBAA, Germany, FINEEC, Finland, GAC, Germany, HCERES, France, IAAR, Kazakhstan, IQAA, Kazakhstan, ZEvA, Germany, AAQ, Switzerland, ACPUA, Spain, the need to maintain the relevance of external quality assurance. This argument appears in six out of 13 self-ECBE, Belgium (European), NAA, Russian Federation, and UKÄ, Sweden. assessment reports that reflect on the need to diversify the portfolio of external QA activities, and primarily 14 Belgium (an agency operating throughout Europe), Croatia, Denmark, Finland, France, Germany, Kazakhstan, Russian Federation, Spain, Sweden, Switzerland. in the reports of national quality assurance agencies (e.g., Croatia, Sweden, Finland). As external QA in higher 15 More information about ENQA Agency Reviews, including full reviews, is available at https://www.enqa.eu/the-review-process/. education matures, agencies are discussing the need to reconsider the value of their activities for higher education 16 ENQA conducts agency reviews also outside the EHEA. In 2020, one such review has taken place in the Hong Kong Council for Accreditation of Academic and Vocational Qualifications, Hong Kong. institutions and potentially offer new activities that are close(r) to the needs of stakeholders. 17 With the exception of UKÄ, Sweden, whose self-assessment report was sent to ENQA prior to the outbreak of the Covid-19 pandemic in Europe. 18 See AQU Catalunya, A3ES, NOKUT & QQI 2021, ‘Reflecting on the future of European Quality Assurance’, webinar on 27 May 2021, https://www.aqu.cat/en/Studies/ 19 ENQA Agency Reviews require an agency under review to submit a self-assessment report (SAR). The report “provides the agency with an opportunity to reflect on how it conferences-and-workshops/Reflecting-on-the-future-of-European-Quality-Assurance. See also Hopbach, A & Flierman, A 2020, ‘Higher education: a rapidly changing world aligns with the ESG and to gather key documentation to support this. Additionally, the SAR is an opportunity to initiate the discussions on the agency’s current challenges” and a next step for the Standards and Guidelines for Quality Assurance in the European Higher Education Area’, in: ENQA, 2020, ‘Advancing quality in higher education: (Guidelines for ENQA Agency Reviews, p. 10). celebrating 20 years of ENQA’, ENQA: Brussels, pp. 29¬36. https://www.enqa.eu/wp-content/uploads/Advancing-quality-in-European-higher-education-celebrating-20-years-of-ENQA.pdf. 20 Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), 2015, Brussels, Belgium, p. 7. 70 71 Recent major developments21 in the external QA systems of some examined case studies (Germany, France) or at Although this seems not so long ago, the formal request for a revision came from the EHEA Ministers the European level (introduction of European University Alliances through the European Universities Initiative22) for Education already in 201225 and the subsequent revision process involved substantial stakeholder are another identified cause of possible diversification of external QA activities. As one agency notes, “the major consultation, including on an almost-final draft version of the proposed new text. As such, stakeholders change […] simultaneously opens new horizons and possibilities to the agencies” (SAR 1). Specifically in the were well aware of the changes and started to adapt to them already well before the formal endorsement of case of Germany, agencies are facing increased competition to provide programme and system (institutional) the final version in 2015. Furthermore, for some agencies the 2015 version of ESG contained elements that accreditations, as these can be conducted by any EQAR registered agency that has also been approved by the required significant changes to their standards and processes, whereas for others very little adjustment German Accreditation Council. In a shrinking market where higher education institutions shift from programme was needed, particularly as the most substantial changes to the ESG were to Part 1, which relates to to system accreditations and therefore undergo far fewer processes, the German QA agencies have been forced internal QA. For those latter agencies there has not been a period of adaption to the new standards, and to consider alternative offers in external QA to the existing types of evaluations. instead they have already been looking beyond them. As a result of this and other drivers (see below) there is an ongoing discussion regarding the extent to which the current ESG allow for flexibility beyond Furthermore, in Germany, the option for agencies to provide so-called ‘alternative procedures’23 provides or outside of the established traditional formats of external QA26. While the E4 Group27 (as some of the another example of future diversification of external QA, where institutions are given an opportunity to consider key European stakeholder organisations and co-authors of the ESG) have recently reiterated the scope alternative paths toward accreditation other than programme or system accreditation. In line with this, one for flexibility and innovation afforded by the ESG, they also recognise that a further revision may soon be agency listed as an opportunity the development of a procedure that will advise, support and above all encourage needed in order to reflect the changing nature of the higher education landscape28. higher education institutions to self-engage in these new types of activities. Despite the importance of the European framework, it should also be noted that many agencies (though Next, several new dimensions to external quality assurance activities are being explored, such as the social by no means all of them) primarily serve a national purpose and are therefore subject to and must respond dimension of higher education and engagement in implementation of Sustainable Development Goals24. to country-specific drivers and pressures, including political developments. A recent study by Elken and Following this, agencies are considering developing new external quality assurance activities that will evaluate Stensaker noted that agencies ‘tend to have a strong national orientation and where they are still under to what extent institutions consider these new dimensions into higher education. One agency provides an substantial control and influence by national authorities’29 and that this might result in a ‘possible growing example of a new external quality assurance activity under development that “must serve as a model for quality tension between the domestic and European roles and responsibilities of quality assurance agencies’30. assurance initiatives [of that agency] oriented by the Sustainable Development Goals, promoting gender equality or labelling non-discriminatory practices in research and higher education” (SAR 14). Maturity of external QA Digitalisation of higher education provision gained significant momentum due to the Covid-19 pandemic since One of the key motivations for many agencies to develop or diversify their external QA processes is the early 2020. Following this, one agency mentioned an opportunity to develop an accreditation procedure that need to ensure continued relevance and added value for the institutions and programmes within their will specifically evaluate all aspects of ‘digital in education’. More specifically, the agency explains that this jurisdiction. In systems with a long history of external QA, institutions and programmes have already accreditation will evaluate “the quality of the strategy of the digitalisation in the case of teaching and learning, been through multiple rounds of external QA. Agencies recognise that without changes in the approach personnel resources, technics, didactic layout and the quality assurance of digital programmes” (SAR 6). each subsequent round has potentially less impact and risks becoming a bureaucratic exercise and that they therefore have to offer something different (for example ASHE, UKÄ and FINEEC). Furthermore, for The relevance of big data, data management, and automation of information flows between various actors in agencies that operate in an ‘open market’ where institutions or programmes are not restricted to a single higher education (institutions, governmental bodies, students and QA agencies) is another driver in diversifying QA agency, agencies need to develop a competitive advantage by offering something that sets them apart the activities of QA agencies. In the context of this research, this identified driver should be analysed in relation from others operating in the same system or field (for example IAAR, AQAS and ZEvA). to the above discussed changing nature of agencies as organisations. Agencies reported plans to enlarge their portfolios of activities, which might or might not result in entirely new external QA activities, but for sure will This issue is seen not only at the level of QA agencies, but also in the reviews of agencies themselves. result in a wider scope of (existing) activities. ENQA, as the main provider of agency reviews in the EHEA, has recently revised its own review guidelines and in 2021 has diversified its services by launching, in cooperation with EQAR, a methodology for Reflections targeted reviews31, which will be available for agencies that have already had at least two consecutive successful reviews against the ESG. The above examples provide a practical illustration of how QA agencies are diversifying their EQA activities in order to meet the most pertinent challenges in their specific contexts. However, they are also a reflection of a general trend of development in external QA, namely the increasing calls for innovation and diversification in 25 EHEA, 2012, ‘Bucharest Communique: Making the Most of Our Potential: Consolidating the European Higher Education Area’, p. 2. https://ehea.info/Upload/document/ ministerial_declarations/Bucharest_Communique_2012_610673.pdf. EQA, some aspects of which have been amplified in the past 18 months as a result of the Covid-19 pandemic. 26 See for example Hopbach, A & Flierman, A 2020, ‘Higher education: a rapidly changing world and a next step for the Standards and Guidelines for Quality Assurance in the The following paragraphs offer some brief reflections on the drivers behind the currently observed changes. European Higher Education Area’, in: ENQA, 2020, ‘Advancing quality in higher education: celebrating 20 years of ENQA’, ENQA: Brussels, pp. 29¬36, https://www.enqa.eu/ wp-content/uploads/Advancing-quality-in-European-higher-education-celebrating-20-years-of-ENQA.pdf. and Loukkola, T 2020, ‘European quality assurance framework: why all the attention?’, https://eua.eu/resources/expert-voices/188:european-quality-assurance-framework-why-The European QA framework all-the-attention.html, and AQU Catalunya, A3ES, NOKUT & QQI 2021, ‘Reflecting on the future of European Quality Assurance’, webinar on 27 May 2021, https://www.aqu. cat/en/Studies/conferences-and-workshops/Reflecting-on-the-future-of-European-Quality-Assurance. The starting point for external QA in the EHEA is the ESG. The current version of ESG came into effect in 2015. 27 Consisting of ENQA (European Association for Quality Assurance in Higher Education), EUA (European University Association), EURASHE (European Association for Institutions in Higher Education) and ESU (European Students’ Union). 28 E4 Group, 2020, ‘The ESG in the changing landscape of higher education’, https://www.enqa.eu/wp-content/uploads/E4-statement_The-ESG-in-the-changing-landscape-21 In Germany, the entry into force of the Interstate Study Accreditation Treaty in 2018, and in France, the Research Programming Law recently giving HCERES the “responsibility of-higher-education_Final.pdf. for coordinating evaluation bodies and validating the evaluation processes of other evaluation bodies, such as the CTI and the CEFDG” (SAR 9). 29 Elken, M & Stensaker, B 2020, ‘Innovative practices in higher education quality assurance. A study of new activities, tasks and roles in six quality assurance agencies in 22 More information is available at: https://ec.europa.eu/education/education-in-the-eu/european-education-area/european-universities-initiative_en Europe’, Oslo: NIFU, p. 34. https://nifu.brage.unit.no/nifu-xmlui/bitstream/handle/11250/2720712/NIFUarbeidsnotat2020-7.pdf?sequence=1&isAllowed=y. 23 See https://www.akkreditierungsrat.de/de/akkreditierungssystem/alternative-verfahren/alternative-verfahren (in German). 30 Ibid., p. 8. 24 See https://sdgs.un.org/. 31 https://www.enqa.eu/wp-content/uploads/Guidelines-for-ENQA-Targeted-Reviews.pdf. 72 73 This new approach is intended to reduce the burden on agencies by looking only at specific standards of Conclusions the ESG as well as putting more focus on the enhancement aspect of the review process (as opposed to the compliance aspect). The information gathered from the analysis of the self-assessment reports by agencies used as the basis for this paper shows that QA agencies are actively considering the options to diversify their portfolio Covid-19 pandemic of external QA activities. The motives for this include the need to remain relevant and useful for higher education institutions, and changes in the QA systems in which they operate. Both of these aspects are It is probably fair to say that QA is not a fast-paced field. Changes in approaches to external QA, compounded in some cases by an opening up of the QA market in their respective systems, meaning whether introduced at the initiative of an agency or at the behest of other authorities, take time to some agencies find themselves working in a competitive environment. Furthermore, there have been introduce and usually (and quite correctly) follow substantial periods of evidence gathering and many broader developments in higher education in the EHEA, to which QA agencies are trying to stakeholder consultation, with pilot and transition phases. While the Covid-19 pandemic caused respond through their activities. unprecedented challenges for QA agencies, it also demonstrated that changes can be agreed and implemented rapidly when needed. It also presented an unusual opportunity for innovation, with In parallel, there is evidence of a growing discussion in the QA community about the room for innovation agencies forced to rethink their usual processes, including the standard timelines, the format of, or within the current QA framework of the EHEA. Although European stakeholder representatives and even need for, a site visit, and reflection on which are the essential and non-essential elements of policy contributors, such as ENQA, have long talked about the diversity of external quality assurance their work. In some cases, the suspension of the usual regulations for accreditation or other external approaches across Europe, it is nonetheless possible to observe a convergence around a handful of review have allowed for experimentation with activities and formats that might otherwise have most common approaches (external QA at programme or institutional level, emphasis on compliance taken years to develop. It is important to note here that a clear distinction must be drawn between or enhancement, generic or subject-specific approaches etc.33). Furthermore, following the guidance the emergency response to the consequences of the pandemic and any subsequent strategically set out by the ESG, external QA is largely based on the peer review approach in the four-stage format chosen developments to change previous formats. Nonetheless, it is clear that many agencies of self-assessment, external review usually including a site visit, report, and follow-up34. Agencies that are evaluating the lessons learnt over the past 18 months and are refining some aspects of the move away from this format have risked exclusion from the groups of approved agencies in the EHEA emergency actions for incorporation into their regular processes32. While this may not specifically (i.e. ENQA membership and EQAR listing)35. This demonstrates an ongoing tension between the desire fall under the definition of ‘diversification’ of external QA activities (as it is rather an adaptation of to experiment and innovate and the wish to remain part of the ENQA and EQAR structures, with the existing activities) it nonetheless appears to have opened minds to the potential for change. legitimisation that that bestows, not forgetting that external QA in line with the ESG is one of the key commitments of the Bologna Process to which national governments have agreed. Developments in European higher education All this raises a question about when are conditions right for innovation to be beneficial rather than In order to remain fit-for-purpose, external QA must keep pace with broader changes in European disruptive? At what point does the balance tip in favour of the desire to shake up the status quo? The higher education. Ongoing developments that must be reflected in approaches to external QA scope of this paper does not allow for a more detailed exploration of this, but it could form the basis include: the European Universities Initiative and other forms of internationalisation that deepen for further investigation, including the application of a theoretical framework based on organisational universities’ strategic cross-border cooperation, increased prominence of micro-credentials theory. and other units of learning that respond to calls for flexible study paths and life-long learning, digitalisation and online education (which has been on the agenda for many years but is now at the Note: the paper is work in progress and part of ENQA Agency Reviews’ thematic analyses. forefront due to the Covid-19 pandemic), as well as attention towards other cross-cutting issues such as diversity and inclusion, sustainability, and increasing links and interaction between the Goran Dakovic, Reviews Manager, ENQA three higher education missions of education, research and service to society. External QA has Anna Gover, Deputy Director, ENQA traditionally focused on learning and teaching, and indeed the ESG apply specifically to this area of activity. While many agencies have for some time also looked at higher education research activities, others are finding this area now being brought into their remit (e.g. . Additionally, some agencies have explored how they can specifically support and evaluate universities’ work in sustainability and societal engagement (for example, FINEEC and ACPUA). Some of these developments have resulted in revision of or addition to agencies’ own standards, whereas others have prompted agencies to add further services to their portfolios, including branching into consultancy services or offering other review options that are not specifically related to the ESG or system-level education standards. This demonstrates not only the changing nature of external QA activities, but also the changing nature of quality assurance agencies as organisations. In the quest to remain relevant and responsive (and in some cases, competitive), agencies may start to look quite different and the diversity of profiles across the EHEA is likely to become even more pronounced. 33 For other reflections on convergence and diversity in external QA see Hopbach, A & Flierman, A 2020, ‘Higher education: a rapidly changing world and a next step for the Standards and Guidelines for Quality Assurance in the European Higher Education Area’, in: ENQA, 2020, ‘Advancing quality in higher education: celebrating 20 years of ENQA’, ENQA: Brussels, pp. 29¬36, https://www.enqa.eu/wp-content/uploads/Advancing-quality-in-European-higher-education-celebrating-20-years-of-ENQA.pdf. 32 See for example presentations and discussions from two ENQA webinars on the topic of online quality assurance, which present case examples from ENQA members and affiliates. Part 1, April 2021, https://www.enqa.eu/events/enqa-online-members-forum-online-quality-assurance-experiences-from-enqa-members/ and Part 2, May 2021, 34 See ESG Standard 2.3. https://www.enqa.eu/events/enqa-webinar-online-quality-assurance-experiences-from-enqa-members-part-2/. 35 A prominent example was that of the Swedish quality assurance agency in 2013. Their ENQA membership was restored in 2021. 74 75 76 77 Slovenian Quality Assurance Agency for Higher Education Miklošičeva cesta 7, SI-1000 Ljubljana, Slovenija T: +386 1 400 5771 E: info@nakvis.si NAKVIS is a full member of the following European and other international associations: Slovenian Quality Assurance Agency for Higher Education (NAKVIS) was established in 2010 for accreditations and external evaluations in higher education and higher vocational education, as well as for development in this field. It operates responsibly, professionally, impartially and independently in line with European and global trends. Through membership in international associations, it strengthens its reputation and ensures comparability and international visibility of the Slovenian higher education area. It is committed to continuously improving quality, including cooperation with and counselling to all stakeholders in tertiary education. Document Outline ABOUT THE AGENCY BRIEF OVERVIEW OF THE AGENCY 'S WORK IN 2020 TENTH ANNIVERSARY OF THE AGENCY OVERVIEW OF SIGNIFICANT EVENTS AT THE AGENCY AND THE HIGHER EDUCATION COUNCIL AFFECTING THEIR ACTIVITIES AND COOPERATION WITH STAKEHOLDERS DEVELOPMENT OF THE ACCREDITATION AND EVALUATION CRITERIA OR CRITERIA FOR EXTERNAL QUALITY ASSESSMENT DEVELOPMENT OF IN-DEPTH SELF-EVALUATION AND COOPERATION OF THE AGENCY WITH STAKEHOLDERS INTERNATIONAL VISIBILITY OF THE AGENCY ANALYTICAL AND DEVELOPMENT ACTIVITY OF THE AGENCY WITH A VIEW TO FUTURE DEVELOPMENT STRATEGIC DEVELOPMENT OF THE SLOVENIAN QUALITY ASSURANCE AGENCY FOR HIGHER EDUCATION FOR THE 2021–2025 PERIOD I. INTRODUCTION II. OVERVIEW OF STRATEGIC GOALS ACHIEVED IN THE 2017-2020 PERIOD III. STRATEGIC OBJECTIVES FOR THE 2021-2025 PERIOD IV. CONCLUSION GUIDELINES FOR DISTANCE EVALUATION, CEENQA Klemen Šubic and Nataša Kramar BACKGROUND FOR WRITING GUIDELINES Klemen Šubic and Nataša Kramar GUIDELINES FOR DISTANCE EVALUATION, GUIDELINES BY THE CEENQA NETWORK PROCEEDINGS OF THE CONFERENCE Prof. dr. Peter Purg NEW ROLE AND PARADOXES OF QUALITY IN THE NATIONAL PROGRAMME FOR HIGHER EDUCATION 2021–2030 Prof. dr. Peter Verovšek WORK OF THE NAKVIS COUNCIL DURING THE PANDEMIC Prof. dr. Nejc Šarabon THE EXPERIENCE OF IMPLEMENTING INSTITUTIONAL RE-ACCREDITATION REMOTELY, THROUGH THE EYES OF THE PRESIDENT OF THE GROUP OF EXPERTS IN THE PROCEDURES OF THE TWO MAJOR SLOVENIAN UNIVERSITIES Diversification of external quality assurance in the EHEA - Reflection following the analysis of ENQA Agency Reviews conducted in 2020—2021