Borut Čampelj, PhD, Eva Jereb, PhD, Uroš Rajkovič, PhD
Introduction of the Hierarchical Self-assessment Model for Upgrading Digital School Strategy
Received 28. 1. 2020 /Accepted 19. 2. 2020 Scientific article UDC 004:37+519.816
KEY WORDS: hierarchical multi-attribute decision model, .school, self-assessment, informatisation, modelling
ABSTRACT - Headmasters and teachers have different perceptions of school informatisation, and the consequences are unsystematic planning and implementation of activities at individual and school level. This paper presents a comprehensive hierarchical self-assessment model for school informatisation. The designed indicators are organised hierarchically as a tree, from more complex to more simple. Sub-indicators at the highest level are: "School and the environment", "Teachers and on-line communities" and "Students and the environment". The tree representing the indicators has a complexity of up to 10 levels and is highly diversified with 106 leaves (basic indicators). The user self-assesses the current situation at the school by choosing one of the four possibilities for each basic indicator. Then the model itself determines the values of indicators at higher levels. The model was also tested in schools during four-hour workshops. The participants (54) assessed the model as a promising qualitative tool for understanding comprehensive informatisation and objectively determining the current school situation, which could lead to a higher level of further planning and implementation of school informatisation.
Prejeto 28. 1. 2020 /Sprejeto 19. 2. 2020 Znanstveni prispevek UDK 004:37+519.816
KLJUČNE BESEDE: hierarhični večparametrični odločitveni model, .šola, samoevalvacija, informatizacija, modeliranje
POVZETEK - Ravnatelji in učitelji imajo različno predstavo o informatizaciji .ole, posledice pa so lahko precej nesistematično načrtovanje in izvajanje dejavnosti na ravni posameznika in šole. V članku je predstavljen celovit hierarhični model samoevalvaci-je informatizacije na ravni šole. Oblikovani kazalniki so organizirani hierarhično kot drevo, od bolj kompleksnih kazalnikov do preprostejših. Kazalniki na zgornji ravni v drevesu so: »Šola in okolje«, »Učitelji in spletne skupnosti« in »Učenci in okolje«. Drevo kazalnikov ima globino tudi do 10 ravni in je zelo raznoliko s 106 listi (osnovni kazalci). Uporabnik sa-mooceni trenutno stanje na šoli, tako da pri vsakem osnovnem kazalniku (list na drevesu) izbere eno od štirih možnosti. Nato model sam določa vrednosti kazalnikov na višjih ravneh (odspodaj navzgor). Model je bil testiran tudi na šolah v obliki 4-urnih delavnic. Udeleženci (54) so model ocenili kot obetavno kakovostno orodje za določanje trenutnega stanja na posamezni šoli, kar bi lahko vodilo do višje stopnje nadaljnjega načrtovanja in izvajanja informatizacije šole. Model je uporabnikom omogočil poenotenje pogleda na trenutno stanje na šoli in razumevanje celovite informatizacije.
1 Introduction
Society expects schools to provide conditions for further development and to upgrade the quality of activities through an open and innovative learning environment in cooperation with different stakeholders (European Commission, 2013b). They should include innovations and opportunities that ICT brings and offers as motivational and creative environment (Loveless, 2008). ICT is embedded in all school activities, but in most cases partially and unsystematically, because the activities are not planned, organised and implemented comprehensively as headmasters or teachers have differing
44
Revija za ekonomske in poslovne vede (1, 2020)
perceptions of school informatisation without a coherent view of the current situation in the school. If schools are aware of the advantages and disadvantages of ICT use, then the integration of ICT in teaching, learning and administrative processes becomes one of the major tasks of a modern and change-oriented school (Balanskat, Blamire and Kefala, 2006; Bocconi, Panagiotis and Punie, 2012). The contributing conditions of school informatisation refer to the individual teacher level (digital competences, professional ICT development) and the school level (school vision, leadership, organizing) (Vanderlinde, Aesaert and Van Braak, 2015). In the new Digital education action plan, the European Commission set three priorities (Making better use of digital technology for teaching and learning; Developing digital competences and skills; Improving education through better data analysis and foresight) to meet the challenges and opportunities of education in the digital age (European Commission, 2018).
Although the national or regional strategies of ICT in education are upgraded constantly (Bassi, 2011), this is not always the case at school level. For example, Slovenia is one of the first countries where the "Computer Literacy Programme" at the national level had started in 1993 (Batagelj and Rajkovic, 1996), and has been updated several times and also renamed as "school informatisation". In 2008, the national project "E-education" started and the Slovenian schools have been encouraged to prepare comprehensive action plans for ICT and to continue them later on. The achieved level of the school informatisation strategies and provided support for schools in Slovenia has been visible by international studies (European Commission, 2013a, p. 141). But the studies are not detailed enough to highlight the exact situation at the individual school and areas to be improved. The school ICT strategies (digital school strategies) are not comprehensive and accepted by all employees, nor are they clearly integrated into annual work plans of schools and individuals. For example, the survey ICILS 2013 (Fraillon, Ainley, Schulz, Friedman and Gebhardt, 2014, p. 181) showed that only 35% of teachers have a common set of expectations about how ICT should be used in classrooms and only 31% have a common set of rules. Schools need to be given recommendations and guidelines on ICT to become competitive and to encourage the development processes (Becta, 2008; Davies, 2005).
The goal of our development project presented in this paper is the development of a self-assessment model for the level of school informatisation, which can contribute to a higher quality of teaching and learning, as well as other school activities based on a systematic approach. The indicators should be effectively designed and organised with qualitative connections. In practice, the model needs to encourage a comprehensive and detailed determination of the current situation of the individual school, support the unification of the understanding of informatisation between different stakeholders and, finally, based on the collected and organised data and ideas, lead the school in identifying realizable goals and concrete changes of plans. The context of the development project is connected to the needs of schools in Slovenia (e-competent teacher standard, e-material, school autonomy and leadership, etc.), but the results could have implications for international policy making and sharing.
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	45
2 Literature review
The development project linked the theory and practice of ICT in education, self -assessment process in education and the theory of decision-making process, particularly the methodology of the Hierarchical Multi-Attribute Decision Model.
2.1 Indicators of ICT in education
Based on the international theoretical and practical background and our own experience (Campelj and Rajkovic, 2008), we designed and organized a set of comprehensive indicators for school informatisation. A practical example has been developed around 2002 for schools in UK (Becta, 2008). The review of the development in other countries shows the importance of different aspects of ICT, like the analysis which factors are related to the use of ICT for teaching and learning (Vanderlinde, Aesaert and Van Braak, 2015). The future development and activities should be more measured and adaptive, taking account of the multidimensional nature of technology (Hammond, 2014).
Also, the results of various international development projects supported us in upgrading the existing models and ensuring sustainability. Eight general indicators and 28 sub-indicators were defined to support innovative pedagogies with ICT (Bocconi, Panagiotis and Punie, 2012). The first two international studies on digital competences of pupils and ICT in schools have been completed on the basis of the designed indicators (Wastiau et al., 2013; Fraillon, Ainley, Schulz, Friedman and Gebhardt, 2014).
The analysis of the trends over the last decade in 10 major journals in the field of instructional design and technology (West and Borup, 2014) are relevant in the process of the development of indicators. The latest teaching scenarios and learning approaches increasingly includes the personalization and learning analytics, for example collecting information from students about their personal learning behaviour or personal learning styles (Tseng, Chu, Hwang and Tsai, 2008).
The development of different frameworks on digital competences also contributes in designing the indicators of school informatisation. Since 2013, a detailed framework for the development of digital competence of all citizens at EU level, DigComp, has been developed and upgraded by the European Commission (EC), Joint Research Centre (JRC), Institute for Prospective Technological Studies Seville (IPTS) (Carretero, Vuorikari, and Punie, 2017). Another framework on pedagogical teacher digital competences (DigCompEdu) was developed by JRC (Redecker, 2017), which also integrated the results of certain countries that have developed their own e-compe-tence frameworks and models of teacher training, for example "E-competent teacher standard" in Slovenia (Kreuh, 2012). In 2015, JRC-IPTS developed another European framework for digitally-competent educational organisations called DigCompOrg (Kampylis, Punie and Devine, 2015), where 74 indicators at school level were introduced.
46
Revija za ekonomske in poslovne vede (1, 2020)
2.2 Self-assessment in schools
In order to ensure a higher level of qualitative activities in schools, it is necessary to implement both external evaluation ("how others see me") and self-assessment ("how I see myself') (Brejc and Zavasnik, 2010). There is a need to develop comprehensive approaches and examples of self-assessment, because the external evaluation cannot measure all parameters and external experts are not familiar with the details of everyday activities of individual schools. A sustainable self-assessment could be one of the most appropriate ways to specify the current situation and guarantee proper changes and improvements in schools (Hill and Ekey, 2010). Each individual must accept that only measurable indicators can verify his overall performance and they should be trusted and integrated into everyday work (MacBeath, 2000).
A comprehensive self-assessment has no beginning or end, and should be upgraded constantly. Effective self-assessment is analytical, using quantitative and qualitative data. In this process, we need to preserve and strengthen successful activities and, on the other hand, improve the weak activity or correct any errors (Musek, 2007). There is a lack of interpretation and analysis of the objectives and results in the form of effective discussion of different stakeholders at schools (Arlestig, 2008). Finally, the synthesis is the key for efficient external or internal evaluation (House, 1973). The achievable goals should be identified and then concrete steps provided and implemented.
Despite various development projects, awareness in schools that team cooperation between principals and all employees is a basic condition for further development is not strong enough (Hallerstrom, 2006). The practitioners of self-assessment should be not only teachers and headmasters or other employees but also students and parents or even local authorities.
2.3 Hierarchical Multi-Attribute Decision Model
As comprehensive school informatisation is a complex problem, the simplification of its indicators and their hierarchical organisation shall be a useful approach. A decision-making model such as the Hierarchical Multi-Attribute Decision Model - HMADM (Triantaphyllou, 2000; Turban, Aronson and Liang, 2004) used to assess the current situation can be used for both: the qualitative analysis of the existing situation and the interpretation of the results based on which the existing situation can be upgraded.
Figure 1: Abstract example of HMADM
Source: Own.
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	47
Figure 1 shows an example of the abstraction of HMADM. It consists of Xn attributes (indicators) and utility functions Fi as aggregation from the lower sub-attributes (sub-indicators). The model is based on a selected list of major attributes (indicators). In order to reduce the complexity of the decision model with respect to the number of attributes (indicators) and their interrelations, a hierarchical structure (tree) is employed. Attributes (indicators) at the higher levels of the tree are dependent on the attributes (indicators) at lower levels.
Based on the position of the attributes in the hierarchy (as a tree), it is possible to distinguish between:
□	the basic attribute - basic indicator (leaf of the tree; X1, X2, X3 and X5 in Figure 1) and aggregated attribute - complex indicator (X4 and, as the root of the tree, X6 in Figure 1). For each aggregated attribute, there is a corresponding aggregated utility function (F1 and F2 in Figure 1) defined by its dependence on all sub-attributes at a lower level, for example: X4 = F1 (X1, X2, X3).
3 Methodology
Based on the literature review and our own experience, the domain of school informatisation is extremely complex. ICT is embedded in all school activities: teaching and learning as pedagogical parts, but also administration and other services (psychologist, library, kitchen, etc.). The methodology of our development project addresses:
□	Design of indicators which covers all school informatisation activities and the qualitative approach of constructing the links between the indicators. This should support schools in understanding comprehensive informatisation and adapt the priorities;
□	On the basis of concrete data and cooperation of all stakeholders, the development of a transparent procedure of simple but also detailed determination of the current situation which could lead to a higher level of further planning and implementation of digital school strategy. This should increase the possibility of better understanding the assessment process and identifying possible changes in school informatisation.
3.1 Qualitative model as a tree of indicators
We identified the Decision Expert Methodology - DEX (Bohanec and Rajkovic, 1990) based on HMADM as a qualitative method of modelling and analysing. The discrete attributes (indicators) are usually represented by words rather than numbers for leaves (basic indicators) and the corresponding utility functions for aggregated indicators are defined by decision rules and not only by mathematical function (such as linear or weighted-sum). The range of possible values of the utility functions is therefore presented in the form of tables. A tree of indicators of school informatisation has been constructed step by step. We used the procedure of decomposition of complex indicators to more identifiable and independent sub-indicators. Each indicator
48
Revija za ekonomske in poslovne vede (1, 2020)
has up to 3 sub-indicators and we rather built a tree with greater depth. This is one of the major differences from other existing models. Namely, the complex indicators are further broken down to up to 10 levels, if necessary. For each leaf (basic indicator), a descriptive four-range measuring scale is developed and thus the indicator is more easily evaluated. For each aggregated indicator, a discrete utility function (values from 1 to 10) is defined based on the lower level. Therefore, the tree of indicators should be better adapted to human cognitive abilities and comprehensive information processing (Lindsay, 1977) as the individual indicator is not fragmented into too many sub-indicators (not more than 3), and also the corresponding utility function is not too complex or unclear. With this method, each indicator is simple and precisely defined, but there are a lot of indicators. This helps schools to efficiently and in detail monitor the progress of not too complex and manageable priority areas of school informatisation.
Our approach has been supported in practice by a tool for progressive development and testing of HMADM, computer program Dexi, the expert system framework for multi-attribute decision making process (Bohanec, 2014).
Because of its complexity, the tree of indicators is presented in three figures (Figure 2, 3 and 4), including a total of 174 indicators. The major indicator "Level of school informatisation" is divided into three sub-indicators: "School and the environment" (Figure 1), "Teachers and e-communities" (Figure 2) and "Students and the environment" (Figure 3). In all three figures, there are three numbers for each indicator representing the examples of self-assessment results of three users (headmaster or teachers) of the model (the numbers will be further described in section 4.1). For example, the headmaster (Figure 2) selected one of the four possible values for each leaf. Then the model itself (based on the utility functions) determined the values of the attributes at the higher levels with decision rules (as in Figure 2, the headmaster's score is 6).
In this article, the indicators are not explained in detail, because their name and their hierarchical organization provide a sufficient enough explanation. But the organizational part of the model is explained for the sub-tree "School and the environment".
A concrete example of the descriptive four-point scale of the leaf "Leadership" (in Figure 2, located in the sub-tree "E-competent school" - "Vision and plan" - "Vision" - "Design" - "Content" -"E-competences" - "Employees") is:
1.	There is no specific vision/strategy for e-competences of school leadership, there is a general mention of the use of ICT.
2.	The vision/strategy contains basic e-competences: text design and spreadsheet design (annual plans, reporting templates ...), communication (e-mail, web, videoconferences ...) and other skills, e.g. for presentation purposes.
3.	The vision/strategy contains a major part of e-competent headmaster standard or leadership (critical use at work and school's management, communication, safe use, production of e-materials, etc.), it also contains personal development and lifelong learning supported by ICT.
4.	The vision/strategy contains all parts of e-competent headmaster standard or leadership, professional (leadership) and personal development (teamwork, personal
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	49
growth) and development of lifelong learning, international recommendations.
Figure 2: Sub-tree "School and the environment": indicators and three self-assessment results (Headmaster, Teacher 1, Teacher 2)
Source: Own.
For example, in Figure 2, the headmaster and one teacher scored the indicator "Leadership" with 3 and one teacher with 4.
As example, Table 1 defines the utility function of the aggregated indicator "School and the environment". We mentioned that the possible values of each utility function are discrete from 1 to 10. The utility function of the indicator "School and the environment" is defined by 100 possible options, because it has two sub-indicators, "School" and "Environment", and both of them have possible values from 1 to 10 (100 = 10 * 10). First, we set the weights: "e-competent school"- 60% and "Environment" - 40%. Then we adjusted some individual values and 9 examples are underlined in Table 1. The utility function is discrete and could be adjusted to the priorities and situation of the individual school. This is one of the major reasons why we selected this qualitative method. In Table 1, three lines are shadowed as these are three results of self-assessment, presented in Figure 2.
50
Revija za ekonomske in poslovne vede (1, 2020)
Table 1: Utility function of the aggregated indicator "School and the environment"
Option	E-compe-tent school	Environment	School and environment
1	1	1	1
2	1	2	1
3	1	3	2
4	1	4	2
5	1	5	3
6	1	6	3
7	1	7	3
8	1	8	3
9	1	9	3
10	1	10	3
11	2	1	2
12	2	2	2
			
38	4	8	6
39	4	9	6
40	4	10	6
41	5	1	3
42	5	2	3
43	5	3	4
44	5	4	5
			
Option	E-compe-tent school	Environment	School and environment
65	7	5	6
66	7	6	7
67	7	7	7
68	7	8	7
69	7	9	8
70	7	10	8
71	8	1	5
72	8	2	5
			
78	8	8	8
			
91	10	1	5
92	10	2	6
93	10	3	7
94	10	4	7
95	10	5	7
96	10	6	8
97	10	7	9
98	10	8	9
99	10	9	10
100	10	10	10
Source: own.
The sub-tree "Teachers and e-communities" is presented in Figure 3, including indicators and three results of the self-assessment (Headmaster, Teacher 1, Teacher 2).
Figure 3: Sub-tree "Teachers and e-communities": indicators and three results of the self- assessment (Headmaster, Teacher 1, Teacher 2)
Source: Own.
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...
51
The sub-tree "Students and the environment" is presented in Figure 4, including indicators and three results of the self-assessment (Teacher 1, Teacher 2, Teacher 3).
Figure 4: Sub-tree "Students and the environment": indicators and three results of the self-assessment (Teacher 1, Teacher 2, Teacher 3)
Source: Own.
3.2 Transparent introduction of the model as self-assessment
One of the most important conditions of progress and real changes is an objective overview of the existing situation at the school. Partly, it can be determined by a standardised external evaluation, but a detailed and comprehensive determination could be done only by self-assessment (Blanchard, 2002). Based on theory and existing practice, we developed a continuous loop of self- assessment of school informatisation, covering three major activities, as shown in Figure 5.
Figure 5: Areas of the self- assessment process
Source: Own.
Self-evaluation questionnaire covers all areas of school informatisation and encourages the systematic approach of collecting, analyzing and assessing various aspects. The school would reveal the details of its strengths and weaknesses, not missing anything important. In the self-evaluation questionnaire, the indicators are determined as
52
Revija za ekonomske in poslovne vede (1, 2020)
measurable and efficient, and are interconnected in a meaningful way to support schools in the understanding of comprehensive informatisation and adapting the priorities and prediction for further processes of school informatisation.
Reflection as a dialogue between teachers, headmasters and other staff about the results and ideas obtained by using the self-evaluation questionnaire should be encouraged . It is recommended that self-evaluation questionnaire is used in pairs or triples to encourage reflection between them. After they self-assess current situation, they need to be encouraged to reflect on the results with all participants. The participants explain any different results about the current school situation and understand others' opinions as well as the reasons for the differences. Finally, the shift of the debate should be encouraged towards unification and comprehensive understanding of the current level of school informatisation.
Further planning (digital school strategy): all strengths and weaknesses of the school informatisation (which were identified and discovered during reflection) could be used in the synthesis and determination of priority areas. Those should be transferred to real, effective and qualitative changes of further plan and must be measurable, not too complex and adopted by all employees and other stakeholders. In practice, the debate shall be focused to define necessary activities to upgrade the quality of processes at school.
4 Testing and results
The developed self-assessment model of the school informatisation level was tested in more stages. The first stage was a verification stage with 18 experts and the results were used to upgrade the model. We present and discuss the final testing which is the most interesting for readers. We validated the model in five primary schools -authentic environments - with 54 teachers, ICT coordinators, headmasters and others. Four-hour workshops were held at each school to determine the current situation of school informatisation by using the self-evaluation questionnaire, reflect on the results, and articulate specific priorities and possible changes of future plans (digital school strategy).
4.1 Example of the results
We present one example of self-assessment results, because the tree of indicators is extremely complex. It refers to the results of a headmaster and two teachers from the same school, presented in the sub-tree "Teachers and e-communities" (Figure 3). The headmaster assessed the sub-tree "Teachers and e-communities" with 6, and the teachers with 5 and 4. In reflection, they identified and discussed the indicators with significant difference, for example:
□ The indicator "Plan" in the sub-tree "Teachers" - "Lessons" was scored by the headmaster with 6 and by both teachers with 8: the headmaster expressed the opinion that the teachers do not fully take into account the recommendations of the
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	53
national curriculum or school guidance where digital competences and expected learning outcomes are explained for every subject;
□	The indicator "Evaluation of lessons" was assessed by the headmaster with the lowest score (1), while one teacher assessed it much higher (6). The headmaster was sure that the teachers had not done any proper regular evaluation of the lessons,
□	They had different views about the current school situation in the sub-tree "Training and R&D", namely the "Training" was assessed by the headmaster with the highest score (10), but the "R&D" with a much lower one (3). They exchanged their views, and tried to agree and find common ground about the actual situation. Through the discussion, they discovered that they should re-assess the situation again, because they were unaware of certain facts.
Together with the other participants, they identified the most problematic areas which should be improved in the future:
□	Active participation of students in the planning of lessons;
□	Teachers should be informed about innovative teaching approaches more intensively (not only through training, but also from magazines, e-communities and other sources) or should exchange the best and worst practices in the school and from other schools;
□	Teachers should self-assess their own lessons, learn from other teachers (peer learning) and be more motivated by school leadership;
□	More teachers should be involved in development activities and participate in collaborative projects with other teachers at the school and from other schools;
□	Teachers should be more familiar with and active in e-communities.
The next step was determining concrete future activities of the above identified areas.
4.2 Evaluation of the model and discussion
The 54 participants tested the model first and then assessed it, which is important for the validation of the developed model. We use a simple descriptive statistics, because our research activity was the development of a comprehensive model and not a study. The participants assessed the model and workshops very positively (on average, 4.4 out of 5; standard deviation 0.7). A thematic content analysis was used to organise the qualitative and quantitative data about the usefulness and effectiveness of the model at schools in four groups:
1. The model supports a detailed determination of the overall situation of school informatisation:
□	All participants (100%) answered that the tree of indicators (questionnaire) is a useful tool for assessing the current situation of school informatisation. Most of them (96%) expanded the range of possibilities of ICT use in schools. All participants (100%) were sure that all school informatisation areas in the model are effective, and meaningfully designed and organized. The majority (93%) answered that no area is too detailed or too vaguely described. The users stressed that
54
Revija za ekonomske in poslovne vede (1, 2020)
they determined the current school situation, and also specified and assessed their own work in an organized manner. The four-level descriptive scales for the tree leaves are reasonable and simply understandable for 78% of the participants, however, 22% of the participants complained about excessive description or certain less understandable parts of the descriptive scales. The descriptive four-point scales are also qualitative, which allowed the users to assess the exact situation based on the individual basic indicator. The results allow us to conclude that this was one of the crucial decisions in our development project to support a simple and efficient but also detailed self-assessment.
2. The model encourages a unified understanding of the comprehensive school informatisation:
□ All experts (100%) agreed that they unified the understanding of "comprehensive school informatisation". The self-evaluation questionnaire (tree of indicators) encouraged half of the participants (50%) to change the priorities of school informatisation. The remaining 50% of the participants also expected that they would change priorities in the future because of the participation at the workshop. Some participants particularly stressed that almost no one had an overall view of school informatisation before the workshops. All participants were sure that the model could support not only school leadership but also teachers, pupils, parents and others to understand the school informatisation processes. The agreement among all participants contributes to the comparability and further monitoring of school informatisation. Graph 1 answers the question: "Who should use the self-evaluation questionnaire in the future?"
Graph 1 : Users of the self-assessment questionnaire
Headmaster Teacher	Pupil	Parents School council
Source: Own.
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	55
The main reason why the model should not be used by parents is that the self-evaluation questionnaire (tree of indicators) is too detailed, and should be adjusted and simplified for them.
All participants answered that the possible values from 1 to 10 of the aggregated indicators provide a sufficient dispersion of the results, which helps to more easily identify the relevant specific differences and issues for discussion between school employees.
1.	The model supports the schools in upgrading their digital school strategy and priorities with more concrete activities based on objective data;
2.	The participants reviewed the school's yearly plans (e. g. digital school strategy) and 83.3% of them determined that school informatisation is not comprehensive, although schools do have school infomatisation plans. One of the workshop results shows that the participants defined concrete (and not too general) weak and strong points about the current school informatisation situation as a basis for further planned manageable activities at individual and school level;
3.	The model is a contribution to the development of school self-assessment and school leadership in Slovenia.
Self-assessment is not a new development in Slovenian schools, but for all participants, it represented the first comprehensively developed self-assessment. The opinion of 98% of the participants is that the self-assessment workshop should be implemented again, preferably within a period of one year. They answered that the model and workshops also meets the needs which are articulated in the latest results of the development of school leadership: autonomy, accountability, distributed leadership, professional standards, evaluation, capacity building, policy self-assessment (Kikis -Papadakis, Kollias and Hatzopoulos, 2014).
The major limitation of our work is that we validated the model on rather small samples in primary schools (in Slovenia, the lower secondary level is included in the primary school). The major critical aspects of the developed model depend on:
□	The model itself: for some individual users, there could be too many indicators in the tree or some descriptive four-point scales for the basic indicators could be less understandable or even incomprehensible, and therefore, a different approach should be provided, like simplification of the tree with less indicators;
□	The users because of lack of knowledge about school informatisation, lack of self-assessment competences and lack of personal competences (team work, critical view of co-workers, peer learning). But some participants also lack a sufficiently developed "reflection skill", others were too self-confident about their competences and did not want to pay attention to the opinions of their colleagues or the objective view of the current school situation. It was also obvious at the workshops that self-assessment is effective only if the users take the time for details.
56
Revija za ekonomske in poslovne vede (1, 2020)
5 Conclusion
The original contribution is the development of a comprehensive organizational and information model for the self-assessment of school informatisation supported by ICT and intelligent methodologies. The model supports the schools in organizing and more efficiently using the existing information and competences for further development at the school level. Based on testing, a simple descriptive statistics allows us to determine that the participants assessed the model very positively. It was found that the model is a very promising qualitative tool for assessing the current situation, which could lead to a higher level of further planning and implementation of school informatisation. The thematic content analysis also showed that it enables users to unify the view about the current school situation and to understand comprehensive informatisation, as well as encourages the participation of different stakeholders in the self-assessment process. The results and our discussion clearly indicates that the model provides support to organise and to efficiently use the existing information and competences for further development at the school.
Based on the results, we can conclude that we contributed to the achievement of the development project objective by:
□	Articulating and developing a comprehensive hierarchical model for a complex problem by designing the indicators which cover all activities of school informatisation, together with a qualitative approach of constructing the links between the indicators. It is a comprehensive upgrade of the existing approaches (non-hierarchical), where the indicators are consistent with the latest developed recommendations and follow the contribution conditions of central measurement concepts in many ICT studies. The model also follows trends, technologies and challenges for European schools. The effective information techniques were used with an emphasis on the artificial intelligence methods that enable transparency and explanation. The existing various comparisons of the hierarchical models supported our decision to use the discrete utility function and not mathematical functions to interconnect the indicators in the tree;
□	Validating the model with transparent self-assessment on the basis of concrete data and cooperation of all stakeholders: determination of the current situation which could lead to a higher level of further planning and implementation of school informatisation (digital school strategy). This procedure increased the possibility of better understanding the assessment process and identifying possible changes. We supported the needs of Slovenian schools to set common expectations and provide the rules of ICT in teaching and learning. At the school level, the model highlights the needs for flexible organization and development of all areas of school activities continuously. At the teacher level, the model contributes to the use of data in the further development of teachers' approaches and professional development. At the student level, the model supports the learning analytics to understand and optimise learning and the environments in which learning takes place. The workshop participants performed their first self-assessment in that extent, and it has been assured
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	57
that the process was as qualitative, sustainable and continuous as expected. If an individual context can create supportive environments, school self-assessment has an important role in terms of the significance of the bottom-up approach, sustaina-bility, further development, focused improvement, risk taking, etc.
Dr. Borut Čampelj, dr. Eva Jereb, dr. Uroš Rajkovič
Uporaba hierarhičnega modela samoevalvacije za nadgradnjo digitalne šolske strategije
Ravnatelji in učitelji imajo različno predstavo o informatizaciji šole, posledici pa sta lahko precej nesistematično načrtovanje in izvajanje dejavnosti na ravni posameznika in šole (Bocconi, Panagiotis in Punie, 2012). Evropska komisija je v novem akcijskem načrtu za digitalno izobraževanje določila prednostna področja za spopadanje z izzivi in priložnostmi izobraževanja v dobi digitalne družbe (Evropska komisija, 2018).
Načrti informatizacije šol (digitalne šolske strategije) niso celoviti in jih ne sprejemajo vsi zaposleni. Na primer, raziskava ICILS 2013 (Fraillon, Ainley, Schulz, Friedman in Gebhardt, 2014, str. 181) je pokazala, da ima le 35 % učiteljev enaka pričakovanja z uporabo IKT v učilnicah in le 31 % skupne pristope oz. pravila. Namen in cilj razvojnega projekta je:
□	oblikovanje kazalnikov, ki zajemajo vsa področja informatizacije šole, in kvalitativnega pristopa gradnje povezav med kazalniki, kar lahko pomaga šolam pri razumevanju celovite informatizacije;
□	na podlagi konkretnih podatkov in sodelovanja vseh deležnikov bi razvoj preglednega in enostavnega postopka podrobnega določanja trenutnega stanja lahko spodbudil in pripomogel k doseganju višje ravni nadaljnjega načrtovanja in izvajanja.
Predstavljeni razvojni projekt povezuje teorijo in prakso na področju informatizacije izobraževanja, na področju samoevalvacije in na področju odločanja z uporabo večparametrskega hierarhičnega modeliranja. Od leta 2013 Evropska komisija razvija podrobne okvire za razvoj digitalnih kompetenc, in sicer: okvir digitalnih kompe-tenc za evropskega državljana (Carretero, Vuorikari in Punie, 2017), okvir pedagoških digitalnih kompetenc učiteljev (Redecker, 2017) ter okvir za digitalno kompetentne izobraževalne organizacije (Kampylis, Punie in Devine, 2015). Okviri združujejo tudi rezultate iz nekaterih držav, ki so razvile lastne okvire, kot je npr. »Standard e-kompe-tentni učitelj« v Sloveniji (Kreuh, 2012).
Trajna samoevalvacija je lahko eden najprimernejših načinov za določitev trenutnega stanja in zagotavljanje ustreznih sprememb in izboljšav na šoli (Hill in Ekey, 2010). Celovita samoevalvacija nima svojega začetka niti konca, ampak jo je treba nenehno nadgrajevati. Učinkovita samoevalvacija je analitična, uporablja kvantita-
58
Revija za ekonomske in poslovne vede (1, 2020)
tivne in kvalitativne podatke (Musek, 2007). Ker je celovita informatizacija šole kompleksen problem, je poenostavitev kazalnikov in njihova hierarhična organiziranost koristen pristop. Model odločanja na podlagi hierarhičnega modela z več kriteriji oz. kazalniki (Triantaphyllou, 2000; Turban, Aronson and Liang, 2004) je mogoče uporabiti tako za kakovostno analizo obstoječega stanja kot tudi interpretacijo rezultatov, na podlagi katerih je mogoče obstoječe stanje nadgraditi.
Na podlagi teorije in obstoječe (lastne) prakse smo razvili kontinuirano zanko samoevalvacije informatizacije posamezne šole, ki zajema tri glavne dejavnosti:
□	Reševanje samoevalvacijskega vprašalnika, ki zajema vsa področja informatizacije šole in spodbuja sistematičen pristop zbiranja, analiziranja in ocenjevanja različnih pogledov.
□	Treba je spodbujati refleksijo in razmišljanje med učitelji, ravnatelji in drugim osebjem o rezultatih in idejah, pridobljenih z vprašalnikom za samoevalvacijo. Pomembno je, da si obrazložijo različne rezultate in poglede o trenutnem stanju ter razumejo druga mnenja in razloge za razlike.
□	Nadaljnje načrtovanje: vse prednosti in slabosti informatizacije šole, ki so bile ugotovljene in odkrite med refleksijo, je treba uporabiti pri sintezi in določitvi prednostnih področij. Potrebne spremembe bi morale biti prenesene v realen, učinkovit in kakovosten nadaljnji načrt.
Pri razvoju smo uporabili ekspertno metodologijo odločanja DEX - Decision Expert Methodology (Bohanec in Rajkovič, 1990), ki temelji na hierarhičnem večpa-rametrskem modelu odločanja HMADM (Hierarchical Multi-Attribute Decision Model), kot kvalitativno metodo hierarhičnega modeliranja (npr. v obliki drevesa) in za analizo informatizacije posamezne šole. Gradnja drevesa kazalnikov informatizacije šol je potekala korak za korakom.
Pri postopku razgradnje kompleksnejših kazalnikov (najkompleksnejši kazalnik je v korenu drevesa) so nastajali enostavnejši (manj kompleksni), bolj prepoznavni in neodvisni podkazalniki. Vsak kazalnik ima največ 3 podkazalnike, da se drevo ni širilo preveč v širino, ampak bolj v globino. To je ena večjih razlik v primerjavi z drugimi obstoječimi modeli. Kompleksni kazalniki se namreč po potrebi razčlenijo na do 10 ravni. Na vsakem listu na drevesu (osnovni kazalnik) se razvije 4-stopenjska opisna lestvica, ki jo je mogoče meriti, zato je kazalnik tudi lažje ovrednotiti. Na vsakem kazalniku, ki ni list, je definirana diskretna funkcija koristnosti (v vrednosti od 1 do 10) na podlagi vrednosti kazalnikov na nižji ravni.
Drevo kazalnikov je prilagojeno človekovim kognitivnim sposobnostim in celoviti obdelavi informacij, da posamezni kazalnik ni razdrobljen na preveč podkazalnikov na naslednjem nižjem nivoju (ne več kot 3). Diskretni kazalniki so po navadi predstavljeni z besedami in ne s številkami na listih drevesa (osnovni kazalniki), ustrezne funkcije koristnosti pri združenih kazalcih pa so določene z odločitvenimi pravili in ne le z matematičnimi funkcijami (na primer linearno ali uteženo vsoto). Obseg možnih vrednosti funkcij koristnosti je zato predstavljen v obliki tabele. Pri tej metodi je vsak kazalnik preprost in natančno določen, vendar pa zato obstaja veliko kazalnikov. To
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	59
šolam zagotavlja učinkovitost, saj je tako lažje natančno spremljati napredek ne preveč zapletenih in obvladljivih prioritetnih področij informatizacije šol.
Predstavljeni pristop je bil v praksi podprt z orodjem za napreden razvoj in testiranje HMADM, računalniškim programom Dexi (Bohanec, 2014), ki zagotavlja sistemski okvir za postopek odločanja z več kriteriji.
Drevo kazalnikov je predstavljeno v treh poddrevesih, njegova kompleksnost skupaj vsebuje 174 kazalnikov. Glavni kazalnik (koren drevesa) »Raven informatizacije šole« je razdeljen na tri podkazalnike: »Šola in okolje«, »Učitelji in e-skupnosti« in »Učenci in okolje«. Vsak kazalnik pa je ponovno razdeljen na do 3 podkazalnike. Sa-moocenjevanje torej poteka tako, da uporabnik izbere eno od štirih možnih vrednosti na vsakem listu drevesa. Nato model sam (na podlagi funkcij koristnosti) določi vrednosti kazalnikov na višjih ravneh na podlagi odločitvenih pravil modela.
Model smo testirali na 5 osnovnih šolah - v avtentičnih okoljih, kjer je sodelovalo 54 učiteljev, koordinatorjev IKT, ravnateljev in drugih. Na vsaki šoli so potekale štiriurne delavnice: določitev trenutnega stanja informatizacije šole s pomočjo vprašalnika za samoevalvacijo, razmislek o rezultatih in opredelitev prioritet ter možnih sprememb načrtov.
Na delavnicah so uporabniki uporabljeni model na koncu tudi ocenili, in sicer:
1.	model podpira podrobno objektivno določitev konkretnega stanja informatizacije
na posamezni šoli:
n Vsi udeleženci (100 %) so odgovorili, da sta drevo kazalnikov in samoevalva-cijski vprašalnik koristni orodji za oceno trenutnega stanja informatizacije na šoli. Večini (96 %) sta razširila pogled nad možnostmi uporabe IKT v šolah. Vsi udeleženci (100 %) so bili prepričani, da so vsa področja informatizacije šole v modelu učinkovita ter smiselno zasnovana in organizirana. Večina (93 %) je odgovorila, da nobeno področje ni preveč podrobno ali nejasno opisano. Rezultati so pokazali, da je bila to ena ključnih odločitev v našem razvojnem projektu za podporo preprostemu in učinkovitemu, a tudi podrobnemu modelu samoevalvacije.
2.	model spodbuja poenotenje razumevanja celovite informatizacije šole:
n Vsi strokovnjaki (100 %) so se strinjali, da so poenotili razumevanje »celovite informacijske šole«. Samoevalvacijski vprašalnik (drevo kazalnikov) je polovico udeležencev (50 %) spodbudil k spreminjanju prioritet informatizacije šole. Toda tudi preostalih 50 % udeležencev pričakuje, da bodo spremenili prioritete v prihodnosti. Vsi udeleženci so bili prepričani, da je model lahko dobra podpora ne le vodstvu šole, temveč tudi učiteljem, učencem, staršem in drugim, da razumejo celovit proces informatizacije šole.
3.	model omogoča podporo šolam, da nadgradijo načrte s konkretnejšimi dejavnostmi na podlagi objektivnih podatkov:
n Udeleženci so pregledali letne načrte šole in 83,3 % jih je ugotovilo, da informatizacija šole ni vključena celovito v šolski delovni načrt. Eden izmed rezultatov delavnic pokaže tudi, da so udeleženci opredelili konkretna (in ne preveč splošna) šibka in močna področja trenutnega stanja informatizacije šole kot
60
Revija za ekonomske in poslovne vede (1, 2020)
osnovo za nadaljnje načrtovane vodljive dejavnosti na ravni posameznika in šole.
4. model je prispevek k razvoju samoevalvacije in vodenja šol:
n 98 % udeležencev meni, da je treba samoevalvacijsko delavnico ponoviti večkrat, po možnosti v roku enega leta.
Rezultati in razprava so pokazali, da model daje podporo pri organizaciji in učinkoviti uporabi obstoječih informacij in kompetenc za nadaljnji razvoj informatizacije
šol. Na podlagi rezultatov lahko sklepamo, da smo prispevali k doseganju namena in
cilja razvojnega projekta na sledeča načina:
□	Opredelili in razvili smo celovit hierarhični model zapletene problematike z oblikovanjem kazalnikov, ki pokrivajo vse dejavnosti informatizacije šole, in uporabili kvalitativni pristop gradnje povezav med kazalniki. Različna primerjava hierarhičnih modelov je podprla našo odločitev, da je smiselno za povezovanje kazalnikov v drevo uporabiti diskretno funkcijo koristnosti in ne matematične funkcije.
□	Model so validirali strokovnjaki z uporabo transparentne samoevalvacije na podlagi konkretnih podatkov in s sodelovanjem vseh zainteresiranih strani -je objektivna opredelitev trenutnega stanja, ki lahko privede do višje stopnje nadaljnjega načrtovanja in izvajanja informatizacije šol. Ta postopek je povečal možnost boljšega razumevanja postopka ocenjevanja in prepoznavanja možnih sprememb.
LITERATURE
1.	Arlestig, H. (2008). Communication between Principals and Teachers in Successful Schools. Umea: Umea Universitet.
2.	Balanskat, A., Blamire, R. and Kefala, S. (2006). The ICT impact report, A review of studies of ICT impact on schools in Europe. Brussels: European Schoolnet.
3.	Batagelj, V. and Rajkovič, V. (1996). Information Technology Project in Slovenian Schools. In: Van Weert, T. J. (ed.). Proceedings of the First Euro Education Conference, Aalborg, pp. 9-15.
4.	Bassi, R. (2011). ICT in Education (ICT4E) Policies and Plans worldwide. Retrieved 1/28/2020 from the Internet: https://www.eldis.org/document/A64501.
5.	Becta (2008). Schools - Leadership and Management, England. Retrieved 1/28/2020 from the Internet: http://webarchive.nationalarchives.gov.uk/20110130111510/http:/schools.becta.org.uk/ index.php?section=lv&catcode=ss_lv_mis_im03&rid=14734.
6.	Blanchard, J. (2002). Teaching and Targets, Self-evaluation and School improvement. London: Routledge Falmer.
7.	Bocconi, S., Panagiotis, G. K. and Punie, Y. (2012). Innovating Learning: Key Elements for developing Creative Classrooms in Europe. Seville: European Commission, Joint Research Centre, Institute for Prospective Technological Studies.
8.	Bohanec, M. and Rajkovič, V. (1990). DEX: an expert system shell for decision support. Sistemica, 1, pp. 145-157.
9.	Bohanec, M. (2014). DEXi: Program for Multi-Attribute Decision Making, User's Manual, Version 4.01. Ljubljana: Jožef Stefan Institute.
10.	Brejc, M. and Zavašnik Arčnik, M. (2010). Samoevalvacija v šolah, Zasnova in uvedba sistema ugotavljanja in zagotavljanja kakovosti vzgojno-izobraževanih organizacij (vrtcev in šol) - kviz. Ljubljana: Republiški izpitni center Slovenije.
11.	Carretero, S., Vuorikari, R. and Punie, Y. (2017). DigComp 2.1: The Digital Competence Framework for Citizens with eight proficiency levels and examples of use. Seville: European Commission.
Borut Campelj, et al.: Introduction of the Hierarchical Self-assessment Model...	61
12.	Čampelj, B. and Rajkovič, V. (2008). Primer indikatorjev za (samo)evalvacijo e-kompetentnosti šole. In: Orel, M., Matjašič, S. and Kosta, M. (ur.). Zbornik Mednarodna konferenca Splet izobraževanja in raziskovanja z IKT - SIRIKT 2008. Kranjska Gora Ljubljana: ARNES, pp. 134-142.
13.	Davies, B. (2005). (Ed) Strategic leadership, Essentials of School Leadership, London: Paul Chapman.
14.	European Commission (2013a). Survey of Schools: ICT in Education. Brussels: European Commission. Retrieved 1/28/2020 from the Internet: https://ec.europa.eu/digital-agenda/sites/ digital-agenda/files/KK-31-13-401-EN-N.pdf.
15.	European Commission (2013b). Opening up Education: Innovative teaching and learning for all through new Technologies and Open Educational Resources. Brussels: European Commission. Retrieved 1/28/2020 from the Internet: http://eur-lex.europa.eu/legal-content/EN/TXT/ PDF/?uri=CELEX:52013DC0654&from=EN.
16.	European Commission (2018). Digital education action plan. Brussels: European Commission. Retrieved 1/28/2020 from the Internet: https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan_en
17.	Fraillon, J., Ainley, J., Schulz, W., Friedman, T. and Gebhardt, E. (2014). Preparing for Life in a Digital Age, The IEA International Computer and Information Literacy Study - ICILS, International Report. Amsterdam: International Association for the Evaluation of Educational Achievement (IEA).
18.	Hallerstrom, H. (2006). Rektors normer i ledarskapet for skolutveckling - Principal's Norms in Leadesrhip Practice for School Development. Lund: Lunds Universitet.
19.	Hammond, M. (2014) Introducing ICT in schools in England: Rationale and consequences. British Journal of Educational Technology, 45, No. 2, pp. 191-201.
20.	Hill, B. C. and Ekey, C. (2010). The Next-Step Guide to Enriching Classroom Environments: Rubrics and Resources for Self-Evaluation and Goal Setting for Literacy Coaches, Principals, and Teacher Study Groups, K-6. Portsmouth, NH: Heinemann.
21.	House, E. (1973). School Evaluation: The Politics and Process. San Francisco: McCutcheon.
22.	Kampylis, P., Punie, Y. and Devine, J. (2015). Institute for Prospective Technological Studies Promoting effective digital-age learning: a European framework for digitally-competent educational organisations. Luxembourg: Publications Office European Commission.
23.	Kikis - Papadakis, K., Kollias, A. and Hatzopoulos, P. (2014). Beyond Uniformity: on School Leadership Policy Development in Europe. Conference Proceedings, International Conference The Future of Education, 4th Edition, Florence, Italy, pp. 379-383.
24.	Kreuh, N. (2012). The way towards e-competency. E-Centre of the E-Education Project, No. 7.
25.	Lindsay, P. H. (1977). Human Information Processing: An Introduction to Psychology. Harcourt College Pub.
26.	Loveless, A. M. (2008). Creative learning and new technology?. In: Sefton - Green, J. (ed.). Creative learning. London: Arts Council England, pp. 61-72.
27.	MacBeath, J. (2000). Self-evaluation in European schools. A story of change. London: Routledge.
28.	Musek Lešnik, K. (2007). Kakšna je dobra samoevalvacija - What is a successful self-evaluation. Ljubljana: Inštitut za psihologijo osebnosti.
29.	Redecker, C. (2017). European Framework for the Digital Competence of Educators: DigCompEdu. Punie, Y. (ed). Brussels: European Union.
30.	Tseng, J. C. R, Chu, H. C., Hwang, G. J. and Tsai, C. C. (2008). Development of an adaptive learning systems with two sources of personalisation information. Computers & Education, 51, pp. 776-786.
31.	Triantaphyllou, E. (2000). E. Multi-criteria Desicion Making Methods. A Comparative Study, Boston: Kluwer Academic Press, pp. 5-72.
32.	Turban, E., Aronson, J. and Liang, T. P. (2004). Decision Support Systems and Intelligent System, 7th edn. New Jersey: Prentice Hall, pp. 558-601.
33.	Vanderlinde, R., Aesaert, K. and van Braak, J. (2015). Measuring ICT use and contributing conditions in primary schools: ICT use and contributing conditions. British Journal of Educational Technology, 46, No. 3, pp. 1056-1063.
62	Revija za ekonomske in poslovne vede (1, 2020)
34.	Wastiau, P., Blamire, R., Kearney, C., Quittre, V., Van de Gaer, E. and Monseur, C. (2013). The Use of ICT in Education: a survey of schools in Europe. European Journal of Education, 48, No. 1, pp. 11-27.
35.	West, R. E. and Borup, J. (2014). An analysis of a decade of research in 10 instructional design and technology journals. British Journal of Educational Technology, 45, pp. 545-556.
Dr. Borut Čampelj, predavatelj na Univerzi v Novem mestu, Fakulteti za ekonomijo in informatiko, ter raziskovalec in sekretar na Ministrstvu za izobraževanje, znanost in šport. E-naslov: borut.campelj@guest.arnes.si
Dr. Eva Jereb, redna profesorica in raziskovalka na Univerzi v Mariboru, Fakulteti za organizacijske E-naslov: eva.jereb@um.si
Dr. Uroš Rajkovič, izredni profesor in raziskovalec na Univerzi v Mariboru, Fakulteti za organizacijske E-naslov: uros.rajkovic@um.si