Revija za elementarno izobraževanje Journal of Elementary Education ISSN 2350-4803 (splet/online) ISSN 1855-4431 (tisk/print) Revija za elementarno izobraževanje Odgovorni urednik: red. prof. dr. Matjaž Duh (Univerza v Mariboru, Pedagoška fakulteta, Slovenija) Urednica za področje družboslovja: izr. prof. dr. Silva Bratož (Univerza na Primorskem, Pedagoška fakulteta, Slovenija) Urednica za področje humanistike: izr. prof. Sonja Starc (Univerza na Primorskem, Pedagoška fakulteta, Slovenija) Urednik za področje naravoslovja in doc. dr. Tomaž Bratina (Univerza v Mariboru, Pedagoška informatike: fakulteta, Slovenija) Tehnična urednika: doc. dr. Jerneja Herzog (Univerza v Mariboru, Pedagoška fakulteta, Slovenija Jan Perša (Univerza v Mariboru) MEDNARODNI UREDNIŠKI ODBOR dr. Renate Seebauer (Pädagogische Hochschule Wien, Avstrija), dr. Ligita Stramkale (Latvijas Universitate, Riga, Latvia), dr. Herbert Zoglowek (UiT The Arctic University of Norway, Troms0, Norveška), dr. Maria Aleksandrovich (Akademia Pomorska w Slupsku, Poljska), dr. Nevenka Tatkovic (Fakultet za odgojne i obrazovne znanosti, Sveučilište Jurja Dobrile u Puli, Hrvaška), dr. Grozdanka Gojkov (Učiteljski fakultet Univerziteta u Beogradu, Srbija), dr. Jelena Prtljaga (Visoka škola strukovnih studija za obrazovanje vaspitača »Mihailo Palov« Vršac, Srbija), ddr. Jürgen Kühnis (Pädagogische Hochschule Schwyz, Švica), dr. Marie Fulkova (Pedagogicka fakulta, Univerzite Karlove, Praha, Češka), dr. Vera Janikova (Pedagogicka fakulta, Masarykova univerzita, Brno, Češka), dr. Oliver Holz (Faculty of Economics and Business, KU Lueven, Belgija), dr. Ljubica Marjanovič Umek (Filozofska fakulteta, Univerza v Ljubljani, Slovenija), dr. Janez Vogrinc (Pedagoška fakulteta, Univerza v Ljubljani, Slovenija), dr. Milena Valenčič Zuljan (Pedagoška fakulteta, Univerza v Ljubljani, Slovenija), dr. Mateja Pšunder (Filozofska fakulteta, Univerza v Mariboru, Slovenija), dr. Majda Schmidt Krajnc (Pedagoška fakulteta, Univerza v Mariboru, Slovenija), dr. Sonja Rutar (Pedagoška fakulteta, Univerza na Primorskem, Slovenija), dr. Tina Štemberger (Pedagoška fakulteta, Univerza na Primorskem, Slovenija) NASLOV UREDNIŠTVA Revija za elementarno izobraževanje, Uredništvo revije Revija za elementarno izobraževanje Koroška cesta 160, SI-2000 Maribor, Slovenija, e-pošta: rei.pef@um.si, http://rei.um.si ZALOŽNIK Univerzitetna založba Univerze v Mariboru Slomškov trg 15, 2000 Maribor, Slovenija e-pošta: zalozba@um.si, http://press.um.si, http://journals.um.si Članki se referirajo v: SCOPUS (Elsevier Bibliografhic Databases), DOAJ, ERIH PLUS, EBSCO (EBSCOhostweb), Ulrich's Periodicals Directory, IBZ (Internationale Bibllographle der Zeitschriftenliteratur), Proquest, dLib.si, DKUM, COBISS (Co-operative Online Bibliographic System and Services). Članki v reviji so recenzirani. Revija za elementarno izobraževanje je revija, ki jo izdaja Univerzitetna založba Univerze v Mariboru v soizdajateljstvu Pedagoške fakultete Univerze v Mariboru, Pedagoške fakultete Univerze na Primorskem in Pedagoške fakultete Karlove Univerze v Pragi. V njej so objavljeni prispevki s področja vzgoje in izobraževanja zlasti na predšolski in osnovnošolski stopnji. Avtorji prispevkov z znanstvenega vidika pišejo o problemih, ki zadevajo vzgojo in izobraževanje. Namen revije je spodbujati objavo znanstvenoraziskovalnih člankov. Revija za elementarno izobraževanje izhaja štirikrat letno. V njej so objavljeni prispevki v slovenskem ali angleškem jeziku oz. nemškem jeziku. Prispevke pošljite na naslov uredništva ali po e-pošti na naslov: rei.pef@um.si Journal of Elementary Education Editor-in-Chief: prof. Matjaž Duh, PhD (University of Maribor, Faculty of Education, Slovenia) Editor for Social Sciences: assoc. prof. Silva Bratož, PhD (Universitiy of Primorska, Faculty of Education, Slovenia) Editor for Humanities: assoc. prof. Sonja Starc, PhD (Universitiy of Primorska, Faculty of Education, Slovenia) Editor for Nature and Information assist. prof. Tomaž Bratina, PhD (University of Maribor, Sciences: Faculty of Education, Slovenia) Technical Editors: assist. prof. Jerneja Herzog, PhD (University of Maribor, Faculty of Education, Slovenia) Jan Perša (University of Maribor) INTERNATIONAL EDITORIAL BOARD Renate Seebauer, PhD (University College of Teacher Education, Vienna, Austria), Ligita Stramkale, PhD (Faculty of Education, Psychology and Art, University of Latvia, Riga, Latvia), Herbert Zoglowek, PhD (Norwegian Arctic University, University of Tromso, Alta, Norway), Maria Aleksandrovich, PhD (Faculty of Social Science, Pomeranian University in Slupsk, Slupsk, Poland), Nevenka Tatkovic, PhD (Faculty of Educational Sciences, Juraj Dobrila University of Pula, Pula, Croatia), Grozdanka Gojkov, PhD (Teacher Education Faculty, University of Belgrade, Belgrade, Serbia), Jelena Prtljaga, PhD (Preschool Teacher Training College »Mihailo Palov«, Vršac, Serbia), Jürgen Kühnis, Phd, (The Schwyz University of Teacher Education, Goldau, Switzerland), Marie Fulkova, PhD (Faculty of Education, Charles University, Prague, Czech Republic), Vera Janikova, PhD (Faculty of Education, Masaryk University, Brno, Czech Republic), Oliver Holz, PhD (Faculty of Economics and Busines, KU Leuven, Belgium, Ljubica Marjanovič Umek, PhD (Faculty of Arts, University of Ljubljana, Ljubljana, Slovenia), Janez Vogrinc, PhD (Faculty of Education, University of Ljubljana, Ljubljana, Slovenia), Milena Valenčič Zuljan, PhD (Faculty of Education, University of Ljubljana, Ljubljana, Slovenia), Mateja Pšunder, PhD (Faculty of Arts, University of Maribor, Maribor, Slovenia), Majda Schmidt Krajnc, PhD (Faculty of Education, University of Maribor, Maribor, Slovenia), Sonja Rutar, PhD (Faculty of Education, University of Primorska, Koper, Slovenia), Tina Stemberger, PhD (Faculty of Education, University of Primorska, Koper, Slovenia) EDITORIAL OFFICE ADDRESS Journal of Elementary Education, Editorial Board of Journal of Elementary Education Koroška cesta 160, SI-2000 Maribor, Slovenija, e-pošta: rei.pef@um.si, http://rei.um.si PUBLISHED BY University of Maribor Press Slomškov trg 15, 2000 Maribor, Slovenia e-mail: zalozba@um.si, http://press.um.si, http://journals.um.si Articles appearing in this journal are abstracted and indexed in: SCOPUS (Elsevier Bibliografhic Databases), DOAJ, ERIH PLUS, EBSCO (EBSCOhostweb), Ulrich's Periodicals Directory, IBZ (Internationale Bibliographie der Zeitschriftenliteratur), Proquest, dLib.si, DKUM, COBISS (Co-operative Online Bibliographic System and Services). Journal of Elementary Education is a peer-reviewed journal, open access journal that publishes scientific articles primarly but not limited to the area of elementary school education. JEE is published four times yearly and accepts articles in Slovene, English and German. JEE is published by the University Press University of Maribor with cooperate Faculty of Education University of Maribor, Faculty of Education University of Primorska and Charles University, Faculty of Education, Prague. Articles may be sent electronically to: rei.pef@um.si Revija za elementarno izobraževanje Journal of Elementary Education Volume 12 Number 2 Junij 2019 Kazalo / Table of Contents Prispevki / Articles E-learning Materials for the Slovene Language in Elementary Schools 139 (6th-9th Grade) E-gradivo %a slovenski jezik v osnovni šoli (6. — 9. razred) Alenka Valh Lopert & Mihaela Koletnik The National Assessment of Mathematics in High Schools in Italy with 155 Slovene as the Language of Instruction Nacionalni preizkus znanja iz matematike na višjih šolah s slovenskim učnim jezikom v Italiji Daniel Doz Using Mark Chagall's Visual Art in Teaching Visual Arts 177 Uporaba umetniškega dela pri poučevanju likovni umetnosti (Marc Chagall) Dubravka Kuščevic, Marija Brajčic, Luce Sipic Students' Attitudes towards Didactic Characteristics in Graphic Design 199 Classes Stališča dijakov do didaktičnih značilnosti pri pouku grafičnega oblikovanja Martina Kač Nemanič Explaining Preferences for Illustration Style and Characteristics in Eraly 215 Childhood Razlogi za preference različnih likovnih stilov in značilnosti v zgodnjem otroštvu Marijana Županic Benic & Kristina Kalic REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE JOURNAL OF ELEMNTARY EDUCATION Vol. 12, No. 2, pp. 139-154, Junij 2019 FE1 E-learning Materials for the Slovene Language in Elementary Schools (6th-9th Grade) Potrjeno/Accepted 13. 3. 2019 Objavljeno /Published 28. 6. 2019 Keywords: the Slovene language, mother tongue, elementary school, e-learning resources, Slovenščina na dlani Ključne besede: slovenski jezik, materni jezik, osnovna šola, e-učni viri, Slovenščina na dlani UDK/UDC 811.163.6:373.3 Alenka Valh Lopert1 & Mihaela Koletnik1 1 University of Maribor, Faculty of Arts, Slovenia Korespondenčni avtor/Corresponding author alenka.valh@um. si Abstract/Povzetek The theoretical part of the paper presents the project Slovenščina na dlani (Slovene in the palm of jour hand), which is intended for the establishment of an interactive learning environment for the Slovene language (as mother tongue) in elementary and secondary schools. In this paper we limit the research to elementary school. In the empirical section, we focus on freely available e-learning materials for the Slovene language from the 6th to the 9th grade of elementary school and present the results of an analysis of 10,118 tasks according to their types. The findings indicate an unbalanced representation of different types of tasks, with short answer types in the majority, the task type that is least popular among students. By introducing the most advanced linguistic technology into learning processes, we try to overcome the limitations of existing e-resources for learning the Slovene language. E-gradivo za slovenski jezik v osnovni šoli (6. — 9. razred) V teoretičnem delu prispevka bo predstavljen projekt Slovenščina na dlani, ki je namenjen vzpostavitvi interaktivnega učnega okolja za učenje slovenščine kot materinščine v osnovnih in srednjih šolah. V tem prispevku se omejujemo na osnovno šolo. V empiričnem delu se osredotočamo na prosto dostopna e-gradiva za učenje slovenščine od 6. do 9. razreda osnovne šole in predstavljamo rezultate analize 10 118 nalog glede na njihov tip. Ugotovitve kažejo na neuravnoteženo zastopanost različnih tipov nalog, pri čemer prevladujejo naloge s kratkim odgovorom, ki so med učenci najmanj priljubljene. Z uvajanjem najsodobnejših jezikovnih tehnologij v učne procese poskušamo premagati omejitve obstoječih e-gradiv za učenje slovenščine. DOI https://doi.org/10.18690/rei. 12.2.139-154.2019 Besedilo / Text © 2019 Avtor(ji) / The Author(s) To delo je objavljeno pod licenco Creative Commons CC BY Priznanje avtorstva 4.0 Mednarodna. Uporabnikom je dovoljeno tako nekomercialno kot tudi komercialno reproduciranje, distribuiranje, dajanje v najem, javna priobčitev in predelava avtorskega dela, pod pogojem, da navedejo avtorja izvirnega dela. (https://creativecommons.org/licenses/by/4.0/). EH University of Maribor Press 140 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION Introduction The article will explore question types in existing E-leaming resources for the Slovene language as mother tongue, and will ascertain which types should be integrated—as being the most desirable, but currently under-used—into a typical textbook to maximise learning opportunities for learners. The project Slovenščina na dlani was implemented in August 2017, and its main aim is the creation of an interactive learning environment for the Slovene language in elementary and secondary schools. The creators of the project, however, have set up the project in order to overcome the limitations of existing e-resources for the Slovene language learning, so it is no longer just a transfer of the printed workbooks into an electronic form. The present article focuses on a review of existing, freely accessible materials for teaching/learning the Slovene language from the 6th to the 9th grade of elementary school and the results of an analysis of current e-learning resources according to question types. The findings show an unbalanced representation of different question types, with a dominant type that is least popular among students. In what follows, the second phase of the project, secondary school materials, will be taken into consideration, using the same criteria. Slovene experts have carried out research (O kakovosti 2018, p. 8) on the quality of Slovene textbooks, discussed the content-didactic and professional aspects of textbooks at the primary level of elementary school within various subjects, including textbooks for the Slovene language. Four criteria of assessment have been taken into consideration: (1) Thinking—a textbook promotes problem/critical/creative thinking. (2) Vocabulary, language—a textbook encourages the enrichment of the vocabulary to a sufficient extent and includes appropriate professional terminology, as well as illustrating the professional concepts with an example or an illustration, i.e. with an image, a model etc. (3) Content—the examples used in the textbook are related to everyday life, the textbook enables meaningful cross-curricular integration and deals with current scientific knowledge and topical issues. (4) Learning, knowledge—a textbook meets the requirements of an appropriate level of student development and includes an appropriate level of complexity; it enables the inclusion of the student's pre-knowledge and a pre-knowledge check; it increases the difficulty for the progress of the learners; it encourages active forms of learning, A. Vaïh Liïpert & M. Koletnik: E-ïearning Materials ^ for the Slovene 'Language in Elementary Schools (6th— 9th Grade) 141 learning by research and collaborative learning; it allows learners to reflect on their own learning, and enables self-evaluation of knowledge. According to teachers' responses, the researchers stated (O kakovosti 2018, p. 9), on the one hand, that a high percentage of textbooks for the Slovene language at the primary level of elementary school do the following: (1) stimulate problem solving (79%), critical (78%) and creative thinking (82%); (2) promote the enrichment of vocabulary (89%), give professional examples with illustrations (85%); (3) use examples from everyday life (85%), allow cross-curricular integration (82%); (89%), allow the inclusion of student's pre-knowledge (82%), allow students to check their pre-knowledge (68%), and enable student progression (70%). On the other hand, in the opinion of teachers, the following characteristics of textbooks are represented in a lower percentage: (1) textbooks promote active forms of learning (57%), (2) enable self-assessment of knowledge (50%); (3) allow students to reflect on daily learning (only 6%); and (4) discuss contemporary scientific findings (52%). The findings of the research coincide with the main guidelines for the preparation of new learning materials for the Slovene language, and we will strive to achieve the highest possible level of consideration of these goals. On the Project Slovenščina na dlani The project Slovenšščina na dlani (Gradivo za izobraževanje 2017; Internet source: http://projekt.slo-na-dlani.si/sl/) is co-financed by the Ministry of Culture of the Republic of Slovenia and by the European Union, i.e. the European Social Fund. It is carried out at the University of Maribor, including three faculties: the Faculty of Arts, the Faculty of Education and the Faculty of Electrical Engineering, Computer Science and Informatics. It will last four years, until the end of September 2021. During the project period, fourteen educational institutions (primary and secondary schools) from all over Slovenia will participate in order to improve the responsiveness of the formal education system and increase its attractiveness. 142 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION The purpose of the project is to create an innovative, interactive learning e-resource by introducing the most advanced linguistic technology into the learning processes, on the one hand. On the other hand, we seek to offer students an e-environment that is highly automated. It will provide (for certain tasks) an almost unlimited number of individualised exercises and adapt automatically to the needs of each individual user, while being interactive in the sense of automatically providing information to the user on his/her achievement and performance and automatically directing him/her among tasks, using the advantage of gaming elements from the computer environment to motivate students. The target group comprises students from the 6th to the 9th grades of elementary school and secondary school students, who can produce longer texts, but with deficits in language expression and understanding, especially in the case of non-literary texts. It is precisely these deficits that are targeted in our new learning e-resource, Slovenščina na dlani. The learning e-environment will be freely accessible and adapted for use on tablets and smartphones. It will be available to anyone, even without registration, but registration is desirable for use in schools, so that students can take full advantage of all the potential of e-environment adaptation to each individual user. The e-exercises will be divided into five content sets: (1) orthography (use of commas, capital letters, words written together and apart, etc.), (2) grammar (use of prepositions, conjunctions, pronouns, etc., difficulties in conjugation, declension and comparison), (3) idioms, proverbs, (4) texts (reading and summarizing), and (5) explanation of linguistic topics. Theoretical insight Common to all the free e-learning materials for Slovenian language in elementary school (those reviewed for this paper) from the 6th to the 9th grades is that they are designed to provide students and teachers with many (diversified) question types. They differ from each other according to how they are compiled, according to the level of student activity in solving tasks, according to predicted response, and according to demand, namely the taxonomic level. We point out—in our area, the best known—three different classifications of types of tasks that differ in the way they are sorted. A. Vaïh Liïpert & M. Koletnik: E-ïearning Materials ^ for the Slovene 'Language in Elementary Schools (6th— 9th Grade) 143 As early as 1965, the Slovenian psychologists Ivan Tolicic and Leon Zorman had classified and described the question types in their Testi %nanja in njihova uporaba v praksi (Tests of Knowledge and Their Application in Practice, 1965). They divided the question types into two groups with several subtypes, using as criteria for the division the composition of the question types and the manner in which students answer them. The first group lists question types for which the answer needs to be written, that is, question types requiring the reproduction of learning material (complement question types and the short answer). In the second group, there are question types in which the answer should be selected (optional type/with subtypes), the linking and editing type, and the alternative type (with subtypes) (Tolicic, Zorman 1965, pp. 26—57). They also mention question types with free answers that require a written response but do not classify these in any group. In 1968 (pp. 81—84), Zorman also wrote about interpretational question types, which are more demanding and complex on the level of content and form in comparison to other question types. These are suitable for identifying complex educational components (e.g. knowledge of cause-effect relationships or the ability to develop hypotheses), but among pupils with less reading ability, these are very unpopular, as they usually contain a great deal of text. In her paper Pisno preverjanje in ocenjevanje pri pouku slovenskega je^ika v tretjem triletju osnovne sole (Written examination and evaluation in Slovene language classes in the third trimester of elementary school, 2008), Slavist Tamara Vomer divides the question types into two groups: In the first group, they are classified according to the level of understanding of the text—i.e. factual question types, in which the individual is expected to copy the answers literally from the given text; concluding question types, which require the individual to conclude on an answer that is not literally given in the text, and the critical/creative/applicable question types, requiring a certain pre-knowledge. In the second group, she classifies question types according to the learner's activity-question types of a closed or objective type, requiring the individual to choose an answer from the multiple answers given or specify a specific answer; question types of a semi-open or partially objective type, which the individual needs to answer or solve independently, with the type of answer being specified by the instructions, or question types that are composed of several different question types or subquestions that are not necessarily interdependent: question types of an open or subjective type (all with several subtypes), which are characterized by an individual showing his/her own ideas and creativity or giving his/her opinion (Vomer 2008, pp. 34, 35). 144 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION In 2002, psychologists Barica Marentič Požamik and Cirila Peklaj divided assignments into two groups in the Preverjanje in ocenjevanje %a uspešnejši študij (Verification and Assessment for More Successful Study). The first ones are essay assignments, i.e. question types in which an individual responds freely, with a longer answer (a discussion essay, an essay with an element of role-play), and the second group comprises objective question types, which are further subdivided into open-type tasks (question types with short answers and complement question types) and closed-type questions (question types of the optional and alternative type, and the question type of connecting and editing) (Marentič Požarnik, Peklaj 2002, pp. 73, 74, 87). In her Master's thesis (2017), the Slavist Maja Hadner concludes that, despite these classifications, some question types cannot be subsumed under the given types. Therefore, she upgraded existing classifications and question types according to the activity they require from the individual. The new, extended classification thus encompasses 21 question types, some types from existing classifications being also renamed. Existing E-Learning Resources for Learning the Slovene Language in Elementary School The following section presents materials for learning the Slovene language in elementary schools from the 6th to the 9th grades—freely accessible on the Internet (Internet source: http://projekt.slo-na-dlani.si/sl/). The study took place in 2017, from June until December. We were interested in question types and exercises, considering non-literary texts and covering the following topics: the Slovene language in general, the historical development of the Slovene language and its contemporary situation as well as its social and functional varieties, phonetics, orthography, lexicology, word classes, sentence elements and complex sentences etc. The analysis includes the following textbooks, workbooks and additional interactive materials: 1: Gradim slovenski je^ik 6 — dodatno gradivo k učbeniku (Additional material to the Textbook, 6th grade); 2: Gradim slovenski je^ik 6 — dodatno gradivo k delovnemu zvezku (Additional material to the Workbook; 6th grade); 3: Gradim slovenski je^ik 6 — dodatno gradivo k samostojnemu delovnemu zvezku (Additional material to the autonomous Workbook; 6th grade); A. Vaïh Liïpert & M. Koletnik: E-ïearning Materials ^ for the Slovene 'Language in Elementary Schools (6th— 9th Grade) 145 4: Slovenščina v oblaku: interaktivno gradivo za slovenščino v 6. razredu osnovne šole (Interactive material; 6th grade); 5: Od glasov do knjižnih svetov 6 (6th grade); 6: Slovenščina, Vaje za 6.-9. razred (Exercises: 6th—9th grade); 7: Devetka.net — Zbirka spletnih nalog (Collection of interactive tasks); 8: E-učbeniki SIO.si: 8. razred (8th grade); 9: Projekt slovenščina za 8. razred — Neumetnostna besedila (Part: Non-literary texts: 8th grade); 10: E-učbeniki SIO.si: 9. razred (9th grade). Methodology E-resources were reviewed and analysed according to the question types formulated and completed by Hadner (2017, pp. 25—33; terminology and typology of question types were also adjusted according to Internet source: https://docs.moodle.org/34/en/Category:Questions) that occur in all materials, i.e. textbooks, workbooks and additional interactive materials. Individual question types were checked and counted, and then percentages of all question types in the analysed materials were calculated. Within the analysis, 21 different question types were examined: 1 Fill in the blank questions—this question type requires the student to complete or finish a statement by writing in the missing words, phrases or short clause. 2 Short answer questions—this question type requires only a brief/short response, i.e. with a word or phrase; no reply with a sentence is necessary. 3 Multiple choice questions (one correct answer)—this question type requires only one correct answer from the answers offered. 4 Multiple answer questions (two correct answers)—this question type requires two correct answers from the answers offered. 5 Multiple answer questions (answers where several are correct—this question type requires several choices, i.e. at least three answers from the answers offered. 6 Multiple choice questions with varying degrees of correctness—the individual is expected choose the most appropriate/correct from the offered answers. 7 Multiple choice questions with an incorrect answer—the individual is expected to identify the wrong answer from the answers offered. 8 Multiple choice questions with the best answer—this question type offers several correct answers, the best of which should be chosen. This question type is very similar to the optional type of question with varying degrees of correctness. 9 Matching questions—this question type requires connecting individual elements to meaningful relationships. 10 Editing questions—this question type requires organization of given elements by a certain criterion. 11 Alternative questions—this question type requires an answer or a definition with one of the two responses offered (for example, true/false, yes/no). 12 Alternative questions with argumentation— this question type is composed of an alternative question type, and in addition, it 146 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION requires a justification for the selected answer. 13 Open questions—this question type requires a longer answer formed in the student's own words. 14 Oral responses—this question type requires a response that does not need to be written. 15 Gap-filling-this question type requires a specific element to be inserted into the word or text (for example, missing letters or punctuation marks). 16 Mark correct answer (of the multiple-choice answers)—this question type requires that something be marked, underlined, circled or coloured in. 17 Error Correction—this question type requires that the incorrectly written word, text or incorrectly written punctuation be corrected. 18 Ordering questions—this question type requires that the given elements be classified or distributed according to a given criterion. 19 Reading—this question type requires that the word or text be read carefully. 20 A type including in-class activities—this question type requires classroom activities among students. 21 A type involving other resources—this question type provides for the use of other materials and usually relates to a particular topic or substance and seeks to bring it closer to a student's real-life situation. (Hadner 2017, pp. 25-33). The total number of question types analysed within the e-resources was 10.118. Thus, we sought to obtain data on the representation of individual question types in e-materials and compare the results with those question types that turned out to be the most common in Slovene language learning. Question Type Analysis of Exercises in Existing E-Learning Resources for Learning the Slovene language The tables present a set of tasks that represent at least 70.0% of all tasks in individual e-resources. Tasks representing less than 4.0% of tasks are not representative and therefore not included in the tables. Table 1: Gradim slovenski jezik 6 (additional material to the Textbook; 6th grade). Question types Number of tasks Percentages (%) Short answer questions 36 46.8% Fill-in-the-blank questions 12 15.6% Multiple choice questions 10 10.4% Gap-filling 5 6.5% Alternative questions 4 5.2% Total: 77 tasks. The most common question types appear in 84.5% of all tasks, while other question types appear in 3.9% or less. A. Vaïh Liïpert & M. Koletnik: E-ïearning Materials ^ for the Slovene 'Language in Elementary Schools (6th— 9th Grade) 147 Table 2: Gradim slovenski jezik 6 (additional material to the Workbook; 6th grade). Question types Number of tasks Percentages (%) Short answer questions 97 60.3% Fill-in-the-blank questions 13 8.1% Open questions 10 6.6% Multiple choice questions 8 5.0% Total: 160 tasks. The most common question types appear in 80.0% of all tasks, while other question types appear in 3.8% or less. Table 3: Gradim slovenski jezik 6 (additional material to autonomous Workbook; 6th grade). Question types Number of tasks Percentages (%) Short answer questions 111 59.0% Alternative questions 20 10.6% Multiple choice questions 14 7.5% Open questions 13 6.9% Total: 188 tasks. The most common question types appear in 84.0% of all tasks, while other question types appear in 3.2% or less. Table 4: Slovenščina v oblaku: (Interactive material; 6th grade). Question types Number of tasks Percentages (%) Multiple choice questions 10 14.7% Mark correct answer 9 13.2% Open questions 8 11.8% Short answer questions 7 10.3% Reading 5 7.4% Ordering questions 5 7.4% Fill-in-the-blank questions 4 5.9% Multiple answer questions (several correct answers) 4 5.9% Total: 68 tasks. The most common question types appear in 75.9% of all tasks, while other question types appear in 4.4% or less. Table 5: Od glasov do knjižnih svetov 6. Question types Number of tasks Percentages (%) Short answer questions 27 44.3% An oral response 21 34.4% Alternative questions 4 6.6% Fill-in-the-blank questions 3 4.9% 148 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION Total: 61 tasks. The most common question types appear in 90.2% of all tasks, while other question types appear in 3.3% or less. Table 6: Slovenščina, Vaje za 6.-9. razred (Exercises, 6th—9th grade). Question types Number of tasks Percentages (%) Short answer questions 567 36.2% A classroom activity—listening 398 25.4% Alternative questions 344 22.0% Multiple choice questions 130 8.3% Total: 1566 tasks. The most common question types appear in while other question types appear in 2.0% or less. 91.9% of all tasks. Table 7: Devetka.net (Collection of interactive tasks). Question types Number of tasks Percentages (%) Multiple choice questions 1003 35.7% Short answer questions 564 20.1% Alternative questions 404 14.4% Fill-in-the-blank questions 196 7.0% Matching questions 162 5.8% Total: 2807 tasks. The most common question types appear in while other question types appear in 3.1% or less. 83.0% of all tasks. Table 8: E-ucbeniki SlO.si: 8. razred (8th grade). Question types Number of tasks Percentages (%) Short answer questions 600 31.3% Multiple choice questions 290 15.1% Alternative questions 196 10.2% Mark correct answer 141 7.4% Reading 123 6.4% Open questions 104 5.4% Fill-in-the-blank questions 89 4.6% Total: 1919 tasks. The most common question types appear in 80.4% of all tasks, while other question types appear in 3.1% or less. A. Vaïh Liïpert & M. Koletnik: E-ïearning Materials ^ for the Slovene 'Language in Elementary Schools (6th— 9th Grade) 149 Table 9: Projekt slovenščina za 8. razred (Part: Non-literary texts; (8th grade)). Question types Number of tasks Percentages (%) Short answer questions 239 31.9% Open questions 84 11.2% Reading 77 10.3% Alternative questions 76 10.2% A type involving other resources 58 7.7% Multiple choice questions 49 6.5% Alternative questions with argumentation 32 4.3% Total: 749 tasks. The most common question types appear in 82.1% of all tasks, while other question types appear in 3.5% or less. Table 10: E-ucbeniki SlO.si: 9. razred (9th grade). Question types_Number of tasks Percentages (%) Short answer questions 860 34.1% Multiple choice questions 269 10.7% Reading 243 9.6% Mark correct answer 232 9.2% Alternative questions 212 8.4% Fill-in-the-blank questions 142 5.6% Open questions 138 5.5% Total: 2523 tasks. The most common question types appear in 83.1% of all tasks, while other question types appear in 3.0% or less. Interpretation of Question Type Analysis It can be noted that simpler question types are repeated very often, while those that are more demanding, and require more complicated mental processes, are rare. Simpler question types are—unquestionably—necessary for accustoming students to thinking, but too frequent usage of these types does not stimulate in-depth thinking and leads to a decline in motivation to answer this type of question. During the process of analysis, particularly while determining the question types, we discovered a gap in the classification of question types, which we want—as much as possible— to fill with expanded question types. 150 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION The review covered ten sets of e-resources, and 10.118 tasks were analysed: (1) in the additional material (77 tasks) to the textbook Gradim slovenski je^ik 6 the Short answer questions type is presented in 46.8% of tasks (36 tasks); (2) in the additional material (160 tasks) to the workbook for the same text-book the Short answer question type prevails, with 60.3% of tasks (97 tasks); (3) similarly, in the additional material (188 tasks) to the autonomous workbook for the same text-book, i.e. 59.0% (111 tasks); (4) in the interactive material (68 tasks) for the text-book Slovenščina v oblaku for 6th grade, Multiple choice questions amounted to 14.7% (10 tasks); (5) in the text-book Od glasov do knjižnih svetov 6 (61 tasks), the Short answer question type was present in 44.3% (27 tasks); (6) among the exercises for the text-book Slovenščina (1566 tasks), the Short answer question type occupies 36.2% (567 tasks); (7) the collection of interactive tasks (2807 tasks) accompanying the text-book Devetka.net includes 35.7% (1003) of the Multiple-choice question type; (8) in the e-text-book (1919 tasks) E-učbeniki SIO.si for the 8th grade, 31.3% of tasks (600 tasks) comprise the Short answer question type; (9) in the Non-literary texts section (749 tasks) of the text-book Projekt slovenščina for 8th grade, the Short answer question type prevails again at 31.9% (239), as well as in (10) the e-text-book E-učbeniki SIO.si for the 9th grade, in 34.1% (860 tasks). The results of the e-resources analysis reveal a shortage of the following question types: Gap-filling, Mark correct answer of the multiple-choice answers, Error Correction, Ordering questions, Alternative questions with argumentation, Editing questions, Multiple answer questions (two correct answers), Multiple answer questions (several correct answers), and Multiple choice questions with an incorrect answer. Since all these are question types that are more than suitable for e-environments, and they also enable the realization and achievement of higher taxonomic levels (use, analysis and synthesis), both the results of the analysis and the teachers' perceptions will help us to design appropriate e-resources within the framework of the project Slovenščina na dlani. Bernik (2011, p. 209) emphasizes, according to all the analyses carried on Slovene and foreign textbooks, in his case, for history, that "/.../ an ideal textbook cannot be made, because students have different styles of learning, and teachers have different teaching methods, so they should have different methodologically designed textbooks /.../''. This is undoubtedly true for textbooks of the Slovene language as well. That is why, when composing e-materials and tasks, we will—as far A. Vaïh Liïpert & M. Koletnik: E-ïearning Materials ^ for the Slovene 'Language in Elementary Schools (6th— 9th Grade) 151 as possible—take into account different types of perception, which is divided into five senses: visual perception, auditory (hearing) perception, kinaesthetic (movement, touch) perception, olfactory (smell) perception and gustatory (taste) perception (known as the VAKOG model). Depending on the different types, we could also suggest words (verbs for instance), used in speech patterns within the instructions, while remaining fully aware of the different effect of a given message on different groups of recipients. Conclusion The results show that, in the majority of e-resources, the type Short answer questions is most often represented, i.e. in 30.72% of cases (3108 tasks; in individual sources 10.3%—60.3%), followed by Multiple choice questions in 35.7% (1783 tasks; in individual sources 5.0%—35.7%) and Alternative questions in 10.52% (1064 tasks; in individual sources 5.2%-22.0%). In extreme contrast to the most frequent question type (Short answer questions) present in e-resources, student answers to the questionnaire about their most/least liked question types revealed that the Short answer question type is the least popular among the elementary school population (Internet source: http://projekt.slo-na-dlani.si/sl/). Considering the typology of questions comprises 21 different types, it is reasonable to follow the guidelines in the preparation of e-resources in the future, which will focus on a more balanced representation of all question types. Some very good suggestions about question types for textbook writing have been offered by experts in the analysis of history textbooks (Bernik 2011, pp. 208—9), some of which could be used in preparing the Slovenian language learning material. Let us support them with concrete proposals related to the Slovenian language: content in the form of comics (exercises for interjections), instructions for project work (mini projects: search for linguistic—spelling/morphological—mistakes in online editions of newspapers, in e-mails, advertisements etc.), work with pictorial materials (to write down all the nouns in the picture, adjectives used with—selected—nouns, verbs-activities of people/animals in the picture etc.), blank maps (map of Slovenia—to enter multi-word geographical names with respect to capital letters etc,), a glossary of terms (a glossary of unknown/new words that can be supplemented by students themselves), social games (dominoes for word-formation, phraseology etc.), Internet addresses (search for language manuals, language counselling etc.). 152 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION References Bernik, V. (2011). Metodološka struktura izbranih slovenskih in tujih osnovnošolskih učbenikov zgpdovine. In: D. Trškan (Ed.), Žrojarjev opornik, pp. 183—210. Ljubljana: Znanstvena založba Filozofske fakultete (Historia: znanstvena zbirka Oddelka za zgodovino Filozofske fakultete v Ljubljani; 19). Retrieved from https://hdl.handle.net/11686/file17922 (Accessed 10th February 2019.) Hadner, M. (2017): Pregled, primerjava in analiza delovnih zvezkov zap°uk slovenskega jezika v srednjih šolah. Master's thesis. Maribor: Filozofska fakulteta, Oddelek za slovanske jezike in književnosti. Marentič Požarnik, B., Peklaj, C. (2002): Preverjanje in ocenjevanje za uspešnejši študij. Ljubljana: Center za pedagoško izobraževanje Filozofske fakultete. Koletnik, M., Valh Lopert, A. (2018): Tipologija nalog v aktualnih e-okoljih za slovenščino v OŠ. In: N. Ulčnik (Ed.), Slovenščina na dlani 1. Zbornik. [E-source]. Maribor: Univerzitetna založba Univerze. Retrieved from http://press.um.si/index.php/ump/catalog/book/341 (Accessed 10th February 2019.) pp. 55-59. O kakovosti slovenskih učbenikov. Mnenja učiteljev. Prvo poročilo. (2018). (Za kakovost slovenskih učbenikov — KaUč). Retrieved from http://kauc.splet.arnes.si/files/2018/11/0_kakovo— sti_slovenskih_ucbenikov_mnenja_uciteljev_2018.pdf (Accessed 10th February 2019.) Pulko, S., Zemljak Jontes, M. (2013): E-izobraževanje kot kombinirano izobraževanje pri pouku slovenščine — E-learning as blended learning in the teaching of Slovenian. In: M. Orel (Ed.), Sodobni pristopi poučevanja prihajajočih generacij [zbornik referatov] — Modern approaches to teaching the coming generation [conference proceedings]. 743—755. Polhov Gradec: Eduvision. Retrieved from http://e— duvision.si/Content/Docs/Zbornik%20prispevkov%20EDUvision%202013_splet.pdf. (Accessed 10th February 2019.) Slovenščina na dlani. Analiza trenutno aktualnih e-okolij za poučevanje jezikov (primarno e-okolij za slovenščino) (vsebinsko poročilo junij 2017-december 2017). Pulko, S., Zemljak Jontes, S., Koletnik, M., Šek Mertuk, P., Valh Lopert, A. Retrieved from http://projekt.slo-na-dlani.si/sl/ (Accessed 10th Sept. 2018) Slovenščina na dlani 1 [E-source]. Zbornik, 2018. (Ed.) Ulčnik, N. Maribor: Univerzitetna založba Univerze. Retrieved from http://press.um.si/index.php/ump/catalog/book/341 (Accessed 10th February 2019.) Toličič, I., Zorman, L. (1965): Testi znanja in njihova uporaba v praksi. Ljubljana: Zveza delavskih univerz Slovenije. Valh Lopert, A. (2017): Lektoriranje kot učinkovita metoda učenja maternega jezika (za študente nesloveniste). In: M. Jesenšek (Ed.), Med didaktiko slovenskegajezika in poezijo: ob 80-letnici Jožeta Lipnika = When Slovene language didactics meets poetry: Jože Lipnik 's 80th anniversary. pp. 76—93. Maribor: Univerzitetna založba. Verdonik, D., Krajnc Ivič, M., Ulčnik, N. (2017): Gradivo za izobraževanje učiteljic in učiteljev slovenščine v okviru projekta Slovenščina na dlani. Maribor: [s. n.] Vomer, T. (2008): Pisno preverjanje in ocenjevanje pri pouku slovenskega jezika v tretjem triletju osnovne šole. Preverjanje in ocenjevanje: specializirana strokovna pedagoška revija, 5(2/3), pp. 11—38. Zorman, L. (1968): Preverjanje in ocenjevanje ter opazovanje učencev v šoli. Ljubljana: DZS. Internet sources Projekt slovenščina za 8. razred OS (8th grade): E-gradiva, slovenščina, Projekt slovenščina za 8. razred OŠ: http://www.ssers.mb.edus.si/grad— iva/w3/slo8/000_mapa/index.html Neumetnostna besedila. A. Vaïh Liïpert & M. Koletnik: E-ïearning Materials ^ for the Slovene 'Language in Elementary Schools (6th— 9th Grade) 153 Slovenščina v oblaku (6th grade): Slovenščina v oblaku 6: http://www.devetletka.net/gradiva/slovenscina/6; Dodatna gradiva: https://mauthor.rokus.com/svo6/demo/. E-učbeniki SlO.si: 8th grade: http://eucbeniki.sio.si/slo8/index.html; 9th grade: http://eucbeniki.sio.si/slo9/index.html. Gradim slovenski jezik 6: Gradim slovenski jezik 6 (6th grade): http://www.devetletka.net/index.ph— p?r=downloadMaterial&id=2421&file=1 Od glasov do knjižnih svetov 6 (6th grade): http://www.devetletka.net/gradiva/slovenscina/6 Slovenščina (vaje 6.-9. razred) (6th—9th grade): http://www2.arnes.si/~osljjk6/slovenscina_ps/sl— ovenscina_6_9r.htm. Zbirka spletnih nalog Devetka.net: http://devetka.net/index.php?r=1&p=36&t=794&n=1— &sid=fe13-8b19ba11d098aaf4f4ddade8dd9a http:/ /projekt. slo-na-dlani. si/sl/ https://docs.moodle.org/34/en/Category:Questions Authors Alenka Valh Lopert, PhD Associate professor, University of Maribor, Faculty of Arts, Koroška cesta 160, 2000 Maribor, Slovenia, e-mail: alenka.valh@um.si Izredna profesorica, Univerza v Mariboru, Filozofska fakulteta, Koroška cesta 160, 2000 Maribor, e-pošta: alenka.valh@um.si Mihaela Koletnik, PhD Full professor, University of Maribor, Faculty of Arts, Koroška cesta 160, 2000 Maribor, Slovenia, e-mail: mihaela.koletnik@um.si Redna profesorica, Univerza v Mariboru, Filozofska fakulteta, Koroška cesta 160, 2000 Maribor, e-pošta: mihaela.koletnik@um.si 154_REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE JOURNAL OF ELEMNTARY EDUCATION Vol. 12, No. 2, pp. 155-176, Junij 2019 The National Assessment of Mathematics in High Schools in Italy with Slovene as the Language of Instruction Daniel Doz Liceo Scientifico Statale »France Prešeren« with Slovene teaching language Korespondenčni avtor/Cokresponding author doz_daniel@yahoo.it Povzetek/ Abstract In the following paper, we analyzed the question types in the INVALSI national mathematics assessments in Italian high schools with Slovene as the language of instruction. Through a statistical analysis, we found that closed-type questions were more frequent than open-type questions. A greater presence of closed-type questions could lead to the issue of guessing and cheating. Moreover, a greater quantity of closed-type questions could lead to a partial evaluation of knowledge, since procedures and other mathematical competences are not considered. In our research, we also considered the topics of the questions that were presented in national assessments. Nacionalni preizkus znanja iz matematike na višjih šolah s slovenskim učnim jezikom v Italiji V prispevku predstavljamo tipologijo vprašanj v vsedržavnem preverjanju znanja iz matematike INVALSI. S pomočjo statistične analize lahko ugotovimo, da so vprašanja zaprtega tipa pogostejša od vprašanj odprtega tipa. Prisotnost večjega števila vprašanj zaprtega tipa lahko privede do problema ugibanja in prepisovanja. Poleg tega pa lahko večje število vprašanj zaprtega tipa privede to nepopolnega preverjanja znanja, saj se pri teh vprašanjih ne oceni postopkov in drugih matematičnih kompetenc. Potrjeno/Accepted 27. 2. 2019 Objavljeno /Published 28. 6. 2019 Keywords: mathematics, Slovene education in Italy, national examinations, testing Ključne besede: matematika, slovensko šolstvo v Italiji, nacionalni preizkus znanja, testiranje UDK/UDC 37.091.276:51(450) DOI https://doi.org/10.18690/rei.12.2.155-176.2019 Besedilo / Text © 2019 Avtor(ji) / The Author(s) To delo je objavljeno pod licenco Creative Commons CC BY Priznanje avtorstva 4.0 Mednarodna. Uporabnikom je dovoljeno tako nekomercialno kot tudi komercialno reproduciranje, distribuiranje, dajanje v najem, javna priobčitev in predelava avtorskega dela, pod pogojem, da navedejo avtorja izvirnega dela. (https://creativecommons.org/licenses/by/4.0/). ■il^B lEHil University of Maribor Press 156 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION Introduction National assessments have the purpose of evaluating learning outcomes, which are based on criteria and expectations set by various national education authorities (Benavot & Tanner, 2008). These assessments should describe the level of achievement and competences of the education system in its totality, or a specific part of it, such as 11-year-old pupils (Kellaghan & Greaney, 2001). National assessments also play an important role in providing national policymakers with objective information about the status of the education system (Benavot & Tanner, 2008). Since they are typically subject-oriented (mathematics, language, science etc.), and since they evaluate a particular grade level, they are an important diagnostic tool, used to identify which areas in the school systems need more attention (ibid.). Discussions regarding standardized tests have long been part of the overall concern in relation to their ambiguity and validity (Powell & Gillespie, 1990). Despite this, the spread and growth of such tests has not diminished in recent years. On the contrary, such tests are still very much in use, and the growing concern is that they are no longer a true representation of a student's knowledge, but focus instead only on the techniques needed to pass the tests. In fact, as Roberts (2006) suggests, learning has been focused on the assessments themselves, on "passing tests", as it were. Assessments are usually divided into two main categories: formative and summative (Garrison & Ehringhaus, 2010). Summative assessments are periodic tests of student knowledge. Their aim is to measure students' knowledge and acquisition of standard curricular content. They are also tools to evaluate the effectiveness of curricula and school programs (Leung, Leung & Zuo, 2014). Examples of such summative assessments, as Garrison & Ehringhaus (2010) explain, include state assessments, end-of-unit or chapter tests, semester exams and other types of tests. According to the authors, this kind of assessment is helpful for the school and teacher, specifically when they need to make instructional adjustments and interventions. Formative assessments, on the other hand, can be used in classroom practice as a tool to understand how to adjust both teaching and learning. This kind of assessment can be formal or informal; in both, it is nevertheless important that the teacher give students feedback that shows the presence of a 'gap' between the level of the assessed work and the required standard (Taras, 2005). D. Dor: The 'National Assessment of Mathematics in High Schools in Italy with Slovene as the 'Language of Instruction 157 The Italian national examination of knowledge, INVALSI, is a summative assessment. Since 2010, grade 10 students in Italian high schools are supposed to take an examination in both mathematics and the Italian language (Italian is replaced by Slovene in schools with Slovene as the language of instruction). The questions in these assessments can be either open- or closed-type. There are several reasons for preferring one type to the other: for example, closed-type questions are easier to grade; on the other hand, open-type questions do evaluate students' knowledge and abilities in a more complete way. When students answer closed-type questions, there is a greater chance of guessing, since in the INVALSI examinations, there are no penalties for answering a question incorrectly. In order to establish which question type (open- or closed-type) was more frequent in the INVALSI assessment of knowledge of mathematics over the last 8 years, we decided to conduct a study based on seven INVALSI mathematics assessments for grade 10 students in Italian high schools. We were additionally interested in finding out which mathematical topic was more common in the INVALSI national assessments of mathematical competence. In particular, we wanted to understand whether there had been a shift in interest over the last three years from more theoretical topics (functions, equations, radicals etc.), to more applied topics (statistics, data representation, modeling with linear functions etc.). If some mathematical topics were being neglected by the national assessment of knowledge, the issue could emerge of more marginal students' knowledge of certain mathematical content, which could lead to mathematical illiteracy and to the impossibility of a deeper understanding of mathematical topics. In fact, teachers that base their programs on assessment of knowledge could omit some of the more theoretical content and concentrate only on those mathematical topics that are somehow "hot" and statistically more common in the examinations. Theoretical Framework Which evaluation? In education, evaluation is necessary in order to gauge what students understand and what they can do (Kartianom & Mardapi, 2017). National evaluations are done to measure the level of knowledge and competence of students, as well as to diagnose the status of the school system (Bansilal, 2017; Kartianom & Mardapi, 2017), and the national competences and achievements in specific subjects (Sulistyaningsih & Sugiman, 2016). National assessments, which are equal for the whole population, guarantee objective information about student knowledge (Cankar, 2008). 158 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION Assessment practices are rapidly transforming, since today we use more open-ended problems, hands-on problems, essays and information technology (such as computer simulations of real-world problems) (Linn, Baker & Dunbar, 1991; Stecher et al., 1998). On the other hand, some national assessments also include closed-type questions, such as multiple-choice questions, as can be seen in the INVALSI examination (Quadro di Riferimento, 2017). The INVALSI assessments in mathematics do not take into account open-ended problems, or essays. There are no hands-on problems that might be evaluated just by solving the INVALSI assessment of knowledge. Hence, if we compare the structure of the INVALSI examinations with the ideas present in Linn, Baker & Dunbar (1991) and in Stecher et al. (1998), we could conclude that the INVALSI national assessments do not evaluate students' abilities, knowledge and competences in the ways described by the authors, since they diagnose only the student's ability to solve a certain type of problem. As stated by Cankar (2008), some national examinations assess only the cognitive achievements of students, and only in specific subjects. In the case of the Italian assessment, the INVALSI examinations evaluate only student knowledge of mathematics and languages. Nevertheless, national assessments of mathematics might help teachers and educators to modify, and hence improve, their teaching (Felda, 2018). Moreover, national assessments are useful in identifying the strong and weak points of teaching mathematics, and they help to monitor the developing factors in education, such as lesson programs, textbooks and teachers' learning and training (Magajna & Zakelj, 2011; Zakelj, Ivanus Grmek, Cankar, 2012). Parveva, De Coster & Noorani (2009) stated that national assessments can be divided into three groups: - in the first group we consider all the national assessments that have the goal of grading students' knowledge: the results of national assessments also have an impact on the grades of individual students. This kind of assessment also has an impact on the future choices of students, determining, for example, which school a student can attend; - in the second group, there are national assessments that are done mainly to assess the school system and identify reforms regarding schooling; D. Dor: The 'National Assessment of Mathematics in High Schools in Italy with Slovene as the 'Language of Instruction 159 - in the third group, we consider those assessments which are a supplement to teaching processes. Based on the results of the examinations, educators and schools can identify their weeks points and teaching needs, while seeking improvements. In this category, we could place the national assessments in Italy. Some authors are concerned by the fact that students who are not motivated perform worse in national and international assessments than those who are motivated (O'Neil Jr., Sugrue & Baker, 2010). In this respect, Cankar (2008) states that national assessments may seem to many a waste of time, since they are "not for a grade". In Italy, the national assessment of knowledge does contribute partly to the final grade in grade 9 (INVALSI, 2010), but with the new school reform, the INVALSI examinations have only a diagnostic function and not a grading one (Studenti, 2019a, 2019b). In grades 10 and 13, the INVALSI examinations are obligatory, but they do not influence the final grading of the student (Studenti, 2019; Studentville, 2019). Considering these affirmations and the conclusions of Cankar (2008), we can observe that many students are demotivated while sitting the INVALSI assessment tests, which is congruent with what was stated by the Italian educator Daniele Novara (Repubblica, 2018). Open- and closed-type questions Stankous (2016) points out that the issue of measuring student performance has often been at the center of several debates. In order to evaluate students' knowledge, teachers can prepare tests with different typologies of questions: open (constructed-response) and closed (selected-response) questions. In open questions, students must construct their own responses, organizing and applying their knowledge (Powell & Gillespie, 1990). Preparing such questions is easy, while grading them seems to be much more difficult, since clear criteria and scoring tables are difficult to prepare (ibid.). In these types of questions, there is less chance of students answering by guessing. In closed questions, on the other hand, students need to select the answers among various alternatives. Grading such questions is much easier and faster, but their preparation is time consuming. Guessing in closed-type questions is also an important issue (Klufa, 2015). Students, who face a multiple-choice question with 4 options, have a 25% chance of guessing the correct answer. If there is no penalty for a wrong answer, students are more likely to guess a correct answer, in order to collect more points (Espinosa & Gardeazabal, 2010). 160 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION National assessments can have both open- and closed-type questions. Scoring assessments with multiple-choice questions is cheaper than those with judgement-based tasks, but the gains to student learning are greater (Wiggins, 1990). Similarly, Stankous (2016) has shown that constructive-response tests do more to encourage student learning than multiple-choice tests. The author affirms that open-type tests are more reliable and valid than closed-type tests and that student success cannot be measured by multiple-choice tests alone. In her research, the author found that many teachers want their students to recognize questions and question types, memorizing the correct answers, so that they can "meet certain educational performance standards" (ibid., p. 315), forgetting this information when the tests are over. Conversely, Roberts (2006) thinks that multiple-choice tests could be used to enhance the learning process. On the other hand, students might leave the multiple-choice testing having assimilated false or incorrect knowledge, (Roediger III & Marsh, 2005). We would like to present an example of the application of concerns voiced by Stankous (2016) about mere memorization, an example taken from the INVALSI examinations. Example 1: Example of a multiple-choice question form the INVALSI examinations. School year 2010-11 Term 1037 + 1038 it's the same: A. □ 2075 B. □ 1075 C. □ 11 • 1037 D. □ 1037'38 School year 2011-12 Term a37 + a38 it's the same: A. □ 2a75 B. □ a75 C. □ a37 (a + 1) D. □ a37'38 School year 2014-15 Term a43 + a44 it's the same: A. □ a44'43 B. □ a43 (a + 1) C. □ a87 D. □ 2a87 In Example 1, we note that the structure (or the "Question type", Stankous, 2016) is similar, and the answers are constructed on the same basis; the numbers involved are also often repeated. Students could just "recall" the answer form previous exercises and "drills", forgetting how to get to the correct answer. D. Dor: The 'National Assessment of Mathematics in High Schools in Italy with Slovene as the 'Language of Instruction 161 Topics in Mathematics Assessment Mathematics literacy is defined by the OECD (2018, p. 51) as follows: "an individual's capacity to formulate, employ and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to recognise the role that mathematics plays in the world and to make the well-founded judgments and decisions needed by constructive, engaged and reflective citizens". Following this basic concept, there are four categories of mathematical knowledge and the mathematics program that are assessed in the PISA and PISA-D. These are (OECD, 2018): • change and relationships; • space and shape; • quantity; • uncertainty and data. The OECD document (ibid.) states that the proposed problems should be challenging and based on real situations. The categorization of the content into the four presented categories is important for the development and selection of items, but some problems might be transversal and would thus fit into more than one category: for example "space and shape", "change and relationships" and even "quantity". The interdisciplinarity of the proposed problems is also important, in order to underline the coherence of mathematics as a discipline (ibid.). The situation in Italy: the national INVALSI examinations Italian schooling comprises five school levels: kindergarten, primary school (five years, from level 1 to 5), middle school (three years, from level 6 to level 8), high school (five years, from level 9 to level 14), and university. After the third year of middle school, students are supposed to pass a state exam, in order to proceed to a high school; after the fifth year of high school, students must sit the "Tsame di stato" (State Exam) to get their diploma. The National System for Evaluation (Sistema Narionale di Valuta%ione) works inside the National Institute for the Educational Evaluation of Instruction and Training (INVALSI). The INVALSI also works under the supervision of the Ministry for Education, University and Research (Ministero deU1stru%ione, Universita e Ricerca, 162 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION MIUR); its aim is to investigate and periodically assess student knowledge on the whole Italian national territory, as proposed in the decree D.Lgs. n. 286/2004. In the directive of the MIUR 76/2009, INVALSI must assess the level of knowledge among primary school pupils, as well as middle school and high school students. Student knowledge is measured through standardized tests of Italian language and mathematics (Martignone, 2016; Quadro di Riferimento, 2017). As written in the Quadro di Riferimento (2017), these tests are currently applied in the second and fifth years of primary school (levels 2 and 5), the third year of middle school (level 8) and the second and fifth years of high school (level 10 and 13). INVALSI prepares the examinations considering the curricula for primary and secondary schooling, i.e. "Indica%ioni Na%ionali per il curricolo della scuola dell'infan%ia e delprimo ciclo di istru%ione" and the "Indica^oni Na%iona& e Linee guidaper le scuole secondarie di secondo grado" (Martignone, 2016; Quadro di Riferimento, 2017). The decree MIUR-MEF n. 211 from the 7th October 2010 and the directive MIUR n. 57 from 15th July 2010 are the two documents that regulate the curricula in Italian high schools. In these documents it is stated that students, at the end of five years' schooling, should know the basic concepts and methods used in mathematics, not only from a theoretical point of view, but also to model and describe various phenomena from the real world. It is also stresses that students should be able to use logical and coherent argumentation and apply mathematical concepts in everyday life. In these documents it is also stated that, while formulating problems for students in high schools, it is important to show them the connections between theoretical knowledge and other sciences (economics, sociology, technology, physics, biology, etc.) or the real world. The documents invite teachers to show students how to use formal language and how to prove theorems, how to analyze data and predict the evolution of phenomena, use mathematical knowledge in other sciences, introduce new concepts using elements form the history of mathematics, history of science, technology and cultural development. During the second year of high school, students at all Italian high schools are required to write a national assessment of Italian language and mathematics. However, these standardized tests: - cannot evaluate students' metacognitive or non-cognitive achievements, such as that embodied in "the students develop a positive attitude toward mathematics" (Quadro di Riferimento, 2017); - cannot evaluate students' ability to argue, prove or solve certain problems (that would require more time or a greater number of steps), formulate D. Dor: The 'National Assessment of Mathematics in High Schools in Italy with Slovene as the 'Language of Instruction 163 hypotheses, or model real-world situations and analyze them from a mathematical point of view (Quadro di Riferimento, 2017); - are objective; hence, they do not take into account the affective and conative aspects, which are also important in evaluating student work and assessing competence (Quadro di Riferimento, 2017). Until the school year 2017-18, the INVALSI examinations for schools with Slovene as language of instruction were printed. Every year, the printed dossier had a variable number of questions, which were divided into different items (Quadro di Riferimento, 2017). Questions in the INVALSI examinations, as expressed in the Quadro di Riferimento (2017), can be - closed: multiple choice questions, where there are four alternatives (only one answer is correct); true-false questions, composed of several subquestions (see Example 2); - short, open questions: these require a simple, rapid open answer, such as the result of a computation, or some graphic answers (see Example 3); - open: these require simple argumentation or short computations; - clo^e: the student is required to complete a sentence, computation or expression. Example 2: Example of a multiple-choice question from the INVALSI examination for the school year 2017-18. The result 16100: 2 is the same: A. □ 899 B. □ 8100 C. □ 1650 D. □ 2399 Example 3: Example of a short answer question from the INVALSI examination for the school year 2017-18. The equation is given (2k — 3)x + 1 — k = 0, with unknown X in real number k. The equation solution is 1, if k =......... Students were allowed to use their calculators during the tests, but not devices that could connect to the internet, wireless or Bluetooth. They could also use compasses, rulers and goniometers. 164 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION In the Quadro di Riferimento (2017) it is also stated that simple language must be used in writing the problems proposed to the students; no dialectal or regional expressions are used, and the testers try to avoid useless technological jargon. Pictures are used only when particularly explicative; the data provided in the problems are mainly taken from real data and statistics. Questions are also equally distributed among the various topics. The proposed problems have two dimensions: the cognitive dimension and the topic dimension. Topics are divided into four categories: numbers, geometry, algebra and data analysis (and probability). The cognitive dimension is divided into three categories: - knowing: the student understands the facts, concepts and procedures; - applying: the student should know how to apply their knowledge and acquired concepts to solve problems and answer questions; - reasoning: the student solves problems related to complex and unfamiliar contests. Empirical Research Aim of the Research The aim of the present research is to analyze several INVALSI examinations in mathematics in the Slovene language, in order to understand the typology of their questions and the mathematics field to which each question is related. This research could have practical applications: teachers and students could be informed of the mathematical topics that are more common on the INVALSI examinations and the question types. On the other hand, this research could show which knowledge and competence in mathematics Italy requires from its students. We wanted to investigate whether there have been changes in the types of questions and the interest field from 2011 to 2017. And, if so, what these differences were. Our research questions were the following: - Is there a prevalent typology for questions on the INVALSI examinations? Are there more closed-type questions (e.g. true-false questions or multiple-choice questions) or open-type questions (e.g. short answer questions)? D. Dor: The 'National Assessment of Mathematics in High Schools in Italy with Slovene as the 'Language of Instruction 165 - Is there any mathematical topic that appears more often in the INVALSI examinations than others? And if so, has there been any change in the topics of interest over the years? - Have theoretical fields, such as functions, set theory and logic, become gradually less common on the INVALSI mathematics examinations? Methodology Research method In the research, we used the descriptive statistical method and the non-experimental method for causal analysis. We decided to use these methods, because they are best suited to answer the research questions. Statistical sample In the research we considered seven INVALSI mathematics examinations for the second year of high schools with Slovene as the language of instruction, i.e. all the examinations from the school year 2010—11 to the school year 2016-17. We omitted from our sample the INVALSI examination from the school year 2017-18 because it was almost identical to the one from 2013—14. The samples cannot be found online, since they are prepared ad hoc for schools with Slovene as the language of instruction; the samples in Italian can be found on the website https://www.engheben.it/prof/materiali/invalsi/seconda_superiore_matematica. htm. These versions are similar to those in Slovene. Analysis of the Data We first looked at the types of questions appearing on the various INVALSI examinations. Next, we sought to identify the mathematical topic to which the question referred, in order to understand which topic is the most popular. The collected data was analyzed using the descriptive statistical method, expressing frequency of appearance. All data was analyzed with the help of the statistical software Jamovi. 166 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION Results and Discussion The Typology of the Exercises In Table we present the typology of the various questions on the INVALSI examinations from the school year 2010—11 to the year 2016—17. In the table we present the number of questions (and sub-questions) for each type: multiple-choice, true-false, long answer questions (a procedure is evaluated and scored), fill-in the blank, short answers (only a numerical answer is required, no procedure is evaluated), connect the terms, and the total number of questions on the examination. In brackets, we give the percentage of that type for that year's examination. Table 1: Typology of exercises in various school years. School year Multiple-choice True-false Long answer Fill-in the blank/ complete Short answer Multiple-choice and discussion Connect Total 2010-11 23 3 3 1 11 0 0 41 (56.1%) (7.3%) (7.3%) (2.4%) (26.8%) (0.0%) (0.0%) (100.0%) 2011-12 21 4 2 4 11 3 0 45 (46.7%) (8.9%) (4.4%) (8.9%) (24.4%) (6.7%) (0.0%) (100.0%) 2012-13 28 (63.6%) 2 (4.6%) 0 (0.0%) 1 (2.3%) 11 (25.0%) 2 (4.6%) 0 (0.0%) 44 (100.0%) 2013-14 13 (34.2%) 6 (15.8%) 0 (0.0%) 2 (5.3%) 16 (42.1%) 1 (2.6%) 0 (0.0%) 38 (100.0%) 2014-15 18 (42.9%) 4 (9.5%) 0 (0.0%) 2 (4.8%) 16 (38.1%) 2 (4.8%) 0 (0.0%) 42 (100.0%) 2015-16 17 (42.5%) 3 (7.5%) 0 (0.0%) 3 (7.5%) 15 (37.5%) 2 (5.0%) 0 (0.0%) 40 (100.0%) 2016-17 14 (35.0%) 6 (15.0%) 2 (5.0%) 4 (10.0%) 13 (32.5%) 0 (0.0%) 1 (2.5%) 40 (100.0%) Total 134 28 7 17 93 10 1 290 (46.2%) (9.7%) (2.4%) (5.9%) (32.1%) (3.4%) (0.3%) (100.0%) From Table 1, we can see that the most frequent type of question on the INVALSI mathematics examinations for the second year of high school is the multiple-choice question (46.2%). The second most frequent type is the short answers (32.1%), followed by true-false questions (9.7%). In the last three years, there has not been any significant increase in the frequency of multiple-choice questions, or true-false ones, but it is still clear that the most frequent type of question is the multiple-choice; it is also the most frequent in each school year examination, with the only D. Dor: The 'National Assessment of Mathematics in High Schools in Italy with Slovene as the 'Language of Instruction 167 exception being the school year 2013—14, when short answer questions were the most frequent. In order to understand which type of question was the most frequent, we grouped multiple-choice, true-false and connect in the category "closed-type questions", and long answer, short answer, fill-in the blank, and "multiple-choice and discussion" in the category "open-type questions". These results are shown in Table 2. Form the analysis, we see that closed-type questions (56.2%) are more frequent than open-type ones (43.8%). In particular, in the school years 2010-11 and 2012-13, closed-type questions were significantly more frequent than open-type ones, whereas on the other examinations, the frequency was slightly higher. No significant increase in the frequency of closed-type questions can be seen in the last three years. Every year, the number of closed questions was greater than or equal to those of the open-type. Table 2: Open- and closed-type questions on the examinations. School year Closed-type questions Open-type questions Total 2010-11 26 (63.4%) 15 (36.6%) 41 (100.0%) 2011-12 25 (55.6%) 20 (44.4%) 45 (100.0%) 2012-13 30 (68.2%) 14 (31.8%) 44 (100.0%) 2013-14 19 (50.0%) 19 (50.0%) 38 (100.0%) 2014-15 22 (52.4%) 20 (47.6%) 42 (100.0%) 2015-16 20 (50.0%) 20 (50.0%) 40 (100.0%) 2016-17 21 (52.5%) 19 (47.5%) 40 (100.0%) Total 163 (56.2%) 127 (43.8%) 290 (100.0%) As presented, closed-type questions are easier to correct, but they have several limitations. A unavoidable limitation of multiple-choice questions is that it is possible for students to guess the correct answer (Burton, 2001). Furthermore, closed-type questions cannot completely evaluate students' knowledge and abilities. 168 REVIJA ZA ELEMENTARNO IZOBRAŽEVANJE/ JOURNAL OF ELEMENTARY EDUCATION The Mathematical Topics We first sought to understand which mathematical topics are covered by the national INVALSI examination; we therefore analyzed the content of each question and sub-question. From the content of such questions, we decided to distinguish among the following topics: - statistics: sampling, data organization, data representation (tables, graphs, etc.), relative and absolute frequency, percentage frequency, means, standard deviation, variance; - probability: definition of classical probability, events, disjoint events, dependent and independent events, computation of probability of elementary events, a priori and a posteriori probability, binomial tree; - geometry: principal geometrical shapes and objects, definitions, relations between geometrical objects, congruence, parallelism, perpendicularity, geometrical constructions, segments, distance in the plane, measuring with the ruler, angles (internal, external, vertically opposite angles, supplementary, complementary, explementary, etc.), measuring with the goniometer, translations, rotations, symmetries, similarity, Pythagorean theorem, Euclid's theorem and equivalency, Thales' theorem and similarities, perimeters and areas of plane shapes (square, rectangle, triangle, parallelogram, circle, trapezoid, etc.); - arithmetic: natural numbers, integers, rational and real numbers, operations between numbers, properties of the operations, representations of numbers on the number line, fractions, proportions and applications, powers and their properties, square roots and cube roots, decimal and scientific notation, rounding and positions, numerical expressions, symbolic expressions; - approximation: approximation of real and rational numbers, operations with approximated numbers, working with big or small numbers, scientific notation; - diagrams: line diagrams, histograms, bar diagrams, pie charts, representation of numbers, reading diagrams, interpretation of a diagram; - Solid geometry: solids (cube, parallelepiped, cone, sphere, cylinder, pyramid, prism, etc.), volume and surface area, diagonals and sections; - analytic geometry: coordinate plane, points and coordinates of points, distance between points, areas of shapes in the coordinate plane, triangles in the coordinate plane; D. Dor: The 'National Assessment of Mathematics in High Schools in Italy with Slovene as the 'Language of Instruction 169 - algebra: unknowns, polynomials, factorization of polynomials and binomials, equations, equalities, inequations and inequalities, squares and square roots of expressions; - percentages: interpretation of percentages, discounts and marketing, application of percentages; - functions: definition of functions, plots of functions, roots of functions, modeling with functions, domain and codomain of functions, polynomial and rational functions, parabola and hyperbola; — linear functions: definition of linear function, plot of linear functions in the coordinate plane, lines, intersection of lines, parallel lines and perpendicular lines, slopes and lines through two points, equation of a line in the coordinate plane, modeling with the linear function, increasing rate; — logic: propositions, logical connectives, negations, conjunctions, disjunctions, implications and equivalences; — set theory: sets, elements, operations with sets (union, intersection, etc.), complementary set, power set, cardinality of a finite set. Our findings are shown in Table 3. In the last column we presented the relative frequency percentage of the topic among all 290 questions analyzed. Table 3: Content of INVALSI examinations. 2 3 4 5 6 7 1 o 1 1 ^ 1 3 ^ 1 5 1 6 •a 1 1 1 1 1 1 b*