c e p s Journal | V ol.12 | N o 2 | Y ear 2022 29 Textbooks and Students’ Knowledge Barbara Japelj Pavešić* 1 and Gašper Cankar 2 • In Slovenia, textbooks are an integral part of the curriculum. Nation - ally certified textbooks guarantee both teachers and students that they provide all of the necessary knowledge in each subject. There are many available certified textbooks for each subject and teachers must decide which will be the source of instruction for their students. Our research question is whether groups of students who use different textbooks as their mandatory learning resource differ in their knowledge and their attitudes towards learning. We linked existing data from several sources and explored the scope of the use of different textbooks for mathemat - ics and science subjects in primary schools. Data on student knowledge measured independently by National Assessments (and the Trends in International Mathematics and Science Study were used, and differ - ences in knowledge and attitudes to learning between students who are taught using different textbooks were explored. Although the study has considerable limitations due to missing data, the results of the analyses indicate some profound differences in knowledge and attitudes between groups of students using different textbooks. These findings could serve as a guide for teachers when choosing the optimal available textbook for their students and, even more so, as support for improving the criteria in the national system of validation of textbooks in the future. The link between the use of textbooks and student learning outcomes also high - lights the need to systematically collect information on the use of text - books among students and follow the effects on achievement in order to improve the quality of future textbooks. Keywords: certified textbooks, knowledge, attitudes, mathematics, science 1 *Corresponding Author. Educational Research Institute, Slovenia; barbara.japelj@pei.si. 2 National Examinations Centre, Slovenia. doi: 10.26529/cepsj.1283 30 textbooks and students’ knowledge Učbeniki in znanje učencev Barbara Japelj Pavešić in Gašper Cankar • V Sloveniji so učbeniki sestavni del kurikuluma. Nacionalno potrjeni učbeniki učiteljem in učencem zagotavljajo, da bodo z njimi dosegli vse - binske cilje učnega načrta pri vsakem predmetu. Potrjenih učbenikov za določene predmete je na voljo več in učitelji se odločijo, kateri bodo obvezni vir pouka za njihove učence. Vprašanje je, ali se razlikujejo zna - nje in stališča učencev, ki uporabljajo različne učbenike kot svoj obvezni učni vir. Združili smo obstoječe podatke iz več virov ter raziskali obseg uporabe različnih učbenikov za matematiko in naravoslovne predmete v osnovni šoli. Uporabili smo obstoječe in medsebojno neodvisne podatke o znanju učencev z nacionalnih preizkusov znanja ter mednarodne razi - skave znanja matematike in naravoslovja, TIMSS. Preverili smo razlike v stališčih do učenja med učenci, ki se učijo z različnimi učbeniki. Čeprav ima študija precej omejitev zaradi nekaterih manjkajočih podatkov, so rezultati analiz nakazali nekatere bistvene razlike med skupinami učen - cev, ki uporabljajo različne učbenike, v znanju in stališčih. Ugotavljamo, da bi bile lahko te opora učiteljem pri izbiri najboljšega mogočega raz - položljivega učbenika za svoje učence, še bolj pa podpora za dvig meril v nacionalnem sistemu potrjevanja učbenikov v prihodnje. Povezanost med uporabo učbenikov in učenjem učencev vzpodbuja tudi potrebo po sistematičnem zbiranju informacij o uporabi nacionalno potrjenih učbenikov med učenci ter povratnih informacijah o njihovih učinkih na pouk za izboljšanje kakovosti prihodnjih učbeniških gradiv. Ključne besede: potrjeni učbeniki, znanje, stališča, matematika, naravoslovje c e p s Journal | V ol.12 | N o 2 | Y ear 2022 31 Introduction Textbooks are a fundamental written source of knowledge and often represent a factor in educational research that explains differences in students’ knowledge at different levels of the education system (Oakes & Saunders, 2004). As an ‘embodiment’ of the prescribed national curriculum for an individual school subject and year of schooling, they transfer the requirements and expec - tations regarding student outcomes at the national level into teaching practice (Oates, 2014). As a resource for teaching, textbooks present teachers with a set of learning objectives and their transformation into didactic presentations of the material intended for teaching. As a learning tool, they are a fundamental and reliable source of information for students as they acquire new knowledge, while offering the teacher the opportunity to focus attention on improving ped - agogy and effective learning (Oates, 2014). The role of textbooks is multifaceted and crucial for pedagogical practice. Textbooks link the expected, implemented and achieved curriculum as different curricular levels are understood by international large scale assess - ments (Johansson, 2003). The expected curriculum represents a national con - sensus on the content to be presented and taught to students, the implemented curriculum describes the process of teaching this material to students in the classroom, and the achieved curriculum is students’ knowledge as demon - strated by their achievements. In the present study, we were interested in how the use of a textbook affects students’ knowledge. Specifically, we investigate how the attitude towards textbooks at three levels in the curriculum in Slo - venia compares to the situation in other countries, and whether it is possible to describe a relationship between students’ achievement and textbooks, and, consequently, certain characteristics of teaching practices. In general, the research literature suggests that textbooks should be con - sidered as an important parameter in education studies and textbook choice a relevant factor for education practice. However, comparative studies of the re - lationship between different textbooks and student achievement are rare, with most of the available reports concerning learning mathematics. Researchers observe that teachers’ choice of textbooks substantially impacts students’ math - ematics achievement, while individual textbooks differ in their effects. This im - pact was found to be cumulative over the school years and significantly linked to the problem-solving strategies students use from as early as Grade 1 (Sievert et al., 2021; Van den Ham & Heinze, 2018). Researchers observing relationships between achievement, curriculum and textbooks in mathematics found little variation in achievement between 32 textbooks and students’ knowledge different textbooks, but some variation between the curriculum that was im - plemented before and after a school reform in the US (Blazar et al., 2020). They assumed that textbooks became increasingly similar due to the reform, as it changed local curricula to the unified National Core Curriculum in many states. However, researchers suggest that excellent teaching and high student achievement cannot be achieved by the improvement of textbooks alone. A similar message about small differences between students’ knowledge accord - ing to the use of different textbooks could be concluded for the case of Slovenia, where the national curriculum is already prescribed for all students. Textbooks should also be seen as a pedagogical tool, as they can lend support to teachers when they are choosing teaching strategies. Examining how teachers’ subject knowledge relates to their use of textbooks, Mili and Winch (2019) claim that textbooks can be a powerful pedagogical tool for highly trained teachers, as opposed to being a teaching script in the hands of poorly qualified teachers. Regarding open educational resources, which are increas - ingly attractive for facilitating distance learning and reducing education costs, research on their effect on student achievement shows that teachers matter more than textbooks. The results of an analysis by Hardin et al. (2019) reveal that traditional textbooks do not necessarily ensure greater academic success, even in the hands of well-trained instructors, thus suggesting a need to devote resources to teacher training and support in the use of different textbooks. Comparisons within Slovenian mathematics education have shown that teachers in Slovenia lag behind other countries in teaching decimals and frac - tions, as these topics are covered later (Mullis et al., 2016). Another very im - portant message emerges from recent research on the relationships between textbook content and mathematics learning, focusing on how children learn about rational numbers (i.e., fractions, decimals, percentages) (Siegler & Op - penzato, 2021). It is known from previous research that fraction knowledge in the fifth grade uniquely predicts mathematics achievement five years later in secondary school, even after controlling for socioeconomic status, IQ, read - ing comprehension and whole number arithmetic knowledge. However, recent findings suggest that children’s learning depends on items and problems that are presented (or absent) in their textbooks. Differences in achievement were strongly associated with differences in the composition of different types of ex - ercises in the textbooks used. More demanding textbooks with a diverse range of different problems seem to elicit higher achievement. In national and inter - national assessments of mathematics in Slovenia, extremely low achievement in items that are not presented in textbooks in a similar form, or not presented at all, is traditionally observed, as we will demonstrate later. c e p s Journal | V ol.12 | N o 2 | Y ear 2022 33 The research problem The importance of textbooks in educational research is growing as the research literature shows a trend towards the increased use of textbooks as the primary source of teaching, especially in mathematics at Grade 8 (Mullis et al., 2012). This could mean that the content taught in class is increasingly limited to the presentations and explanations of concepts given in selected students’ text - books. In other countries, the intensive use of textbooks as a primary sources of teaching has not lead to increased effectiveness in learning, especially not in science. Data from the Trends in International Mathematics and Science Study (TIMSS) show that mathematics textbooks are rarely used or read from in Slo - venian classrooms. From this we assume that mathematics teachers in Slovenia view the textbook as the basic catalogue of expected knowledge students should acquire, both during lessons and through independent learning at home. This leads to our basic research question of whether student outcomes, as measured by the TIMSS and National Assessment, are related to textbook choice. An - other problem of mathematics and science education in Slovenia is the decline in motivation to learn. Since publishers put a lot of effort into making textbooks attractive for students – and since they compete to encourage teachers, parents and students to choose their textbooks for compulsory learning material – we will explore whether more attractive textbooks actually motivate students to learn mathematics and science more, and whether mathematics and science teachers achieve greater motivation to learn among their students by choosing a specific textbook. The research question is: how does the choice of textbook relate to students’ motivation for learning mathematics and science (whether they like learning these subjects), their self-confidence in both subjects, and their values associated with mathematics and science? Data source The study has two parts. In the first, we review reports about use of text - books from international data provided by successive editions of the TIMSS, in which Slovenia has participated for several decades. These international large- scale assessments measure students’ knowledge and teaching factors among representative samples of national populations of Grade 4 and Grade 8 students every four years. The samples for all TIMSS studies in each participating coun - try follow a two-stage random sample design, with a sample of schools drawn as the first stage and one or more intact classes of students enrolled in a specific grade selected from each of the sampled schools as the second stage (Laroche 34 textbooks and students’ knowledge et al., 2016). For example, in 2015, the Slovenian sample included 4,800 Grade 4 students and 4,600 Grade 8 students from 150 primary schools. The students were assessed in mathematics and science, and reported about themselves and their learning by answering questionnaires, while questionnaires were also an - swered by the students’ school principals and teachers (257 fourth-grade class teachers, 471 mathematics teachers and 859 science teachers). The data were linked into a database for each study cycle and appropriate weightings were created to accommodate complex sampling design. Statistical analyses of the weighted student data enable us to calculate nationally representative popula - tion estimates. With the questionnaires, TIMSS studies over the years have col - lected a variety of comparative data about textbooks. We combined the TIMSS data with a national database on use of text - books in each Slovenian school from the national portal Trubar. Until 2019, this portal was intended to inform students and parents about the printed or paper textbooks that were selected as compulsory for each grade and subject at each school. Schools were obliged to enter the chosen list of teaching mate - rial into the central national database Trubar. The chosen textbooks were pro - vided to schools nationally and were available in the school textbook library for students. In the present paper, we used data on the use of textbooks from the Trubar web portal for the 2016 school year. The data lacks validity after 2016, as schools were no longer obliged to report data. The other part of the present study includes National Assessment (NA) data. Held in Grades 6 and 9 of primary school, NA is a low-stakes external assessment of Slovenian language, mathematics and a third school subject that is by default the first foreign language in the sixth grade or one of the selected third subjects in the ninth grade. NA is based on the curriculum and provides students, parents, teachers, schools and others with additional information about each student’s knowledge in addition to the teacher’s school grades. Data on the use of textbooks from the Trubar database were linked to anonymised NA achievements in Mathematics (2016), Chemistry (2015), Biol - ogy (2017) and Physics (2016), again to explore differences in achievement due to the different textbooks used. Method Our report starts with a summary of TIMSS studies conducted in 2003, 2007, 2011 and 2015 on use of textbooks, as reported by either students or their teachers and principals. This gives us a good overview of the use of textbooks in Slovenia. c e p s Journal | V ol.12 | N o 2 | Y ear 2022 35 Next we selected variables and indices from the international data - base of TIMSS 2015 that reflect the use of textbooks, as well as data on student achievement, and linked them to the national database on the use of textbooks (Trubar). At the level of the individual sample unit (the student), we obtained information on which textbooks were prescribed at the student’s school in 2015 for use in mathematics and science subjects, together with all of the data from the international study TIMSS 2015. This enabled the calculation of predictive models and differences in achievement in mathematics and science of eighth graders according to the use of different textbooks. For Grade 4, the Trubar portal contained much less data on the prescribed printed textbooks. This is due to the fact that for lower grades, a large selection of workbooks and digital textbooks or materials are freely available, and these were therefore most fre - quently prescribed as compulsory textbooks by schools. Since they were not lent by school libraries, the selections were not reported nationally. The results of the analysis for Grade 4 are therefore limited to the available data. Similar analyses were repeated for National Assessment (NA) data from Grade 9. The aim of studying the relationships between textbooks and knowledge was to determine whether the research could identify some effects; it was not to measure the effectiveness of individual textbooks. Therefore, the textbooks are shown under codes in order to prevent the disclosure of publishers. Correlation and regression analyses were performed using specific software developed for complex TIMSS assessment data (IDB Analyzer and SPSS) or using the pro - gram R in case of NA data. Results Overview of the use of textbooks in Slovenia until 2015 First assessment during the stepwise implementation of the extensive school reform of compulsory schooling, 2003 In 2003, an extensive school reform was implemented in the lower grades of most primary schools, but only in a sample of schools for Grades 6 to 9. The TIMSS 2003 study provides Slovenia with an opportunity to examine differences between the old and the reformed school system, which prolonged compulsory schooling by one year and renewed curricula and didactic approaches. In the TIMSS, principals of national representative samples of primary schools in more than 30 participating countries were asked about the lack of textbooks in their schools for students in mathematics and science classes. Slovenian principals of 36 textbooks and students’ knowledge the vast majority (87%) of eighth-grade students confirmed that their schools do not feel limited in teaching due to a lack of textbooks; principals of 11% of the students judged that classes suffer little, and principals of only 2% of the students thought that classes suffer seriously from a lack of textbooks. The mean interna - tional percentage of students in schools where classes do not suffer from a lack of textbooks was 41%, less than half of that in Slovenia (TIMSS Almanacs 2003, 2005a, p. 183). This shows that Slovenian students had ample access to new text - books for the reformed curricula immediately after the reform was implemented in practice. Differences in achievement between students from schools suffering from a shortage of textbooks and those not suffering from such a shortage cannot be estimated, as there are too few students in the last group, but the mean math - ematics and science achievement across other countries decreases slightly with an increased shortage of textbooks for school instruction. In Slovenia, mathematics and science teachers of all students confirmed that they use a textbook for teaching (TIMSS Almanacs 2003, 2005b, p. 151). The international average shows higher achievement when teachers use textbooks for teaching, but there are differences between countries. It was not possible to determine the differences in achievement in Slovenia, as there were not enough students with teachers who did not use textbooks for teaching. However, in some lower achieving countries, the students whose teachers did not use a text - book achieved lower scores than students who were taught by teachers using textbooks. In Belgium and England, both high-achieving countries, teaching with textbooks was associated with higher student knowledge. In Slovenia, mathematics teachers of 68% of the students taking the TIMSS stated that the lack of textbooks was not an obstacle to teaching math - ematics at all, teachers of 14% of the students regarded the lack of textbooks as a minor obstacle to teaching, teachers of another 14% of the students perceived the lack of textbooks as significantly or severely limiting teaching, and teachers of the remaining 3% of students did not use textbooks. Student achievement did not differ between the first three groups according to the teacher responses, but it did decrease in the group of students whose teachers felt severely limited in their teaching by the lack of textbooks. Across all countries, the last group of students whose teachers felt severely limited by the lack of textbooks is more than twice as large as in Slovenia and the first group is half as small, while math - ematics achievement does not differ substantially between groups of students experiencing more or less limited teaching due to the lack of textbooks (TIMSS Almanacs 2003, 2005b, p. 199). In science (TIMSS Almanacs 2003, 2005c, p. 226), teachers felt least lim - ited by the shortage of textbooks in Hungary and the Netherlands, and most c e p s Journal | V ol.12 | N o 2 | Y ear 2022 37 limited in the poorer countries of Ghana, Indonesia and Morocco. In Slovenia, 63% of the students had science teachers who reported that the lack of text - books for science did not limit them in teaching. Despite the high share, teach - ing science was at least slightly limited for 14% of the students, quite limited for 8% and very limited for 3% due to lack of textbooks. In total, a substantial share of the participating students (a quarter of all of the Slovenian students) experi - enced limited teaching of science due to a lack of textbooks. Reformed compulsory school before the first systematic review, 2007 The TIMSS study in 2007 was the first reflection of the fully implement - ed newly reformed school system in Slovenia. The principals of the schools at which 81% of all of the eighth graders participating in the study were enrolled confirmed that they did not have any problems with a shortage of teaching ma - terials, which was an extremely high proportion compared to other countries. Internationally, the average share of students in schools without a shortage of teaching materials was 50% (TIMSS Almanacs 2007 , 2009a, p. 79). According to these data, the availability of textbooks increased during the full implementa - tion of nine-year reformed primary schooling in Slovenia. Science teachers reported how often they asked students to read about the taught content from a textbook (TIMSS Almanacs 2007, 2009b). The share of students asked by teachers to read from the textbook each lesson was less than 10% in three countries: Norway, Sweden and Slovenia (4.2%). Judging by the science achievements in Slovenia, more frequent reading from a textbook did not increase student achievement. On the contrary, slightly higher achieve - ment was shown by students from whom teachers never required reading from a textbook than by students who read from a textbook during at least half of the lessons. This result can be explained by the national practice of teaching sci - ence and the nature of textbooks for these subjects, which are intended more to explain the content to students when they learn individually after school than as a working source for following the lessons in school. Unlike mathematics textbooks, science students mostly do not need to practise solving problems from textbooks, neither in school nor after school. Textbooks in schools ten years after the reform, 2011 From the trends in the proportions of students whose teachers con - firmed that they use a textbook as their primary source of instruction and not as a secondary source (Figure 1), it can be seen that teaching in Slovenia has increasingly relied on textbooks since the reform of compulsory school. In 2003, teachers of only about half of the students based teaching primarily 38 textbooks and students’ knowledge on textbooks, whereas in 2011, in all of the observed subjects, teachers of 90% of the students or more viewed the textbook as the primary source of their teaching. Figure 1 Trends in the share of students whose teachers use a textbook as their primary source of teaching, 2003–2011 Mathematics teachers of 90% of the participating students in Slovenia stated that a textbook is their primary source of preparation and implementa - tion of lessons (TIMSS Almanacs 2011, Mathematics teacher background data with mathematics achievement). Comparable international data on the use of textbooks and the measured knowledge of students do not show that teaching strictly linked to the prescribed textbook is more effective. There was no differ - ence between the achievements of students whose teachers use a textbook as a basic or secondary source in Slovenia. The results suggest that the way the text - book is used does not in itself lead to large differences in students’ knowledge. In science, a textbook was used as a secondary source by 15% of students (TIMSS Almanacs 2011, Science teacher background data with science achieve - ment). Students’ knowledge was higher when teachers used textbooks as a secondary source of teaching than when students were taught strictly from a textbook in several high-achieving countries, including Taiwan, Finland, Hong Kong and Hungary, and a slight trend is also observed in Slovenia. The frequency of reading from textbooks and other materials in science lessons has increased in Slovenia since 2007 . The share of students whose teach - ers require reading every lesson or every other lesson has more than doubled, and the share of students who have never been asked to read something from a textbook in class has halved. Contrary to expectations, the gap in knowledge between students who never or at least sometimes read in class has widened. Those who never read showed a trend towards higher achievements. The c e p s Journal | V ol.12 | N o 2 | Y ear 2022 39 explanation could be that these students experience active learning in school and learn from textbooks individually at home to review the content and deep - en their understanding. After implementation of the reviewed changes in curricula, 2015 In the TIMSS conducted in 2015, Slovenian students demonstrated in - creasing knowledge in all areas, especially in science subjects. At the same time, the international comparison of curricula confirmed that a larger amount of content was expected to be taught to Slovenian students in science lessons than in most other countries. For the first time, mathematics achievement in the eighth grade, which had previously been largely stagnant, increased signifi - cantly. The problematic part of the research findings refers to the trend of nega - tive student attitudes towards learning, with a further decline in liking learning mathematics and science to less than a quarter of all students, and to interna - tionally comparable extremely low self-confidence of students, their valuing of knowledge, their sense of belonging to school, and their perception of engaging teaching of their teachers (Japelj Pavešić & Svetlik, 2016). General relationships between textbooks, teaching and student knowledge The relationships between the frequency of reading from textbooks, achievement and science teaching strategies were first checked by correlation analyses. The results showed insignificant or weak correlations between the fre - quency of reading from textbooks and other material in lessons and student achievement, the frequency of the teacher’ s linking new content to everyday life, the teacher asking students to solve challenging problems, the teacher encour - aging discussion among students, or the teacher connecting subject knowledge with other areas, as well as the school’s commitment to the academic success of students and the teachers’ commitment to teaching by experimenting (correla - tion coefficients are less than .18) (Table 1) 40 textbooks and students’ knowledge Table 1 Correlation between reading textbooks in lessons and characteristics of teaching science, Grade 8 (s.e.) → Correlation coefficients ↓ A B C D E F G H I How often the teacher\ Relates the lesson to students’ daily lives A (.04) (.04) (.04) (.04) (.05) (.05) (.04) (.02) How often the teacher\ Asks students to explain their answers B .36 (.04) (.04) (.04) (.05) (.04) (.04) (.01) How often the teacher\ Asks students to complete challeng- ing exercises that require them to go beyond the instruction C .20 .35 (.04) (.04) (.04) (.04) (.04) (.02) How often the teacher\ Links new content to students’ prior knowledge D .37 .41 .18 (.04) (.06) (.04) (.04) (.01) How often the teacher\ Encour- ages students to express their ideas in class E .33 .41 .28 .36 (.04) (.04) (.04) (.02) School emphasis on academic success F -.16 -.13 -.24 -.09 -.16 (.04) (.05) (.03) How often the teacher\ Asks students to read their text- books or other resources G .06 .15 .18 .15 .11 -.10 (.05) (.02) Teachers emphasis on science investigation H -.24 -.26 -.29 -.22 -.29 .10 -.14 (.01) Science achievement I .01 -.01 -.05 .00 .01 .07 -.02 .01 Differences in achievement in Grade 4 according to textbooks An analysis of data from the Trubar database for Grade 4 in 2015 reveals that there are no data on prescribed printed textbooks for science lessons for about 30% of students, and that there are no data on prescribed printed textbooks for mathematics for a very large share, 81% of students. This could be attributed to the large number of digital or electronic textbooks already in use in 2015, as well as to the fact that no records of the prevalence or prescribed use among students were kept. According to school policy, all students should have prescribed text - books. Therefore, students without data in the Trubar database most likely used online textbooks. Among the reported printed textbooks there were four for the subject science and seven for mathematics. The results of the analyses of textbook use in Grade 4 should be read with caution, taking into account the fact that printed textbooks were used by a decreasing share of students. A third of students were learning science from one textbook and a fifth from another, while about a third were learning from several different sources, c e p s Journal | V ol.12 | N o 2 | Y ear 2022 41 each of which was much less widespread. Two facts were revealed by the analy - sis: (a) differences in the knowledge of students learning from different text - books were not significant, and (b) the knowledge of students without a printed textbook did not differ from the knowledge of students who had a printed text - book. The knowledge of different content sets of science was similar among students with different textbooks. It is therefore not possible to identify any textbook as more effective in learning either individual science content or sci - ence content in general (Figure 2). Figure 2 Percentages of students and science achievement by prescribed textbooks, TIMSS 2015 Grade 4 In mathematics, the results are limited to one-fifth of the entire student population. Of these students, half had one particular textbook and the other half one of six other different textbooks. The differences in knowledge are greater than in science. They show a favourable picture of higher knowledge among students who use the most frequently used textbook, but also a significant decrease in knowledge among students with another particular textbook TM 2(4). In the area of numbers, the difference between students having one of other textbooks is 22 points (on a scale with an average of 500 and a deviation of 100 points). Follow - ing the recommendations for assessing differences by calculating the effect size d (Cohen, 1992), which is the quotient between the difference and the total stand - ard error, we find that the above difference is large (d = 3.3; d > .8 is considered a large effect). However, this does not present a problem, because the textbook that proved to be less effective was only used by a very small group of students (2.5%). There is a similarly large difference between the achievement of students who learn from the predominant printed textbook and the achievement of 81% of students who learn from other learning sources. The latter group scored 14 points lower and the effect size was again defined as high (d = 3.4) (Figure 3). 42 textbooks and students’ knowledge Figure 3 Percentages of students and mathematics achievement by prescribed textbooks, Grade 4 Due to the large differences in knowledge, we checked the number of topics taught based on teachers’ reports in the TIMSS. The teachers indicated whether the students had already been taught the list of topics covered by the international TIMSS assessment, which is internationally agreed to reflect cur - ricula from the participating countries. The reports partly limit our interpre - tation, as some topics were not covered by the Slovenian curriculum, while there were some topics that were not included in the international tests but were taught to students in Slovenia. Nevertheless, from the share of students whose teachers reported which topics had been taught to these students, it is possible to estimate the total level of taught content. In science (Figure 4), the share of students who had been taught all of the topics ranges from about half to 85% and varies considerably between textbooks. With the textbook that also showed the highest level of student knowledge (TS 3(4)), most students learned all of the content, whereas the lowest number of students had the opportunity to learn all of the listed content with the textbook that is associated with the lowest achievement. It is problematic that the predominant science textbook does not demonstrate the maximum amount of material learned. c e p s Journal | V ol.12 | N o 2 | Y ear 2022 43 Figure 4 Proportions of students taught science content from the TIMSS test by prescribed textbooks, TIMSS 2015 Grade 4 The pattern is different in mathematics (Figure 5). All of the textbooks except one show comparable shares of students taught all mathematical topics included in the TIMSS assessments. Figure 5 Proportions of students taught mathematics content from the TIMSS test by the prescribed textbook, TIMSS 2015 Grade 4 A side result of this comparison is the large difference between the in - dividual content areas of mathematics. Only between 30% and 60% of the stu - dents had learned the geometry content expected by the TIMSS test, whereas almost all of the students had learned the content from the area of data. This 44 textbooks and students’ knowledge large difference was not expected, as the same national curricula is requested to be taught to all students. The lower coverage could be explained by the fact that in the TIMSS tests, the geometry items included calculations and meas - urements with units as well as an understanding of the concept of area, which are taught in Slovenia later, from the fifth grade onwards. Similarly, the area of numbers also covers some concepts in the fourth grade that are not taught to Slovenian students, such as calculation with simple fractions and decimal num - bers. Nevertheless, for mathematics, we can conclude that the number of topics taught to students using sources other than printed textbooks does not deviate from the number taught with a printed textbook. Teaching fourth-grade students using different textbooks From the comparison of the frequency of certain activities required by the teacher in the classrooms of fourth-grade students in Slovenia in 2015, we find significant differences in teaching with different textbooks. Teachers’ data on their teaching of students were linked with information on prescribed text - books at the student level and averaged for Slovenia. In mathematics, students more often state or explain their opinions if they have the first textbook, where - as teachers talk more often if they use the second textbook (Figure 6). Figure 6 Frequency of activities in mathematics lessons by prescribed textbooks, TIMSS 2015 Grade 4 In science, the differences are smaller but still noticeable, especially in the frequency of activities associated with experimentation (Table 2). One sci - ence textbook (TS 3(4)) stands out by the highest frequency of most teaching activities, especially planning experiments and other experimental activities, but also for the more frequent presentation of new content by the teacher. A c e p s Journal | V ol.12 | N o 2 | Y ear 2022 45 second textbook (TS 4(4)) stands out for the lowest frequency of all activities, especially the teacher’s explanations, while teachers also reported that the stu - dents were least likely to have to memorise facts and procedures when they used this textbook. Table 2 Frequency of approaches to teaching science by prescribed textbooks, Grade 4 Frequency of activities (some lessons or half of the lessons) TS1(4) TS2(4) TS3(4) TS4(4) Listen to me explain new science content some some some some Have students memorise facts and principles some some some some Read their textbooks or other resource materials some some some half Observe natural phenomena such as the weather and describe what they see some some some some Watch me demonstrate an experiment or investigation some half half half Use evidence from experiments or investigations to support conclusions half half half half Present data from experiments or investigations half half half half Conduct experiments or investigations half half some half Interpret data from experiments or investigations half half half half Design or plan experiments or investigations half half half half Differences in fourth graders’ attitudes towards learning according to the textbook As part of the TIMSS survey, the students were also asked to rate their attitudes towards learning mathematics and science, reporting their level of agreement with statements about learning each subject. Four scales of attitudes were modelled from the students’ responses for each subject: liking learning mathematics and science, self-confidence, valuing mathematics and science, and perception of engaging teaching by their subject teachers. The students were assigned values on each scale that had international means of 10 points and standard deviations of 2 points. Higher values on the scale describe a more positive attitude. An analysis of the comparison of mean scores on the attitudinal scales by the prescribed textbooks follows the opposite pattern as demonstrated knowledge (Figure 7). The use of the mathematics textbook TM 2(4) shows the highest student motivation to learn, and TM 3(4) and TM6(4) the lowest. We already know that the textbook TM 2(4) is associated with the lowest level of knowledge among students, and textbooks TM 3(4) and TM6(4) with the 46 textbooks and students’ knowledge highest. We can thus confirm the presence of an internationally known para - dox between achievement and attitudes (the attitudes-achievement paradox, Min et al., 2016), which shows higher achievement in groups of students with lower attitudes and, vice versa, lower achievement in groups of students with higher attitudes. This is significant in the across country comparisons and has been measured and reported several times for the international large scale as - sessments TIMSS and PISA. It is strongly present in the countries of East Asia, where they try to explain it to some extent with a culture of modesty in report - ing personal attitudes. It is also clearly seen in the case of the Slovenian com - parison of student attitudes towards the use of a mathematical textbook in the fourth grade. Roughly speaking, students with higher achievement develop less positive attitudes towards learning mathematics and science than students who are less successful in learning. Figure 7 Liking learning vs. achievement by prescribed textbooks, TIMSS 2015 Grade 4 The paradox of the conflicting results in achievement and motivation for learning is repeated in fourth-grade science. The trends fall from the high - est motivation to learn when using the textbook TS 4(4), where achievement is lowest, to the lowest motivation for learning with the textbook TS 3(4), which shows the highest achievement (Figure 7). Since the graphs for both subjects are drawn in the same axis ratio, we also see a direct comparison between the large differences in attitudes towards learning mathematics and the small differences in attitudes towards learning science between the textbooks for both subjects. Note, however, that direct comparisons of absolute differences on the scales of both attitudes and achievement are not possible, as each scale is modelled separately. c e p s Journal | V ol.12 | N o 2 | Y ear 2022 47 Differences in the knowledge of Grade 8 students using different textbooks The TIMSS collected data for mathematics and natural science, which in the Slovenian education system includes biology, physics, chemistry and ge - ography. Below we limit ourselves to mathematics and the first three typical natural science subjects. In the first step, we observed the shares of students who studied accord - ing to a particular textbook and their achievements in the TIMSS for biology, chemistry, physics and mathematics. The overall science achievement should not be used, as textbooks are created for each school subject separately. In mathematics, we were able to examine overall knowledge and knowledge in individual fields of mathematics, which was reflected in the scales of individual types of achievements. The differences between achievements seem large, but are not statistically significant due to the large differences in the proportion of students using textbooks. The presented figures (8 and 9) include self-con - fidence intervals that single out some achievement differences as significant. The differences in achievement in biology are not statistically significant among students who were using different textbooks (Figure 8). Nor is there any difference between students who have or do not have a prescribed printed text - book. In chemistry, where more textbooks (six) were used, three of which were used by most students, the achievements differed between students more than in biology. There are no statistically significant differences between the three most common textbooks, but there is a significant difference in achievement between TC 1 (8) and TC6(8) in favour of TC 1 (8), which also shows the highest average achievement of students. Figure 8 Achievements in biology and chemistry by different textbooks, TIMSS 2015 Grade 8 48 textbooks and students’ knowledge Even in physics (Figure 9), where two thirds of the students used one textbook, two other textbooks were used by 13% and 1% of the students, re - spectively, and almost a fifth of the students do not use any printed textbook, the average achievement does not differ significantly among all of these four groups of students. In mathematics, two thirds of all of the students used one of the textbooks. Students achievement does not differ significantly because the groups using the other five textbooks are small and the confidence intervals wide (T able 3). Nevertheless, there is a trend for the achievements in each math - ematics topic to follow the pattern of differences in the overall achievement among the students who learn with the aid of different textbooks. The data do not show that the students achieved a noticeably higher or lower level of knowl - edge in individual mathematics topics with one of the textbooks. It is, however, interesting to note that the achievement of the fifth of the students without a prescribed printed textbook was not the lowest. Figure 9 Achievements in physics and mathematics by different textbooks, TIMSS 2015 Grade 8 c e p s Journal | V ol.12 | N o 2 | Y ear 2022 49 Table 3 Proportions of students and mean mathematics achievement by content areas in 2015 by different textbooks, Grade 8 Prescribed textbooks % of students Mathematics (st. n.) Algebra (st. n.) Number (st. n.) Geometry (st. n.) Data (st. n.) TM1(8) 2.8 509.3 (10.4) 495.0 (7.9) 517.9 (8.6) 519.1 (9.5) 514.0 (12.4) TM2(8) 65.6 518.3 (2.6) 500.7 (2.9) 525.8 (3.0) 523.6 (3.3) 527.2 (3.2) TM3(8) 7.8 507.4 (7.1) 488.2 (7.6) 514.5 (8.6) 512.5 (7.4) 513.0 (9.7) TM4(8) 2.0 524.2 (9.7) 509.3 (9.8) 530.0 (11.8) 537.8 (11.7) 530.3 (8.9) TM5(8) .5 536.8 (1.4) 512.2 (4.7) 542.7 (4.2) 534.1 (4.0) 546.9 (1.3) TM6(8) .7 504.9 (4.2) 484.3 (3.5) 511.0 (4.3) 508.9 (6.5) 513.1 (6.4) NoTM(8) 20.7 513.7 (4.7) 494.0 (4.6) 521.2 (5.1) 520.4 (5.0) 521.7 (5.4) We were able to compare mathematical achievement according to the cognitive aspect. For science, achievement on cognitive scales is only available for the overall science score and is therefore not useful for our analysis. Figure 10 shows the average achievement on three cognitive scales – knowledge of facts, use of knowledge and reasoning – according to the textbook used in the classroom. The average achievement in these scales again follows a common pattern of differences between textbooks. Since most of the differences are not significant, we are unable to confirm that the textbook TM 2(8), which is used by two thirds of the students, is a more or less successful tool for acquiring mathematical knowledge than other textbooks. Figure 10 Achieved scores on cognitive scales of mathematics by textbooks, TIMSS 2015 Grade 8 Differences in teaching students with different textbooks in Grade 8 Teaching with different textbooks was compared with the teacher’s re - port on the frequency of certain activities with students in classes on a scale of 1 50 to 4 (4 = every or almost every lesson, 3 = about half of the lessons, 2 = at some lessons, or 1 = never). There are noticeable differences between lessons in all science subjects (Figure 11). In biology, the classes differ most for students who have a textbook and those who do not. In physics, these activities are least often carried out by students using the textbook TP 1 (8), and most often by those who use TP 3(8) or are without a textbook. In chemistry, one textbook stands out, as students are very often (every lesson) asked to read from it and remember the facts. When observing data, we should remember that students may have their own textbooks or online textbooks that are not reported on the school level. Figure 11 Frequencies of teaching activities in biology and physics by prescribed textbooks, Grade 8 Note. 1 = never, 2 = some lessons, 3 = every second lesson, 4 = every or almost every lesson. textbooks and students’ knowledge c e p s Journal | V ol.12 | N o 2 | Y ear 2022 51 Figure 12 Frequencies of teaching activities in mathematics by prescribed textbooks, Grade 8 Note. 1 = never, 2 = some lessons, 3 = every second lesson, 4 = every or almost every lesson. In mathematics, the pattern of differences is different. T extbooks TM 1 (8) and TM5(8) deviate the most, while the textbook TM 2(8), which is used by the vast majority of the students, does not stand out in any activity (Figure 12). Although the shares of students using other textbooks are small, the analysis clearly shows that the teaching methods vary according to the textbook used. Differences in attitudes of the students towards learning and textbooks used The students’ views are measured by a scale with an international av - erage of 10 points and higher values for more positive views. An analysis of differences in the students’ attitudes towards mathematics and specific science subjects broken down by the textbooks used shows that the self-confidence of students has the highest values across all textbooks (Figure 13). Other attitudes do not follow the same pattern and differences are not always statistically signif - icant. Across most of the textbooks, self-confidence is followed by the students’ perception of the engaging teaching of mathematics, which also measures the teacher’s direct attitude towards the student. Surprisingly, the textbook TM 1 (8) achieved almost the same high score in the students’ perception of the teacher’s commitment to teaching as the self-confidence of the students, which is much higher than the average of the students using other textbooks. Liking learning of mathematics shows a pronounced decline in the textbook TM6(8). 52 textbooks and students’ knowledge The data about students who do not have a printed textbook are also in - teresting. The values of all of the students’ attitudes, except for somewhat higher self-confidence, are very similar and reach the mean values of the students who have textbooks. Therefore, the data do not show that the use of specific textbooks increases motivating factors for students (see t-test statistics in the Appendix, Table 1), but only that the textbooks differ in their influence on attitudes. Figure 13 TIMSS attitudes toward mathematics by prescribed textbooks, TIMSS 2015 Grade 8 Figure 14 TIMSS attitudes toward science subjects by prescribed textbooks, TIMSS 2015 Grade 8 The differences in liking learning science subjects and self-confidence in science subjects between students with different textbooks are very small, with the exception of chemistry. Students prefer to learn chemistry and are confident c e p s Journal | V ol.12 | N o 2 | Y ear 2022 53 in their chemical knowledge when using the textbook TC6(8) (see t-test statis - tics in the Appendix, Tables 2–4). Use of different textbooks and National Assessment National Assessment (NA) items are derived from the curriculum and reflect the requirements and expectations of the curriculum even more accu - rately than TIMSS. Therefore, differences in the performance of students using different textbooks could point even more directly to the unequal coverage of the curriculum in the textbook used by the students. As mentioned above, the data on the use of textbooks were linked to anonymised achievements of NA in Mathematics (2016), Chemistry (2015), Bi - ology (2017) and Physics (2016). For the most widely used textbooks, we looked at achievement in each item and checked whether there were differences in achievements between groups. Table 4 shows an example of items for physics with correct proportions for different textbooks. Table 4 National Assessment results by four textbooks on selected items in Physics 2016 (Grade 9) Proportion correct (PC) Content information Item Total TP1(9) TP2(9) TP3(9) TP4(9) Max diff. Content Taxonomy* Item type** 01 .53 .58 .40 .54 .45 .18 Light UA MC 02 .88 .90 .82 .89 .82 .08 Force KR MC 03 .59 .59 .63 .59 .56 .06 Force UA MC 04 .30 .33 .19 .29 .37 .18 Force UA SA 05 .67 .73 .61 .67 .67 .12 Pressure, density, buoyancy KR SA 06 .14 .14 .11 .14 .15 .05 Pressure, density, buoyancy UA SA 07 .46 .47 .44 .47 .42 .05 Work, energy, heat UA MC 08 .60 .63 .52 .60 .61 .11 Motion UA MC 09 .54 .59 .46 .54 .42 .17 Motion UA MC 10 .46 .39 .39 .47 .53 .14 Space UA MC 11 .95 .96 .93 .95 .92 .04 Space KR MC 54 textbooks and students’ knowledge Proportion correct (PC) Content information Item Total TP1(9) TP2(9) TP3(9) TP4(9) Max diff. Content Taxonomy* Item type** 12.1 .50 .52 .50 .51 .46 .07 Sound and waves KR SA 12.2 .52 .54 .48 .52 .57 .10 Sound and waves KR SA 13.a .37 .46 .34 .37 .28 .18 Light KR MC 13.b .28 .50 .14 .27 .22 .36 Light KR MC 14.1 .32 .34 .20 .32 .33 .14 Force UA SA 14.2 .40 .42 .27 .40 .36 .15 Force UA SA 15.a .70 .73 .65 .71 .67 .08 Pressure, density, buoyancy PS MC 15.b .41 .48 .46 .41 .32 .16 Pressure, density, buoyancy PS MC Note. *UA – understanding and application; KR – knowledge and recognition; PS – problem solving.**MC – multiple choice; SA – short answer The results for all of the school subjects are presented in Figures 15– 20 by distributions of national proportions of correct answers for individual items (each vertical line), according to the textbooks prescribed to students by schools. Figure 15 Proportions of correct answers in NA in Mathematics (2016) for students using three different textbooks, Grade 9 c e p s Journal | V ol.12 | N o 2 | Y ear 2022 55 Figure 16 Proportions of correct answers in NA in Biology (2017) for students using three different textbooks, Grade 9 Figure 17 Proportions of correct answers in NA in Chemistry (2015) for students using three different textbooks, Grade 9 56 textbooks and students’ knowledge Figure 18 Proportions of correct answers in NA in Physics (2016) for students using four different textbooks, Grade 9 An analysis of the proportion of correct answers in each subject shows that the differences between groups are usually not large and that more difficult items appear more difficult in all groups. The differences between groups are sig - nificantly smaller than the differences between the difficulties of different items. For chemistry and biology, the differences between the textbooks are negligible in most cases: the group labels on the graphs are virtually overlapping. In math - ematics, the differences are small, but systematic. The data for physics show larger variations from item to item, but it is more difficult to extract a systematic pat - tern. It should be noted that there could be several reasons for the differences between groups. Since they were not selected at random, it cannot be excluded that the groups are different in their initial abilities and these differences have been reflected in the observed deviations. On the other hand, some of the differ - ences may also be due to actual differences in the treatment of specific content in textbooks and, consequently, to the impact of the textbook in achieving the knowledge associated with particular content. For example, there is a very large difference in Item IT 15 (source: NEC, 2016) in physics (Figure 16). Example IT15: How is an image displayed in an eye and how is it displayed in a camera obscura? Select the correct answer. A If it is flipped in the eye, it is flipped in the camera obscura. B If it is not flipped in the eye, it is flipped in the camera obscura. C If it is flipped in the eye, it is not flipped in the camera obscura. D If it is not flipped in the eye , it is not flipped in the camera obscura. c e p s Journal | V ol.12 | N o 2 | Y ear 2022 57 IT15 was a multiple choice item, so we can examine the frequency of the correct answers (answer A) as well as the frequencies of the three incorrect answers (B, C and D) (Table 5). Table 5 Proportion correct in Item IT 15 by prescribed textbooks Textbook A B C D NR TP1 (9) 49.8% 19.7% 28.3% 1.8% 0.4% TP2 (9) 13.7% 28.6% 39.8% 16.8% 1.2% TP3 (9) 26.9% 29.8% 37.7% 4.9% 0.6% TP4 (9) 21.6% 27.9% 41.1% 9.5% 0.0% Note. Adapted from NA Physics, 2016. The task was difficult for the students and the correct answer was not even the most common. We see quite a lot of variation from the response pat - terns. For the first textbook TP 1 , the proportion of correct student answers is much higher than for the other textbooks and the correct answer is the most common answer. TP 3 and TP 4 have similar patterns of answers. Special atten - tion is given to TP 2, where the students were more likely to choose answer D, which was a very rare choice in the other textbooks. Although the factors influ - encing the specific response of each group cannot be determined without sub - stantive analysis of the textbooks, the differences are clear and suggest that, at least in the case of Item IT 15 , such analyses could be successfully implemented and conclusions reached. Discussion Comparisons between groups of students who learn mathematics and science with the aid of different textbooks show that in the fourth and eighth grades, textbooks are an essential element of teaching that shapes the teacher’s teaching and knowledge as well as the attitudes towards the subject and the knowledge of the students. Although the comparison, especially for Grade 4, was limited by incomplete data and, particularly in mathematics, restricted to the small part of the population that uses a printed textbook for learning, the results give rise to two dilemmas. The use of different textbooks can lead to differences in student out - comes. Teachers should be aware of the relationships between achievement, attitudes and a specific textbook in order to select the best textbook for their 58 textbooks and students’ knowledge students. With this in mind, we cannot overlook the hypothesis that influence is actually bi-directional. A particular textbook can lead to greater achievement among students, but it is also possible that highly skilled teachers choose a par - ticular textbook more often than teachers who are less experienced in teaching the subject. This is also reflected in the comparison of teaching, where teach - ing differs in the frequency of learning opportunities between textbooks. Once again, the relationship is not causal, but most probably mutual. Teachers of stu - dents who demonstrate greater autonomy in learning mathematics are more likely to choose a textbook that supports and facilitates this method. If a par - ticular textbook does not allow so much independent learning or has a poorer presentation of content, the teacher must devote more time to his/her interpre - tation of the content and less time to the active participation of students. Different textbooks encourage a positive attitude towards learning dif - ferently. This information is essential for the Slovenian education system, where there is a trend of growing negative attitudes towards knowledge, learning and school. For teachers, it is essential that this information is not taken out of con - text, thus neglecting the connection between textbooks and student achieve - ment, as it has been demonstrated that textbooks with higher enjoyment result in lower mathematics achievement among the same students. Teachers who choose a textbook that motivates students to learn best and is the most popular among students could unwittingly choose a textbook that does not enable the highest knowledge to be achieved. This prompts publishers and authors of text - books to try to design effective textbooks that facilitate high knowledge acquisi - tion together with more joy in learning. Until this is achieved, the dilemma for teachers remains, as some textbooks will in themselves inspire more joy, while others will probably lead to higher knowledge. The most recent data for 2015 suggests that greater access to online textbooks and materials leads to generally higher knowledge among learners. In 2015, this was true for science subjects, but less true for mathematics. If we combine the findings on textbooks with information about teach - ing in Slovenia, we can see that teaching is increasingly based on the selected textbook, especially in the case of mathematics in the upper grades. This in itself would not be problematic if textbooks reliably presented all of the curricular content and objectives. In 2011, there was a significant update of the curriculum in Slovenia, at least for mathematics and science subjects. New editions of text - books were supposed to replace old copies in the school library rental system. However, such changes take time and, at least in 2015, the available textbooks for students were still not all fully aligned with the new changes. Given that les - sons are primarily based on textbooks, this means that students with the earlier c e p s Journal | V ol.12 | N o 2 | Y ear 2022 59 editions of textbooks were taught according to a slightly outdated curriculum. In fact, lower achievement in topics introduced in the eighth grade during the 2011 curriculum revision was observed in 2015 among Grade 8 students, espe - cially high achievers, based on the example of an extremely poor knowledge of algebraic expressions in mathematics. These topics received a great deal of attention in the revision process and were included in the revised curriculum. However, a workshop with teachers participating in the project KAUČ (Quality of Slovenian Textbooks) revealed that the content was not taught in schools, mainly because it was not presented in the editions of textbooks mostly still used in schools even in 2018. The new editions of textbooks did not replace the old ones also due to the economic crisis and a lack of funding from 2012 on. As a consequence of these problems on the systemic level, eighth-grade students missed the opportunity to learn the concept of formula, which is much needed for their age. According to the TIMSS in 2015 (TIMSS Almanacs, 2017), only 14% of Slovenian students were able to substitute variables with numbers and calculate the value of a formula, compared with 20% in some European coun - tries (Malta, Lithuania), 40% in Russia and more than 60% in Asian countries. National Assessment results provide an independent additional source of information. Comparing results between groups of students using different textbooks on specific items in most cases showed small differences. This is an indication that different textbooks performed similarly on most items. How - ever, not all of the differences were small. We presented the example of an item in physics and the results suggest that it is possible to search for meaningful explanations in the content of the textbooks. Conclusion Since the last extensive reform of compulsory school, we have experi - enced the evolution of textbooks and their use. Data from international stud - ies tell us that teaching is increasingly based on textbooks, as more and more teachers view textbooks as the primary source of their teaching. The relatively small differences in student achievement suggest that we cannot determine any of the textbooks used in this research as being more effective in achieving a higher level of knowledge. However, we did find differences between textbooks. The proportions of students whose teachers have chosen certain textbooks for their instruction vary between subjects. In chemistry, for example, three text - books are roughly equally popular, while a single textbook is most popular in mathematics. Further research could be undertaken to better understand the relationships between textbooks, teaching factors and students’ attitudes, 60 textbooks and students’ knowledge as we observed statistically significant differences in the attitudes of students using different textbooks towards learning and knowledge. This information can be important in the process of the teacher’s assessment and selection of a textbook for his/her students. Further qualitative analyses of each textbook would be necessary in order to determine which characteristics could lead to an improvement in students’ attitudes. In Slovenia, where we have struggled for many years with negative attitudes and resistance to learning, the message of the present study that there are differences in the liking of learning across dif - ferent textbooks is good news and should be explored further. When analysing the link between achievement and textbooks we were limited by the available data. International research is gathering less and less information on the precise use of printed textbooks, as their use has proved to be a universal condition for teaching and, on the other hand, the increasingly diverse use of different sources for teaching and learning cannot highlight the most effective approach at the international level. The textbook is primarily a reflection of the national curriculum and the orientation of each education system towards its specific objectives, which vary greatly between countries; therefore, the use of textbooks is not always comparable between countries. Data spanning more cohorts would also help to confirm our observations and generalise our findings. Despite the limitations imposed by the data, we found that teaching with different textbooks determines the way students are taught, students’ attitudes and achievement in different ways. Research into the im - pact of textbooks could benefit from more accurate data, with additions on the use of online textbooks, as well as qualitative reports from teachers regarding which textbooks are actually used in lessons with students. We expect a greater overview of the use of textbooks from an improved record of the use of materi - als resulting from the renovation of the Trubar portal. Acknowledgement The article was produced as part of the project “Za kakovost slovenskih učbenikov” (For the Quality of Slovenian Textbooks, https://kauc.splet.arnes.si), which is co-funded by the Republic of Slovenia and the European Union from the European Social Fund. c e p s Journal | V ol.12 | N o 2 | Y ear 2022 61 References Blazar, D., Heller, B., Kane, T., J., Polikoff, M., Staiger, D. O., Carrell, S., Goldhaber, D., Harris, D. N., Hitch, R., Holden, K., L., & Kurlaender, M. (2020). Curriculum reform in the common core era: Evaluating elementary math textbooks across six U.S. states. Journal of Policy Analysis and Management , 39(4), 1019. https://doi.org/10.1002/pam.22257 Foy, P ., & Olson, J.F . (2009a). TIMSS 2007 user guide for the international database : School background data almanac by mathematics achievement– 8th grade. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College. https://timss.bc.edu/TIMSS2007/PDF/T07_Almanacs.zip Foy, P ., & Olson, J.F. (Eds.)(2009b). TIMSS 2007 user guide for the international database: Science teacher background data alman ac by science achievement – 8 th grade. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College. https://timss.bc.edu/TIMSS2007/PDF/T07_Almanacs.zip Foy, P ., Arora, A., & Stanco, G. M. (Eds.) (2013). TIMSS 2011 user guide for the international database: TIMSS Almanacs 2011 . TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and International Association for the Evaluation of Educational Achievement (IEA). https://timss.bc.edu/TIMSS2011/downloads/T11_G8_Almanacs.zip Foy, P . (Ed.) (2017). TIMSS 2015 user guide for the international database : TIMSS Almanacs 2015. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and International Association for the Evaluation of Educational Achievement (IEA). https://timssandpirls.bc.edu/timss2015/international-database/downloads/T15_G8_Almanacs.zip Foy, P. (Ed.) (2017). TIMSS 2015 user guide for the international database :TIMSS international database 2015. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College and International Association for the Evaluation of Educational Achievement (IEA) . https://timssandpirls.bc.edu/timss2015/international-database/downloads/T15_G8_SPSSData_pt4.zip Hadar, L. L. (2017). Opportunities to learn: Mathematics textbooks and students’ achievements. Studies in Educational Evaluation, 55, 153–166. https://doi.org/10.1016/j.stueduc.2017.10.002 Van den Ham, A. K., & Heinze, A. (2018). Does the textbook matter? Longitudinal effects of textbook choice on primary school students’ achievement in mathematics. Studies in Educational Evaluation , 59, 133–140. https://doi.org/10.1016/j.stueduc.2018.07.005 Hardin, E. E., Eschman, B., Spengler, E. S., Grizzell, J. A., Taiyib Moody, A., Ross-Sheehy, S., & Kevin M. Fry. (2019). What happens when trained graduate student instructors switch to an open textbook? A controlled study of the impact on student learning outcomes, Psychology Learning & Teaching, 18(1), 48–64. https://doi.org/10.1177/1475725718810909 IDB Analyzer. (n.d.). http://www.iea.nl/data Japelj Pavešič, B., & Svetlik, K. (2016). Znanje matematike in naravoslovja med osmošolci v Sloveniji in po svetu. Izsledki raziskave TIMSS 2015 [Knowledge of mathematics and science among eighth-grade pupils in Slovenia and worldwide. Results from the TIMSS 2015 survey]. Pedagoški inštitut. http://timsspei.splet.arnes.si/?page_id=714 62 textbooks and students’ knowledge Johansson, M. (2003). Textbooks in mathematics education: A study of textbooks as the potentially implemented curriculum. Luleå University of Technology. https://www.diva-portal.org/smash/get/diva2:991466/FULLTEXT01.pdf LaRoche, S., Joncas, M., & Foy, P . (2016). Sample design in TIMSS 2015. In M. O. Martin, I. V . S. Mullis, & M. Hooper (Eds.), Methods and procedures in TIMSS 2015. TIMSS & PIRLS International Study Center, Boston College. http://timssandpirls.bc.edu/publications/timss/2015-methods.html Martin, M. O. (Ed.) (2005a). TIMSS 2003 User guide for the international database: TIMSS Almanacs 2003. School background data almanac by SCIENCE achievement – 8th grade. TIMSS & PIRLS International Study Center, Boston College. https://timss.bc.edu/timss2003i/userguide.html Martin, M. O. (Ed.) (2005 b). TIMSS 2003 user guide for the international database: TIMSS Almanacs 2003. Mathematics teacher background data almanac - 8th grade. TIMSS & PIRLS International Study Center, Boston College. TIMSS Almanacs 2003. https://timss.bc.edu/timss2003i/userguide.html Martin, M. O. (Ed.) (2005 c). TIMSS 2003 user guide for the international database: TIMSS Almanacs 2003. Science teacher background data almanac - 8th grade . TIMSS & PIRLS International Study Center, Boston College. TIMSS Almanacs 2003. https://timss.bc.edu/timss2003i/userguide.html Martin, M. O., Mullis, I. V . S., Foy, P ., & Hooper, M. (2016).  TIMSS 2015 international results in science. Boston College. http://timssandpirls.bc.edu/timss2015/international-results/ Mili, & Winch, C. (2019). Teaching through textbooks: Teachers as practitioners of a discipline? Theory and Research in Education, 17(2), 1–21. https://doi.org/10.1177/1477878519862547 Min, I., Cortina, K. S., & Miller, K. F. (2016). Modesty bias and the attitude-achievement paradox across nations: A reanalysis of TIMSS. Learning and Individual Differences , 51, 359–366. https://doi.org/10.1016/j.lindif.2016.09.008 Mullis, I. V . S., Martin, M. O., Foy, P ., & Arora, A. (2012). TIMSS 2011 international results in mathematics . TIMSS & PIRLS International Study Center. Mullis, I. V . S, Martin, M. O., Foy, P ., & Hooper, M. (2016). TIMSS 2015 international results in mathematics . TIMSS & PIRLS International Study Center. http://timssandpirls.bc.edu/timss2015/international-results/ Oakes, J., & Saunders, M. (2004). Education’s most basic tools: Access to textbooks and instructional materials in California’s public schools. Teachers College Record , 106(10), 1967–1988. https://doi.org/10.1111/j.1467-9620.2004.00423.x Oates, T. (2014). Why textbooks count . University of Cambridge, Local Examinations Syndicate. https://www.cambridgeassessment.org.uk/Images/181744-why-textbooks-count-tim-oates.pdf Portal Trubar (n.d.). https://paka3.mss.edus.si/Trubar/Javno/default.aspx Siegler, R. S., & Oppenzato, C. O. (2021). Missing input: How imbalanced distributions of textbook problems affect mathematics learning. Child Development Perspectives , 15(2), 76–82. https://doi.org/10.1111/cdep.12402 Sievert, H., van den Ham, A. K., & Heinze, A. (2021). Are first graders’ arithmetic skills related to the quality of mathematics textbooks? A study on students’ use of arithmetic principles. Learning and Instruction , 71, 101401. https://doi.org/10.1016/j.learninstruc.2020.101401 c e p s Journal | V ol.12 | N o 2 | Y ear 2022 63 Appendix Table 1 Mathematics textbooks and attitudes – t-test results* Mathematics Textbook Student value mathematics Student like mathematics Student confidence in mathematics Engaging teaching reference group compared group diff_se diff_t diff_se diff_t diff_se diff_t diff_se diff_t 1 2 0.39 -0.07 0.41 -0.45 0.39 0.36 0.46 -0.81 1 3 0.39 -0.49 0.43 -0.69 0.41 -0.17 0.48 -1.14 1 4 0.44 0.45 0.45 0.23 0.46 0.47 0.52 -0.38 1 No MT 0.39 0.12 0.42 -0.46 0.40 0.26 0.47 -0.97 2 3 0.08 -2.16 0.15 -0.77 0.11 -1.92 0.14 -1.24 2 4 0.24 0.96 0.21 1.35 0.25 0.31 0.25 0.71 2 No MT 0.08 0.98 0.11 -0.09 0.08 -0.46 0.11 -0.79 3 4 0.24 1.59 0.25 1.59 0.29 0.99 0.28 1.23 3 No MT 0.08 2.81 0.13 0.84 0.10 1.67 0.14 0.63 4 1 0.44 -0.45 0.45 -0.23 0.46 -0.47 0.52 0.38 4 2 0.24 -0.96 0.21 -1.35 0.25 -0.31 0.25 -0.71 4 3 0.24 -1.59 0.25 -1.59 0.29 -0.99 0.28 -1.23 4 No MT 0.24 -0.63 0.22 -1.35 0.25 -0.44 0.25 -1.02 Note. No MT = Group of students without prescribed Mathematics textbook.*Textbooks TM5 and TM6 were used by a very small share of the students, so a t-test is not available. Table 2 Physics textbooks and attitudes – t-test results Physics Textbook Student like physics Student confidence in physics Engaging physics teaching reference group compared group diff_se diff_t diff_se diff_t diff_se diff_t 1 2 0.86 0.30 0.74 0.75 0.81 0.69 1 3 0.86 0.14 0.74 0.74 0.80 0.54 1 No PT 0.87 0.11 0.75 0.61 0.82 0.50 2 3 0.15 -0.96 0.13 -0.12 0.16 -0.81 2 No PT 0.20 -0.81 0.18 -0.54 0.21 -0.75 3 1 0.86 -0.14 0.74 -0.74 0.80 -0.54 3 No PT 0.17 -0.10 0.17 -0.48 0.19 -0.14 Note. No PT = Group of students without prescribed Physics textbook. 64 textbooks and students’ knowledge Table 3 Chemistry textbooks and attitudes – t-test results Chemistry Textbook Student like chemistry Student confidence in chemistry Engaging chemistry teaching reference group compared group diff_se diff_t diff_se diff_t diff_se diff_t 1 2 0.18 -0.73 0.15 -0.63 0.23 -0.12 1 3 0.16 -0.34 0.15 -1.34 0.18 -0.30 1 4 0.14 -0,46 0.16 -0.98 0.14 0.67 1 No CT 0.16 -0.72 0.18 -1.15 0.18 0.17 2 3 0.19 0.41 0.17 -0.65 0.25 -0.09 2 4 0.18 0.40 0.16 -0.33 0.23 0.54 2 No CT 0.19 0.08 0.19 -0.60 0.25 0.24 3 2 0.19 -0.41 0.17 0.65 0.25 0.09 3 4 0.16 -0.06 0.17 0.34 0.18 0.83 3 No CT 0.16 -0.39 0.18 -0.01 0.19 0.43 4 3 0.16 0.06 0.17 -0.34 0.18 -0.83 4 No CT 0.16 -0.34 0.19 -0.30 0.18 -0.36 Note. No CT = Group of students without prescribed Chemistry textbook. Table 4 Biology textbooks and attitudes – t-test results Biology Textbook Student like biology Student confidence in biology Engaging biology teaching reference group compared group diff_se diff_t diff_se diff_t diff_se diff_t 1 2 0.15 0.81 0.12 1.34 0.15 1.72 1 No BT 0.24 0.35 0.23 0.31 0.23 0.54 2 No BT 0.24 -0.16 0.23 -0.41 0.22 -0.59 Note. No BT = Group of students without prescribed Biology textbook. c e p s Journal | V ol.12 | N o 2 | Y ear 2022 65 Biographical note Barbara Japelj Pavešić is a researcher at the Educational Research Institute, involved in international large-scale assessments in education, na - tionally coordinating Trends in International Mathematics and Science Study, TIMSS, and The OECD Teaching and Learning International Survey, TALIS. With the background in mathematics and statistics, her field of research is the statistical modelling of complex data to explain the knowledge and learning of mathematics and science of students K-13. Gašper Cankar, PhD, is a researcher and methodology expert work - ing on problems of knowledge assessment in a National Examinations Centre (NEC, Slovenia) methodologically supporting general and vocational Matura examinations and national assessment in Grades 6 and 9. Some of the topics he works on include error of rating process, value added models of knowledge, interplay between socioeconomic background and educational outcomes and Simpson’s paradox.