ACTA BIOLOGICA SLOVENICA
LJUBLJANA 2010 Vol. 53, Št. 2: 99-108
Biological knowledge of Slovenian students in the living systems content area in PISA 2006
Znanje slovenskih učencev na vsebinskem področju Živi sistemi v raziskavi PISA 2006
Jelka Strgar
University of Ljubljana, Biotechnical faculty, Department of Biology, Večna pot 111, SI-1000 Ljubljana, Slovenia Correspondence: jelka.strgar@bf.uni-lj.si
Abstract: In the PISA 2006 program Slovenian students exceeded the international average of scientific achievements, while in the field of living systems performed below the national average. The purpose of our analysis was to determine in which areas of biology Slovenian students are weak and in which they are strong, and to determine their biological knowledge in comparison to 23 other European countries, which, by various criteria, are the most comparable to Slovenia and interesting for us. We analysed 24 tasks that tested the biological knowledge in PISA 2006. We found that student achievement in Slovenia wasn't poor in any of the tested biological topics. There are two factors that can explain lower achievements resolving individual tasks: (1) Question type: Students poorly answered the open-constructed response questions that require independent formulation of coherent responses. They were more successful with complex multiple choice questions and multiple choice questions, where they had to choose the correct answer from several given suggestions; (2) Difficulty: Students didn't perform as well with higher cognitive level tasks which required using knowledge. This suggests that in the period when these students were receiving elementary education, their biology teaching focused on developing knowledge of biological content (knowledge of science) and competence explaining phenomena scientifically, while the development of other knowledge (knowledge about science) and the competence to draw evidence-based conclusions (using knowledge) was inadequate. It is therefore in the hands of all in Slovenia involved in biological education, especially teachers of biology, to give students more opportunities for problem-solving, rather than only focusing on content.
Key words: PISA 2006, biology, science, knowledge, competencies, Slovenia
Izvleček: Slovenski učenci so v raziskavi PISA 2006 presegli mednarodno povprečje naravoslovnih dosežkov, vendar so bili na področju biologije slabši kot na drugih naravoslovnih področjih. Namen naše analize je bil ugotoviti, na katerih področjih biologije imajo slovenski učenci šibko in na katerih močno znanje ter kakšno je njihovo biološko znanje v primerjavi s 23 drugimi evropskimi državami, ki so po različnih kriterij najbolj primerljive s Slovenijo oziroma zanimive za nas. Analizirali smo 24 nalog, ki so v PISA 2006 preverjale biološko znanje. Ugotovili smo, da so bili dosežki učencev v Sloveniji pri vseh preverjenih bioloških temah dobri. Dejavnika, s katerima je bilo mogoče pojasniti slabše dosežke pri posameznih nalogah, sta bila tip naloge in njena težavnost. Učenci so slabše reševali naloge odprtega tipa, pri katerih so morali samostojno oblikovati smiseln odgovor. Uspešnejši so bili pri nalogah kompleksnega izbirnega tipa in izbirnega tipa, kjer so morali med danimi odgovori izbrati pravilnega. Učenci so slabše reševali naloge višje kognitivne stopnje, ki so zahtevale uporabo znanja. Iz tega sklepamo, da je bil v obdobju, ko so bili ti učenci vključeni v osnovnošolsko izobraževanje, pri biologiji poudarek na razvijanju poznavanja biološke vsebine (knowledge of science) in kompetence znanstveno razlaganje pojavov (explaining phenomena scientifically), pomanjkljivo pa je bilo
razvijanje znanja o naravoslovnih znanostih (knowledge about science) ter kompetence uporaba naravoslovno-znanstvenih podatkov in preverjenih dejstev (using scientific evidence). V rokah vseh, ki se v Sloveniji ukvarjajo z biološkim izobraževanjem, predvsem pa učiteljev biologije, je, da usposobijo učence tudi s tega vidika, ne samo z vidika vsebinskega znanja.
Ključne besede: PISA 2006, biologija, naravoslovje, znanje, kompetence, Slovenija
Introduction
In 2006 Slovenia participated for the first time in the Program for International Student Assessment (PISA). The first PISA study in 2000 focused on reading literacy, the second in 2003 on mathematical literacy, and the third in 2006 on scientific literacy, which means the proportion of science tasks was significantly larger than the proportion of mathematical and reading tasks. The result was the first extensive and complete collection of internationally comparable data on the science competencies of students in 57 countries from all over the world.
The PISA differs from other assessments (for example from TIMSS, which is also international) in that it is not directly linked to the science curricula of any of the participating countries, but collects data about the competences that students will need for effective functioning in their adult professional and personal life, and are important both for individuals and for the whole society. According to the OECD definition (PISA 2006 2007), scientific literacy is "a capacity to use scientific knowledge, to identify questions, explain scientific phenomena, and to draw evidence-based conclusions in order to understand and help make decisions about science-related issues." The term 'literacy' indicates focusing on the application of knowledge and abilities (Bybee 2008). The tasks are structured in a manner that enables the assessment of the ability to solve tasks related to life situations, and hence not limited to knowledge of a specific subject (Straus et al. 2007).
The PISA includes students who, in most participating countries, are approaching the end of compulsory education. The age of participants is precisely specified, ranging from 15 years and 3 months to 16 years and 2 months. This means that the PISA 2006 students were born in 1990. In the study 6,595 Slovenian students took part (Straus et al. 2007).
Science achievements of Slovenian students in PISA 2006
Slovenian students in PISA 2006 scored the average science achievement of 519 points (Tab. 1), therefore exceeding the average achievement of 500 points of all the participating students by 19 points (Achievements of students in PISA 2006).
The PISA 2006 science assessment evaluated students' knowledge in two areas: knowledge component and competency component. Knowledge component comprises two categories - knowledge about science (which includes scientific inquiry and scientific explanations) and knowledge of science (which includes Earth and space systems, living systems, and physical systems). The results of Slovenian students in the category of knowledge about science was as many as 9 points below the national average, while in the category of knowledge of science they showed a good knowledge in the areas of Earth and space systems (15 points above average), and physical systems (12 points above average). In the area of living systems, covering biology, Slovenian students ranked lower than the national average by 2 points (Tab. 1). On the international scale the biggest difference between knowledge about science and knowledge of science in PISA 2006 was 29 points. In the case of Slovenian students this difference was 17 points in favour of knowledge of science. These results are not a clear indicator of whether one or the other of these two categories of knowledge leads to higher overall scientific achievements (Straus et al. 2007).
In the competency component (explaining phenomena scientifically) our students achieved 4 points above the international average (Tab. 1). The other two competences are less developed, but the differences are not significant: in the competence of identifying scientific questions they rank 2 points below the international
average, and in the competence of drawing evidence-based conclusions 3 points below the international average. In many countries with the highest achievements the students are particularly successful in the competence of drawing evidence-based conclusions. It turns out that in all
Table 1: Tabela 1:
Science	519
Competency component
1.	Identifying scientific questions
2.	Explaining phenomena scientifically
3.	Drawing evidence-based conclusions Knowledge component
1.	Knowledge about science
2.	Knowledge of science
- Earth and space systems - Living systems - Physical systems
Objective analysis
The purpose of our analysis was to determine the strong and weak areas of biological knowledge of Slovenian students, and where this knowledge ranks in comparison with other countries. These results allow us to extract the characteristics of biology teaching in Slovenian primary schools during the period when these students were in elementary school, and identify any necessary changes.
Material and methods
Tasks included in the analysis
Of the total set of tasks in the area of living systems, i.e. tasks related to biological topics in PISA 2006, one part did not test the knowledge
of the successful countries the students are quite strong in this competence. The conclusion is that the ability to draw evidence-based conclusions, i.e. interpret and apply scientific data and verified facts, is particularly characteristic of highly developed science literacy (Straus et al. 2007).
rankings
-2 4 -3 -9
15 -2 12
of (biology) science, but verified other components of scientific literacy (knowledge about science, identifying scientific questions, explaining phenomena scientifically, drawing evidence-based conclusions) based on biological content. These tasks are not included in our analysis because in this case biology was just a basis from which students demonstrated their knowledge and competencies not linked specifically to biology, but to any scientific topic. Students had to show knowledge of science in the content area of living systems in 27 tasks, of which 24 were used for our detailed analysis.
Countries selected for comparison with Slovenian achievements
There were 57 countries participating in PISA 2006, while for the purpose of our analy-
Comparison of the overall achievements of Slovenian students in science with the achievements of individual scientific competencies and in the various science areas (adapted from Štraus et al., p. 51). Primerjava skupnega dosežka slovenskih učencev pri naravoslovju z dosežki pri posameznih naravoslovnih kompetencah in na posameznih naravoslovnih področjih (povzeto po Štraus et al., p. 51).
Overall achieve- The difference between the overall achievement	ment and the achievements on individual
sis we presented the comparative performance of Slovenia and 23 other European countries, which are, according to various criteria (similar cultural background and history), most comparable with Slovenia and interesting for us. These were: Austria, Belgium, Switzerland, Czech Republic, Germany, Denmark, Spain, Estonia, Finland, France, Great Britain, Hungary, Ireland, Iceland, Italy, Lithuania, Latvia, Neder-land, Norway, Poland, Portugal, Slovakia, and Sweden.
Data analysis
The overall achievements of the selected 23 countries in each of the 24 questions that tested the knowledge of (biology) science were calculated. The significant differences between the achievements of Slovenian students with regard to biological topic and the type of question were determined by one way ANOVA and Chi-square test (P < 0.05).
Results and discussion
1.	General achievements of Slovenian students in the area of living systems
The achievements of Slovenian students in 24 biological questions ranged from 22.4% to 89.5% (Fig. 1). Within this, the majority of tasks placed between 45.0% and 77.9%. The upward deviation was demonstrated in one task with 89.5% and downward in a group of five tasks where the achievement was less than 37%.
2.	Achievements of Slovenian students in the area of living systems in terms of content (biological topic)
The 24 analysed tasks tested the knowledge of six areas of biology. The tasks were classified into four groups that contained 4 to 8 tasks each. We formed thematic groups: (1) ecology with 7 tasks; (2) physiology with 4 tasks; and (3) health with 8 tasks. The remaining three biological ar-
Figure 1: Achievements of Slovenian students in 24 tasks testing competency knowledge of science based on the
content area of living systems Slika 1: Dosežki slovenskih učencev pri 24 nalogah, ki so preverjale kompetenco znanje naravoslovja z vsebinskega področja živi sistemi.
Figure 2: Medians, quartiles, and extreme values of the achievements of Slovenian students in tasks testing the
knowledge of science based on the content area of living systems. Slika 2: Mediane, kvartili in skrajne vrednosti dosežkov slovenskih učencev pri nalogah, ki so preverjale znanje naravoslovja z vsebinskega področja živi sistemi. Tematska področja: ekologija, fiziologija, zdravje, drugo.
eas were combined with the fourth group of varied content with 5 tasks. Such a division of tasks is not fully in line with the official topics of the PISA study in the area of living systems, which are: Health, Natural Resources, Environmental Quality, Hazards, and Frontiers of Science and Technology (PISA 2006 2007), but it seemed sensible from the standpoint of biological content, and, additionally, it gave us the minimum number of tasks for statistical data processing in each group.
The achievements of Slovenian students are considerably dispersed (Fig. 2) in all three subject areas (ecology, physiology and health). In the fourth group, which includes tasks from various other areas of biology, the achievements are more level, which is attributed to the fact that all the tasks in this group were of the same
type, in this case complex multiple choice questions. Students gave the best answers in the areas of physiology and health (median = 60.5 and 59.8), while the achievement in ecology is lower (median = 45.7), but the differences among the achievements in individual thematic areas are not statistically significant (ANOVA, P > 0.05)
3. Achievements of Slovenian students in the area of living systems with regard to the type of question
The 24 tasks used in our analysis were presented in three different ways (8 tasks each): in the form of the complex multiple choice questions (a set of options from which the student chooses one); in the form of multiple choice questions (4 or 5 suggested answers of which
Figure 3: Medians, quartiles, and extreme values of the achievements of Slovenian students in tasks testing the knowledge of science based on the content area of living systems. CMCQ - complex multiple choice questions, MCQ - multiple choice questions, Open - open-constructed response questions.
Slika 3: Mediane, kvartili in skrajne vrednosti dosežkov slovenskih učencev pri nalogah, ki so preverjale znanje naravoslovja z vsebinskega področja živi sistemi. Tipi nalog: C - naloge kompleksnega izbirnega tipa, M - naloge izbirnega tipa, O - naloge odprtega tipa.
only one is correct); and in the form of open-constructed response questions (which require a longer written answer).
Achievements of Slovenian students with regard to the type of task are not so dispersed (Fig. 3) as the achievement with regard to biological topic. Students gave the best answers to multiple choice questions and slightly less appropriate answers to the complex multiple choice questions (median = 69.3 and 57.5), but this difference is not statistically significant (ANOVA; P > 0.05). Students showed significantly lower achievements answering open-constructed response questions (median = 45.1; ANOVA; P = 0.004).
Other research, for example National Assessment of Knowledge in Slovenia, showed that students have difficulties in formulating responses to open-constructed response questions, which applies not only to biology, but to all areas (Jagodnik et al. 2009). Poor performance on these questions thus indicates lack of ability to formulate the knowledge into sensible short answers rather than lack of content knowledge.
4. Comparison of Slovenian students ' achievements depending on topic and type, and deviations from the average of 24 countries
A detailed analysis showed that the achievements of Slovenian students were mainly influ-
100
Questions
Figure 4: Achievements of Slovenian students in 24 tasks testing competency knowledge of science based on the content area of living systems, and relative deviations of these achievements compared to students of 23 European countries. Shown according to the relative deviation value.
Slika 4: Dosežki slovenskih učencev pri 24 nalogah, ki so preverjale kompetenco znanje naravoslovja na vsebinskem področju živi sistemi in relativno odstopanje dosežkov slovenskih učencev v primerjavi z učenci 23 evropskih držav. Prikaz po velikosti relativnega odstopanja.
enced by the type of task, and not by the biological content that a task tested (x2 test, P = 0.03). Some tasks deviated from these findings, in other words, their result was influenced by biological content and not by the task type. Only from those tasks it is therefore possible to identify the strong and weak areas and other characteristics of biological knowledge of Slovenian students.
With regard to the biological topic, students were less successful in resolving four tasks: two in ecology, and one in physiology and health. The biological content of these tasks can be found in the Slovenian education curricula for primary school, which means that our students must have been exposed to the content. In our view the content should not have been too difficult for the students. But the difficulty increased as a result of the way the questions were presented. All four questions the Slovenian students had trouble with required the utilisation of knowledge. We concluded that students in Slovenia don't lack biological knowledge (knowledge of science), but other knowledge and competencies, namely
knowledge about science and the competence to draw evidence-based conclusions.
We also arrived at the same conclusion when we analysed the relative deviations of the achievements of Slovenian students comparing to 23 European countries comparable to Slovenia for all 24 tasks testing the knowledge of science in the content area of living systems (Fig. 4). None of the tasks in which the Slovenian students exceeded the average (right side of the graph) was, in our opinion, cognitively challenging. This means that upward deviations of achievements of Slovenian students can be largely explained by lower levels of cognitive difficulty of tasks. On the other hand, it is our view that the tasks Slovenian students performed below average (left side of the graph) are cognitively more demanding, though one of them was impossible for Slovenian students to answer since Slovenian schools don't address the content in the way the question anticipated. Our finding that students in Slovenia don't lack biological knowledge (knowledge of science),
but other knowledge and competencies, is consistent with what Straus et al. (2007) wrote in the PISA 2006 National report for the entire area of science. They say that in order to effectively solve science tasks a sequence of three processes is required: (1) students must first recognize the problem; (2) use their knowledge to explain phenomena scientifically; (3) and finally interpret and apply the results. Traditional teaching of science is often directed at another process, namely the scientific interpretation of phenomena or, in other words, the acquisition of key science knowledge. From the achievements of Slovenian students Straus et al. (2007) concluded that the Slovenian elementary schools focus precisely on these work methods. Such teaching doesn't lead to comprehensive science literacy because later in life the student's ability to successfully use these data and arguments will be limited despite good command of the theory.
In Slovenia, the current elementary school curriculum for the area of living systems, at least at the primary school level, considers the need for students to develop both content knowledge and the process of science, as they followed the recommendations in Benchmarks (American Association for the Advancement of Science 1993). But the curricula for biology at higher levels of primary school do not. They are instead content-focused, with few process goals, and it is therefore not surprising that our students in PISA 2006 showed relatively good content knowledge of biology (knowledge of science and competence explaining phenomena scientifically), and, on the other hand, a considerable lack of other competences and knowledge about science.
The PISA 2006 results not only offer insights into the current status, but also allow for predictions of how today's 15 year olds will function as adults citizens who will need to use their knowledge and abilities in new situations and make decisions on issues related to, for example, the common good of humankind, wellness, the environment, human ecology, and other applications of biology (Bybee 2008). Education in the 21st century should consider the changing circumstances. DeHard Hurd (2001) claims that today's data is more qualitative than quantitative in nature. The intellectual skills required to do research in today's biology are mostly those of
problem-solving. It is therefore in the hands of all in Slovenia involved in biological education, especially teachers of biology, to give students opportunities to utilise their knowledge and not only focus on content.
Conclusions
The analysis of Slovenian students' achievements in 24 questions in the area of living systems in PISA 2006 showed that:
1.	The achievements of Slovenian students were good in all of the tested biological topics.
2.	There are two factors that could explain the poor performance in specific tasks:
-	Question type: Students were less successful in answering open-constructed response questions where they had to independently formulate a coherent response. They were better with the complex multiple choice questions and multiple choice questions, where they had to pick a correct answer among several suggestions
-	Difficulty: Students were less successful with higher cognitive level tasks that required the use of knowledge. This suggests that in the period, during which the students were given elementary education, biology teaching focused mainly on developing knowledge of biological content (knowledge of science) and competencies in explaining phenomena scientifically, and less on developing other knowledge (knowledge about science) and competences in drawing evidence-based conclusions (use of knowledge).
The sample of tasks that we were able to include in the analysis was predetermined by the PISA 2006 structure. Because it's so small, our findings should be viewed with caution. In addition, due to a small number of available tasks the achievements of the students were analysed only with regard to the content and type of the task, not taking into account a third possible aspect - proficiency level. Each task was in fact classified in one of six proficiency levels on the basis of substantive considerations relating to the nature of the underlying competencies (PISA 2006 2007).
From the standpoint of biology, a very important issue remains unresolved: what is the reason that Slovenian students achieved significantly lower results in the area of living systems than in the areas of Earth and space systems, and physical systems. This information could be obtained by an overall interdisciplinary analysis aimed at discovering the essential characteristics of biology, physics, and chemistry teaching in Slovenian elementary schools. A comparison would point to the aspects of teaching that strongly deviate in biology and are the likely reason for poorer performance of the students.
Povzetek
Slovenski učenci so v raziskavi PISA 2006 dosegli povprečni naravoslovni dosežek 519 točk, kar presega povprečje vseh sodelujočih držav (500 točk). Znotraj naravoslovja so bili slovenski učenci nadpovprečni na področju fizike, kemije in geografije, medtem ko so bili na področju biologije rahlo podpovprečni. Namen naše analize je bil ugotoviti, na katerih področjih biologije imajo slovenski učenci šibko in na katerih močno znanje ter kakšno je njihovo biološko znanje v primerjavi s 23 drugimi evropskimi državami, ki so po različnih kriterij najbolj primerljive s Slovenijo oziroma zanimive za nas.
Iz celotnega nabora nalog, ki so bile v raziskavi PISA 2006 povezane z biološko tematiko, je bil del takih, ki so na bioloških vsebinah preverjale znanje o naravoslovnih znanostih (knowledge about science) in ne naravoslovnega znanja (knowledge of science) samega. Teh nalog nismo analizirali, ker je bila biologija v tem primeru samo tema, na kateri so morali učenci pokazati kompetenci znanstveno razlaganje
(scientific explanations) ali znanstveno raziskovanje (scientific enquiry), ki nista vezani samo na biologijo, temveč na katerokoli naravoslovno znanost. Kompetenco znanja (knowledge of science) na področju biologije so učenci morali pokazati pri 27 nalogah, od katerih smo jih 24 uporabili za našo analizo. Naloge smo razvrstili v tri tematske skupine: ekologija s 7 nalogami, fiziologija s 4 nalogami in zdravje z 8 nalogami. Preostalih pet nalog smo združili v vsebinsko raznoliko četrto skupino.
Ugotovili smo, da dosežki učencev v Sloveniji pri vseh preverjenih bioloških temah dobri. Dejavnika, s katerima je bilo mogoče pojasniti slabše dosežke pri posameznih nalogah, sta bila tip naloge in njena težavnost. Učenci so slabše reševali naloge odprtega tipa, pri katerih so morali samostojno oblikovati smiseln odgovor. Uspešnejši so bili pri nalogah kompleksnega izbirnega tipa in izbirnega tipa, kjer so morali med danimi odgovori izbrati pravilnega. Učenci so slabše reševali naloge višje kognitivne stopnje, ki so zahtevale uporabo znanja. Iz tega sklepamo, da je bil v obdobju, ko so bili ti učenci vključeni v osnovnošolsko izobraževanje, pri biologiji poudarek na razvijanju poznavanja biološke vsebine (knowledge of science) in kompetence znanstveno razlaganje pojavov (explaining phenomena scientifically), pomanjkljivo pa je bilo razvijanje znanja o naravoslovnih znanostih (knowledge about science) ter kom-petence uporaba naravoslovno-znanstvenih podatkov in preverjenih dejstev (using scientific evidence). V rokah vseh, ki se v Sloveniji ukvarjajo z biološkim izobraževanjem, predvsem pa učiteljev biologije, je, da usposobijo učence tudi s tega vidika, ne samo z vidika vsebinskega znanja.
References
American Association for the Advancement of Science, 1993. Benchmarks for Science Literacy. New
York, Oxford University Press, 418 pp. Bybee, R.W., 2008. Scientific Literacy, Environmental Issues, and PISA 2006: The 200 Paul F-Bran-
dwein Lecture. J Sci Educ Technol, 17, 566-585. DeHard Hurd, P., 2001. The Changing Image of Biology. The American Biology Teacher, 63 (4), 233-235.
Dosežki učencev v raziskavi PISA 2006. http://www.pei.si/UserFilesUpload/file/raziskovalna_deja-
vnost/PISA/PISA2006/Rezultati_raziskavePISA2006.pdf (15. 9. 2010) Jagodnik, A., Sopčič, B., Strgar, J., Volčini, D., Zupan, A., 2009. Analiza dosežkov nacionalnega preverjanja znanja iz biologije ob koncu tretjega obdobja. In: Rigler Šilc K. et Novak, M. (eds.): Nacionalno preverjanje znanja. Letno poročilo o izvedbi nacionalnega preverjanja znanja v šolskem letu 2008/2009. Ljubljana, Državni izpitni center, pp.179-191. http://www.ric.si/mma_bin.php/$fileI/2009121612174940/$fileN/Letno%20poročilo%202009%20 Tisk.pdf (15. 9. 2010)
PISA 2006, 2007. Science Competencies for Tomorrow's World. Vol. 1 - Analysis. OECD, 383 pp. Straus, M., Repež, Maša, M., Stigl, Simona, (eds.), 2007. Nacionalno poročilo PISA 2006: naravoslovni, bralni in matematični dosežki slovenskih učencev, Pedagoški inštitut, Ljubljana, 223 pp.