Letn i k XXV, števi I ka 5-6, 2014 Revija za teorijo in raziskave vzgoje in izobraževanja Šolsko polje Evidence from the PISA Study on Educational Quality in Slovenia and Other Countries ed. Mojca Straus Šolsko polje Revija za teorijo in raziskave vzgoje in izobraževanja Letnik XXV, številka 5-6, 2014 Šolsko polje je mednarodna revija za teorijo ter raziskave vzgoje in izobraževanja z mednarodnim uredniškim odborom. Objavlja znanstvene in strokovne članke s širšega področja vzgoje in izobraževanja ter edukacij-skih raziskav (filozofija vzgoje, sociologija izobraževanja, uporabna epistemologija, razvojna psihologija, pedagogika, andragogika, pedagoška metodologija itd.), pregledne članke z omenjenih področij ter recenzije tako domačih kot tujih monografij s področja vzgoje in izobraževanja. Revija izhaja trikrat letno. Izdajajo Slovensko društvo raziskovalcev šolskega polja. Poglavitni namen revije je prispevati k razvoju edukacijskih ved in interdisciplinarnemu pristopu k teoretičnim in praktičnim vprašanjem vzgoje in izobraževanja. V tem okviru revija posebno pozornost namenja razvijanju slovenske znanstvene in strokovne terminologije ter konceptov na področju vzgoje in izobraževanja ter raziskovalnim paradigmam s področja edukacijskih raziskav v okviru družboslovno-humanističnih ved. Uredništvo-. Valerija Vendramin, Zdenko Kodelja, Darko Strajn, Alenka Gril in Igor Z. Žagar (vsi: Pedagoški inštitut, Ljubljana) Glavni urednik-. Marjan Šimenc (Pedagoški inštitut, Ljubljana) Odgovorna urednica-. Eva Klemenčič (Pedagoški inštitut, Ljubljana) Pomočnica odgovorne urednice-. Mojca Rožman (Pedagoški inštitut, Ljubljana) Urednik recenzij za objavo-. Igor Bijuklič (Pedagoški inštitut, Ljubljana) Uredniški odbor. Michael W Apple (University of Wisconsin, Madison, USA), Eva D. Bahovec (Filozofska fakulteta, Univerza v Ljubljani), Andreja Barle-Lakota (Urad za šolstvo. Ministrstvo za šolstvo in šport RS), Valentin Bucik (Filozofska fakulteta. Univerza v Ljubljani), Harry Brighouse (University of Wisconsin, Madison, USA), Randall Curren (University of Rochester, USA), Slavko Gaber (Pedagoška fakulteta. Univerza v Ljubljani), Milena Ivanuš-Grmek (Pedagoška fakulteta. Univerza v Mariboru), Russell Jacoby (University of California, Los Angeles), Janez Justin t (Pedagoški inštitut, Ljubljana), Stane Košir (Pedagoška fakulteta. Univerza v Ljubljani), Janez Kolenc t (Pedagoški inštitut, Ljubljana), Ljubica Marjanovič-Umek (Filozofska fakulteta. Univerza v Ljubljani), Rastko Močnik (Filozofska fakulteta. Univerza v Ljubljani), Zoran Pavlovič (Svetovalni center za otroke, mladostnike in starše, Ljubljana), Drago B. Rotar (Fakulteta za humanistične študije. Univerza na Primorskem), Harvey Siegel (University ofMiami, USA), Marjan Šetinc (Slovensko društvo raziskovalcev šolskega polja, Ljubljana), Pavel Zgaga (Pedagoška fakulteta. Univerza v Ljubljani), Maja Zupančič (Filozofska fakulteta. Univerza v Ljubljani), Robi Kroflič (Filozofska fakulteta, Univerza v Ljubljani), Marie-Helene Esteoule Exel (Universite Stendhal Grenoble III) Lektor (slovenski jezik), tehnični urednik, oblikovanje in prelom: Jonatan Vinkler Lektor (angleški jezik)-.)zson^cn([on¥>2£son Izdajatelja-. Slovensko društvo raziskovalcev šolskega polja in Pedagoški inštitut © Slovensko društvo raziskovalcev šolskega polja in Pedagoški inštitut Tisk-. Grafika 3000 d.0.0.. Dob Naklada: 400 izvodov Revija Šolsko polje je vključena v naslednje indekse in baze podatkov: Contents Pages in Education; EBSCO; Education Research Abstracts; InternationalBibliography of the SocialSciences (IBSS); Linguistics and,Language Behavior Abstracts (ELBA); MithiathitralEditcation Abstracts; Pais International; ProQttestSocialSciencesJournal, Re-sec trch into Higher Edna ition Abstr, icts; Sod J Services Abstn icts; Sociologic, ilAbstn icts; Worldwide Politic, J Science Abstracts Šolsko polje izhaja s finančno podporo Pedagoškega inštituta in Javne agencije za raziskovalno dejavnost Republike Slovenije. Tiskana izdaja: ISSN 1581-6036 Izdaja na zgoščenki: ISSN 1581-6052 Spletna izdaja: ISSN 1581-6044 Letnik XXV, številka 5-6, 2014 Revija za teorijo in raziskave vzgoje in izobraževanja Šolsko polje Evidence from the PISA Study on Educational Quality in Slovenia and Other Countries ed. Mojca Straus Contents/Vsebina 1 EDITORIAL/U VODNIK 5 Mojca Straus ■ The Timeless Questions About Educational Quality 7 2 PAPERS/RAZPRAVE 11 Darko Strajn ■ The PISA Syndrome: Can we Imagine Education without Comparative Testing? 13 Urska Stremfel ■ Slovenia on its Own Way Towards Improving PISA Results 29 Christine Sàlzer. and Manfred Prenzel* Looking Back at Five Rounds of PISA: Impacts on Teaching and Learning in Germany 53 Pierre Brochu ■ The Influence of PISA on Educational Policy in Canada: Take a Deep Breath 73 Maria Stephens andAnindita Sen ■ Comparing U.S. States' Mathematics Results in PISA and Other International and National Student Assessments 87 Ana Kozina and Ana Mlekuz ■ The Predictive Power of Attribution Styles for PISA 2012 Achievement: International and National Perspective 101 Mojca Straus ■ (In)equalities in PISA 2012 mathematics achievement, socio-economic gradient and mathematics-related attitudes of students in Slovenia, Canada, Germany and the United States 121 3 ABSTRACTS/POVZETKI 145 4 R H V1H W / K H C h X /J I A 161 Slavko Gaber (ur.) (2.014), Finska v vrhu znanja2030 (Darko Strajn) 163 5 AU'I I IO RS/AVTOR.) I 167 I editorial/uvodnik The Timeless Questions About Educational Quality Mojca Straus Concerns about the quality of education systems have now substantially shifted from the questions about the quantity and quality of resources, such as school buildings and accessibility, to the questions about the outputs of the educational process, such as student achievement. Achievement has become one of the key indicators used in evaluating the quality of education systems. Furthermore, these questions are not constrained to the local contexts but are globalized in the sense that the outputs of educational systems, working in different societal and economical contexts, are compared. To address the comparative information needs in the process of the educational quality control, several large scale assessments have been launched in the last decades primarily to provide an information base from which the hypotheses about stability and change in education can be tested. Some questions remain the same throughout these decades, such as how well do students in a particular country perform in comparison with students from other countries, do they reach expected levels of achievement and what should be expected of them. Further questions pertain to the methodology of the studies and validity of the usage of their results in more general contexts. The quality of an education system proves to be a complex concept that needs constant attention at all levels of the system. This thematic issue is devoted to findings emerging from the latest cycle of the Programme for International Student Assessment (PISA). Slovenian and foreign authors present views and reactions to the PISA methodology and results in the efforts for assessing the quality of the education systems in their respective countries. In his article, Darko Strajn discusses relatively recent criticisms of PISA. The criticisms focus on the ranking of results that inscribe PISA as the foun- dations of the neoliberal market competition entering the education field. Since the initiation of PISA, there were many discussions about the impact of the study's ranking results on the educational policies and processes around the world. In his analysis, the author explores different views 011 these impacts 011 the general understanding of the meaning and working of education as well as educational policy development. In the next article, Urska Strcmfcl addresses the impact of Slovenia's below average results in reading literacy on the country's educational policy. Using policy analysis, the author provides insight into the first steps of the process of improving the Slovenia's PISA results. 'Ihc author discusses the importance of having nationally defined educational priorities and goals in order to be able to actually derive a well-defined policy problem and to find the appropriate policy solution to this problem, for example by drawing lessons from the successful results of other participating countries. One of the countries from which Slovenia might decide to learn from is Germany. In the last twelve years since the first, so-called PISA shock in 2000, Germany has successfully improved its PISA results. In their article, Christine Sael/.er and Manfred Prcnzel describe three major aspects of Germany's educational development; a thorough diagnosis of the problems in the country's educational system, an intense discourse between all relevant actors, and the implementation of nationwide, overarching programmes to improve teaching and learning. These elements and their impact on German students' PISA results are analyzed. Rased on the PISA 2012 results, it is evident there has been a positive educational development in Germany. Another country, Canada, has been considered very successful in PISA since its beginning in 2000. However, the recent downward trend in the country's results have initiated the call for action, 'ihc issues arou nd Canada's PISA results and reflections of different educational actors are presented by Pierre Brochu, Ihe author analyzes the important considerations in the efforts of finding the appropriate levers for changing the observed negative trend in Canada's student achievement. In the United States of America, the educational policy is developed at the state level. Maria Stephens and Anindita Sen address considerations arising when three U.S. states - Connecticut, Florida, and Massachusetts - derived comparisons of states' results from the PISA data as well from data of other international studies. When different assessments sometimes indicate different or even contradicting results about the educational quality, the important question is what specific factors might explain the observed differences. VI. SI KAl'S ■ nil- II VIII I'SS QLU'S'I IONS A KOU I I IXC A I ION Al. QL'A 1,1 I'Y Ana Kozina and AnaMlekuz studied the relationship between PISA 2011 mathematics achievement and attribution styles. In their article, they use national as well as international perspective for investigating students' attributions of causes for success and failure on the PISA 2012 mathematics achievement test in relation to actual test score. They conclude that attribution for success should be considered in educational setting for example in communicating praises for students' success in a manner promoting effort. In the final article, Mojca Straus explores the roles of socio-economic background and mathematics-related attitudinal factors in explaining achievement ¡11 mathematics literacy of the PISA 2012 study for Slovenia in comparison with Germany, Canada and the Uniced States. Mathematics-related self-beliefs are shown to be stronger predictors of achievement than students' drive and motivation and similarities are observed between the Slovene and German students' responses as well as between die Canadian and the United States students' responses. 'ihe articles in this issue show that data from international assessments of student achievement represent a rich source of information on education systems in the world. However, thorough understanding of the design, methodology and implementation of the assessments is of vital importance for making valid and useful interpretations of the results. The general steps in conducting an international comparative assessment are that participating countries agree on the population of students and the curriculum domain to be assessed, and 011 an instrument to assess achievement in the chosen domain. Ihe instrument is administered to a representative sample of students in each country and comparative analyses of the data are carried out. Ihese analyses are intended to provide information about the educational quality in the form of comparisons ofstudents' scores or sub-scores on an international achievement test. An important part of this is understanding the reasons for observed differences between and within the countries from the collection of the background data, especially in the areas where weaknesses in achievement are identified. 'Ihe story of educational quality control does not end with the publication of the international or national comparisons of participating countries' results. After the PISA 2012 results were published in lace 2cn, countries started with additional qualitative and/or quantitative studies designed to unravel the origins of the observed weaknesses in order to set up and carry out the appropriate remedial actions. The contributions in this issue show examples of such analyses and the interpretations of the findings. It is shown that the internationally comparative data are most often used for the functions of descriptive comparisons and trend analyses and that it is more difficult to provide answers about the causes of any observations. When the results of the studies are used, one needs to be careful in drawing conclusions. There is an abundance of caveats that could diminish the validity of these conclusions rangingfrom the samplingdesign and response rates, coverage of domain and instrument design, data collection procedures, motivation of students and other respondents, technical procedures in data analysis and, not least, inappropriate causal inferences. As evident from the contributions in this issue, the data from an assessment of student achievement do not, by themselves, convey messages about the quality of education or evaluation of the reforms that have been implemented. Ihe data collected need to be interpreted with a reference to relevant comparisons, for example to the goals of education in a particular country or to the results of other countries. However, setting absolute standards in education is difficult. To try to set realistic standards for educational system comparisons with other relevant countries are essential. As shown in this issue, this is important in die largest education systems ¡11 the world and even more so in Slovenia. The overall problem with analyses of the assessment data is how to address the imminent questions 011 the educational quality and effectiveness without reporting information that is easily misunderstood and/or misused. It is very difficult to determine abundant factors within or outside the education system that influence achievement. Moreover, conclusions from an assessment rarely offer clues about causal inferences, 'fhey can, however, be useful as circumstantial support for the conclusions about the determinants ofachicvcmcnt or as asource of inspirations for finding possible levers of improvement in further research, fhere are, nonetheless, important reasons for the usefulness ofsuch studies. Not unimportantly, assessments are relatively inexpensive compared to other aspects of managing education, such as implementing curriculum changes that involve substantial professional development of teadiers. further, it is easier to mandate assessment requirements at the system level than it is to take actions that involve actual change in what happens inside the classroom. Such studies are therefore useful for getting the overall picture of the status of the things in education. And, as a consequence of media attention given to the international assessments, international studies can help education to become a priority among the areas that need policy makers' attention. i papers/razprave The PISA Syndrome: Can we Imagine Education without Comparative Testing? Darko Strap Introduction Unlike "normally" by citing academic books and journals - I am starting this article by recalling relatively recent criticisms of OECD's PISA testing addressed to wider public. The fact that this criticism is recent does not imply that it is also entirely new. The logic of this criticism, which has been detectable almost ever since the inception of PISA - and indeed since much earlier pioneering IEA studies like FIMS, TIMSS, and so on in more than just governmental settings - had been conducted, has gone public on a grand scale. The Guardian, Tuesday 6th May 2014, published a letter addressed to PISA director Dr Schleicher under the title "OECD and Pisa tests are damaging education worldwide." The letter was signed by many distinguished academics from universities (mostly American and European) and some other interested public personas. This academic public gesture had a quite strong echo in world press. However, answers by the PISA director and by members of a global network, consisting of researchers, who actually work on designing and implementing PISA testing, were much less published in the world press. Another case of recent public criticism of PISA is Erwin Wagenhofer's film documentary Alphabet (2013), which actually commences with a strong point on how educational achievements of Shanghai schools were under the influence of PISA testing. The type of education, which is adapted to achieving high scores in PISA testing, especially in the fields of mathematics and natural sciences, presumably - as it is stated at the beginning of the film - flattens children's creativity, ability to think critically and independently. Both of these critical statements aimed at policy makers, and even more to the broader public, expose what they see as a dubious nature of ranking of results that inscribe PISA into the foundations of the neoliberal extension of market competition to all avenues of life. However, exactly the rankings, as they are presented in league tables in a somewhat quick succession once in three years, made PISA so "popular" and influential. Therefore, any abandoning of such presentations of the results seems quite unimaginable. On the other hand, a dilemma on whether these rankings are conscqucnccs or causes of what has been seen as educational transformation in favour of global ncolibcralism seems pertinent, but hard to answer. In this paper, I shall just briefly discuss the main lines of argument in the above mentioned public outcries against PISA and in the next step I shall take a look at some examples of academic deliberations 011 PISA testing. further on, I will be exploring on the paradigmatic level for "deeper" reasons for such disputes and insuperable differences, concerning cultural, methodological and theoretical aspects of these considerations. At the end of the paper, I shall try to open questions on how PISA testing nevertheless makes sense. Questions and Answers The views, which are expressed in Tí)e Guardian letter (Andrews, 2014), represent an important step in discussions about standardised testing precisely because they arc communicated to a larger public. This means that we can take them to be an attempt to make an impact 011 public policies, as well as trying to influence a critical understanding of such procedure as PISA testing and its results. In all fairness to the signatories' good intentions, it should be noted that they do not a priori reject the very method of testing itself and, in spite of the rather harsh criticism; they give suggestions on how PISA shou Id proceed in its work to attain socially and educationally more acceptable impact. The signatories assert that PISA "/.../has contributed to an escalation in such testing and a dramatically increased reliance on quantitative measures," which has, in their view, resulted in many negative effects. Just three years assessment cycle shifts attention to short-term policies, which arc mostly inappropriate in various cultural contexts. PISA is further, in the signatories' opin ion, tot) focused on measurable aspects and so it "takes away attention from the less measurable or immeasurable educational objectives." PISA is then, among other problematic effects, blamed for an increase of "public--private partnerships," which sustain for-profit educational services in America and project them also in Africa. After avowing some more harmful consequences of PISA, such as it is conducted for last n years, the authors of the Guardian letter make seven "constructive ideas and suggestions." Since my intention is not to deal with the whole spectrum of problems, which these "ideas and suggestions" touch upon, let me only mention that the first suggestion requires from Of,CD to "develop alternatives to league tables" and to "explore more meaningful and less easily sensationalised ways of reporting assessments outcomes." Hie letter is concluded by questioning the legitimacy of OECD as an organisation for becoming a "global arbiter of the means and ends of education." Ihc authors of the letter find that the "international competition for higher test scores" harms diversity among cultures and traditions. A direct answer to these allegations under the title "OECD's PISA under Attack!" signed by almost 400 above all "researchers of school performance" (as they chose to present themselves) from all continents is without any doubt an illustration of the fact that the academic sphere is divided on most questions raised in The Guardian letter. Of course, I have no intention to judge who is right in this dispute. 'ihe answer to The Guardian letter is obviously an upshot of a quite quick reaction, 'ihere-fore, the answer mainly succeeds in demonstrating that, at least, there is a strong misunderstanding on the matter between members of research communities, which are supposed to know what is there to know about the testing of school achievement. Still, I would dare to say that the answer seems somewhat weak. It essentially boils down to this assertion: "PISA student assessments, like other similar kinds of tests around the world, have the same function of a thermometer in medical diagnostic." (Ichi-no, 2014) We can take this as a statement on PISA being essentially just a "neutral" instrument, ihe medical metaphor, which is further elaborated, seems to be unsatisfactory as an answer. Reside this, as it appears to me, the answer imputes to The Guardian letter an intention, which it did not have, saying that it was "clearly aimed at excluding comparable evidence of student performance from educational decision-making." Ihc "coming out" into the open public space of the two academic groupings points towards a need to rethink the role of PISA testing not only in order to fight social battles in the academic arena, but also in order to distinguish between research results and its (ab)uses, and then to at least recognize differences in justifiable approaches to such complexities as educational in- lliesisjiatoriesol ihemsyveri.o'/¿s fe/^i./ v leiicrprol13hlyme.ini 10.address no 1 jusi ihe academic community and therefore they picked a linguistic sliorl-cni 10readers. Still it should l>c pointed out that mctaphors can l>c tricky Let 11 ie cite jusi 011cexample oi many similar notices loi which early examples can Lie loiiud also in Plato Socrates dialogues: Metaphor i« helpful land own indispensable: aswhick ro rhinkahemr abstract phenoiiv 011a. bur one should be e irefill nor to niisrake the metaphors for the 'reality they riy ro describe. ; Boers. IVmeeheleer. 1997, p. it} Perceptions of PISA's "cultural impact" actually vary since most authors are aware that there are other agencies of a global "cultural homogeni-zation" that might have benefited from PISA, which indeed tends to be "culture-blind." Educational systems and their elements (like curricula, teaching methods, school management, and so on) of course change, and, of course, they arc always making part of cultural context, "/.../for many countries in the world that has happened is a shift in what could be called the topography of education. Between the early nineteenth century and the early twenty-first century, the map of education* itself has changed. Its contents, its institutions, and che people who populate it have been reconfigured." (Cowcn, loii, p. 30) A quick "meta-analysis" of PISA impacts would probably show that educational systems still conform to their local social and cultural contexts - which are in their turn changing either in a progressive or conservative direction - in spite of responding to some "incitements" from PISA results. China's case is typical in this respect. ... our analysis of the reasoning surrounding the PISA results reveals rliar. there is a profound discrepancy between local political actors and stakeholders on 1 he one hand and independent reseat chers and ova -seas professors on die 01 her. I lie discourse ecni ring on 1 he PISA 2009 results has reshaped die éducation discourse in ( 'him. I he ease ol C 'hi na is particularly interesting lor education discourse analysis, because the pre-PISA discourse had been ch.iracicii/cd by the criticism o( the exam oriented education and the .scepticism ol the cilcctivcncss ol the education reform, (/.hang, Akbik, 1011,p. 2.6) I am leaving many other aspects of the "cultural problem" of PISA open, since the above-mentioned facets are maybe sufficient to exemplify the type of the problem. Paradigmatic Divide Ivpistemological questions will always represent issues for differences among researchers. Such questions, of course, open problems of methods, which are unavoidably intellectually funded. Undoubtedly "the syndrome" of PISA consists of many components. As we can gather from many debates, these components are: conceptual differences, political perceptions, and cultural contexts. However, fundamentally PISA is linked to knowledge as is any education-related phenomenon, which means that it cannot avoid paradoxes of "knowledge about knowledge." Philosophy for centuries searched for a universal model of knowledge. Hence, at least two broad different "paradigms" of reflexive knowledge persist. Philosophers - of course with immense number of nuances - basically agree that these different paradigms could be identified as a difference between cm- piricism and rationalism from 17th Century, or as the diiferenee between positivism and transcendentalism (or constructivism), or as the gap between Anglo-American philosophy and Continental philosophy. Some would also argue that the split between the two basic paradigms is rooted in Antiquity - for instance in the unfinished dialogue from Plato, Par-menides, which left readers with unanswered questions on the relation between the part and the whole - others, would see this split in mediaeval logics, and so on. In modernity and postmodern icy, chcrc were many attempts co overcome che divide, buc ic looks chac such attempts mostly contribute to just new elaborations of the rift. One of the modern manifestations of the divide - between positivism and dcconstruccion - was highlighted by Stanley Cavell, who certainly made a few steps towards creating a field of mutual understanding. 'And I cite then positivism's and deconsuuction's| claims ro what may be seen as die discovery of the oiiiynariness of writing over voice, of system over individual intervention, of sign over word since the appeal to mathematical logic for its algorithmic value is an appeal to its sublime insciipiion.il powers (ol alignment, rewriting, ucranon. subsiim-lion. and so on). Positivisms inscripiionalii v may be seen as in scrvicc ol a hoinogcni/.aiion ol ihc field ol sense"(Cavcll. 1994. p. 8;) Cave lis success in bridging the gap between two "universes" of thought made a strong impression in such fields as culture, or, to be more precise, in film theory, as well as in some trends in philosophy itself. We are still waiting for "a Cavell" in the realm of the scientific mind. As ic is well known, "positivism" is closely associated with (positive or "exact") sciences. Especially thanks to rcccnc possibilities to acquire and manage large amounts of data, positivism is also re-occupying the space of social sciences, which chrough che work of Durkhcim, cricical philosophers, cxisccncialiscs, and so 011, was for a long period a domain of chinking about the world in terms of the notion of tocality. PISA is just one of the phenomena in research that makes use of the "positivist" methodologies, which carve out their problem field from the social and cultural complexity. Such methodologies, 110 matter how well elaborated or specific in their founding they may be, lay claim that the knowledge, which they acquire by applying their rules and "cools", is cercain as it is firmly "evidence-based." Usually users of such methodologies — viewed as "partial" by a range of anti-positivist critics - do not hesitate to give the "we don't know" kind of answers for any problems, which arc considered to be outside of their methodological framework. However, this insisting on a particular insight, "based 011 facts," is seen as a synecdoche: the way PISA test results are presented strongly suggests that mathematics and natural sciences stand for entire knowledge, as well as, that such knowledge is crucial for economic development. Of course, such a supposition can probably not be proved, since such categories of knowledge as historic memory and artistic sense have their role in any social system, and they operate within the economy in a broader sense of the word. On the level of theory, the differences will probably never be settled, since anti-positivists will always insist on an attribute of "instrumental ity" of such methods as the ones, used in PISA. This brief and very superficial explanation of the paradigmatic gap can be taken as just one aspect of many reasons for "misunderstandings" between advocates and adversaries of PISA. However, by taking into account such sophisticated aspects of the differences, one can still find data - no matter how much thev arc seen to be ideologically constituted, or no matter how they represent only a reduced picture of the "reality," and so on - as representing something. Of course, one is free to decide what they represent. Any decisions of actions in changing the profile of a national education depend on complex local contexts. In spite of credible reproaches, regarding what is voiced as "homogenisation," there is always a space in local policies to advocate "good traditions" against mismatched changes. Conclusion It is a truism to say that theoretical and practical constituents of education have always been ingredients of larger social movements. They mark conflicting issues in the politically determined power relations in the educational field. In countless discourses, education keeps recurring as a crucial agency of the social emancipation, both from class or gender oppression and from other forms of cultural exclusion, but also as a precondition for self-accomplishment of an individual. A huge intellectual input into developments, processes and events in educational systems is an inherent force of social-educational movements. As an end of neol iberal-ism is anxiously hoped for, there is a huge helping backlash of emancipatory educational discourses. However, in light of the question on whether PISA is the cause or effect of structural institutional shifts, adaptations in the economy, and so on, another question on the full pertinence of PISA as a main object of such criticism is relevant. Scholarly volumes of books - let alone journal articles and other not strictly just academic publications - that deal with the role of education in social reproduction and in movements for social change are growing almost exponentially.* Ihe out- 4 I •oriiistancv. I niwlfwroiosomr fifteen book reviews for ImcnattwitiRififv of those, who may hope lor a good job,' in Finland, 7 $'A> in Poland and Hungary about 70%, but they also take into account the research results of l'ISA and TIMSS, which allow valid performance comparisons of educational systems and empirically lit analysis oi national systems." (Gaber ei al„ 1009, p. 84-85) Such comments by researchers of education are not very rare. PISA, therefore, makes possible critical analysis, which even runs against its assumed "ncolibcral and homogenising objectives." No matter how well any such criticism is founded, no matter how strong its arguments are, it should be recognised that even so the testing and the acquired data make such criticism and its conceptual achievements possible. Of course, one would like to see more dialogue between different "schools" of comparative research, as well as some pondering on the effects of such presentations as, for instance, the league tables, within PISA organisation itself. On the other hand, one should be aware that controversies in as much as possible unrestrained democratic public space generate breakthrough new ideas and social movements. And this holds true whether controversies arc resolved or not. References Alphabet (2013) Documentary film. Directed by: Prwin Wagenhofer. Germany: Pandora film. Andrews, P. et al. (1014) OPCD and Pisa tests are damaging education worldwide. The Guardian [Online] Tuesday 6 May 2014. Available from: http:/Avww.theguardian.com/educati0n/2014/may/06/ oecd-pisa-tests-damaging-education-academics (Accessed: 411' August 2014). Boers, F„ Dcmcchclccr, M. (1997) A Few Metaphorical Models in (Western) Economic Discourse. In: I.icbcrt, W;-A„ Rcdckcr, G. & Waugh, I.. R. (eds.) [Discourse and Perspective in Cognitive Linguistics. Amsterdam: John Benjamin Publishing Co, Carvalho, L. (2012) Hie Fabrications and Travels of a Knowledge-Policy Instrument. European Educational Research Journal. 11 (2), pp. 172188. Soi.s ko poni-.. i i i\ik \ x v, s i hvii.ka 5-6 Cavell, S. (1994) A pitch of philosophy: autobiographical exercises. Cambridge, Massachusetts, London: Harvard University Press. Cowcn, R. (1011) Edging closer to the hero, the barbarian, and the stranger. A note on the condition of comparative education. In: Triih 1er, D., and Barbu, R. (eds.) Education Systems in Historical, Cultural, and Sociological Perspectives. Rotterdam: Sense Publishers. Debruge, P. (ion) Film Review: 'Alphabet'. Variety |Onlinc| December 3, 1013. Available from: http://varicty.c0m/1013/film/revicws/film-rc-vicw-alphabct-i200877133/ (Accessed: 5th August, 1014). Fabre, M. (ion) Eduquer pour un monde problématique. La carte et la boussole. Paris: Presses universitaires de France. Gaber, S., Marjanovič Umek, L., Podlesek, A., Sočan G. and Fašner, V. (1009) Analiza razlik v dosežkih učencev/dijakov ter analiza primarnih in sekundarnih učinkov družbenih razlik na dosežke učencev/dijakov [Analysis of Differences in Student Performance and of Primary and Secondary Effects of Social Differences!. Šolsko polje. XX (1-1), pp. 83-125. Hanberger, A. (2014) What PISA intends to andean possibly achieve: A critical programme theory analysis. European Educational Research Journal [Online]. 13 (1), pp. 167-180. Available from: http://dx.doi. 0rg/i0.2304/eerj.2014.13.2.167 . (Accessed: 8th August, 2014). Hillgers, M. (2013) Embodying neoliberalism: thoughts and responses to critics. Social Anthropology/AnthropologieSociale. 21 (1), pp. 75-89. Hyunjoon, P. (2014) No matter how high your test score is, you arc still bad: Korean education's responses to PISA. Revue internationale d'éducation de Sèvres |Onlinc|, Colloque: L'éducation en Asie en 2014: Quels enjeux mondiaux. Available from: http://rics.rcvucs. org/3749 (Accessed: nth August, 2014). Ichino, A. et al. (2014) OECD's PISA under attack! [Onlinc| Available from: http://www.eale. nl/ab0Ut%2ceale/OF,CD'S%20PISA%20 under%ioattacki.htm (Accessed: 4th August 2014). Laval, Ch. (1004) L'école n'est pits une entreprise. Le néo-libéralisme à l'assaut de l'enseignement public. Paris: Editions La Découverte. Lawn, M. (1011) Standardizing the European education policy space. European Educational Research Journal. 10 (1), pp. 159-272. T.icssmann, K. P. (2006) Theorie der Unbildung. Die Irrtiimerder Wissensgesellschaft. Wicn: Paid Zsolnay Verlag. Mangez, F., and Hilgers, M. (2012) The Field of Knowledge and the Policy Field in Education: PISA and the production of knowledge for policy, European Educational Research Journal. 11 (1), pp. 189-205. O EC D (2010) PIS/l 2009 Results: Overcoming Social Background - Equity in Learning Opportunities and Outcomes (Volume II). Paris: OF,CD publishing. Parmenter, T.. (1014) Globalization in Japan: Education Policy and Curriculum. In: Ncllv, P., Stromquist, N. P. &. Monkman, K. (eds.) Globalization and Education / Integration and Contestation across Cultures - 2nd Edition. I-an ham, Boulder: Rowman & Eittlcficld. Pereyra, M. A., Kotthoff, H.-G, & Cowcn, R. (ion) PISA under examination: Changing knowledge, changing tests, and changing schools: An Introduction to the Comparative Puzzle. In: Pereyra, M. A., Kotthoff, H.-G. &c Cowcn, R. (eds.) PISA under Examination. Rotterdam: Sense Publishers. Spring,.], (2.008) Research on Globalization and Education. Review of Ed-ucationalResearch. 78 (2), pp. ^0-161. Schleicher, A. (201}) 'Use data to build better schools'. TED-conference published 21st February 2013. Available from: https://www.youtube. com/watch?v=7Xiiir87iisl74 (Accessed: 26chjulv 2014). Straus, M„ and Markelj, N. (2011) Bralna, matematična in naravoslovna pismenost dijakinj in dijakov 1. letnikov srednjih šol v Sloveniji v raziskavi PISA 2009 [Reading, mathematics and science literacy in 1st class of upper secondary school students in Slovenia in PISA 2009]. Šolsko polje. XXII (5-6), pp. ^5-68. Zhang, Ch. and Akbik, A. (2012) PISA as a legitimacy tool during China's education reform: Case study of Shanghai. TranState Working Papers, No. 166 |Onlinc|. Available from: http://hdl.handlc. net/10419/64810 (Accessed: 4th August 2014). Slovenia on its Own Way Towards Improving PISA Results Urska Stremfel Introduction PISA (Programme for International Student Assessment) becomes a prevalent assessment of the national education systems in the last decade (Hopmann et al., 2007; Pereryra et al., 2011; Meyer and Benavot, 2013). PISA results, presented in comparative achievement scales, provide an insight into how one educational system performs in comparison to other systems and also how one educational system contributes to the achievement of common goals of particular group of participating countries (e.g. European Union (EU) member states together decided a benchmark to have less than 15% of low achievers1 in PISA by 2020) (Council of the EU, 2009). Since PISA results and results of other international comparative assessment studies1 often becomes incorporated in the national educational targets, PISA also helps to identify how successfully participating countries follow their national priorities and goals.' There is one additional insight that PISA allows. The design of PISA, which is conducted in cycles, enables the monitoring of changes in students' outcomes over time. Such changes indicate how successful education systems have been in developing the knowledge and skills of 1 PISA provides a profile of students'performance using six proficiency levels. The low-achievers are students, who do not reach the proficiency level 2, which present a baseline level of literacy at which students begin to demonstrate the competencies that will enable them to actively participate in life situations (OECD, 2010a). 2 E.g. Trends in International Mathematics and Science Study (TIMSS), Progress in International Reading Literacy Study (PIRLS). 3 E.g. Slovenian White Paper on Education (2011, p.25) states "At the state level we need to state and map out a clear path towards the goal, that performance of Slovenian students in international comparative assessment studies are at the top, that mean at least in the upper third of the students' achievement of the developed countries". 15-year-olds. All countries seeking to improve their results can therefore draw and learn lessons from those that have succeeded in doing so since 1000, when the PISA was first conducted (OECD, 2010a, p.n). fhe importance that PISA has gained in the assessment and development of national educational systems is often understood in terms of transnational policy making (Meyer and Bcnavot, 1013). If we understand the policy making as the solving the policy problems of / for society (T.ass-well, 1951), we can also argue that it can be understood as transnational problem solving (Scharpf, 1997). Ihac means that PISA helps participating countries to understand the weakness of their national educational systems (in international comparative perspective) and also provide the environment for finding the right solution of perceived problem. Despite some theoretical reservations towards considering comparative achievement scales as the legitimate source of policy making (e.g. Kodclja, 1005) and exploiting their results for politically motivated changes at the national level (e.g. Strcmfel, 1013), PISA has become widely accepted that these comparative achievement scales (called also league tables) present an important source of the identification of national policy problems and finding policy solutions in participating states (see e.g. Grck, 2010). As such comparative achievement scales, if appropriately used, can present an important source not only for the assessment, but also for the development of national educational systems.4 Although one of the formally stated goals of PISA is to create an internationally comparative evidence base for educational policy development and implementation (Wiseman, 101?, p.304), Waldow (2009) recognized that headline news about PISA is often more about "shock"5 over the assessment results than what the assessment information contributes to discussions about long-term educational reform and improvement. Iheoretical and empirical researches (see Strcmfel, 201?) show that participating countries become especially attentive to the PISA results when they perform below international (OECD, EU) average. Ihat effect was experienced also in Slovenia. When the PISA 2009 reading literacy results were published and for the first time since Slovenia had been participating in international comparative assessment studies, it showed that Slovenian students perform below international (OECD, EU) average, the perception of the Slovenian educational system as a successful system 4 For more theoretical insi^ln abom the role die evaluation plays in die development ol pub lie policies see knsree l.ipieer s Phillips a nd Oehs[1 explain rharedncarion policy shock happens whe 11 rheiv is a deviation from rhe norm, ofre 11 invoKi 11« mediocre or low performance i.e. below expc cra-tioruC was marred at the level of experts, policy makers, practitioners and general public (Interviews by author, ion). PISA ion results confirmed the underperformance of Slovenian students in reading literacy and emphasized the need for improvement of the performance of Slovenian students in that domain. 'ihe aim of the,article is through the understanding of PISA as transnational policy making, using the Slovenian PISA 2011 results, is to show how the policy problem of below average results is identified by participating member states and to illustrate how the policy solutions for the improvement of students' performance in PISA could be found. In order to illustrate the policy framework of improving PISA results, the article as a case study takes into consideration PISA reading literacy results (the domain in which Slovenia perform below Of.CD and PU average) and students performance at the Proficiency level 2 (the level which Slovenia togedier with other J'.U member states chose for defining a common benchmark "to reduce the percentage of low-achieving students to 15% by 1010")." A research question die article addresses is "How to find a way towards improving Sloven ia's PISA results according to the concept of transnational policy making and policy learning theory?'' ihe article addresses die research question in die framework of policy analysis studies. The concept of transnational policy making (in terms of governance of problems and transnational policy promotion) and theory of policy learning (in terms oflesson-drawing) are employed in order to provide an in depth insight into the process of defining and solving policy problems in che contemporary educational policies. Theoretical dispositions are further elaborated 011 in the case of Slovenian PISA 1012. results in reading literacy and trends in other participating PU member states from 2000 onwards. Ihe empirical data for the case study were gathered by the analysis of die OPCD and J'.U official reports, as well as an analysis of the respective Slovenian legislation and strategic documents. In order to provide an additional understanding of the reception of transnational policy making at the national level, the data gathered by interviews with Slovenian and F.U representatives (policy makers, researchers, practitioners) from 1008 to 1012" and the results of the survey about the reception of 6 By takiilg into consideration llie policy approaches lor ¡niprovini; the PISA results, the articlc docs not take into consideration the more substantive and pcdagogicalapproaclies for improving PISA ivsnlrs. Data oarhnvd through srmi-smiiTurod iiirervkw piwnr an additional somvi ofinfor-iivirioiiand wiTiMiscd only rod,lrityrhosr open issues rharwo wrro 111a hit-ro idonri ty from ourarwlysesol official documaus. member stares. OI.Ol) is often called vthc dnb of world s most advanced contiinc-s- iOi ;c;l). 141. I .ible i: Idem i licai ion ol lie most succcsslul HU member suits in following ihe HI,' hcnchtn.nk PISA cycle/ member state 2000 2003 2006 200^ 2012 1 rend (200020 r 2} A . 111 l>/9'\i 1 *' 71\> i>).y\. -, 4lu Belgium IJ.'j'ii i7<;"o ly 4»o 17.7% 16.1% l.?"o Bulgn ri.i. 51.1% 41.0% W-4"i ->>1» Crojtia il.y>5 ii.y'ó lt.'7-o Cm eh Republic 17.5'V. 14-S'V. ( AjH IIV Denirurk 17'51\l if-.yv KÓA. 15.2% l+.iMi PsLonia iyi'% 13.3'V »> 1% 1 ¡nía nd — • .0;; i 4-S*. 8.1« 11 & I 'm nee li.1% i-y'ó 11.7 "i ly.i<"a iS.trt ->7üó Germany 21.6% 22.3% 10.0% 18.5% 14.5% 8.1% f ¡reeee 144"" 177"» ii.yi i.S'V. Hungm ii.7";i i97";i Irekuid 11 'A II.:« 11. rv 17. y.íA. 1.4« Italy lH.9'\> •-in". 26.4',, 11 - ni.yv ■o.fA, 1 .arvia 21.2% 17.ÍA, -, a. 131% Lithuania -5""" 14.V0 ll.l"i 1 .nxemSui-g 21.7 "i 11 .•)"<, lf.'j"o 11.1% Neilu rLmds n.V'ii lyl".. 14. V'.> l+r,*, Poland 2}. 2% 16.8% 16.2% 15.0% I0.6% 12.6% Portugal 1 Mi'« 11 v.. I4.9"" T7.6% iMW -x„ Romania 53 5*» 4*: 4°" i í"- 4 Slovakia ' 4 :•"•• . tf 1% Sloven in ié.5°S n.r'i ll.l'ii. Spain 11.1% 15.7"i i').6"b iS.3% Sv.( den 11A IyV'11 I74".. :i.7"i -Tar'.. United Kingdom 18.4« if. iA, Source: OHCD (2013 a). those who succeed. However, which educational system could be considered as successful according to the PISA results? Although we agree that there is no one way to answer this, in this article, we adopt the OECD (2.010a, p. 14) understanding of successful states as not just top-scoring, but especially those ones which are rapidly improving from the first PISA cycle in 1000 onwards. OECD (2010a, p. n) explains: "The design of PISA does not just allow for a comparison of the relative standing of countries in terms of their learning outcomes; it also enables each country to monitor changes in those outcomes over time. Such changes indicate how successful education systems have been in developing the knowledge and skills of 15-year-olds. All countries seeking to improve their results can draw encouragement - and learn lessons - from those that have succeeded in doing so in a relatively short period of time." fable 1 shows the trends of the EU member states' PISA reading literacy performance since 1000 in order to identify those member states, which were the most successful in improving the results of their low-achieving students and the most successfully follow the EU benchmark "to reduce percentage of low-achievers in PISA to 15% by 1010"." The Table 1 shows that among 18 EU member states, 18 of them have been participating in all PISA cycles (2000-2012.). For these member states trends in die percentage of low-achieving students are presented. It shows that n of the member states succeed in reducing the percentage of their low achieving students, and in 5 of them, these percentages from 2000 to 2012 increase. The table also shows that the most successful EU member states in reducing the percentage of their low-achieving PISA students in reading literacy are Poland (12,6%), and Germany (8,1%). OECD (2010a) claims that success of such a diverse group of countries in raising the level of their students' performance in reading i ndicates that improvement is possible regardless of a country's context and where it starts out from. Similarly, European Commission (201?) recognizing that the EU as a whole is lagging behind in its challenge to reduce the share of low achievers in reading, points out that this trend does, however, disguise large differences found between and within EU member states. By indicating the concrete member states and their improvement, the European Commission does not only exert the pressure 011 some member states 011 the basis of "naming and shaming" but also indicate the countries, from which the lessons could be drawn. Ihe European Commission (2013, p. 5) states: "'Hie reasons why some member states succeeded in significantly reducing the share of low achievers may serve as an inspiration for other countries that are struggling to overcome similar challenges or even face a deteriorating situation." 11 Although Ol.(• 0 idenrifie«trends in result« of parricipiiri 11c countries on a special methodology ;seeOl'.C ID. which measnlettendsoiilvbetween thecycle s when the marh was a main resting domain, wc have present trends hom the 1 -..-. -. onwards which is also ihttcsialilished practice ol ihe hi h 1 able2:Overview ol lhe perlonn.uifc ol ihc mosi surresslnl tneinberin following the EE1 benchmark by different indicators Indicator; member stare Slovenia Poland Genua nv Percentage of bwaehievcrs .1 ~ r,r,': Peicenrageoftawaohicvcrs; 2--f 1": it.t% Tj.ffli Mi'' Percentage of bwaehievcrs ;dii ierencc i-.-ii-iuoo": ll.A'',, 8.1% Percentage of high achievers ¡iuo'j) 5-9"0 S.8"0 ill and loth percentiles ...... 160 points 284 points ( iji>Ih L\v< (11 *>:jih rind 1:ji h ik rccn 1 ic^ 1 nftpoinLs (blunge in pip belwecn y<;tli and i-uli pereentiles ;iou-lo"") 122 poin 1 s iS poinl s poinls 47 points Proportion oftoral variation explained by betwci n-school variance (i - -) Proportion ofroral variation explained by between-school variance (10 - % (-5.4-0 W"" 67.1% Change ill proportion oi total variation explained by between-school variance i' 1: JO-I'...... V fV S.l"« Rt'lri Lion ship be [ v.( (11 k riding per 1« 1 ianc( .1 ltd the PISA index ol economic.social .ntd cull 111,1 Istai ns (FSC ;Si .•.-.. Relationship between reading performance and the PISA index of economic, social and culturn 1 starusilSCS) ; 10.-9) Change in relationship between reading pcriormance and the PI SA index oi economic,social and cultural status b.SCS^ i' 1: JO-I'...... 4' J poinl s ;9 points 1 point pomis 44 points S points Diliercncc npet lorm.ince be twee n n.nIve student rind snideni^wii i iinimgivinl background ....... DilFerencc m performance between native srudi nrsnndsrudrnrswith immigrant background Change in diifcrenee 1 n pcrformance between native students and stndenls with itriniigrant background .. ■..-) S4poiniv points ■ IS points Source: OECD iioioa: 1015a; 2015b). Since some authors (see Stremfel et al., 2014) argue that one indicator cannot provide enough insight in the functioning of the individual educational system, Table 2 shows how in EU member states, which states succeed the most in following the EU benchmark (reduce a number of low-achievers in reading literacy), the which trends in other indicators have changed. Table i shows that Poland and Germany, which succeed the most in following EU benchmark (reducing the percentage of low-achievers), were not as successful in other selected indicators. Estimating which of them would be the most appropriate to learn from in order to improve Slovenian PISA results and successfully follow the EU benchmark is therefore a comprehensive task, llic review of trends in different indicators shown in Table i, first of all requires that a learning country (Slovenia) define concrete goal about which set of indicators it would like to improve upon. One single benchmark (defined at the EU level) is too broad and cannot provide that focus and learning the state should find itself. Even OJ'.CD (2010a, p.4) recognized that "PISA results suggest that the countries that improved the most, or that are among the top performers, are those that establish clear, ambitious policy goals (...)." Conclusions If a new mode of governance in the EU is viewed as governing, steering and supervising actors (Kooiman, iooi, p. i), for them to participate in collcctivc policy problem solving and thus achieve the pursued goals jointly (Pierre and Peters, 2000), the highlighted lack of clarity of educational goals both at the supranational and the national level: (a) opens up room for political manipulation of international organisations (RorrAs and Radaclli, ion) or (b) present a huge obstacle 011 the way of improving the results on the basis oflcsson-drawing. llic widc-ness and openness of goals (and consequently their lacking clarity) allows the development of legitimate, reasonable and good policies and the (imaginary) common good in the context of social learning (BorrAs and Conzelmann, 2.007) and therefore pursuing a specific not necessary evidence-based educational model. With apparent PISA neutrality EU and Of.CD steers the member states towards achieving specific educational goals. The EU benchmark (reducing a number of low achievers to 15% by 2010) facilitates assessments and comparisons of member states' achievements (output-oriented governance and governance by comparison) in pursuing the common EU goals. PISA comparative achievement scale thus exerts dual pressure 011 the EU member states, 'liie primary pressure to perform well is related to securing the international competitiveness of the state. Ihe secondary pressure to perform well is related to avoiding the blaming and shaming by the 1 European Commission anel by other member states for not attaining com- 111011 agreed goals (Alexiadou, 2007; loannidou, 1007). Once a member srate perceives a policy problem (related to lack of economic competitiveness) following its ranking on PISA achievement scale, the best models for solving the problems in question (governance of problems) have commonly already been developed at die OECD level. In the article, this is shown using the example of OECD Economic Review for Slovenia (1011). In the case that member states follow these recommendations, the presented dynamics facilitates the deepening of the OECD cooperation in the field of education towards what is preferred by the OECD (internationalpolicy promotion), while the member states have over the past few years - in the circumstances of the economic crisis - been following the OECD more so than before, aiming to maintain their competitiveness within the knowledge-based economy (also see Tsarouhas, 2009). However, it is also necessary to be aware of the fact that actors have different sources for a critical appraisal of the knowledge provided by international comparative assessment studies and an effective use of that knowledge for development of dieir national educational systems. I11 such a context, deep and careful reflection about the nature of knowledge and its mobilisation within public policy is essential. This raises a question of whether the use of (international) comparisons as a mode of governance has not resulted in excessive legitimacy of knowledge they produce and whether it is time for actions towards a diversity of knowledge types, communicated by means of knowledge-based governance tools (Delvaux and Mange/, 2010). The main implications of understanding PISA as transnational problem solving would therefore be that the expert knowledge, which the PISA and other international comparative assessment studies provide, should be used at the national level in accordance with neopositivist and critically rational means of "speaking truth to power" and not in accordance with the interpretative and neopragmatic means of "making sense together" (Hoppc 2011, p. 55).12 In this authors opinion, the role of national experts is to assess what data (from PISA and international comparative assessment studies) and proposals for solving the identified policy problems arc to be taken as legitimate and definite in implementing the changes and improvements in the national system (Wiseman 2010, p. 9). That was already recognized, when the OECD Economic Review for Slovenia ii Experts and die expert knowledge would thus be used in an instrumental sense of making Lite right decisions and not for tile advocacy ol political decisions and die ideology of ¡supranational a nd national": political acrors (Stone. 2 -••-. I ones, i.-.-.j; Nassehi and i\ m» IS'i 1 r: ^ I--" '.U.J.I 15Í in,v i-7 in íi.6) ÍU.5) s1 ift'.i' 5»» Science «4 -51 ; -5' Sis 1-" 1 [i.. 1 : .0 'a-''- Canada also stands out not only for attaining high results but also because of the considerable equity in achievement (OECD, 2011b), as shown in Table 3. 'ihe country has been cited as a model for permitting students to reach their full potential as constructive and reflective citizens regardless of the school they attend (OECD, 2013b), as demonstrated by the many measures of equity used by PISA: a low proportion of low achievers; a relatively small achievement gap between high and low achievers; asmall proportion of variance explained by between-school diffcrcnc-es; a weak relationship between performance and socioeconomic status; and a small gap between students from an immigrant background and those born in the country (Ecvin, 2012). I able 5: PISA 1012 Ma 1 hein.it irs Selcctcd Measures oi Iiquiiy. Canada and die OECD ( 1 n.id.1 OF. CD Proportion of iyyear-old.s below level i 14".. iVVi Gap bctivec n 90th a nd1.Hi percentiles points points Proportion of total va nation explained by benvec n-school variance TP.4% Percentage ol variance explained ay socioeconomic status 9.4°o 14.6% (_iap between non-immigrant and lir>i-s>ciicraLion immigrant students ■6 poults -45 points Ihese results are particularly interesting given that Canada is the only OECD country without a central (federal) ministry/department of education; since, by definition, centralization can facilitate the creation of equitable education policies and help to ensure equitable resource allocation. Other countries with a federal presence in education such as the United States or Germany (discussed below) have generally achieved average performance on PISA since 2000 with far less equity than Canada. However, as argued by Wallner (2013), the high degree of equity in Canada may well be a consequence of decentralization, as the Canadian svs- Soi.sko poi j i;, i i. i'\i k x\v, s i mvii.ka 5-6 terns allow provinces and territories to adapt their policies, curricula, and resource allocation to the specific needs of their populations, That being said, there is a measure of equity for federal countries that has not drawn a lot of attention internationally, but does warrant a closer look for Canada. In 2000, the gap in reading between die lowest and highest achieving provinces was 49 points. In 2012, the gap was 4s points, suggesting slightly more equity. However, this equity came at die cost of achievement, in that the highest achieving provinces reached 15 points less than in 2000 and the lowest achieving province, 6 points less. PISA Results in Other Countries Other federal countries participating in PISA have generally shown results much lower than Canada's. Germany, the United Kingdom, the United States, and Spain have all seen their results close to the OECD average, while Australia and New Zealand have been slightly above. Since the mid-1950s, Germany has stood out as a world leader in higher education and as one of a handful of countries where compulsory education has been well established for the past half century (UNESCO, 2000). However, the initial PISA 2000 results created what has since been referred to as "PISA shock." fhc OECD PISA 2000 ranking had a huge impact in die country to die point where it "stopped the complacency and self-confidence with which Germany had looked ac its education system for too long" (Der Spiegel, as cited in Dráger, 2012, p. 5). l acing results that placed the country below die OECD average, both orders of government (federal and hinder) proposed urgent reforms, which focused on outputs and Germany's international competitiveness (Martens and Niemann, 2010) and laid great emphasis on empirical research and pedagogic practice (Ertl, 2006). They included a significant increase in student testing, changes to curricula, increases in funding, and additional measures of quality control (Grek, 2009; Anderson, Chin and Yore, 2010; Neumann, Fisher and Kaucrtz, 2010). Interestingly, the most recent PISA results (2012) have confirmed the significant improvement in Germany's PISA average scores and more equity in education outcomes (OECD, 2013c). It is worth noting, however, that the streaming of students (a notable feature of the German education system thac has been strongly criticized in some quarters) remains untouched, fhc earlier PISA results also triggered strong reactions in the United Kingdom. While the UK's participating entities (England, Scotland, and Northern Ireland) each registered areas of positive outcomes, die results were less encou raging in aggregace. Since die first round of PISA, the UK's performance has been portrayed as "at best stagnant, at worst de- dining" (Chakrabarti, ion; Coughlan, ion), with teacher qualifications and school autonomy hcing given, among others, as possible reasons for the lower achievement; nonetheless, this did not lead to concrete policy change (Baird et al., ion). However, it has also been argued that PISA was a catalyst for an increase in testing with an explicit reference to PISA-rc-latedperformance cargccs in Ireland (Brcakspcar, 1012; Figazzolo, 2009). In the United Staces, however, che "very average" results from che early rounds of PISA were largely ignored by the American education community, policy makers, and the media. 'ihis may have been due, in part, co the fact that the PISA sample was relatively small: like Canada, the United States maincains a decentralized education system (albcic wich a significant federal presence), and the PISA samples did not yield results that could be analysed at a state level. While little has actually been done to reform education in America based 011 PISA findings, more attention is being paid to them, as seen by the positions taken by the U.S. Secretary of Education (Duncan, 2013) and expressed at the recent International Summit on the Teaching Profession, PISA-rclatcd discourse in the U.S. has been not 011 "spendingmore" but t)ii "spending more wisely," in recognition of the fact that the United States is second only co Luxembourg in terms ofper-student spendingon education (Paine and Schleicher, 2011). Specifically, in an extensive comparative analysis of PISA results in the United States with those in high performing countries, it has been argued that resources need to be redi-recced co sociocconomically disadvancagcd schools (Merry, 2013), teacher salaries (OECD, 2011a), and programs thac increase ccachcr effectiveness (Hanushck, in Froesc-Gcrmain, 2010, p. 18). In Spain, results have been characterized by lower achievement and lower equity both between regions and between sub-populations with 110 tangible improvement over cimc (OECD 2013d). Idencifying che factors that drive PISA results in high-performing countries is difficult (OECD, 2011a). Education systems are highly complex, and virtually any combination of their elements can be cited ¡11 explaining PISA results (whether strong or weak). As explained by Figazzolo (2009), "Taken as they are, which is (...) very often what happens, PISA results can be used to support A as well as the opposite of A" (p. 28). 'Ifius, it is advisable not to look at systems or factors in isolation, but rather to consider how a combination of factors works to produce high performance — and whether this combination of factors can be replicated in other similar contexts. "Ihis has clearly been how the OECD has elected to portray individual country results and how many countries have used PISA results to further their political agendas. The pressure created by PISA about learn- Soi.sko poi j i;, i i. i'\i k x\ v. s 1vii.k a 5-6 ing from the best (i.e., highest-performing countries based on PISA) has triggered the emergence of a new phenomenon labelled "educational tourism" (Robclcn, ion), where high-performing countries are visited by delegations from lower-performing countries. A case in point is Finland. Ics leading performance in PISA since the initial round has generated an exceptional amount of interest and has been attributed to a variety of factors: non-differentiation (i.e., no tracking or streaming of students); highly qualified and respected teachers; the absence of high-stakes national assessments; and a decision-making process for curriculum and teaching approaches that is decentralized and school-based (Valijarvi et al., 1002; Malaty, 2012). Obviously, odier countries have these factors in place to some extent, but, as with any good recipe, the Finnish secret lies in having the right ingredients, in the righc amount, and in the right context. Not surprisingly, many education stakeholders from around the world wished to emulate Finland's results after the first round of PISA. However, they tended to focus on those factors that furthered specific political, educational, or economic agendas. Teacher unions, for example, cited the absence of a testing regime or the presence of highly educated teachers in Finland (OSSTF, 2007; Figa/./.olo, 2009). Other stakeholders pointed to different factors, such as the absence ofstreaming (as compared to Germany, among others) (Ammcrmüllcr, 2005); a homogeneous population (Entorfand Minoiu, 2005); the flexible curriculu m and school scruccu re (OECD, 2011a); and the late entry point for compu lsory schooling (Mead, 2008). The Finnish model contrasts wicli another successful system, namely that of South Korea, which has been used to justify very different policies (Pearson, 2012). 'Ihese include long study hours (Chakrabarti, 2013); private investment in education (Floyd, 2012); a combination of high expectations and a curriculum thac emphasizes creativity and problem solving (Marginson, 2013); and (unlike Finland) a strong culture of testing (Dal-porto 2013). Hiere are also countries where results have been fluctuating over time. This is the case in Japan, whose stellar results in the initial round of PISA were followed by a decrease in reading in 2003 and another decrease in mathematics in 2006. Japanese policy makers responded swiftly to these declines by initiating a multi-year plan to improve reading followed by the implementation of national tests and a national curriculum review (Ninomiya and Urabc, 2011). Trends: The New Yardstick in Education Effectiveness During the Hrst few cycles of PISA, country ranking was the most commonly reported result in the media (as opposed to average score or the proportion of students at certain performance levels). In 1009, however, the emphasis started to shift, with greater interest being given to changes in a country's results over time. Ihis new focus was in part a response to the changes in the countries participating in PISA from one cycle to the next: with new (often high-performing) countries and economies joining in later cycles, it became increasingly difficult for established con ntries to make sense of their "ranking" over time. To assist 111 better evaluating changes within a country, the OF.CD developed an index of annualized change in performance that takes into account the number of years between each measure. This was provided in the ion International Report (OIvCD, 1013a) and gave participating countries a robust indicator of internal progress (or decline) over time. According to this index (see Table 4), about half of all cou ntries have improved over time in reading since 2000; a number of countries have seen a decrease in mathematics performance since 2005; and a majority of countries have remained stable in science since 2006. I able 4: Number of countries and economies and change in average score overiimeT'lSA 2000-2012; Reierencc Year i _u'i Readingioii compared ioiuuo \iatllenlatics loll Science loll compared to 1005 compared._ - 31 is 19 11 is 37 r, 14 S Note: (author'scalculations'' 1 Number ol rot 1 ntries and economics where average srorc has increased over Lime Number of countries and economies where average score has not changed significantly overtime Number ol rot 1 ntries and economies where average score has decreased overtime As a result of the country's high performance since 2000, reactions to PISA results have been more muted in Canada, than in many other countries. This has extended to the policy area, where policy makers have steered clear of making drastic changes based on limited data or research (Hess, 2008). However, provinces that did not fare as well as expected have introduced some moderate initiatives, based in part on PISA data and which could be characterized as fine-tuning an already strong system. New Brunswick reconsidered its French-immersion program (Dicks, 2008; Cooke, 2009) and Ontario launched both its Literacy and Numer- acy Initiative (to improve reading and mathematics results at the primary level) and its Student Success Initiative (to increase the high school graduation rate) (OF.OD, 2011a). A recent decline in national results, however, may stimulate a stronger response. Described in some quarters as "a national emergency" (Globe and Mail, ion), the weakening of mathematics skills evident in PISA 1012 has created calls for immediate action in two areas: the training of teachers in mathematics and a review of the content of provincial mathematics curricula. Teacher training was, interestingly enough, often cited as a reason for Canada's strong results in early PISA cycles (OECD, 2004). More recently, however, the international Teacher Education and Development Study in Mathematics (TJ:.DS-M), administered in 2008 (CMPC, 2010), pointed out that many Canadian elementary-level teachers lacked knowledge in mathematics and mathematics pedagogy, while those at the lower-secondary level lacked training in assessment. Ihe study also noted that a smaller proportion of mathematics educators (those teaching future teachers) in Canadian universities were specialized in mathematics at the doctoral level compared to the international average. Provincial mathematics curricula came under scrutiny from a number of observers for their emphasis on "new math." Hi is approach lies at the heart of mathematics curricula in most of Canada (with Quebec, whose mathematics scores greatly outranked the rest of the country, a notable exception) and was singled out for favouring discovery learning and problem solving over "basic knowledge and skills" and "daily-use math" (Alphonso, 2013). Lessons Learned At the time of writing this article, several provinces are considering proposals in the two areas of teacher training and curriculum renewal (Alberta Education, 1014; British Columbia Education, 2013; Manitoba Education, 2013; Nova Scotia Education and Early Childhood Development, 2013; Ontario Ministry of Education, 2014), although many initiatives had already been undertaken prior to the release of the PISA 2012 results. Canadian provinces would be well advised to reflect carefully 011 any reforms they may undertake. Judging by the situation in other countries, there appears to be an inclination to push the panic button and implement reforms based 011 limited evidence, loo often, causation and correlation arc confused when discussing PISA results (Mortimorc, 2009), and any outcomes should be validated with other data sources, such as other international studies and pan-Canadian or provincial results. Furthermore, the growing use of trend data, rather than reliance on comparative rankings alone, can significantly, improve the usefulness of PISA results, in particular in those countries such as Canada, the United States, Germany, Italy, or Spain, where PISA results are available at the regional, state, or provincial/territorial level. ihc ultimate goal of education should not be to finish first in the PISA race or improve in international rankings. Instead, education should enhance performance levels and equity for all students (Yore et al., 1010). PISA indicators should be used to attain these goals, by being integrated into national/federal policies and practices (Brcakspcar, ion), not by replacing them. Reform of Canada's education systems should acknowledge that from an international perspective, the country is still regarded as a model to emulate. It would make little sense to implement major changes to education policies across Canada based solely on PISA results when other countries use the same results to justify emulating Canada. Canada also benefits from a very large sample size in PISA, which allows results to be analysed at a fine-grained level. Canada's provinces can thus learn not only from the experience of other countries, but also from their neighbouring provinces (Wallner, ion). In the case of the most recent mathematics results, for example, Quebec appears to have much to impart to the rest of the country, as its results place it among the highest of all PISA participants in the world. Not only is it one of those jurisdictions that elected not to completely redesign its mathematics curriculum to integrate discovery learning into the program of study (Alphonso, 2013), it is also the province with the most mathematics teachers who arc actually specialized in teaching mathematics (CM EC, 1011) and the only province where the proportion of low achievers in mathematics has not increased over the past nine years (Brochu et al., 2013). As another example, students in British Columbia have achieved sustained high performance in reading, science, and problem solving in recent PISA cycles, and that provinces on-going curriculum review is cited as one of the reasons for their success (Sherlock, 2014). ibis paper has attempted to analyse the impact of PISA in several countries where PISA results have garnered considerable attention over the past decade. It argues that silver bullets based on PISA results are not only unrealistic but should be avoided (Hargrcavcs & Shirley, 2012). Education systems are complex entities requiring a thoughtful, systematic, and balanced approach to reform (Sahlberg, 2011). References Alberta Education (2014) Curriculum redesign - Letter from the Minister. Available from: http://education.alberta.ca/departmeiit/ipr/curric-ulum/ministcrcrlettcr.aspx (Accessed 14th March 2014). Alphonso, C. (2013, 3rd December) Canada's falling math rankings renews push for national standards. Globe and Mail. Available from: http://www.theglobeandmail.com/news/national/education/cana-das-falling-math-rankings-renews-push-for-national-standards/arti-clei5755434/ (Accessed 3rd December 2013). American Psychological Association (2010) Today'ssuperheroessend wrong image to boys, say researchers (Press rclcasc|. Available from: http:// www.apa.0rg/news/press/releases/2010/c8/n1ach0-scere0cype-un-healthy.aspx (Accessed 11th December 2014). Ammcrmiillcr, A. (2005) PISA: Whac makes the difference? Explaining the gap in PISA test scores between Finland and Germany. 'Discussion Paper No. 04-04. Centre for European Economic Research. Available from: ftp://ftp.zew.de/pub/zcw-d0cs/dp/dp0404.pdf (Accessed iilh December 2014). Anderson, J. O., Cliiu, M.-H., and Yore, E.D. (2010) First cycle of PISA (2000-2006) - International perspectives 011 successes and challenges: Research and policy directions. International Journal of Science and Mathematics Education. S, pp. 373-388. Anonymous (2007, December 21) PISA 2006 - The politics of testing. Update. 35 (4). Toronto: Ontario Secondary School Teachers' Federation. Raird, J.-A., Isaacs, 1'., Johnson, S„ Stoltart, G„ Yu, G„ Spraguc, '1'., and Daugherty, R. (2011) Policy effects of PISA. Oxford University Centre for Educational Assessment. Available from: http://oucca.edu-cation.ox.ac.uk/wordpress/wp-content/uploads/2oii/io/Policy-Ef-fects-of-PISA-OUCEA.pdf (Accessed nlh E)ecember 2014). Brcakspcar, S. (2012) Hie policy impact of PISA. An exploration of the normative effects of international benchmarking in school system performance. OECD Education Working Papers, No. 71. Paris: OECD Publishing. British Columbia Education (2013) B.C. students among best internationally. (Press release|. Available from: http://www.newsrooni.gov.be. ca/2013/12/bc-studcnts-among-best-internationally.html (Accessed iilI' December 2014). Brochu, P., Deussing, M-A., Houme, K„ and Chuy, M. (2013) Measuring up: Canadian results of the OECD PISA Study. The performance of Canada's youth in mathematics, reading and science. 2012 first results for Canadians aged />. Council of Ministers of Education, Canada. Chakraharti, R. (ion, 2nd December) South Korea's schools: Long days, high results. BBC News. Available from: http://www.bbc.co.uk/ ncws/cducation-25187991 (Accessed ii,h December 2014). Cooke, M. (2009J PISA 200(1: It's not about the Leaning Tower. New Brunswick Teachers' Association. Available from: http://www.nbta.ca/ profession/curriculum/financci4.html (Accessed n1'1 December 2014). Council of Ministers of Education, Canada (2010) Teacher education and development study in mathematics 200S: Canadian report. Toronto: Author. Council of Ministers of Education, Canada (2012) Contextual report, on student.achievement in mathematics. Toronto: Author. Coughlan, S. (2013, December 1) Pisa tests: UK stagnates as Shanghai tops league cable. BBC News. Available from: http://www.bbc.com/ news/ (Accessed nlh December 2014). Dalporto, D. (2011) South Korea's school success. We are teachers. Available from: http://www.wearetcachers.com/hot-topics/spccial-rc-ports/teaching-around-thc-worltl/south-koreas-school-success (Accessed iich December 2014). Dicks, J. (2008) The case for early French immersion: A response to J. Douglas Willms. Second Language Research Institute of Canada: University of New Brunswick. Available from: www.unbf.ca/E2 (Accessed nth December 2014). Dragcr.J. (2013) Accountability as a driver for reform: Tin' "PISA shock" of 2001 - A spotlight on the case of Germany. Harvard University, July 26th, 2012. Available from: http://www.hks.harvard.cdu/pcpg/con-fercnccs/Ju ly_20i 2_Presentations/ Dragcr_Pancl%2o I.pptx (Accessed jil1' December 2014). Duncan, A. (2013) The threat, of educational stagnation and complacency. Remarks of U.S. Secretary of Education Arne Duncan at the release of the 2012 Programme for. International Student. Assessment (PISA) Available from: http://www.ed.gov/news/speeches/threat-educa-tional-stagnation-and-complacency (Accessed nlh December 2014). Entorf, H., and Minoiu, N. (2005) What a difference immigration policy makes: A comparison of PISA scores in Europe and traditional countries of immigration. German Economic Review. 6(1), pp.155-176. Erth, H. (2006) Educational standards and the changing discourse on education: The reception and consequences of the PISA study in Germany. Oxford Review of Education. 32 (5), pp. 619-634. Figaz/olo, L. (2009) Impact of PIS/I 2006 on the education policy debate. Education International. Available from: http://download. ci-ic.org/docs/IRLSD0cumcnts/Rcsearch%20Wcbsitc%20D0cu-ments/ioo^-oooifi-oi-E.pdf (Accessed 11th December 2014). froese-Ciermain, B. (2010) Jl)e OECD, PISA and the impacts on educa-tionalpolky. Canadian Teachers' federation. Available from: http:// www.ctf-fce.ca (Accessed 11th December 1014). Grck, S. (2009) Governing by numbers: the PISA 'effect' in Europe../«?«;-" nal of Education Policy. 24 (1), pp. 23-37. Hargrcavcs, A., and Shirley, D. (2012) 'ihc international quest for educational excellence: Understanding Canada's high performance. Education Canada, fall 2012, pp. 10-13. Hess, P.M. (2008) The politics of knowledge. The Phi Delta Kappan. 98 (s).pp- 354-356. Levin, B. (2012) Cheater equity in education. Phi Delta Kappan. 93 (7), pp.74-76. Lloyd, M. (2012, May 14) Whac we can learn from South Korea? Edutech 'Associates. Available from: http://edutechass0ciates.net/2012/05/14/ what-can-we-learn-from-south-korea (Accessed 1 Ith December 2014). Malaty, G. (2011) PISA results and school mathematics in Finland: strengths, weaknesses and future. Available from: http://www.aka.fi/ fi/A/Suomcn-Akatcnua/Bh>git/Markku-Lcskcla-PISA-ja-olympi-alaiset/Asia-on-tutkittu-httpmathun ipait~griinii_projcct2i_char-lotte_MalatyPaperEditpdf/ (Accessed n1'1 December 2014). Manitoba Education (2013, 3rd December) Province's smaller classes initiative results in rjo new teachers in schools this year. [Press rc-leasel Available from: http://news.gov.mb.ca/news/index.htnilJar-chivc=&itcm=i9797 (Accessed n1'1 December 2014). Marginson, S. (2013, 16th December) Australia falls further behind in PISA test of basic education skills. Higher Education. Available from: http://www.theaustralian.com.au/higher-eciucation/australia-falls-further-behind-in-pisa-test-of-basic-education-skills/story-e6frgc-jx-1226783857196 (Accessed nlh December 2014). Martens, K„ and Niemann, D. (2010) Governance by comparison — How ratings & rankings impact nationalpolicy-makingin education. TranState Working Papers, No. 39. Available from: http://www.staatlich-keit.uni-brcmen.de (Accessed 11th December 2014). Mead, S. (2008) How Finland educates the youngest children. The Early Ed Watch Blog. Available from: http://www.ncwamerica.net/blog/ early-cd-watch/2008/h0w-finland-educates-y0ungest-children-9029 (Accessed ii'1' December 2014). Merry, J.J. (ion) Tracing the U.S. deficit in PISA reading skills to early childhood: Evidence from the United States and Canada. Sociology of Education. 86 (3), pp. 234-252. Mortimore, P. (2009) Alternative models for analysing and representing countries' performance in PISA. Education International Research Institute. Brussels. Ninomiya, A., and Urabc, M. (2011) Impact of PISA on education policy-Ihe case ofjapan. Pacific-Asian Education. 23 (1), pp. 23-30. Nova Scotia Education and Early Childhood Development (2013) Nova Scotia students perform well in international assessments | Press release |. Available from: http://novascotia.ca/news/release/?id= 201312.030 01 (Accessed n'1' December 2014). Neumann, K., Fisher, H.E., and Kauertz, A. (2010) From PISA to educational standards: The impact of large-scale assessments on science education. International Journal of Science and Mathematics Education. 8, pp. >45-563. OECD (2004) What makes school systems perform? Seeing school systems through the prism of PISA. Paris: OECD Publishing. OECD (2010) PISA 2009 results: What students know andean do. Student performance in reading, mathematics and science (Volume I). Paris: OECD Publishing. OECD (2011a) Strong performers and successful reformers in education lessons from PISA for the United States. Paris: OECD Publishing. OECD (2.011b) Education: Bridging the classroom divide. OECD Observer, No. 284. OECD (2013a) PISA 2012 results: What students know and can do (Volume I): Student performance in mathematics, reading and science. Paris: OECD Publishing. OECD (2013b) PISA 2012 results: Excellence through equity: Giving every student the chance to succeed. (Volume II). Paris: OECD Publishing. OECD (2013c) Programme for International Student Assessment (PISA) results from PISA 2012. Country note: Germany. Available from: http://www.oecd.org/pisa/kcyfindings/PISA-2012-results-germany. pdf (Accessed n'1' December 2014). OECD (2013d) Programme for International Student'Assessment (PISA) results from PISA 2012. Country note: Spain. Available from: http:// www.occd.org/pisa/kcyfindings/PISA-2012-rcsults-spain.pdf (Accessed iiLl1 December 2014). Ontario Ministry of Education (2014) New math supports and resources for the classroom Ontario government committed to student, success [Press release]. Available from: http://news.ontario.ca/edu/ l counlrics No.ol !h nclimarkin<^ |ii risdici ions1 No.ol U.S. slate.S .inuin^l ii Lx'nclini.irk-ing jurisdictions 1 'ISA LOTl 64 4 - " , -:"!> . 4' ......• 45 1 MSS PHILS4 1-11 (•-■; I 14 9 1007 « s 1 : 5 49 4 1 '99? >9 ■995 45 1.11 4s ^ <• 9 ii <• 1 0 « 1 "Benchmarking jurisdictions'' refers to subnational entities diat participate independently in ati assessment i.e.,eitlier represent itigati incomplete subset of a nai ions subn.uion.il jurisdictions or those thai linance their own participation in addition to the nation's participation. The OECD does nor separately identify "benchmarking jurisdictions" because until iooji, no subnational jurisdictions participated independently. (Chinas two autonomous states oi 1 long Kong and Macao have participated since 1000 and 1005, respectively and arc instead included in die country 1 Mat liemi ticsis eMmined bccause ii wrisihr (ocas o( 1 he 1:11 PISA cycle. M S I l-'PI 11-AS AMJ A. SI X ■ COMPARING L'.S. S'l A I liS M A I I ll; MAI ICS KI-'SL I I S - count. Additionally. a number oi led era I countries have voluntarily overs,implcd in various years ro provide tor disaggregation within the national data and these casts are noi counted as bench marking jurisdictions.) I or the purposes of this table, we have included the livesiibauion.il jurisdictions th.u were represented in PISA 2009 (011c each from China, the United Arab Emirates IIAE . and Venezuela and two I torn India) and the lour Irom 2011 ^Shanghai-China and the three U.S. states). I he IliA has historically treated the situational jurisdictions ol the Flemish and French communities of Bclgitma and the nations of the I ,-iiired Kingdom as individual education systems, on par with other national systems and these art included in the country count [or 1 I.YISS.md PIR1.S.I lovvevcr, they separately identily other subnatioiv al jurisdictions stiiih as the various Emirates of the UAE, U.S. stares, or Canadian provinces. I his column does not include district ordistrict consortia patticipation. 1 Counts include countries, jurisdictions, and states th.u administered a given years assessment in the primary vear or a follow-up wave (e.g., 1000 PISA in 1001/2 or 200,9 PISA in ioio'. j 'I he counts include participants in -+' and/or S1 'grade I IMSS. 4 Only more recently (1011) has the Progress 111 Reading Literacy Study jPIRLS) been opened lor subnaiiona I participation . I lorida was the U.S. state that participated in PI RES ion. Background on State Education Systems and Assessment Education in the United States is decentralized, with each state having responsibility for governing its own education system. These responsibilities including distributing federal and state funding, establishing policies (such as the duration of compulsory education, requirements for graduation, and minimum teacher qualifications), providing guidance regarding curriculum, conducting student assessments, and ensuring equal access to education for all eligible students. Often, some of these responsibilities — particularly those related to instruction — are further delegated to localities, which manage the operation of schools in their districts. While some aspects of education are very similar across states (e.g., the organization of schools), other characteristics (e.g., policies for compulsory education, demographics, teadier salaries) vary (see Exhibit 2, which provides a brief overview of education in the PISA-participant states). ihe three PISA-participant states, as well as the other U.S. states, have access to a number of different macro-measures of student performance, and for the purposes of this article, we focus on those that can currently be compared across states, including the National Assessment of Educational Progress (NAEP) and the international assessments, PISA and TIMSS (see textbox for information and context 011 other macro-measures of student performance). NAEP is the longest-standing measure of student performance for most states. Hie NAEP 4th- and 8th-gradc assessments in reading and mathematics, which are given every two years, are effectively required, as participation is a condition of receiving Title I funding, which is a primary financial resource (over $14 billion in 1014) for school districts and schools with high percentages of disadvantaged students (Federal Education Budget Project, 1014). The other NAF.P assessments, including those at nth grade and in other subjects, are voluntary. NAEP is designed to measure the knowledge and skills students have acquired in school on content determined through the collaborative input of a wide range of experts and participants from government, education, business, and public sectors in the United States. As the "nation's report card," NAtP is supposed to reflect what U.S. students should know and be able to do. For states, the benefit is the long trend line and the applicability to the U.S. context. libit 2: Overview oi Selcncd Edurailon System Charari eristics in ; U.S. States: 201111 (. -i HI hccl K ill Florida Massaclius< 11 \ Appoints llie1 Suit Supirinkiultnl Suit-Board (jovt 1 nor Suit Boa ml Ann«.» inls iIk Suit Boh id C jovt-rnor (jovt 1 nor (iov( rnor Structure 1 v pi cal 0 rg,a n i /a tio n Flcmenraryc duration {K1 ndcrga rren rl VI iridic school (grade: 6-S 1 1 igh school ¡grade?-ti" rough grade y 1:11 trance d^e Must be 5 by January 1(oi school vear" Must be 5 by September i Localities delermitie C .0 n 11 n 1 1 vo iv cc 1 Ui,i l ioi 1 j-iS f. 16 \'n ol divlru ls -i. 4.1 No ol schools Uy. 4.10- i.S« No ol vllldctllS S54-4J7 \'n ol i c.ic crs V: f-.'i.u1 Srude n r-reael icr ratio 117 TV1 U-7 IV rcenr ot sTudents 1 ;R l'[ I4i Iota 1 expenditureon publicc lemen-lary ,md secoiid.tr>1 education' S<>.of>4,'„-i6,lS6 WJ.64v.965.J6} Average annual salary oi public elementary and secondary teachers >46.15! ^1*1 .'J'J _ am 1 Reference year is 2010-11. TRPL is free and reduced price lunch, indicating students with lower socioeconomic resources. 1 Rclercncc vcar is 2010-11. Sources: N( -E S, 1014: ECS, 1014a: ECS, 1014b. M. SI I'PI 11-AS A\D A. SIX ■ COMPARING L'.S. S'l A I liS MAI IIP'MAI ICS RI'SL'I IS. The international student assessments, PISA and TIMSS, are not rc-quired, though the Common Core initiative—described in the text box— has underscored the value of states' engaging in assessment in an international context. The Common Core initiative, which began around 1008 to increase rigor across state education systems, is both a result of and a driver of states'participation in international assessments. Since the 1995 administration of'TIMSS, a total of 18 states have participated in at least one cycle of 'TIMSS, wich 9 participating in multiple cycles and 9 participating in che most recent 1011 cycle. Additionally, as an indirect measure, states have looked to the estimates produced by the NAEP-TIMSS Linking Studies, the most recent of which used improved mcchodology to estimate 'TIMSS scores for eadi of the 50 states based on their NAJ .P scores and the NATP and TIMSS results from the 9 states participating in both assessments in 1012. and 2011, respectively (NCJ .S, 2013). Ihishas been an important - if less reliable - source of information and significantly less costly than actual participation in international assessments. 1 able Overview of Select t J Cliaraci eristics of Assessment Programs PISA i ii 1 IMSSi II NAIil'ion Frequency livery } years Lvery 4 years livery 1 years largei population 15 years old' Grades 4 and S (j rudes 4. and 12 No. of schools sampled yj - -■-1,00. 7.610 No. oi students sampled' •ll.tjtjo -a____u 175.10-, 1111 die United States, PIS As age-based national sampled included students 1110 s thin grade 10 (71 perceiu in ion), though some were in grade 11 ii-percent':, grade 9 in percent). or 01 her grades (less 1 ban 1 percent). i Tor TIMSS and XAEP the numbers are for grade 8 only. Tor all three assessments, die numbers include siaiepan icip.itus. Sources: Prov.isrtikct al.; ¿015; Mullis el al.,2012: XCI.S.ion. Evolving Scate Assessments and che Context for International Participation All U.S. states also have state assessments. Some states have had assessment systems i or decades, 01 hers initialed 1 he in in 1 he tyyos with ihe passage of si ate account abili-iv laws, and ihe rest developed or expanded ihein under 1 he rcqiiircmcnis of ihe No Child Left Behind Act iNCLB) 2001 iChingos, 2012;. XCLB required that states lesi all si udetus in grades and in one grade in high school in uuuliemai ics and reading. Priono X('1 B.onlv 1; siaiis had assessment sysicms 1 hiscxiensivc(Danii/, 2001, as ,iteil in ("liingos, 1011). State assessments, however, arc in the midst of another major change, as mosi si ales vviih a boost Irorti iticemives irom ilii. lederal level have adopted 1 he Common Core Siaie Siandards, which is an iniii.uivc ihai developed common siandards in core academic subjects, and mosi arc collaborating on the development of assessments oi 1 hose siandards thai will it place their existing systems. I his will mean 1I1.11, lor the iirsi Li inc. there will he comparability in learning siandardsacrosssi.iicsand in per Ion nincc tne.isures.imong.il le.isi some si .lies. Ihe Jack of comparability and variable quality across states lias been an often-cited weakness ot NCLB in the past .e.g., Linn, Baker, and Bctebeiiucr. i002.). "flic main purpose ot the Common ("01c is to increase the rigor of standards and align them with die expectations of education institutions and employers so that students meeting the standards will be ready for college or a career. A major driver ofthe Common Core was die states themselves - the initiative is managed bv the Council ol Chief Suit School Oil icers and the National Governors Association and their expressed need for improved benchmarking namely,"comparingoutcomes to ¡dentil v lop performers or last improvers, learning liovv they achieve great results atid applying those lessons 10 improve...performance' (NGA,ioo8,p. y), with an explicit acknowledgement that the standards and benchmarks should luve an international component. 1 litis not only should the standards be rigorous enough to allow U.S. si 1 idems 1 o compete in 1 he global economy, si .lies shot 1 Id me.ist 1 re pcrlorm.incc in an inicrn.iiion.il context (with implicit favour being given 10 PISA as die assessment ol choice Schneider, 1009 ). hoi tv live states and the IHsirict ol (Columbia (including 1 he three PISA-pariicip.ini si ales') have adopted 1 he Common C lore si 1nd.11 ds in hoi h English language iris and tn.uheinaucs and an additional si .11 c in in.11 hematics only. I wo consortia, die Partnership for Assessment of Readiness for College and Careers (PARCC) and the Smarter Balanced Assessment (SBAC) (Consortium, are the primary groups working 011 the new assessments that will roll out in the 1014-15 school year. Connecticut is signed 011 with SBAC, Massachusetts with PARCXC, and Florida with another private provider.2 One analysis has suggested rhat with a quality implementation of the Common Core 111 mathematics and well-designed assessment tasks particularly at the secondary level, U.S. students would be learning the kind of mathematics that j c* would make them potentially more competitive in PISA (OH CD, 101 >V States have participated in PISA because of its targeting ofstudents ncaring the end of compulsory school and its focus on students' ability to apply the knowledge and skills they have learned cumulatively during their schooling, as well as in other contexts, for solving problems in a real-world context. States have participated in TIMSS, on the other hand, 1 I lorirla has conrracred with American Insnrures for Research (AIR) to develop irs srare assessments. A IRis die home organisation ol tlieauihorsoi this article: diearnliorsarein a separate division andindepetideni o[ 1 hat project i I he relerenceil analysis classified 1 lie PISA icii items :ig,ainsi 1 lie Common Core progression according to where thev sit in the progression oi standards tip to high school level, the degree to which they represent aLtributes ol modeling; and their modeling level Tlieanaf vsis iotuid a decree of conitnonality between the PISA and Common Core constructs leading the authors ro conclude rhar "the high school curriculum in rhe United Srares will arte nd ro model inero a greater degree than has happened in rhe past, a nd it J more snidenrs work 011 more and herrer modeling tasks than they doj today, rhen one could reasonably expect PISA performance to improve" p.y . . M SI l-'PI I l-AS AMJ A. SIX ■ COMPARING L'.S. SI A I liS MAI III' VI Al ICS RI'Sl'l IS. because its grade-based target populations are similar to NAEP (grades 4 and 8) and it also similarly focuses 011 school achievement. Both the national and two international assessments collect data on mathematics and science performance, though the sampling requirements and other features differ (see Exhibit 3). U.S. States' Mathematics Results from PISA, TIMSS, and NAEP While each of the sources of student performance data provides valuable input for U.S. states, interpretation of results across the multiple measures requires careful consideration. (Again, since the state assessments aligned with the Common Core arc not fully in place yet, we focus here on the data available from PISA and how that aligns with data available from TIMSS and NAEP.) On average, students in the United States performed below the OECD average in mathematics literacy, scoring 481 points compared to the OECD mean of 494 in ion (see Exhibit 4 and Kelly et al., 2013). ibis masks variation among the states, however. Students in Connecticut scored an average of 506 points, which was above the U.S. mean though statistically comparable to the OECD mean. Just 11 of the 68 total participating education systems scored higher than Connecticut and its scores were comparable to those of students in 15 other systems, including the United States' partners in the C-8 Canada, Prance, Germany, and the United Kingdom. Students in Massachusetts scored an average of 514 points, which was statistically comparable to Connecticut's mean but above both the U.S. and OECD means. Nine education systems outperformed Massachusetts and its scores bested an additional six education systems than did Connecticut's. In contrast, students in Florida scored 467 points on average, which was lower than both the U.S. and OECD means. Florida's mean score was below that of 38 education systems and statistically comparable to a set of five education systems—Lithuania, Sweden, Hungary, Croatia, and Israel—thac were outperformed by the other two PISA-participant states. 'Ihcsc findings were not necessarily surprising as the two Northeastern states are typically above average in NAEP and Florida is typically below-average, as they were in 2011. Looking across the assessments highlights some differences and generates interesting questions. Connecticut performed above the U.S. mean in mathematics in PISA 1012 and eighth-grade NAEP 2011, and above che international mean in TIMSS 2011 though similar to the OECD mean in PISA 2012. (It should be noted thac the OECD mean in PISA is based only on the scores of the participating OECD education systems whereas the international mean in TIMSS is bascdon all theparticipatingTIMSS education systems, which is a much more diverse group in terms of student outcomes.) However, despite the differences in performance relative to the international means, Connecticut appears to have a greater advantage in PISA than in TIMSS (and NAPP), based just on distance from the U.S. mean. What might account for this advantage? Table 4: Mathematics performance of U.S. 15-year-olds 111 PISA arid eighth-grade students in T LV1SS and XAHP: 2.011 and 2.012 ~ o I'ISA i n jyvcar-olds) '1IMSS :ott nadoS' NAUI'ion : Grade Í Mean score lick Live to US. Relative to OF.C.D Mean score Relative to US. Relative to lilt 1 Mean score Relative to U.S. C .onncvlicui y. {■ 51S 1X7 , Florida 467 17 S Ma vs,uluiscl Is >14 Uniu-dSiau-c 481 Across 494 countries - 561 v:> - 1\>V> IX, - Range across y states Rans;e across 36$; Peru': él, ¡Sliang-countries liai- Chilla^ 4<ír,';Ala.':-i6i Mass jji fGhaiia": 6i> ¡Korea, Rep. oi) if,-.. 1 >Q -199 'Mass.; ' 1 Note: PISA measures mathematics literacy, or the application of mathematics tor solving real-world problems. 1 1.V1SS and XAHP locus more exclusively on school-li.ised m.uhcmaiirs. 1 Hie range is based 011 scores estimated in the XAEP-TIMSS Linking Study; results lor tlie three PISA suites, however, are actual I I M.SS restilts as tliev also participated In TIMSS 2011. Xot applicable + SigtiiIicanily higher tlnm reference at l!le .05 level. Signilirantly lower Lhan rclcrcnre.u ihc.05 level. Not significantly different than reference at die .05 level. Sources: Kelly el a I.. 2.01;; .VI till is et a I.. 2.011; XCHS, 2012; and XCHS, 2013. Massachusetts also performed above the U.S. mean in mathematics 011 all three assessments, as well as above the respective international means for PISA 2012 and TIMSS 2011. Based 011 distance from the U. S. mean, however, Massachusetts appears to have a greater advantage in TIMSS (and NAPP) than PISA. Again, what might account for this particular advantage? M. Si I'PI HAS A \ D A. SIX ■ COMPARING L'.S. S'l A I liS MAI IIP MAI ICS IU'Sl.1 IS. Finally, Florida performed lower than the U.S. mean in mathematics in PISA ion and eighth-grade NAF.P ion, but similar to the U.S. mean in TIMSS ion. On the international assessments, despite being lower than the OF.CD mean in PISA 1011, Florida is above the TIMSS ion international mean. How would Florida's relative standing change if the groups of education systems participating in PISA and TIMSS were comparable? What are some possible explanations for Florida's wcakcr-than-avcr-age performance? Analysis of Differences in Results A first analysis to explore these questions is to examine the similarities and differences in terms of item content, which has been collected through studies comparing the various international assessments with each other and with NAEP.4 Generally speaking, these studies have shown that, overall, there are more similarities between NAFP and TIMSS than between NAEP and PISA, as might be expected given the former two programs' focus on curriculum-based achievement and the Fitter's on literacy (Provasnik ct al., 1013; AIR, ion). For example, PISA differs from T IMSS and NAEP in terms of the distribution of test items across content areas: PISA 1011 had a larger percentage of items that would be considered data analysis, probability and statistics items on the NAFP framework than did NAEP 1011/1013 or TIMSS ion, whereas it had a smaller percentage of items classified as algebra (see Exhibit s).' Additionally, the most recent comparison study identified several topics covered by the NAF.P 1013 item pool that were not covered by the PISA 1011 item pool - i.e., that were unique to NAEP - including: estimation; mathematical reasoning using numbers; position, direction, and coordinate geometry; mathematical reasoning in geometry.; measurement in triangles; experiments and samples; mathematical reasoning with data; and mathematical reasoning in algebra (AIR, 1013). In terms of item complexity, PISA 1011 had a greater percentage of items classified as "moderate" on the NAEP framework than eliel NAEP ion, and a smaller percentage classified as "low" (data not shown, AIR, 1013). 4 See liuji:/ nces.ed.gov/surveys.inierr la uonal/cross-siudv-compari.sons.asp [or a lismigoi Liii_.sc studies through i'jn. 5 This is based 011 results from two studies; one (Lin. Darling, and Dodson, that compared the XAEP 2'.ii and TIMSS i'.ii grade S mathematics items :among other ele-incurs": and another (AIR. that compared the NAI.P 1 u grade Sand I'ISA 1-.11 items ;a mong other ele menrs). I hough dil Fere nt expert panels undertook the studios, the distribution of NAllP grade S mathematics items across eonre nt a reas was assessed (similarly by 1 he 1 yvo groups. l ibit 5:1 )isiribuiion ol iicms across NAI.P 111.11 hem.11 ics content .ire.is Conn-Ill Area«in llicNAF.P Frsmew. >1 k PTSAi.ai' XAF.P; G i,u)( x NAF.P-.11 TTMSSion Grade X Xinnberproperl ie« antlopt-r.il ¡011« Gromerrv ««S 14'" T7''a I7«S iS'S 17"S 9% .Vleacuremonr IC.'V w''n Ti";i Data analysis, probability andsraTisries Algebra ir'o TV'.. 14'., iS"„ 52% 54"i 1 1 his is lused on i he 64 (ol S5) PISA hems 1h.11 were ck.ssllied 10 die NAI.P grade X tramework. Sou ires: Provasnikci aLioi j: AIR. 101 So, theoretically if students in Connecticut - where there appears to be a relative advantage in PISA - have had greater exposure to data analysis, probability, and statistics items or items of similar nature or complexity to PISA items, this might contribute to their relatively strong performance in PISA. On the other hand, if students in Massachusetts - where there is a relative advantage in TIMSS - have had a strong focus on algebra this could partly explain the excellence in TIMSS and NAEP. This could be explored by examining the state standards and assessments in place. A second analysis examines states* scores on die mathematics sub-scales, which in PISA ion included three processes (employ, formulate, and interpret) and four content categories (space and shape, change and relationships, quantity, and uncertainty) to determine if states' relative strengths and weaknesses align with relative areas of emphasis or de-emphasis in the various assessments. Por example, Connecticut was comparatively strong in items requiring interpretation, of which there were a larger percentage in PISA 2012 than 111 NAEP 2013 (see Exhibit 6). Items in the interpretation category were a relative strength for all states, however. In terms of the content subscales, there were again similar patterns among the PISA-participant states, with change and relationships (i.e., algebra) and uncertainty (i.e., probability and statistics) as relative strengths and quantity and space and shape as relative weaknesses. It is difficult to relate these results to item distributions in NAEP, however, because in the comparison study on which the data are based, a high percentage of NAEP 2013 items were found not to fit the PISA framework. A third analysis relates to sampling. As PISA uses an age-based sample, sampled students may come from various grades, which is a distinction from TIMSS and NAEP. This feacure of PISA is in keeping with M. SI I'TI IIA'S AMJ A. SIX ■ COMPARING L'.S. S I A I lis' MAI lll; MAI ICS IG'Sl'l IS. 1 able 6: Mathematics performance and percenugcdisiribuiionol items by PISA process and content snbsealcs 1 .mpbv" 1 Vocess suhsca les C 'on te n r suh.scale.s lonnu- Intel'. Into prer Space a lid shape Change and relationships (^iia n-tiry I net -ra inty Me.111 score Connecticut 502 yj4 515 4»7 Ui 5-1 511 i lorkla 4 66 45i! 475 4 46 476 45S 475 Massachusetts 5-9 511 5-4 49f US yj6 525 United Slates 4*'' 47* 49'■ 4S8 47S +*S OF.CD 495 491 497 49'■ 495 495 495 Pt rct nl- PISA 44% ;<»■■ 25«i KA, 15% NAF.P >>'\i -0;; 14« Mi -oii, I lie percentages lor XAHP items in die content categories will not sum to 100 Ix-rause 66 parent o[ the XAI'.P eighth-grade items were lotind not to In the PISA framework. Source: AIR, ion and PISA lnicnuiiemal Data Hxplorer (hiip://ticcs.ed.ge>v/siir-vcys/intei nai ion.il/idc/ its goal to measure the outcomes of learning, rather than schooling/)«-.«? and provides a neutral comparison point internationally. Intra-national-ly, in the case of federal systems with variation in local education policy, this can be a source of some dilferences. For example, analysing the grade distribution of the students who took PISA in ion shows that Connecticut had a larger percentage of students in the nth grade and smaller percentages in the 9th and toth grades than Florida, Massachusetts, or the United States overall (see Exhibit 7). Conversely, Florida had a larger percentage of students in the 9th grade and smaller percentages in the upper grades than the other systems. In other words, a larger percentage of Connecticut's students were exposed to an additional year of schooling than were U.S. students on average or in Massachusetts or Florida. And a larger percentage of Florida's students had not yet been exposed to 10th- or nth-grade mathematics than had students in the other systems. This is due to dilferences in policies on school entry and in grade retention practices. For example, Connecticut has one of the youngest kindergarten entry ages in the United States, allowing students to enrol at 4 years old as long as they will be 5 years old by mid-school year (e.g., January 1) and requiring enrolment at 5 (ECS, 1014; see also Exhibit 1). Other states more typically have cut-offs early in the school year, requiring that students be 5 years old, e.g., by September 1 and not requiring enrolment until 6, as in Florida and Massachusetts. What may then account for Florida's higher Soi. s ko pou h. i i; r\i k \ x v. s i mvii.ka 5-6 rate of yth-grade PISA participants arc generally higher early gracie retention rates than in the other two states (Warren and Sariba, 1012). Of the analyses described, the sampling explanations appear to have the strongest explanatory potential. Table 7: Distribution ot PISA participants by grade: ion Connecticut i:j"h grade Î9 11" ^r.iilo Î4 ,i"gr.,dt- Florida IT (<7 1: - \1assachuscrns I Si IT - United.States 11 71 T7 - * Report ing siandards noi met. Note: Results for Connecticut. Florida, and Massachusetts are for public school students only. Detail maynoi sum lo louds because ol rounding Some apparent differences beiwecn estimates may not bestatisLically signilic.ini. Source: PISA International Data Explorer (http://n<:cs.ed.gov/kinTVs/internanon-abide/). A final analysis questions the differing country populations in PISA and TIMSS: how would states' standings relative to the OE("D/i titer national means change if the assessments included the same group of countries? Restricting the countries in the analyses to only those that participated in both PISA 1012. and TIMSS 1011 at eighth grade, & both the OF,CD and international averages drop. So, while this brings Florida's mean score closer to the PISA OECD mean score (though still statistically significantly below it), it further distances the state's mean score in a positive direction from the TIMSS international mean - essentially, leaving the relative standings unchanged. Conclusion U.S. states' participation in international assessments shows one source of variation in national statistics and also allows states to benchmark themselves to international standards, as has been shown to be an increasing interest over at least the last decade. However, given that states also have access to national assessment data, as well as their own state data and, in some cases, two sources of international data, making sense of the results can be challenging. Analyses described in this paper suggest that opportunity to learn may be an important factor in differing results among assessments - with the amount of schooling related to states' PISA per- ff I his ivpiesenTs iS countries, with rhoonly dilFerenoo hoi 112 r hi pa mo iparion of a II nations ofrhe l-n ¡rod Kingdom in I'ISA versus only l.ngjand in I IMSS. I ho ivforenood analysis is based on dn a inoi shown' obtained Irom iIik PISA InimiaHona] Daia Explorer. m. si i'pi has avd a. six ■ comparing l'.s. s i a i lis' mai 11 i'm a i ics ri'su i i s ... fbrmancc. A next frontier for state participants in international student assessments will be in how they, and their localities, may try to extend the use of data beyond the core benchmarking function to absorb lessons from international partners and inform education policy. References American Institutes for Research (1013) A comparison study of the Program for International Student 'Assessment (PISA) zot2 and the National Assessment of Educational Progress (NAEP) 2013 mathematics tissess-ments. Washington, D.C.: Author. Chingos, M.M. (1012.) Strength in numbers: State spending on K-12 assessment systems. Washington, D.C.: Brown Outer on Education Policy, 'Ihe Brookings Institution. Available from: http://www.bmok-i11gs.edU/~/media/research/files/rep0rts/2012./j1/29%20c0st%20 0f%20assessment%20ching0s/i 1_assessment_chingos_fi11al.pdf (Accessed 7th July 2014). Danitz, T. (2001, 27th February) Special report: States pay $400 Million for tests in 2001. Stateline. Pew Center 011 the States. Education Commission on the States (2014) Early learning: Kintlcrgar-tcn online database [data set]. Available from: http://www.ecs.org/ html/cducationissues/kindergarten/kdb_intro_sf.asp (Accessed 7th July 2014). Education Commission 011 che States (2014) Whac Governors Need co Know: Highlights of State Education Systems. Available from: http://www.ecs.org/clearinghousc/8s/69/8569.pdf (Accessed 7th July 2014). Federal Education Budget Project (2014) Available from: http://febp. newanierica.net/background-analysis/no-child-left-behind-funding (Accessed 7th July 2014)-Kelly, D„ Xie, H„ Nord, C.W, Jenkins, F., Chan, J.Y., and Katsberg, 1). (2013) Performance of U.S. 15-year-old. students in mathematics, science, and reading literacy in an international, context: First look at PISA 2012 (NCES 2014-024). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Available from: http://11ces.cd.gov/ pubs2014/2014024rev.pdf (Accessed 7th July 2014). Linn, R.T.., Baker, E.E., and Betcbenner, D.W (2002) Accountability systems: Implications of the requirements of the No Child I .eft Behind Act of 2001. Education Researcher. 31 (6), pp. 3-16. Provasnik, S., I.in, C„ Darling, D., and Dodson, J. (2013) A comparison of,the 20tt Trends in International Mathematics and Science Study (TIMSS) assessment items and the 2011 National'Assessment of Educational Progress (NAEP) frameworks. Washington, D.C.: AVAR. Available from: http://nccs.cd.gov/nationsreportcard/subject/ about/peif/nacp_timss_comparison_items.pdf (Accessed 7th July 1014). Mullis, I.V.S, Martin, M.O., Foy, P., and Arora, A. (1011) TIMSS2011 international results in mathematics. Boston, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College. National Centre for Education Statistics (1012) Mathematics 2011: National Assessment of Educational Progress at. grades 4 and 8, NCES 2012-45$. Washington, D.C.: NCES, Institute for Education Sciences, U.S. Department of Education. National Centre for Education Statistics (2014) Tables and Figures. Available from: http://nces.ed.gov/quicktables/index.asp. (Accessed 29th September 1014). National Centre for Education Statistics (NCES) (2013). U.S. states in a global context: Results from the 2011 NAEP-TTMSS linking study, NCES 2013-460. Washington, D.C.: NCES, Institute for Education Sciences, U.S. Department of Education. Available from: http://nces.cd.gov/nationsreportcarei/subjcct/publications/studics/ pdf/2013 460.pdf. National Governors Association (2008) Benchmarking for success: Ensuring US. students receive a world-class education. Washington, D.C.: NGA. Available from: http://wwvv.corcstandards.org/as-scts/o8i2BENCHM ARKING.pdf (Accessed 7th July 2014). OECD (2013) Strong performers and successful reformers in education -Lessons, from PISA 20T2 for the United States.. Paris: OECD. Available from: http://www.0ccti.0rg/pisa/kcyfinciings/PISA2012-US-CHAP4.pdf (Accessed 7th July 2014). Schneider, M. (2009) Ihe international PISA test. Education Next., 9, 4. Available from: http://etiucationnext.org/the-international-pi-sa-test/ (Accessed 19th June 2014). Warren, J.R„ and Saliba, J. (2011, May). Public school,grade retention rates in the United- States: Estimates by state, grade, year, and race/ethnicity. Paper presented at the Population Association of America Annual Meeting, San Francisco, CA. The Predictive Power of Attribution Styles for PISA 2012 Achievement: International and National Perspective Ana Kozina and Ana Mlekuz Introduction In the paper, Weiner's attribution theory is used as a framework in explaining the differences between high and low achieving students in PISA 2012 study for international and national analyses. The reasons people give for why they succeeded or failed a task are called attributions. (Heider, 1958, in Nokelainen, Tirri and Merenti-Valima-ki, 2007). Furthermore, attribution theory has been widely recognized as a significant contributor in achievement explaining models (Stroud and Reynolds, 2009). According to Dembo and Eaton (1996, in Stroud and Reynolds, 2009), motivation is constructed from three internal factors, one of them being the students' attributions for success and failure (the other two are: the importance placed on the task and the emotional process associated with the learningprocess). Weiner (1985; 2010) defined attributions more precisely. He distinguished attributions on three dimensions: locus (whether the cause is internal or external), controllability (whether the cause can be subjected to volitional influence) and stability (whether the cause is stable or varies over time). He also identified four common attributions that differ on these dimensions: effort (internal, controllable and unstable), ability (internal, uncontrollable and stable), task difficulty (external, uncontrollable, stable) and luck (external, uncontrollable, and unstable). Additionally, attribution constructs can be classified into three groups: attribution appraisals (explanations assessed following actual or manipulated success or failure in performing a specific task), attribution beliefs (domain specific or domain general beliefs about the causes of success or failure), attribution styles (generalized, stereo- typical patterns ofiattributions and dispositional beliefs) (Dai, Moon and Fcldhusen, 1998, in Nokelainen et al., 1007). The specific attributions that students make affect their expectancy for future performance, persistence in similar tasks, emotional responses, which tasks they choose, and self-efficacy, which is an important characteristic for educational setting (Demo and Eaton, 1996, in Stroud and Reynolds, 2009). Students with an internal locus of control believe that events in life are controlled by their own actions, whereas those with an external locus of control attribute the outcomes of events to outside factors such as luck. In general, people with an external locus of control appear to be prone to a variety of symptoms of stress including emotional distress, job dissatisfaction, burn-out and low self-esteem (Matthews, Deary and Wihitemau, 2009). On one hand, students with attributions showing the internal locus of control (e.g. eifort) will work harder to improve themselves in school. I11 addition to this, those students who attribute their success or failure to external factors (e.g. parents, friends, teachers...) tend not to invest more time in learning. Hie motivational path of causal attribution begins with the interpretation of the event (in our case the mathematics achievement) as success or failure. Following the initial reaction of happiness or sadness, individuals search the reason why this specific outcome has occurred. In the achievement domain, successes and failures are often attributed to an ability factor, an effort factor, the difficulty of the task, luck, mood and help or hindrance from others. When explaining achievement results, individuals attach the most importance to their perceived competences and how hard they tried, ihe attribution theory proposes that people spontaneously engage in such causal thinking in their everyday lives (Graham and Williams, 2009). Studies broadly investigated the relationship between attribution styles and academic achievement (Gibb et al., 2002) stating a significant relationship and significant predictive value of die locus of control for academic achievement (Gibb et al., 2002; Philips and Chilly, 1997), study time and effort (Shell and Husman, 2008). For instance McClure, Meyer, Garisch, Fischer, Weir and Walkey (2011) examined die relationship between attributions for success and failure and academic achievement among students aged 14 and 15 years (as in PISA study), They also measured motivation orientations and cultural differences; therefore European, Asian, Maori and Pacific participants were included in the research. The measure assessed attributions (causes for their best and worst performance only), motivation orientation (doing my best and doing just enough scales), demographic data and achievement data. The results firstly confirmed the self-serving bias, which was already proven in many pre- A kO/.IXA AND A. VI 1,1'kL / ■ I III' PRI'DIC'I IVT POWTR.OI- A'l I R.IHU I'lON SI V IS ... vious studies (e.g. Bong, 2004; Vispoel and Austin, 1995). Students show a self-serving pattern of attributing their highest marks to effort and ability more than their lowest marks, which are mostly attributed to task difficulty. Students who attributed their best marks to internal factors of ability and effort attained higher achievement. On the other hand, students who attributed their best marks to luck, family and friends gained lower achievement scores. Moreover, attributions for their worst marks were also important. Students who attributed their worst marks to ability, effort, high task difficulty and the influence of teachers gained higher achievement scores, whereas students who attributed their worst marks to family and friends gained lower achievement scores. In addition, the regression analyses showed that the students' motivation orientation and attributions is a significant predictor of achievement, accounting for 38 % of the students' achievement scores. Among attributions die strongest positive predictor was attributing the best marks to effort and die worst marks to lack of effort and to the influence or characteristics of the ceacher, while the main negative predictors were attributing the best or worst marks to family and friends and attributing the best marks to luck. Similar patterns were established in primary school students. Klioda-yarifard, Brinthaupt and Anshel (1010) examined the relationships between academic achievement and the child's and the parent's attribution styles in primary school students and their parents. Regarding the connection between attributions and academic achievement, the results were consistent with previous research (Carr et.al. 1991; Stipek and Hoffman, 1988), Students who did not perform well academically tended to show a more negative attributional style (attributing negative events to more stable and uncontrollable causes). Longitudinal effects were tested in Liu, Cheng, Chen and Wu (1009) study, fhey examined the longitudinal effect of educational expectations and achievement attributions on adolescents' academic achievement (secondary school students), ihe results show that high educational expectations and attribution to effort (controllable, unstable attribution) have a positive effect on learning growth rate, while attributions to others have a negative effect on the learning growth rate, furthermore, as already proven in previous research (e.g. Georgiou, 1999), attributions of achievements to effort are positively related to actual achievements, whereas attributions to others are negatively related to achievement. The pattern of perceived control is associated with better self-regulation, knowledge building, question asking, study use and effort (Shell and Husinan, 2008). The study showed that such attributional patterns influence the long-term academic development of adolescents (Schunk, 1991). The relationship between attribution styles and academic achievement can be explained using the concept of self-regulation. According to social-cognitive theory, self-regulation is dependent on the situation and it is not stable. Based on this assumption, Zimmerman (2000) describes self-regulation as cyclical with three phases containing sub processes: forethought (task analyses and self-motivation beliefs), performance (self-control and self-observation), and self-reflection (self-judgement (e.g. self-evaluation and causal attribution) and self-reaction (e.g. sclf-satisfac-tion)). According to their performance in each of these domains, learners have been described as skilled or unskilled learners (Stroud & Reynolds, 2009). Attributions are apart ofthc final stage. Self-reflection begins with self-judgement (individual comparisons of information gained through self-monitoring to extrinsic standards or goals). An individual is motivated to have fast and accurate feedback on his or hers performance as compared to others. Self-judgement leads to attribution interpretations where the learner interprets the reasons for success and failure. Accribution interpretations can lead to positive self-reactions. 'Ihe individual might interpret their failure as the result of too little effort and then increase his or hers efforts. On the other hand, if they interpret their failure as a lack of ability the reaction is likely to be decreased in learning behaviour. Attribution interpretations reveal the possible reasons for learning mistakes and help the learner to find the most appropriate learning strategies. Additionally, they also promote adaptation and self-regulation, which eventually leads to a more positive self-image and enhance intrinsic interest in the task (Nokclaincn et al., 2007). Ellstrtim (2001, in Nokclaincn et al., 2007) goes even beyond that stating that attributions for success and failure affect potential competence. Attribution style has been shown in some studies to alter according to the context (Sarafino, 2006, in Graham and Williams, 2009). 'Ihcrc-fore the focus of this paper is mainly on the educational setting and 011 mathematical achievement, 'ihe paper concentrates specifically on PISA 2012 results and the predictive value of attribution styles on PISA 2012 mathematics achievement. PISA measures attribution styles in die context of the students' drive and motivation in the form of separace questions in the students' background questionnaire. PISA measures drive and motivation using four concepts: perseverance (constructed index based on the students' responses about their willingness to work on problems that are difficult, even when they encounter problems), openness to problem solving (constructed index based 011 the students' responses about their willingness to engage with problems), locus of control/attribution style (constructed index based on the students' responses about whether A KO/.IXA AND A. \ I kl / ■ I III' PKI'DIC'I IVI' POWI'ISOI- A'l I IIIHU I'lON SI V IS ... they attribute failure in mathematics test to themselves or to others; and the students responses about whether they strongly agree that success in mathematics and school depends on whether they put in enough cifort) and motivation to learn mathematics - intrinsic and instrumental (constructed indices based on the students' responses about whether they enjoy mathematics and work hard in mathematics because they enjoy the subject, and whether they believe mathematics is important for their future studies and careers) (OECD, 1013b). In line with the attribution theory, PISA measures attributions on all three dimensions (locus, control, stability). Exposing individuals to academic success or failure and then asking them to report about their feelings and thoughts can measure attribution styles. 'Ihe other possibility is to design a set of items where individuals imagine success or failure and then self-report what their most likely thoughts would be as is the case in the PISA study, ihe present study aims to: (1) Identify the attribution for success question set structure 011 an international level: All constructs that measure drive and motivation in PISA are developed in a form of indices 011 an international level except the question set measuring attribution for success (the students' responses about whether they strongly agree that success in mathematics and school depends 011 whether they put in enough effort) therefore the first aim of this study is to analyse the structure of this question set at the international level in order to construct an index that could be used as predictors in second aim of the study. (2) Analyse predictive power of the attribution for success in mathematics for mathematics achievement 011 an international and national (Slovenia) level. The second aim of the study therefore is to use the newly developed index (indices) as a predicting variable in a regression model for mathematics achievement on an international level. Our basic assumption in line with the theoretical framework is that an internal locus of control predicts higher achievement 011 an international and national level. To test the generalizabilitv of our findings we will use the same regression model 011 an international level (PISA 2012 international data base) 011 national level (Slovene PISA 2012 data base) and additionally in selected EU member states with different average mathematics achievement score. The choice was made based on average students' mathematics achievement score (as presented in international reports), where Netherlands and Estonia are the EU member states with the highest achievement score and Bulgaria and Romania are the EU member states with the lowest Soi. s ko poi.ii-. i i. iai k \ x v. s i mvii.ka 5-6 achievement score'. I11 addition to the international data results and the results for Slovenia and four other countries' results will be analysed in detail. The goal is to test whether the same predictions can be made in high and in low achieving countries. Since attribu-don styles are under the strong influence of culture (e.g. western cultures valuing ability more and eastern countries valuing effort more) (Nokelainen et al., 2007) we have chosen LU member states for the comparisons. Method Participants In the analyses, a PISA international sample is used. PISA samples students aged between 15 and 16 years, disregarding the grade levels or type of institution in which they are enrolled and regardless of whether they are in full-time or part-time education. Therefore, the average age of students included in the survey is 15 years and 9 months (OLCD, 2014). Most countries included in PISA used a two-stage stratified sampling design, which means that the sampling was conducted in two stages. Tire first stage consisted of sampling individual schools, where 15-vcar-old students might be enrolled. A minimum of 150 schools per country were sampled. Ihc second stage of the sampling process consisted of sampling 15 year-old students at the selected schools. Approximately 3515-year-old students were sampled per school with equal probability, however each country then chooses its own modified sampling design (OLCD, 2014). With these sampling procedures the representativeness of the selected test population for each educational system was ensured. PISA 2012 focused on mathematical literacy. There were approximately 510 000 students from 65 countries included in the survey. l;or the purposes of this article elata from the Form B Questionnaire and Slovene, Bulgarian, Romanian, Estonian and Dutch data sets are used (Ar=309 104). Fach student answered a cognitive test and a background questionnaire. PISA 2012 introduced a new rotation design for the stu-tlent questionnaire, which is similar to the cognitive items design. Items are combined in packages, which are distributed over a number of different booklets. Fach student is assigned one of these booklets and therefore receives a limited number of items, whereas all booklets together cover a larger pool of items from different scopes (OLCD, 2013c). i I.von though Cyprus was rhe hi/ member stare with the second lowesr marhcmaries achievemc nt score, irwas nor i nclnded in rhe malvsis since rherewere no avail;! We data for 1 his coimir; 1111he1r11en1aiion.il dai.ibase. A KO/.IXA AND A. M 1 kt / ■ i iir piei;Dici 1 v 'OWl:ROI: Ai 1 :> 5' .'5 48 {'■:>. 4« ,<6 5- J v.'1 44:1 : çs: 43.'' Î4:V Noie: All ibt daui prcseiued in dm lable art calculated using only ilie daia lorsiu-dcms who answered qucsiion S I4; iaiiribuiion lor success. In Siudcni C^ucsiion-11 aire. In Slovenia 3 706 students were included (49% female and 51% male). Ihe average mathematics achievement score for Slovenia is 501, whereas for Netherlands, which is the 1U member state with the highest score, the average students' achievement score is 523 on die odier hand for Bulgaria, the J .U member state with the lowest score, the average students' achievement score is 439. For die data analysis, two programmes were used as follows: SPSS for structures analysis and IDB Analyser for regression analysis. Instruments Background Questionne lires In PISA 2012, students completed a îo-minute student questionnaire, which included questions on their background, attitudes toward mathematics and on their learning strategies (OP-CD, 2013c). 'ihese questions are of vital importance for the analyses of the results. I11 detail, the questionnaire includes: • student and their family background (including their economic, social and cultural capital), • aspects of the students' lives (their attitudes towards learning, their habits and life inside school, their family environment), • aspects of learning and instruction ¡11 mathematics, including the students' interest, motivation and engagement (OPCD, 2013c). Cognitive lests PISA 2012 was composed of a paper-based assessment of the students' mathematics, science and reading literacy and a computer-based assessment of problem solving (NCFS, 2014a). All PISA 2012 cognitive items were organized in clusters. Hie main competency tested in PISA 2012 was mathematical literacy. There were two possibilities to assess the mathematical literacy for countries. The first possibility was a set of 13 booklets, which included items distributed across a range of difficulty. Out of 7 mathematical clusters, 4 were included in these booklets according co a rotated test design, Hie booklets also included 1 reading clusters and 3 science clusters. Moreover, in each booklet chcrc was at least one mathematical cluster. Regardless of a specific countries' choice, the performance of students in all participating countries is represented 011 a common mathematical literacy scale (OJ:: Factor loadings ol auriUnion lor success in maihem.iiiosquestion set It 1 pur nenouglirlForr, I can suceccdin mathematics 1 actor 1 nrcrnal locus of control •7v l.xrerna 1 locus ot control ■■r'7-S Whether or not 1 do well 111 mathematics is completely up to me •«W -.05$ 1 amily demands or o tiler problems prevent me 1 rompul lints a l°l ol I imc inio mv mallieinal ics work 54f. ll I liaddiik reni Icachcrs, 1 would Irv Lardet iti mathematics V- + Ifl wantedro. 1 coulddoivell in mathematics • OT1 1 do badly in mathematics whether or not 1 study for my exams ••¿iî ■519 Regression Analyses For the analysis of the relationship between attribution for success in mathematics and the students' mathematics achievement, regression analysis was used. 'Ihe regression analyses are at the first stage of the analyses conducted on an international level, and further on also 011 a national (Slovene) level followed by international comparisons. We used two stages of multiple regression analyses. In the first stage, only attributions for success in mathematics indices were entered in the model. Furthermore, in the second stage, attributions for failure in mathematics index (perceived self-responsibility for failing mathematics) were added to the model on national and international level. A multicollincarity assumption of predictors in the model was tested with correlation analyses. All indices (internaI locus of control, external locus of control and perceived self-responsibility for failing mathematics) statistically significantly correlate with each other, either weakly or moderately (0,01< r <0.34). Additionally VIFs were significantly below 10 (I.o8) Ink null locusol com rol -16.56* y: Lixtemalloc-iisoi control 51.S4* (0.58) ■Sloven in 0.16 1.0.00.: 0.11 :'o..o'i consranr i~'>79" (t/v InieruallocusoiconLrol -15.17* (i.ir ■0.14* 1 Ixncrnallocus of control 1S.19" : 1 — Nil In rlantls Gonstnnr 5WS" ;vt~.' Inn null locus ol com 10I -14.*?y* (2.21.: 1 '/xrc rnallocus of control yittf* ¡4-4r' -•. If." Psion ia constant 516.10* (1.06; Inn null locus ol com 10I ■ 1S.:j^* :,l.lS: --.iX" V..■.:'! lixiemallocusoi control ij.S'i* ¡1.57': -.51* io.ol" -.15 :o.oi; Romania constant 4H-}S" i ytt) 111 k null locus ol com rol -in,46* : 1 vr -■..11" :.-'-.v1 lixieruallocnsoicontrol iy.oo* : vi>: Bulgaria ..is- ::o.ov -.-9 '.-.Ol cousianL 444.41* : liiTr rnal locus of conrrol -11.71" (1.16! lixicruallocnsoi control 4-96* (1.53; 0.56- .0.01:1 0.14 Noll s: 1 he d.11.1 are weighted with Kin.il Student Weight. R * is adjusted R'. All i he-data [presented 111 this table are calculatcd using only rhe data for students who answered question S'l 43 :.auribuiion lor success'' in Student Quest ioniiaire. Sunisiical-ly signilicant .p > o.os'. cocllirienisare marked with *. tcrnational results of the data analysis show that if internal locus of control increases by one unit, the students' mathematics score increases by 16.6 score points (if external locus of control is constant). If external locus of control increases by one unit, then the students' mathematics score falls for 32.8 score points. Every unit increase in the external locus of control is therefore associated with 32.8 score points fall in the students' mathematics achievement (if the effect of internal locus of control is held constant). On an international level, the model accounts for n % of variance in the students' mathematics achievement score. Likewise, the results of the data analysis for Slovenia show that if internal locus of control increases by one unit, the students' mathematics score increases by 15.3 score points. Therefore, every unit increase in the internal locus of control is associated with 15,3 score points increase in the students' mathematics achievement (if external locus of control is constant). If external locus of control increases by one unit, then the students' mathematics score falls for 2.8.3 score points. Every unit increase in che external locus of control is therefore associated with 28,3 score points fall in the students' mathematics achievement (if internal locus of control is constant). In Slovenia, the model accounts for 8 % of variance of the students' mathematics achievement score. Further comparisons of the countries with the highest and lowest mathematics achievement scores in European Union showed that the regression model, which accounts for the highest percentage of variance (15 %), is the regression model for Estonia. The results of the data analysis for Estonia show that every unit increase in the internal locus of control is associated with 18 score points increase in the students' mathematics achievement (if external locus of control is constant). Every unit increase in the external locus of control is therefore associated with 36 score points fall in the students' mathematics achievement (if internal locus of control is constant). Moreover, the regression model for Romania accounts for the lowest percentage of variance (9%) in analysis. 'Ihe results for Romania show that every unit increase in die internal locus of control is associated with 10 score points increase in the students' mathematics achievement (if the external locus of control is constant). Moreover, every unit increase in the external locus of control is therefore associated with 29 score points fall in the students' mathematics achievement (if the internal locus of control is constant). Table 5 shows that the inclusion of an additional index of attribution for failing mathematics does not add to percentage of explained variance to the original regression model which includes only an attribution for success in mathematics indices. The inclusion of an additional index of a. ko/i va and a. m 1,1-kl 7 • i iii' pio'dici ivi' powi'koi- ai i kibu i ion s'l YI.i'S 1 alilc y. Regression model vvii h ai 11 ihm ion lor (perceived sel i-responsi-bilirv tor failing madiemaries • FAIL\fAT; index included IV ,s.c.; Tmeriiaiional results constant :-• r 1 nrc mal locusofconrrol -16/j i* io.yj -0.154 ip.00; Fxlcrn.il loeuvol control 39 1 AUMA 1 -Mi* -0.04" [rj.-r, 1 ■"'.IT Slovenia consrant Ink rnal locu.s ol ¿ oui rol tixiernal locus ofcontrol -Ii.(I.n: 16.56" ij.iç -r;.T4* aii" ;p.oi. FAIT MAT ■3.75 :.-ai M. .1 Netherlands consi am <¡19.82' fj.77; Internal locut of conirol Fxlcrn.il locusol conirol -is.if (1.19: in.94* i+.iS; -0.17' :c.cil 16 ' p.o,: l'AILMAl Estonia l.lS (0.1JÇ -'-i -.H :..oi' constant sió.51* ÍVH; Internal locut of control -17.94* il-Cl' •c.iS' >02: 1 ;xncr n a 1 loons of control ;-.44' (1.Š7) S.jl* FAT1.MAT -'-*-* c,;/11 ■ ■.i) (■ -'-i. Romania constant 4«. IS* :..i.86) Ink mal locus ol com rol Fxiernal locus of control -■ - »-5=." Ú-Í711 27.97* •an* ,.v:.'-''2 C.17" (o.c;': FAT1.MAT ■"•"J v'-'1' ■'■".> 1 ■ .1 h (s. c,! i (s. e.) V s.c. Bulgaria constant 444.48' Inlcrnjl kvusol onilrol -n.71* (1.15) •0.12* .p.'jl '. External locus of control 0.57' {c.zz) FAIT .MAT :;.T7 ij.yy; s/j'j J.'-'O: Notes: Hie data arc weighted with Tinal Student Weight. FAILMAT is an abbreviation for the index "perceived seli-respotisibility for failing mathematics". R" is adjusted R . All the data presented in this table are calculated using only the data lor students who answered question .ST45 ¡attribution tor success in mathematics) in Student Questionnaire, Statistically sigtiilicatu ;p > 0.05) coeNicients are marked vv il h attribution for failing mathematics accounts for an additional r % only for Bulgaria. For the international data and the rest of the countries (Slovenia, Netherlands, Estonia and Romania), the percentage of variance explained stays the same after the inclusion of additional predictor. Therefore, it can be concluded that the inclusion of the new predictor has not explained a large amount of the variation in students' mathematics achievement scores, fhc attribution for failure (perceived self-responsibility for failing mathematics) is a weaker predictor for the students' mathematics achievement score than the predictors of the attribution for success." Moreover, the predictor attribution for failure (perceived self-responsibility for failing mathematics) is statistically significant in predicting students' mathematics achievement scores only on the international level. Discussion Internal locus of control as rncasu red in PISA study is a significant predictor of higher mathematics achievement 011 international level and based 011 the samples included also regardless of average levels of mathematics achievement (Slovenia, Netherlands, Estonia and Romania). Likewise external locus of control significantly predicts lower mathematics achievement on an international level and in selected countries. The results showed predictive stability - in other words the predictors were significant in all analysed countries. In Slovenia, the students' attribution style explains 8 % of the total mathematics achievement score indicating the relevance of the analysed field. Qt'AI.I II IS IN l-ISA i_ui M A I I 11 VI A I ICS A C.I 11IV I'M I'VI 1 able i: Differences in sample sizes and populaLion esiimai.es based on hill PISA ion samples and form B Questionnaire subsamplcs" Slovenia Germanv Canada I, n ¡red Snares Nnmberofsrudcnrs in full PI SA i". 11 sample 5 1 -'544 49-S Numberoi students in 1 ormB ^sample 1896 1654 ~17> 1665 .Meanachievement based oil lull PIS A sample sot Tv 514 :-•>) 51S ix.S) 4»< 'M- M< j n .u' icvonicnl 1 Vised - hi ForniB sii|,s.iiiip|( ;*-<•■ jiS ii i' 5"> > 4.: 4s4 :;4-:'; Due to this rotation, not all students provided responses for all of the at-titudinal factors to be used in this study. Data on the full set of items of interest was collected through Form B of PISA ion student questionnaire only. In order to analyse the factors derived from these items within a single model, we therefore used data from subsaniples of 15-years-old students that were given Form B during the PISA assessment from each country. Consequently, population estimates based on these subsamplcs may slightly diifer from the estimates obtained from full PISA samples that are available in PISA international reports. The two sets of estimates are presented in Table 1 for comparison. The differences between the full sample and the Form B-subsample estimates for Germany and Slovenia are somewhat larger due to the fact that in these two countries, small subgroups of students were given a shorter version of the cognitive assessment as well as the questionnaire, named UH (une hcurc) instruments. Such versions are available in PISA for students with special educational needs that otherwise could not participate in the PISA assessment. This version of instruments was given to of students in Germany and 1% of students in Slovenia. Exclusion of diese students from the calculations of the mean achievement in a country generally results in an increased estimate. However, UH instruments did not collect data 011 any of the factors selected for the present study, except for the index of socio-economic and cultural status, so students that were given these instruments would not be included in the analysis in any case. As none of the four differences between full-sample mean estimates and the Form B subsample estimates In die whole article, standard circus are ¡yv cii in parentheses. Tile standard errors iiidieaU: the accuracy of the estimate*. for example, if one imagine** rharrhr PISA study had Keen repeated a nil 111 her of rime* with the same sample size* for each country, then in ahonrjji» of cases, the estimates ofrhe means would have fa Ik n within rhr double ran^e indicated by die standard errors Soi. s ko poi.ii-. i i. r\i k x x v. s i mvii.ka 5-6 arc significant', the data from Form B subsamples were deemed of sufficient quality to he used for the present investigation. From here 011, it is only these data that are included in the analysis. Mathematics-related Constructs 1 able 1: Siruciure ol blocks an J indices o I students' maihcmaiics-ixlai-ed an inidcs and opinions Blocki Socio-economic background Index oi socio-economic and cultural stains ;F.SOv Constructed index based oil students responses ik'.iii 1 Ik i p,i 1 ( m v education and occupai ion ,1 nd home possessions Block 2 Mathematicssell-bell* ¡.sand participation in 'i. 11 Ik malicsrt laicdacliviiies luck xol mathematics sell-c ificacv (. .011-11 n. ( d index based on sunk nis : .p>nscs ,1 bout drei r perceive d abi 1 fry 1» sohr a la ns^o of pure and applied niarhcmarics problems 1 ndex of m ath e m ati es sclfco n cc pr 1 ndcx oi ma llienia tics anxiet y Constructed index based on students responses about Hieir perceived competence in mathematics Constructed index based oil students' responses abouL ieeliugs oi stress and helplessness when dealing with mathematics 1 nek xol subjective norm.-, in m,il licmalics (. .011 st nick d index based on stuck nis res|K>ns-esiiboui wliel icil Iua intend 10 use mal Ik ma lies in 1 licit'I in mi' and wliel her si nclenis parents ami pee rsen joy nndva lue mar hematics Block a Studr nrs drive and motivation 1 ndcx oí persevera ncc Constructed index based on students responses ,1 bout rliei rwi II i none.«towork on problemsrliar are difficult, even when they encounter problems Index oi openness lo problem solv iiü> Constructed index based oil students responses about Llieir willingness to engage with problems Index olperceived sell responsibilityiorlail-n^in nuilliemaiics Constructed index based oil students responses about wlic-lhcr they ,i 11 ribule l.iilure in malliemar ics tests 10 themselves or to til hers 1 nclexol ini rin-sic moiivai ion tok'arn 111,11 li-emaiics (. .011 stnick d index based on stuck nis rc spouses .1 bout wliel her they c njov 111.1111e111.il icsand work hard in mathematics because they enjoy the subject 1 ndexof instrumental niorivarion ro learn.sci- ence Constructed index ba.sedon students responses about whether they believe mathematics is impor-tantior their inture studies and careers In PISA ion, background data was collected with the aim to portray important aspects of the affective domain, such as valuing mathematics and being confident in doing mathematics. From the data that were collected via student questionnaires, interval-scaled statistical indices were derived to capture the major constructs related to mathematics achievement. j TesLe(l(all(nvlri£ilicproc<*lurtt>in Ol X I ) ;itsc.«i. M. S'l KAL'S • (IN)I;QL.'AU HI'S IN I-ISA i.ni MAI lll-.M AI ICS ACI11IV I'M l'\ I ... There is a set of indices given in the PISA 2011 database from which it is possible to select the indices that basically capture the major aspects in Aj/.en's (iyyi) theory of planned behaviour. The indices selected for the present study are organized in three blocks; the first block comprises of a single index of socio-economic and cultural status, the second block comprises of indices of students' mathematics-related self-beliefs and the third block of indices of their drive and motivation in mathematics. Descriptions of these indices are presented in Table 2. Concrete items in the PISA 2012 student questionnaires that were used to collect data for the selected indices and data for these items are detailed in OECD (ionb). A statistical index in the PISA database is constructed in a way that for all students in the OJ .CD cou ntries the mean is o and the standard deviation 1 (in computing the mean and standard deviation an equal weight is given to each of the participating countries) (OECD, 2013b and OECD forthcoming). Negative values of the index in the international database therefore do not imply that students responded negatively to the underlying questions, but rather that they responded less positively (or more negatively) than the average response across OECD countries. Eikewise, positive values imply more positive (or less negative) responses than the average response in OECD countries. Socio-economic Gradient Willms (200^) describes that socio-economic gradients comprise of three components, mean level, mean slope and the strength of the relationship between the outcome variable and socio-economic background. The level of the gradient is defined as the expected score on the outcome measure for a student with average socio-economic status. Hie level of a gradient for a country is an indication of its overall performance, after taking into account the students' socio-economic status, f h cslope of the gradient is an indication of the extent of inequality attributable to socio-economic status. Steeper gradients indicate a greater impact of socio-economic status on student performance (greater inequality) while gradual gradients indicate lower impact of socio-economic status (less inequality), ihe strength of the gradient refers to how much individual scores vary above and below the gradient line. If the relationship is strong, then a considerable amount of the variation in the outcome measure is associated with socio-economic status, whereas a weak relationship indicates that relatively little of the variation is associated with socio-economic status. The most common measure of the strength of the relationship is a statistic called R-squarcd, which is the proportion of variance in the outcome measure explained by the predictor variable. Statistical Analyses The main analytical approach for the investigation in this article is linear regression analysis, conducted in a sequence of steps. First we estimate socio-economic gradient using a simple one-predictor model for each of the four countries. Then the model is extended with factors capturing various aspects of students' mathematics-related attitudes, 'ihc appropriate structure of these factors for the final moelcl is derived from preliminary exploratory analyses. For all foilr cou ntrics the same final model is used. Due to the clustering structure of the PISA data - students being sampled within previously sampled schools - the question whether hierarchical modelling needs to be used should be addressed. Since only student-level variables are investigated in our study separately for each of the four countries, it remains to be considered whether the variance of these variables shared between the schools is of interest. Hie impact of clustering on sampling variance is controlled for by Bootstrap procedures of computation. As mentioned, the majority of 15-year-old students in Slovenia attend the first year of their upper secondary education segregated to different educational programs and that the students' selection of these programs tends to parallel their socio-economic background. "Therefore it seems self-evident that the proportion of variance in mathematics achievement as well as other variables between schools is relatively large. The linear regression coefficient of socio-economic background 011 the student achievement provides an estimate of the overall difference in performance due to socio-economic background while multilevel regression moelcl estimates the difference in performance after accounting for the differential attcnel-ancc to schools. Ihc multilevel regression coefficients on socio-economic background may therefore substantially differ from the linear regression coefficients, especially in highly tracked systems. Having four different education systems in our study, the primary interest are the overall differences in the populations of students while differences between schools are left aside. For this reason, the linear regression is used. IBM SPSS 12.0 software is used for the analyses, with the addition of the syntax macros prepared through the IDB Analyzer software (IFA, 2014), which enables calculations of population estimates and standard errors with the use of suitable sample weights and all five plausible values of achievement in the PISA database. Throughout the article, significance of diff erences in mean estimates or in estimates of regression coefficients between countries is tested using the foundations in OF.CD (2009). Testing is carried out at 0.05 level of statistical significance between results for Slovenia and each of other countries. m. S'l RAl'S • (I \) i' QL A 1.1 i'l IS in I-1SA l.'Ti M A I I 11 -A I A i ICS AC i iiivi'mix i ... A final note of caution is in order. When interpreting the results of investigations in this article, it should be taken into consideration that the indices used in the analyses have been derived from students' responses to questions in the background questionnaire and not from, for example, independent observations or other types of objective measurements, litis means chac students' answers depended on the way students understood and responded to questions. Results International Comparisons of Slovene Students' Mathematics Achievement and its Socio-economic Gradient from PISA 2.012. data, we can derive basic comparisons of mathematics achievement and its socio-economic gradient between Slovenia, Canada, Germany and the United States. While these indicators are available in the PISA initial reports (e.g. OUCD, 2.011a), it is important to repeat that this study uses subsamplcs of the original PISA samples within the select-eel countries and, consequently, some of the indicators in Table 1 slightly differ from the initial reports. Table 3: Data on socio-economic gradient in mathematical literacy for Slovenia, Germany,Canada and die United Stales in PISA 2012 Slovenia Germany Canada United S tates Mean sc«."io-economic and cultural status -41 ''J.ol! 0.1Í ;-.-.-4:i Mean score in uiailieuiaiical liieracv 5-5 (1.6: 5'? .1 I 4S4 i-'-- Level of socio-economic $>radienU 5-i >-4; 515 1 4S9 (5.5) Slope ol socio-economic gradien Li 45 41 iU 54 ii-0 5Í- 'M- Sire n'¿illol so;-io-( .'>notni. ^radiniu 16.1 j.-: : 1.1: 14.5 ;i i). Notes: 1 1 evel of socioeconomic gradient is die mean score in mathematical literacy, adjusted lor the mean socio-economic and cultural status .1 :.SCS\ Adjusting for socio-economic: and cultural status takes into account only mean achievement of groups ol students vviili socio-economic and cultural status equal to OHCL) average in each country. 1 Slope ot socio-economic gradient is the score-point change 111 achievement associated with one-tin it increase in socio-economicatid cultural status, j .Strength ol socio-economic gradient is the strength of the relationship between mathematical literacy and socio-economic: and cultural status TSCS) as the percentage ol variance in mathematics performance explained by the socio-economic and cultural slants. First, Table 1 shows differences between the four countries in average socio-economic and cultural status. Average socio-economic and cultural stacus of Slovene 15-ycars-old students (value 0.07) is slightly above the OF,CD average (which is o+), however, ic is the lowest value of the four countries. Socio-economic and cultural status of 15-ycars-old students in (Canada is the highest (value 0.41) and the values for Germany and the United States arc in-between (values 0.21 and 0.16, respectively). The values of mean scores in mathematical literacy show a different pattern. Ihe scores for Germany and Canada are similar, the score for the United States is the lowest and the score for Slovenia is in-between, ihis shows that the socio-economic and cultural status itself does not determine the level of mathematics achievement in a particular country. Furthermore, while one could try to argue that the level of achievement in Slovenia is understandably lower than in Canada and Germany due to lower socio-economic and cultural status of Slovenian students, this is not supported by the level of socio-economic gradient in 'Fable It can be observed that differences between Slovenia and the other countries still exist even when mathematics achievement is adjusted for students' socio-economic and cultural status. Ihese comparisons show that the levels of socio-economic gradients indeed vary between the four countries. Another element of the socio-economic gradient, the slope, also varies between the countries. One can observe that the slopes of socio-economic gradients in Slovenia and Germany are the two highest (45 and 42 points, respectively) and in (Canada and the United States the two lowest (34 and points, respectively). In other words, in Slovenia and Germany a one-unit increase in socio-economic and cultural status is associated with a somewhat higher increase in mathematics achievement than in Canada and the United States. It is interesting that, even though both, average socio-economic status as well as average mathematics achievement of Canadian students arc different from these characteristics of the United States' students, the slopes of the socio-economic gradients arc similar between the two countries. Ihe same can be observed for Slovenia and Germany. 'Ihe percentage of variance in mathematics achievement explained by socio-economic and cultural status is the lowest in (Canada indicating the weakest gradient among the four countries. In Slovenia, the socio-economic gradient seems to be the strongest. However, given the relatively small percentages of variance in mathematics achievement explained by socio-economic and cultural status 111 all four countries, it seems reasonable to expect that there are other factors accounting for the variance in mathematics achievement of students in the selected countries. Some of these factors arc investigated in the next section. 4 I or a 11 indices. OIX U) average is ::•. See section on da i a and methods M.S'l HAL'S • (l\)l'Qi:AI.I I'lI S IN I- ISA 1J.TI M A I I ll-AI A I ICS AC.11II VI'Ml;\ I ... Socio-economic Gradient Together with Self-related Beliefs in Mathematics In this section, we present the structure of (some of) the underly ing factors associated with mathematics achievement in Slovenia in comparison with Canada, Germany and the United States. The model for socio-economic gradient is extended with mathematics-related attitudinal factors. First, descriptive data on these factors are presented in Fable 4. As explained, all factors are derived on an interval scale with a mean value o for OECD cou ntrics and standard deviation of 1. Values presented in Fable 4 are therefore readily comparable. 1 able 4: Mean values of factors5 M arlicma ries-related .self So lifts Slovenia Germany Canada Unired Stares 1 ndex of mathematics self-eliicacy O.l- io.oçi Index of mathematics self-concept ■O.Oi 0,10 ' : ' : 1 _ .< ■! -'.jl ¡o.'-'V 1 ndex oi subjective norms illmallie-malics ':-'.l2 ' : : S l/J.Oi'l Index ol niatfiotnai iesa nxiciv ■0.04 1;1 '.i'i Student1; drive ,ind inolu.n ion 1 ndex ol perscvei.i nc( Index ol openness to problem .solving 1 ndex oi sell re sp,.14 ; - ' .z -'.,11 it..' .11 -0.40 lo.ov Index of intrinsic motivation to learn mathematics Index of instrumental motivation to learn science - MS i'-.r.'vi , 1 .".oi; --14 ;■-■">■ 014 ¡0.-1) .11 0.-8 10.041 'ihc average values of indices of students' attitudes and opinions about mathematics vary between the four countries. 'Ihc levels of sclf-ef-ficacy in mathematics show an interesting distinction between the countries. In Slovenia and Germany, students express high levels of conviction about their capability to cope with certain mathematics tasks (values of 0.33 in both countries), while students in Canada and the United States seem to be less convinced in their capabilities (values 0.13 and 0.17, respectively). ibis finding seems to be in contrast with the result that Canadian 5 Testiiii; of statistical significance- of differences in litis article- is carried 0111 between results lor Slovenia and each oi oilier countries. For brevity, inicrpre unions oi comparisons between other countries are made more gentraliv without tcvliug for significance. Tins resting os 11 ho carried onr using standard errors provided with each estimate. Duo to estimates being based 011 Form B suhsamples only, sta ndard errors a re somewhat larger and loss sign ifioant d ilFeronoes oan be established. I nterpretarions are made as ind n ations oi resirli slot which further. more del ailed invest 1 gat.ioi is seem warranted. students achieve the highest scores in mathematics among the four countries. However, perhaps the more important question is how does sclf-ef-ficacy relate to achievement within individual countries. This will be presented later in this section. A somewhat broader sense of the overall perception of students' personal attributes in mathematics, the mathematics-related self-concept, also varies between the countries, although rankings changed. Slovene students report the lowest self-concept, around the OECD average (value -0.02). German students report slightly higher values than the OECD average (value o.io), with Canadian students reporting also higher (value 0.20), and the highest reports coming from students in the United States (value 0.31). Given that achievement in the United States is the lowest among the four countries, it is difficult to imagine this factor to be positively related to student achievement. However, it needs to be kept in mind that these are average values per country and that there is variation within individual countries in achievement as well as in the background factors. Again, this is examined later in this article. A few additional indices portray clustering of values for Slovenia and Germany together on one side and of values for Canada and the United States on the other. The index of subjective norms captures the belief; of student that specific individuals or groups think they should perform well in mathematics and students' motivation to comply with these groups. German and Slovene students expressed lower than average levels of such beliefs (values -0.12 and -0.23, respectively), and students from Canada and the United States well above average beliefs (values o.?6 and 0.28, respectively). 'Ihc valuing of mathematics in the students' environment, as measured through the index of subjective norms, as well as intrinsic and instrumental motivation to learn mathematics are therefore relatively low in Slovenia and Germany and relatively high in Canada and the United States. fhe index of self-responsibility for failing in mathematics reflects students' perceptions of their personal responsibility for failure in mathematics. Students with high values on this index tend to attribute the responsibility for failure to solve mathematics problems to themselves while students with low values 011 this index are more likely to see other individuals or factors as responsible. While students in Slovenia and Germany report relatively high levels of self-responsibility for failing in mathematics, students' reports show lower levels of this responsibility in Canada and the United States. Similarly, students in Slovenia and Germany report around average levels of perseverance, but students in Canada and the United States report higher perseverance. It is only for the openness for M. S'l KAL'S • (lx)l!Qi:AI.I I'll'S IN PISA i.ni MAI lll-.M A'l ICS ACI11IV I'M l'\ I ... problem solving that students in all four counties give closer reports, all of diem above average. Die index that most stands out from this pattern is mathematics anxiety. While students in Canada and the United States report around average levels of anxiety (values 0.06 anei -0.04, respectively), German students report relatively low anxiety (value -0.20) but Slovene students report the highest levels of mathematics anxiety among the four countries (value 0,13). In summary, among the four countries, Slovene students express the lowest self-concept in mathematics, the lowest intrinsic as well as instrumental motivation to learn mathematics and the lowest level of beliefs that their parents and peers think they should perform well in mathematics. At the same time they express the highest level of self-responsibility for failing in mathematics and the highest mathematics anxiety, ihis in itself is an important message about Slovene mathematics education. Preliminary Regression Analysis In the preliminary analysis, it was first explored which factors, individually or as blocks, explain most variance in mathematics achievement." Ihe results of this analysis showed that for all countries, self-efficacy explains more variance in mathematics achievement than socio-economic and cultural status7. When blocks of indices were entered into the model separately, it was found that a larger amount of variance is explained by the block of mathematics-related self-beliefs (between 28 antl 36 percent for the four countries) than by die block of indices on drive anei motivation (between 7 and 19 percent for the four countries). Furthermore, when both blocks were entered, the amount of variance explained was nearly the same as the amount of variance, explained only by the block of indices of mathematics-related self-beliefs'. From this, it was decided to use only the block of mathematics-related self-beliefs in the regression model. In addition, issues of multicollincarity of the factors in the block of self-beliefs were tested. It was found that there arc relatively large (negative) correlations between self-concept and anxiety (from -0.76 in the United States to -0,61 in Sloven ia). Due to self-concept having larger (positive) correlations with self-efficacy dian anxiety (from 0.39 in Slovenia to 0.5s in Canada), it was decided diat anxiety is kept as the predictor 6 I his was explored tiling a stepwise procedure lor linear regression analysis in SPSS. Also oilier preliminary analyses were carried oui in SPSS. 7 Inasingk-prediciormodel, socio economic and cultural status explained between r.. and I- percenr ofvariance and marhematicsselfetlicacy explained benveen >'", and percent ofva ria nee for the foil r countries. S I he largest increase in a nionnrofvarianeeexplained by adding both blocks of i ndices into ilie model was 1.5percent while self-concept is dropped. (Correlation analysis of the remaining factors showed that they correlate weakly or moderately (-0.47 < r < 0.19). Additionally, variance inflation factors (VIF) were significantly below 10 (1.017 < VIF < 1.42). Other research, however, shows that these concepts are different (Ferla et al. 2009) and have differential impacts on achievement across countries (Morony et al. 2012). Results of Regression Analysis A linear model0 was set up in order to investigate differences in the impacts of selected factors on student mathematics achievement between the four countries, ihe results of regression analysis based on this model are presented in Table 5. I'ablt Relationship between mathematical literary, socio-economic and cultural status and mathematics-related .self-beliefs' Slovenia b ~l r.b; t;'n': R" R- constant 49 s (i-v: IT9/» I.SCS ST 11 - 11.1 MAI III 1 ST TIT ivi SLBNORM -s -0.09 lo.oy -Jo -54 AXXMA1 -1? iix) ■o.io ¡0.0 v -6.1? -71 '-'■J4 o.}4 Cjcrmanv b 71 t;n: R* R" constaiit 50- >4: iOT.o F.SC.S -- i-.t K 14..'! MATHF.FF w ;, 1 QC'AI.I i'll'S in l-ISA i_ui M A I I 11 VI A I ICS A C.I 11IV i'm I'VI I 'n ¡red.Stares h n nb: r n r iv* constant 1 si. s 47? 1-7) o.iii i'o.oo tso.5 it.i it; ma mm i 55 (i-8- 0.4- . -•'-> h.7 ii-7 SUBNORM -14 \l-9) -'j.i6 '...-I'. -7 4 -7 4 anxma1 --1 -j.14 i -05; -S.4 -»•4 -4» 0.41 Wich the model, it was possible to explain from 34 to 41 percent of variance in mathematics achievement in the four countries, seemingly the least in Slovenia '. In all countries, the mean achievements adjusted by the four predictors are closer together than the unadjusted means but the ranking of countries is the same. If four average students with re-gartl to socio-economic and the selected attitudinal factors are taken from each of the countries, than the expected mathematics score is the highest for the (Canadian student, 512 points, for the German student 502 points, for the Slovenian student 495 points and for the student from the United States 479 points. Also, by controlling the factors of the students' self-beliefs in the model, the socio-economic gradient becomes more gradual ¡11 all countries. For example, while the socio-economic gradient 111 mathematics achievement in Slovenia is 4s points (see Fable 3), controlling for students' self-beliefs reduces the gradient to 31 points, 'ihis gradient indicates that if two groups of Slovene students with the same self-beliefs but one with a one-unit higher socio-economic and cultural status arc compared than the higher-status group has on average 31 points higher mathematics achievement. Or in other words, even though students may have the same high or low mathematics-related self-beliefs, the ones with higher socio-economic and cultural status are, 011 average, expected to achieve higher in mathematics. The order of the reduced socio-economic gratii-ents in the regression model remains the same as is the order of gradients obtained from the single-predictor model (see 'Fable 3). 'ihe reduced gradients in Sloven ia and G ermany are the two steepest and in (Canada and the United States the two most gradual of the four gradients. However, analysis showed that believing in one's own capability of solving certain mathematics tasks remains a factor of relatively high impact even when other factors arc controlled. In Germany, Canada and the United States students with similar socio-economic and cultural background and similar levels of subjective norms and mathematics anxiety, but with a one-unit difference in the levels of mathematics self-efficacv have on average over 35 points different scores on PISA mathematics test; tt No sign if cn lit d ifierences were esrabl ¡«hod between esti m ired proportion of varialw tor Slovenia ami Canada. soi sko poi ji;. i i i'\ik xxv, Si i»vilka 5—6 students with higher self-efficacy having higher scores. In Slovenia, the impact of self-efficacy seems to be somewhat smaller; 31 score points." Generally, the index on subjective norms in mathematics, that is, the students' beliefs that their parents and peers value mathematics, was conceptualized to act as a positive predictor in the sense that students with higher values on this index achieve at higher levels (OECD, 2.012). Results in the international PISA reports show that for the overall impact of subjective norms on mathematics achievement, this is true in Canada and the United States where a one-unit increase in this index is associated with an 8-point average increase in mathematics achievement in Canada and a 4-point increase in achievement in the United States. In Slovenia, there is no significant association between subjective norms and achievement but in Germany a one-unit increase in die index of subjective norms is associated with a r vpoint decrease in mathematics achievement (Of CD, 2onb). German students reporting more valuing of mathematics in their personal environment have 011 average lower achievement. When this index is included in the model in the present investigation, its impact t)ii mathematics achievement when other factors are controlled, becomes negative in all four countries." If two groups of students in these countries are compared, having similar socio-economic and cultural status and expressing similar self-efficacy, and mathematics anxiety, than the group reporting higher values of subjective norms have on average lower achievement. Mathematics anxiety presents no surprise as a predictor in the model. As shown by the results in international PISA reports, it has, in general, a negative impact 011 the achievement of at least a 27-point decrease per one-unit of this index in the four countries considered here (OECD, 2013b). ihis impact reduces substantially when other factors in the model are controlled. Ihe decrease in achievement per one-unit increase in anxiety when controlling for other factors is between 16 and 21 scale points'4. Discussion and Conclusion The goal of educational policy and reform in most countries is to raise levels of literacy skills, while reducing disparities among citizens from differing subgroups, like social classes and ethnic groups. In this article, we ad- 1; Sisjiificanceol difference on lie esi ahlishedbei ween 1 lie results lor Slovenia ami Germany and the- results lor Slovenia and Canada 1; Tile impacts of subjective norms and mathematics anxiety range irom a s to 14 point decrease in ach ievcmenr per one-unit increase of the hie tot. 1 he significance ot differences between Slovenia and anvorhrr individual country could not he established. 14 Ihe significance ot differences between Sloven ¡a and a nv other ind ivid nal counrrv could 1101 he established. dressed the issue of social gradient in student mathematics achievement and mathematics-related attitudes in Slovenia in comparison to three other countries, Clanada, Germany and the United States, 'ihe availability and quality of PISA data provided an opportunity to gain further understanding on how differences in socio-economic and cultural background of students along with students' mathematics-related self-beliefs affect student achievement in mathematics. 'ihe international PISA reports showed that, of the four countries, Canada and Germany are top-achieving countries, with mean mathematics achievements significantly above the mean in Slovenia, and the United States with mean mathematics achievement significantly below Slovenia. An overview of data on socio-economic and cultural status and mathc-matics-related attitudes showed Slovene students' socio-economic and cultural status are the lowest among the four countries and that most indices of mathematics-related attitudes of Slovene students are similar to Germany and opposite to Canada and the United States. Standing out from this pattern is the level of mathematics anxiety that, by students' reports, is the highest in Slovenia, average in Canada and the United States and below average in Germany, 1 his can be taken as an indication of the area that needs further research in Slovenia. Such research may reveal the background of the results observed in this study. To investigate how these aspects of student background and attitudes relate to student achievement, we set up a linear model, first only investigating socio-economic gradient and later expanding the model with attitudinal factors. Besides being assessed as outcomes of mathematics education, these constructs can also assist in explaining differences in performance on the PISA mathematics assessment. It was presumed that some of the variation in mathematics achievement observed by socio-economic background may overlap with variation in students' self-beliefs about mathematics. ihere are several interesting findings from die analysis in this study. With regard to socio-economic gradient, this study, as many previous studies, found that there are inequalities in performance in all four countries associated with students' family background. Ihe results also show that the extent of these inequalities varies between the countries. Ihe most gradual socio-economic gradient among the four countries is found in Canada, then the United States, and then Germany and Slovenia.'5 In a similar order, the socio-economic gradient is the weakest - that is, socioeconomic and cultural background of students explains the smallest Tj No prarisric-.il sionificance between the results for Cc rmanyand Sloven ¡a could be established. percentage of variance in mathematics achievement - in Canada (n percent), then United States and Germany (14 percent) and is strongest in Slovenia (16 percent)." For Slovenia, this indicates the importance of research in the area of equity in education, such as the present study, to further illuminate the background for the observed results. A preliminary regression analysis showed that in all four countries the indices of mathematics-related self-beliefs of students, as a block, arc meaningfully stronger predictors of mathematics achievement than the block of indices on drive and motivation. 'Ihis was seen by a larger amou nt of variance explained by the block of mathematics-related self beliefs than by the block of indices on drive and motivation. Furthermore, the single factor explaining the largest amount of variance in mathematics achievement is self eificacy. 'ihis is in line with findings from other studies (e.g. Ferla et al., 2009). As founded by the work of Bandura (1997), this indicates that conviction of one's own capability to perform is closely connected to achievement, in a circular manner where stronger conviction leads to better performance and better performance reinforces convictions. In reverse, if students are not convinced in their abilities to accomplish particular academic tasks, they have a higher probability of underperforniing, even though they may have the ability. Ihis is because they may not put in the self-control and motivation needed to perform die tasks. Zimmerman (2000) showed that self-efHcacy is an important predictor of common motivational outcomes, such as students' activity choices, effort, persistence, and emotional reactions, but that it is sensitive to subtle changes in students' performance context. Our analysis showed that if socio-economic and other attitudi-nal factors in the model are controlled, mathematics-related self-efficacy is still a strong and important predictor of mathematics achievement. Hie analysis further showed that among the four countries, this predictor seems to have the least impact in Slovenia. In efforts to avoid the vicious cycle of self-fulfilling prophecy for students with low self efficacy, further investigations of this phenomenon seem warranted. A plausible hypothesis about the reasons behind this phenomenon may be that Slovene students with relatively high efficacy do not perform as well or students with low efficacy perform better. Given that average efficacy is as high in Slovenia as is in Germany, it seems the first is more likely than the latter. Although this finding needs to be cross-checked with additional information, like policy documents or data from adelitional countries, it seems a reasonable hypothesis that one of the reasons for this phenomenon may t£ I he significance of (inferences could he esrabl ¡shed between the results for Slovenia and r» Canada. come from teachers' practices in giving feedback to students (Zupanc and Brcn, 1010). A finding from preliminary analysis is also that, next to self efficacy, die strongest factor explaining most of the remaining variance in mathematics achievement in all four countries is socio-economic and cultural status. When the attitudinal factors in the model are controlled, the socio-economic gradient varies from 21 to 31 scale points, the two largest being in Slovenia and Germany,1 A notable finding is also that while other factors preserved their conceptualized positive or negative nature of impact on mathematics achievement in the model, the index of subjective norms changed to a negative impact in all four countries. Hie initial PISA results published in the international reports already indicated that the nature of this factor's impact on mathematics achievement varies between the countries; in 18 countries it is positive, in 30 negative and in 17 countries its impact is neutral (OF,CD, 2013b). Hie finding that the impact of this factor in our model is negative in all four countries may be interpreted that the students agreeing with items 'most of my friends do well in mathematics', 'most of my friends work hard at mathematics', 'my friends enjoy taking mathematics tests', 'my parents believe it's important for me to study mathematics', 'my parents believe that mathematics is important for my career' and 'my parents like mathematics' actually responded about the pressure they feel from parents and friends that they have to do well in mathematics, fhis interpretation is substantiated on the negative impact of this factor for students with otherwise similar levels of socio-economic and cultural status, self-efficacy, and anxiety. Ihere may, however, be a reversed causality in this association; somewhat weaker students may feel more pressure than their more successful peers that are otherwise similar to them on other factors. In conclusion, our study confirmed the influences of socio-economic background 011 student mathematics achievement reemphasizing the need for constant and more in-depth research in this area. It seems safe to say that research on equity in education is needed also in other achievement areas. Based 011 comparisons with the other countries this is even more important for Slovenia due to a somewhat stronger impact of socio-economic and cultural status and a weaker mediating impact of mathematics-related self-efficacy 011 student achievement. There are, of course, limitations to generalizing the results of this study. As mentioned, all data arc based on students' reports, fhis may in- Gr.ulieniMn Oiti.kIj ind 1 he I Jtriinl Si ;ii re are significantly smaller ilian in Slovenia. flucncc objectivity and comparability of data across countries as well as within. Also, it is important to be careful in assuming causality from the models. It may well be that the outcome variable - mathematics achievement - influences the levels of predictors as well. For example, evidence of high achievement naturally increases one's conviction of their capability to solve mathematics tasks. Or, parents may exert less pressure for mathematics learning when their children arc high achievers. Since the model included four predictors, the observed impacts of these factors may not only or not at all be direct effects but also due to effects of possible other hidden or unmeasured variables not included in the model. In addition, we assumed only linear relationships in the model while there may be curvilinear relationships between the factors as well as with the outcome variable and additional multilevel influences. However, the findings from this model seem reasonable and informative for future methodology of the national and international educational studies as well as for educational policy. further studies may explore the issues addressed in this article in several directions. First, other countries may be taken into account. This could show general ¡/ability of the present results across different cultures and educational settings. Second, the outcomes in specific mathematical sub-domains could be considered. This could show general ¡/.ability of the results across different process and content-specific achievements, which were organized in PISA in process sub-scales formulating, employing and interpreting and content sub-scales change and relationships, space and shape, quantity, and uncertainty and data. Further, with a larger number of countries or additional factors other methods could be used, like multi-level modelling to explore the proportions of between-country variance explained by the selected predictors, or structural equation modelling to explore the possible causal interrelationships of the selected factors. References Ajzen, I. (1991) ihe theory of planned behavior. Organizational Behavior and Human Decision Processes. 50, pp. 17y-2.11. Ban dura, A. (1997) Self-Efficacy: the Exercise of Control. New York: Freeman. Bandura, A. (1977) Social Learning Theory. New Jersey: Prentice Hall. Bourdicu, P., and Passeron, J.C. (1990) Reproduction in Education, Society and Culture. 2nd Ed. London: Sage Publications. Duru-Bcllat, M. (2004) Social Inequality at School and Educational Policies. Paris: UNFSCO International Institute for Educational Planning. M. S'l RAl'S • (l\)l'Qi:AI.I M I'S IN I-1SA 2 jti M A I I 11 -A I A i ICS AC. 11ii V i'M l;\ I ... Feria, J., Valcke, M., and Cai, Y. (2.009) Acadcinic sclf-cificacy, and aca-dcmic self-concept: Reconsidering structural relationships. Learning and Individua] Differences. 19, pp. 499-505. IF A (2014). IDB Analyzer. Available from: http://www.ica.nlAlata.html (Accessed: 17th October 2014). Lamb, S. and Fullarton, S. (2001) Classroom and school factors affecting mathematics achievement: A comparative study of the US and Australia using TIMSS. Australia: Australian Council for Educational Research. Available from: http://rcscarch.accr.cdu.au/timss_mono-graphs/10 (Accessed: 27th October 2014). Ma, I.., and Bradley, K. D. (2004) 'Ihc development of mathematics achievement in secondary school individual differences and school effects. Paper presented at the Mid - Western Educational Research Association annual meeting, Columbus, OH. Available from: http://www.uky.edu/~kdbrad2/lhe%20Devel0p111ent%200f%20 Math%2oAchievement.pdf (Accessed: 27a' October 2014). Morony, S., Kleitman, S„ Ping Lee, Y„ and Stankov, L. (2012) Predicting achievement: confidence vs self-efficacy, anxiety, and self-concept in Confucian and European countries. International Journal of Educational Research. Available from: http://dx.d0i.0rg/10.1016/j. ijer.2012.11.002 (Accessed: 27th October 2014). Observatory of Economic Complexity (2014) Available from: http://atlas. media.mit.cdu/profile/country/svn/ (Accessed: 27th October 2014). O ECU) (forthcoming) PISA 2012 Technical Report. Paris: OECD Publishing- OECD (2009) PISA Data 'Analysis Manual. Spss. Second edition. Paris: OECD Publishing. OECD (2012) PISA 20T2 'Assessment and Analytical Framework. Mathematics, reading, science, problems solving and financial literacy. Paris: OECD Publishing. OECD (2013a) PISA 2012 Results: Excellence through Equity. GivingEvery Student the Chance to Succeed., Volume II. Paris: OECD. OECD (2013b) PISA 2012 Results: Ready to learn: Student's engagement, drive and self beliefs. Volume III. Paris: OECD. OECD (2013c) PISA 2012 Results: What, students know and can do: Student performance in mathematics, reading and science. Volume I. Paris: OECD. National Examination Centre [Dr/avni izpitni center] (2014) National Assessment of Knowledge. Available from: http://www.ric.si/ national_assessmcnt_of^knowledgc/gencral_information/ (Accessed: 27th October 2014). Pang, X., and Rogers, W.T. (ion) Comparative Examination of the Influence of Selected Factors on Achievement in Grade g Academic and Applied Mathematics. Canada: Queen's Printer for Ontario. Available from: http://www.eqao.eom/Research/pdf/F/Comparative_ Examinati0n_Influence_Fact0rs_Achicvemcnt_G9Math_cn.pdf (Accessed: 171,1 October 1014). Rychcn, D. S., and Salganik, I.. H. (eds.) (1003) Key competencies for.a successful life and well-functioning society. Gottingen: Hogrcfc and Huber Publishers. Statistical OiHce of the Republic of Slovenia (2014a) SI-STAT database. Available from: http://pxweb.star.si/pxweb/Dialog/varval. a.sp?nia=095?20iS&ti=&path=../Datahase/Dem_soc/o9_izo-brazevanje/o7_srednjesol_izobraz/o i_09$3 2_zac_sol_le-ta/&lang=2. (Accessed: 27th October 2014). Statistical OiHce of the Republic of Slovenia (2014b) SI-STAT database. Available from: http://pxweh.stat.si/pxweb/Dialog/varval. asp?ma=09527o6S&ti=&path=../Database/Dem_soc/o9_izo-brazevanje/o4_osnovnosol_izobraz/oi_0 9>27_zac_sol_le-ca/&:lang=2. (Accessed: 27th October 2014). Suan, J.S. (2014) Factors affecting underachievement in mathematics. Proceeding of the Global Summit on Education C □ SII 2014,5 March 2014, Kuala Lumpur, Malaysia. Available from: http://www.kuis. ethi.my/icomm/ist/e-proceedings/ICoo2%2oFACFORS%2oAF-FECnNG%2oUNDERACHIEVEMENT%2oIN%2oMATHE-M AT ICS .pdf (Accessed: 27lh October 2014). Straus, M., and Markelj, N. (2011) Reading, mathematical and scientific literacy of students in grade 1 of upper secondary schools in Slovenia in the PISA 2009 study [Bralna, matematična in naravoslovna pismenost dijakinj in dijakov 1. letnikov srednjih šol v Sloveniji v raziskavi PISA 2009I. Šolsko polje. XXII (5-6), pp. 35-68. Available from: http:// www.pei.si /UserPilesUpload/Hle/digitalna_kn-jiznica/SP/20ii/%C>%Aoolsko%2opolje,%2oletm"k%2oXXII,%2o %C5%Aitevilka%20>-6.pdf (Accessed: 27th October 2014). Will ms, J. D. (200^) Ten hypotheses about socioeconomic gradients and community differences in children's developmental outcomes. Ottawa, Ontario, Canada: Applied Research Branch of Human Resources Development Canada. Willms, J. D. (2006) Learning divides: ten policy questions about the performance and equity of schools and schooling systems. Montreal: UNESCO Institute for Statistics. M. S'l KAL'S • (I \) I'QL A1.1 II IS IN PISA l^Ti VI A'l I 11 i VI A'l ICS AC. 111IV I'M l;\ I ... Zimmerman, B.J. (2000) Self-efficacy: an essential motive to learn. Contemporary Educational Psychology. 25, pp. 82-91. Zupane, D., and Bren, M. (2010) Grade inflation in Slovenia. Contemporary Pedagogy [Sodobna Pedagogika]. 61 (3), pp. 208-228. Zakclj, A., and Ivanuš (Jrmek, M. (2010) Relationship between the results of national assessment of knowledge and socio-cultural background of students, school classes and homework (Povezanost rezultatov pri nacionalnem preverjanju znanjas socialno-kulturnim okoljem učencev, poukom in domačimi nalogami|. Ljubljana: Zavod Republike Slovenije za šolstvo. 3 abstracts/povzetki Abstracts/ Povzetki Darko Strajn The PISA Syndrome: Can we Imagine Education Without Comparative Testing? The article starts by recalling two relatively recent criticisms of PISA testing addressed to wider public: The Guardian, Tuesday 6 May 2014 letter addressed to PISA director Dr Schleicher and Erwin Wagenhofer's film Alphabet. Both of these criticisms, aimed at policy makers and even more, to the broader public, expose the dubious nature of ranking of results that inscribe PISA into the foundations of the neoliberal extension of market competition to all avenues of life. On an another level many disputes, divergent studies, books and articles predominantly in much less agitated discourse ponder the social role, impacts, advantages and shortcomings of PISA and also of other similar assessments of education, done with methods of testing, as well as rankings and benchmarking as consequences of testing. In this context Konrad Liessmann and Christian Laval criticised the "neoliberal attack on public school". Other writers expose a threat of cultural homogenization. Of course, many reflections on PISA are enunciated in the context of post-colonial studies, gender studies and other contemporary forms of critical thinking that are often associated with political anti-globalisation movements, which also include a range of alternative education practices and experiments. These criticisms cannot be easily typified, but they are mainly based on similar, albeit much more elaborated, theses as the main points of The Guardian letter. The deeper reasons of controversy should be seen in the paradigmatic divide, which has its roots in the gap between the continental and Anglo-American philosophy. The outcry against PISA in The Guardian letter is a kind of cumulative effect of the growing bid for emancipatory education, which again strives to return to a composition of educational ideals instead of the aims comprised in more or less utilitarian and technocratic concepts of increasingly visible failure of such neoliberal projections as knowledge society, human capital, and so on. Does all this mean that such comparative testing as PISA, as its most outstanding case, becomes obsolete? In spite of all criticism, the answer should be definitely: "No!" Key words-, testing, criticism, neoliberalism, methodology, emancipation, enlightenment Sindrom PISA: si lahko predstavljamo izobraževanje brez primerjalnega testiranja? Članek najprej spomni na razmeroma novi kritiki testov PISA, naslovljeni na širšo javnost: pismo, objavljeno v časopisu The Guardian, v torek, 6. maja 2014, naslovljeno na direktorja raziskav PISA Dr. Schleicherja in film Erwina Wagenhoferja Abeceda. Obe kritiki, namenjeni oblikovalcem politik, in še več, širši javnosti, izpostavljata dvomljivo naravo razvrščanja rezultatov, ki vpisujeta raziskave PISA v temelje neoliberalnega posplošenja tržne konkurence v vse pore življenja. Na drugi ravni mnogo sporov, različnih študij, knjig in člankov pretežno v obliki veliko manj vznemirjenih diskurzov premišljuje o družbeni vlogi, vplivih, prednostih in pomanjkljivostih raziskav PISA in tudi o drugih podobnih evalvaci-jah izobraževanja z metodami testiranja, kot tudi o razvrstitvah in standardizacijah kot posledicah testiranj. Y tem kontekstu Konrad Liessman in Christian Laval kritizirata »neoliberalni napad na javno šolo«. Drugi avtorji izpostavljajo grožnjo kulturne homogenizacije. Seveda je veliko razmislekov o raziskavah PISA oblikovanih v kontekstu postkolonialnih študij, študij spola in drugih sodobnih form kritičnega mišljenja, ki se pogosto navezuje politična antiglobalizacijska gibanja, ki vsebujejo tudi vrsto alternativnih izobraževalnih praks in eksperimentov. Vseh teh kritik ni mogoče zlahka tipizirati, vendar pa v glavnem temeljijo na podobnih, četudi bolj elaboriranih, tezah kot so poglavitne točke pisma v Guardi-anu. Globlje razloge polemike je treba videti v paradigmatski delitvi, ki korenini v prepadu med kontinentalno in anglo-ameriško filozofijo. Protest proti raziskavam PISA v pismu v Guardianu je nekakšen kumulativni učinek vse višje stave na emancipacijsko izobraževanje, ki si spet prizadeva vrniti v nabor izobraževalnih idealov namesto ciljev izraženih v bolj ali manj utilitarističnih in tehnokratskih konceptih vse bolj vidno zgrešenih neoliberalnih projekcij družbe znanja, človeškega kapitala, itd. Ali vse to pomeni, da je takšno primerjalno testiranje kot ga izvajajo v raziskavah PISA, odvečno in zastarelo? Kljub vsem kritikam, bi moral odgovor vsekakor biti odločni »Ne!« Ključne besede: testiranje, kritika, neoliberalizem, metodologija, emancipacija, razsvetljenstvo Urška Štremfel Slovenia on its Own Way Towards Improving PISA Results Programme for International Student Assessment (PISA) 2009 and 2012 results showed that Slovenian students performed below the Organisation for Economic Cooperation and Development (OECD) average in reading literacy. Additionally Slovenia is a European Union (EU) member state that does not successfully follow the EU benchmark, which states that by 2020, the number of low achieving students in PISA at the EU level should be less than 15%. The article discusses PISA in terms of transnational policy making and transnational problem solving. The article, using the case study of Slovenia, explains triple pressures participating countries face when performing below average in PISA comparative achievement scale (performing below international (OECD, EU) average, non-attain-ingofEU benchmark and common goals, non-attaining of national goals) and how these pressures are translated in the identification of policy problem at the national level and which ways participating countries have at their disposal to find the solution to the perceived policy problem. The article therefore provides policy analysis insight in the first stage of improving PISA results through the lenses of governance of problems (where PISA could be understood as international policy promotion) and policy learning theory (where PISA could be understood as instrument for lesson-drawing). In order to preserve the sovereignty of national state (Slovenia) over its educational system, the article suggests that instead of uncritical reception of international promotion of certain educational model, the more promising alternative for improving PISA results is lesson drawing. Considering lesson drawing, by providing empirical insights on the case study of Slovenia, the article shows how important it is for participating countries to have carefully defined national educational priorities and goals in order to be able to precisely define a policy problem according to its PISA results and to find a policy solution by drawing lessons from other successful participating countries. Key words-. PISA, governance of problems, lesson-drawing, low achievers, Slovenia Slovenija na lastni poti izboljševanja rezultatov v raziskavi PISA Objavi rezultatov raziskave Programa mednarodne primerjave dosežkov učencev (PISA) iz leta 2009 in 2012, sta pokazali, da so bili rezultati slovenskih 15-letnikov na področju bralne pismenosti primerjalno nižji kot povprečno v državah Organizacije za ekonomsko sodelovanje in razvoj (OECD) ter da Slovenija neuspešno zasleduje ciljno vrednost Evropske unije (EU), po kateri naj bi bil odstotek 15-letnikov, ki ne dosegajo temeljne ravni bralne pismenosti do leta 2020 pod 15 %. Rezultati so v slovenskem izobraževalnem prostoru sprožili vprašanja o možnih načinih izboljševanja dosežkov slovenskih učencev. Članek skozi konceptualni okvir analize politik raziskavo PISA obravnava kot obliko transnacionalnega oblikovanja politik in transnacionalnega reševanja javnopolitičnih problemov. V članku pritiske OECD, EU in nacionalnih akterjev po izboljšanju dosežkov učencev v raziskavi PISA tako osvetlimo skozi prizmo nove oblike vladavine v EU na področju izobraževalnih politik, ki raziskavo PISA razume kot primer vladavine javnopolitičnih problemov (oziroma kot primer mednarodne javnopolitične promocije) ter kot priložnost za medsebojno javnopolitično učenje med sodelujočimi državami pri izboljševanju njihovih dosežkov (oziroma kot primer učenja lekcij). Ob upoštevanju navedenih konceptov analize politik članek predstavlja uvid, kako sodelujoče države članice na podlagi podpovprečne uvrstitve v mednarodni primerjalni lestvici dosežkov PISA zaznajo javnopolitični problem (zaradi nedoseganja mednarodnega (OECD in/ali EU) povprečja, zaradi nedoseganja ciljne vrednosti EU ali zaradi nedoseganja nacionalnih ciljev na področju izobraževanja), in katere možnosti so jim na voljo pri reševanju zaznanega javnopolitičnega problema oziroma izboljšanju dosežkov učencev. V članku izpostavimo, da je za ohranjanje suverenosti nacionalne države (Slovenije) nad njenim izobraževalnim sistemom pomembno, da za izboljšanje dosežkov učencev v raziskavi PISA kritično presoja priporočila OECD kot sredstvo transnaciona-lne javnopolitične promocije ter premišljeno išče rešitve in dobre prakse pri drugih sodelujočih državah na podlagi učenja lekcij. Pri tem članek izpostavi pomen jasno opredeljenih nacionalnih prioritet in ciljev kot predpogoja za ohranjanje suverenosti pri iskanju lastnih rešitev zaznanega javopolitičnega problema ter identifikaciji držav pri katerih se lahko zgledujemo pri izboljševanju dosežkov naših učencev. Ključne besede: PISA, vladavina problemov, javnopolitično učenje, nizki dosežki, Slovenija Christine Salzer and Manfred Prenzel Looking Back at Five Rounds of PISA: Impacts on Teaching and Learning in Germany The German results of PISA 2012 were solid showing student performance in all domains significantly above the average of OECD countries. Nonetheless the data still point out some challenges for the next years. If PISA 2012 had been the first round of PISA, nobody in Germany would have been surprised and the overall picture would have been described as not very spectacular. However, given the history of the profound PISA-shock in 2001, the results of PISA 2012 mark a milestone of progress after twelve years of efforts to improve learning outcomes in Germany's educational context. Looking backward on PISA 2000, this paper starts with an analysis of the different aspects of the poor performance of the students in Germany at that time, including a very broad distribution and high correlation with social background and migration. The paper then discusses three major aspects of educational development: First, a thorough diagnosis of the problems in the educational system in Germany using PISA data as well as findings from other studies was important to draw adequate conclusions for measures taking into account different parts of the educational system (including e.g. pre-school or teacher training). Against this background an intense and evidence-based discourse between policy makers, researchers and the public could be started. This discourse led to a common understanding that a higher appreciation of education and educational reforms were of vital necessity. Last but not least a considerable number of nationwide, overarching programmes to improve teaching and learning with respect to educational standards was implemented. All in all the paper argues that findings from PISA have to be interpreted in the light of other types of educational research (e.g., longitudinal design, video studies). An improved public understanding of research on education helps to get acceptance for reforms. Besides political attention and engagement a strategic and systemic view is crucial to the success. Key words-. PISA, PISA-shock, Germany, improvement, education Pogled na dosedanjih pet ciklov raziskave PISA: učinki na poučevanje in učenje v Nemčiji Za Nemčijo so bili rezultati raziskave PISA 2012 ugodni, saj so pokazali dosežke učenk in učencev nad povprečjem držav OECD. Kljub temu pa še vedno nakazujejo nekatere izzive za naslednja leta. Če bi bil cikel PISA 2012 prvi cikel raziskave, nihče v Nemčiji ne bi bil presenečen in splošen vtis o rezultatih ne bi bil preveč spektakularen. Vendar pa glede na velik PISA-šok v letu 2001 rezultati raziskave PISA 2012 predstavljajo prelomnico po dvanajstih letih naporov za izboljševanje dosežkov v nemškem izobraževalnem sistemu. S pogledom na raziskavo PISA 2000 v članku začnemo z analizo različnih vidikov nizkih dosežkov nemških učenk in učencev v takratni raziskavi vključujoč zelo razpršene dosežke in visoko korelacijo s socialno-ekonomskim ozadjem in priseljenskim statusom. V nadaljevanju predstavljamo tri glavne vidike razvoja izobraževanja. Poglobljena diagnoza problemov v izobraževanem sistemu v Nemčiji z uporabo podatkov raziskave PISA kot tudi drugih raziskav je bila pomembna za izpeljavo ustreznih sprememb na različnih ravneh vzgojno-izobraževal-nega sistema (vključujoč na primer predšolsko vzgojo ter izobraževanje in usposabljanje učiteljev). Na teh podlagah se je lahko začela intenzivna in na podatkih temelječa razprava med oblikovalci politike, raziskovalci in splošno javnostjo. Razprava je vodila do skupnega razumevanja, da so vrednotenje znanja in prenova izobraževanja bistvenega pomena za izboljšanje stanja. Ne nazadnje pa se je začelo izvajanje vrste vsesplošnih programov za izboljšanje poučevanja in učenja na nacionalni ravni za doseganje standardov znanja. V članku zagovarjamo, da morajo biti izsledhi raziskave PISA interpretirani v luči drugih vrst edukacijskega raziskovanja (na primer, longitudinalnih raziskav, video študij). Boljše razumevanje javnosti o raziskovanju v izobraževanju pomaga pri sprejemanju nujnosti izvajanja reform. Ob pozornosti in angažiranosti politike pa sta za uspeh pomembni tudi njena strateška in sistemska naravnanost. Ključne besede-. PISA, PISA-šok, Nemčija, izboljšave, izobraževanje Pierre Brocbu The influence of PISA on Educational Policy in Canada: Take a Deep Breath The results from the most recent round of the Programme for International Student Assessment (PISA) revealed that while Canada remains among the top performing countries in the world, it is showing a downward trend in skills. This paper looks at how PISA results have been used since its inception in 2000 to inform education policy in a number of countries, including Canada. It summarizes the Canadian results in the global context and compares and contrasts Canada's results with those in a number of countries of interest focusing on how the initial and subsequent PISA results have been received in these countries. In several cases, PISA was exploited to initiate new education policies, while in others it was used to justify planned or newly implemented reforms. Considering the most recent PISA results in Canada and the call for action from several education stakeholders, this article argues that a federated country like Canada should avoid a "one-size-fits-all" approach to education reform. Furthermore, the author argues that the initial focus ofiPISA results on country ranking should be replaced, or at least complemented, with a look at trends over time where a country would not only judge its progress against other countries but also against itselfi In addition, federal systems like Canada, where education is decentralized, offer interesting opportunities for analyzing the PISA results at a microcosmic level to study factors related to high performance not only in other countries but in other provinces, as these often share similar contexts. Among those lessons learned from PISA over the past decade, the experience in a number ofi countries suggests that as useful as they may be, PISA results on their own are not a sufficient basis for initiating educational reform, as the data needs to be analyzed in a context that extends beyond the assessment itselfi Key words-. PISA, large-scale assessment, education policy, international comparison Vpliv raziskave PISA na izobraževalno politiko v Kanadi: zajemite sapo Rezultati zadnjega cikla Programa za mednarodno primerjavo dosežkov učenk in učencev (PISA) so pokazali, da Kanada sicer ostaja med državami z najvišjimi dosežki na svetu, vendar pa je trend ravni kompetenc učenk in učencev padajoč. V članku obravnavamo, kako so bili rezultati raziskave PISA uporabljeni za oblikovanje izobraževane politike v različnih državah, vključujoč Kanado, od njenega začetka v letu 2000. Rezultati za Kanado so predstavljeni v globalnem kontekstu in primerjani z rezultati v drugih relevantnih državah s fokusom na sprejemanje začetnih in nadaljnjih rezultatov raziskave v teh državah. V več primerih so bili rezultati raziskave uporabljeni za oblikovanje novih politik, v drugih pa za utemeljevanje že načrtovanih ali na novo vpeljanih sprememb. Glede na najnovejše rezultate raziskave PISA za Kanado in zahteve po spremembah v državi s strani različnih deležnikov v izobraževanju v članku poudarjamo, da se mora zvezna država, kot je Kanada, izogniti spreminjanju izobraževanja po modelu »enako za vse«. Avtor nadalje utemeljuje, da bi morala biti začetna pozornost na razvrstitve držav nadomeščena, ali vsaj dopolnjena, s pregledom časovnih trendov, ob katerih država svojih dosežkov ne bi primerjala le z drugimi državami, ampak tudi sama s sabo. Dodatno, zvezni sistemi, kot je kanadski, kjer je izobraževanje decentralizirano, ponujajo zanimive priložnosti za analizo rezultatov PISA na mikrokozmični ravni z raziskovanjem dejavnikov, ki se povezujejo z visokimi dosežki ne le v drugih državah, pač pa tudi v drugih provincah, saj le-te pogosto delujejo v podobnih kontekstih. Med lekcijami iz raziskave PISA v zadnjem desetletju izkušnje v drugih državah nakazujejo, da so rezultati raziskave PISA sicer uporabni, vendar sami po sebi niso zadostna osnova za odločanje in oblikovanje izobraževalnih reform, saj morajo biti analizirani v kontekstu, ki je precej širši od same raziskave. Ključne besede: PISA, raziskave na velikih vzorcih, izobraževalna politika, mednarodne primerjave Maria Stephens and.Anindita Sen Comparing U.S. States' Mathematics Results in PISA and Other International and National Student Assessments In 2012, three U.S. states - Connecticut, Florida, and Massachusetts -participated in the OECD's Program for International Student Assessment as individual entities in order to obtain an international benchmark of student performance. Such subnational participation in international assessments provides value nationally by contributing to a better understanding of the variation in national statistics and, for states, by providing a sense of the global comparative health of their education systems. However, one of the challenges in using the international data is in interpreting it alongside sometimes differing data from other international and national assessment programs in which states also participate. This article thus focuses on the question: What specific factors might explain differences in the PISA 2012 mathematics results of the three U.S. participant states and their mathematics results on other recent international and national assessments? It describes the results of a comparative analysis of four possible factors: (1) differences in the overall content distribution of the items, (2) differences in relative strengths and weaknesses on content and cognitive subscales, (3) differences in sampling, and (4) differences in participating countries. Key words-. PISA, large-scale assessment, education policy, international comparison Primerjave matematičnih dosežkov v nekaterih državah ZDA med raziskavo PISA in drugimi mednarodnimi in nacionalnimi preverjanji Leta 2012 so tri države v ZDA - Connecticut, Florida in Massachusetts - samostojno sodelovale v OECD-jevem Programu mednarodne primerjave dosežkov učenk in učencev z namenom, da bi pridobile mednarodne primerjave dosežkov svojih izobraževalnih sistemov. Tovrstne oblike sodelovanja enot znotraj nacionalnega izobraževalnega sistema v mednarodnih primerjavah so pomembne tudi na nacionalni ravni zaradi boljšega razumevanja različnosti v rezultatih na nacionalni ravni in, za države znotraj ZDA, ugotavljanje stanja v njihovih izobraževalnih sistemih. Vendar pa je pri uporabi podatkov raziskav med večjimi izzivi interpretacija včasih medsebojno neusklajenih rezultatov med različnimi mednarodnimi in nacionalnimi raziskavami, v katerih države sodelujejo. V članku se posvečamo vprašanju, kateri specifični faktorji lahko razložijo razlike med matematičnimi rezultati omenjenih treh držav ZDA v raziskavi PISA 2012 in drugih nedavnih mednarodnih in nacionalnih raziskavah. V članku primerjalno analiziramo štiri faktorje: (i) vsebinske razlike v razporeditvi nalog v mednarodnih preizkusih različnih raziskav, (2) razlike v relativno močnih in šibkih področjih, izkazanih na vsebinskih in procesnih podlestvicah v raziskavah, (3) razlike v metodologiji vzorčenja med raziskavami in (4) razlike v naboru in dosežkih drugih držav, ki so sodelovale v raziskavah. Ključne besede: PISA, raziskave na velikih vzorcih, izobraževalna politika, mednarodne primerjave Ana Kozina and, Ana Mlekuž The Predictive Power of Attribution Styles for PISA 2012 Achievement: International and National Perspective The study explores the predictive power of attribution styles for PISA 2012 mathematics achievement from international and national perspective. For this purpose, Weiner's' attribution theory was used as a framework in explaining the differences between high and low achieving students in PISA 2012 study for both international and national data analyses. The attribution theory (Weiner, 1985, 2010) investigates the process of attributing causes for success and failure and has been widely used as a motivational framework in achievement outcomes models. In the analyses, PISA 2012 samples were used ^=309.140) in order to define the predictive value of attribution styles. In more detail, PISA 2012 measures attributional style with two question sets dealing with: (i) the measurement of attributions for failure in mathematics (constructed index FAILMAT) and (ii) the measurement of attribution for success in mathematics (a set of questions that we combined using factor analyses into two indices: internal locus of control and external locus of control (ELC)). In the analyses, we focus primarily on attribution for success in mathematics and the predictive power of newly developed indices. The national (Slovene) results are com- pared to high and low achieving countries in European Union (Bulgaria, Romania, Estonia, Netherlands) and with international results. The results showed that attributions for success in mathematics is a significant predictor of PISA 2012 mathematics achievement in all selected countries explaining from 7 to 14 % of mathematics achievement variance (8 % in Slovenia; 11% international average). The percentages of explained variances remain high even after the inclusion of additional the index measuring attributional style in the model (FAILMAT). The students' internal locus of control significantly predicts higher mathematics achievement and external locus of control predicts lower mathematics achievement. To article ends with the implications for classroom practise being discussed. Key words: attribution theory, locus of control, students, PISA, achievement, mathematics Napovedna moč atribucijskih stilov za dosežke v raziskavi PISA 2012: mednarodna in nacionalna perspektiva V prispevku na mednarodni in na nacionalni ravni ugotavljamo napoved-no moč atribucijskih stilov (različnih načinov pripisovanja vzrokov uspehu in neuspehu) za dosežke iz matematike v mednarodni raziskavi PISA 2012. Teoretični okvir predstavlja v motivacijski literaturi široko sprejeta atribucijska teorija (Weiner, 1985, 2010), ki razlaga različne načine pripisovanj vzrokov uspehu in neuspehu. V analizah smo uporabili PISA 2012 podatkovne baze - mednarodno podatkovno bazo ter izbrane nacionalne podatkovne baze (Ar=?09 140). PISA meri atribucijski stil z dvema nizoma vprašanj: (i) pripisovanje vzrokov za neuspeh (na mednarodni ravni je oblikovan v indeks FAILMAT) in (ii) pripisovanje vzrokov za uspeh. Slednji sklop vprašanj je podrobneje analiziran v prispevku. V prvem koraku smo s faktorsko analizo analizirali postavke, ki v raziskavi PISA merijo način pripisovanja vzrokov za uspeh. Na podlagi izločevalnih kriterijev smo na mednarodni ravni identificirali dva faktorja: notranji lokus kontrole in zunanji lokus kontrole. Oba indeksa sta bila kasneje uporabljena v regresijskih modelih (multipla regresija) PISA matematičnih dosežkov. Podatki Slovenije so bili primerjani z mednarodnim povprečjem ter izbranimi državami Evropske unije. Primerjalne države smo izbrali glede na njihov povprečni matematični dosežek v raziskavi PISA 2012 (dve najvišje uvrščeni državi Evropske unije: Nizozemska in Estonija ter dve najnižje uvrščeni državi Evropske unije: Romunija in Bolgarija). Rezultati kažejo, da lahko matematični dosežek v raziskavi PISA napovemo iz podatkov o načinih pripisovanj vzrokov za uspeh dijakov. Z regresijskimi modeli lahko v izbranih državah pojasnimo od 7 do 14 % variance matematičnega dosežka (8 % v Sloveniji; 11% mednarodno povprečje). Odstotki var- iance ostanejo visoki tudi po vključitvi dodatnega indeksa atribucijskih stilov: sprejemanje odgovornosti za neuspeh pri matematiki (FAILMAT). Dijaki, ki dosegajo višje vrednosti notranjega lokusa kontrole dosegajo pomembno višje dosežke na PISA testu iz matematike. Dijaki, ki dosegajo višje vrednosti na zunanjem lokusu kontrole dosegajo pomembno nižje dosežke na PISA matematičnem testu. Na podlagi rezultatov so podane smernice za pedagoški prakso. Ključne besede: atribucijska teorija, lokus kontrole, dijaki, PISA, dosežek, matematika Mojca Štraus (In)equalities in PISA 2012 Mathematics Achievement, Socio-economic Gradient and Mathematics-related Attitudes of Students in Slovenia, Canada, Germany and the United States The study aimed at examining the roles of socio-economic background and mathematics-related attitudinal factors in explaining achievement in mathematics literacy of the PISA 2012 study for Slovenia in comparison with Germany, Canada and the United States. The data on these factors are collected through the student background questionnaires accompanying the PISA achievement tests. While (in)equalities in student achievement due to socio-economic background have long been established, it continues to remain relevant to explore to what extent motivational and attitudinal factors can mediate this influence of socio-economic and cultural status. The international context of four countries was considered. Using linear multivariate regression, the study found that while socio-economic and cultural status remains as a strong influence on achievement, students' mathematics-related self-beliefs are stronger predictors of achievement than their drive and motivation. If socio-economic and other attitudinal factors in the model are controlled, mathematics-related self-efficacy is still a strong and important predictor of mathematics achievement in all four countries. Observing students' responses to the questions about attitudes towards mathematics interesting patterns emerged between the four countries; similarities were observed between the Slovene and German students' responses as well as between the Canadian and the United States students' responses, indicating there may exist more general, for example cultural influences on these attitudes outside the educational contexts. The slopes of socio-economic gradients on mathematics achievement varied among the four countries, being relatively high in Slovenia and Germany and relatively low in Canada and the United States. The influence of socio-economic and cultural status therefore shows the same commonalties between the four countries as the atti-tudinal responses. Across all four countries, the mediating impact of factors in the relationship between the socio-economic and cultural status and mathematics achievement was generally similar with exception of mathematics self-efficacy showing a somewhat different impact in Slovenia than in the other countries. Key words-, mathematics achievement, PISA, socio-economic gradient, self-efficacy (Ne)enakosti v matematičnih dosežkih, socio-ekonomskem gradientu in stališčih do matematike v raziskavi PISA 2012 za učenke in učence v Sloveniji, Kanadi, Nemčiji in ZDA V članku raziskujemo vloge socio-ekonomskega ozadja in stališč do matematike pri pojasnjevanju dosežkov pri matematični pismenosti v raziskavi PISA 2012 za Slovenijo v primerjavi z Nemčijo, Kanado in Združenimi državami Amerike. Ti podatki se zbirajo z vprašalniki za učenke in učence, ki spremljajo preizkuse znanja v raziskavi PISA. Medtem ko so (ne) enakosti v dosežkih učenk in učencev zaradi socio-ekonomskega ozadja že dolgo prepoznane, še vedno ostaja relevantno raziskovanje, koliko lahko motivacijski in stališčni dejavniki mediirajo ta vpliv socio-ekonomskega in kulturnega ozadja. Obravnavali smo mednarodni kontekst štirih držav. Z uporabo linearne multivariatne regresije smo ugotovili, da so poleg socio-ekonomskega in kulturnega statusa prepričanja o sebi v povezavi z matematiko močnejši napovednik dosežkov kot vztrajnost in motivacija. Ko v modelu kontroliramo socio-ekonomski in kulturni status ter druge dejavnike, so prepričanja o sebi v povezavi z matematiko še vedno močan in pomemben napovednik matematičnih dosežkov v vseh štirih državah. Analiza odgovorov učenk in učencev na vprašanja o stališčih do matematike je pokazala zanimive primerjave med štirimi državami; med seboj so si podobni odgovori učenk in učencev v Sloveniji in Nemčiji ter odgovori učenk in učencev v Kanadi in Združenih državah Amerike. To morda nakazuje obstoj splošnejših, na primer kulturnih vplivov na stališča učenk in učencev, ki lahko izvirajo izven izobraževanega konteksta. Nakloni so-cio-ekonomskega gradienta na matematične dosežke se med državami razlikujejo. V Sloveniji in Nemčiji je ta naklon relativno visok in v Kanadi in Združenih državah Amerike relativno nizek. Vpliv socio-ekonomske-ga in kulturnega statusa tako kaže podobne primerjave med štirimi državami kot odgovori učenk in učencev o stališčih do matematike. V vseh štirih državah so vplivi dejavnikov na matematične dosežke v splošnem podobni z izjemo zaznane samoučinkovitosti pri matematiki, pri kateri v Sloveniji vpliv na dosežke nekoliko odstopa od vpliva v drugih državah. Ključne besede: matematični dosežki, PISA, socio-ekonomski gradient, zaznana samoučinkovitost 4 review/recenzija Review/ Recenzij a Slavko Gaber (ur.) (2014). Finska v vrhu znanja 2030. Ljubljana: CEPS. Kadar je nekoliko obširnejši policy paper zelo zanimivo in kar napeto branje, ne pa dolgočasno in formalistično razpredanje ter v najhujših primerih še ideologizirano verbalno gestikuliranje, se lahko zazdi, da se »nekaj dogaja« na ravni samih paradigem, osnovnih konceptov in načinov ter shem razmišljanja, če o zadevni realnosti niti ne govorimo. Ne govorimo o besedilu, ki bi ga podpisala kaka oblastna instanca na nacionalni ali mednarodni ravni ali o dokumentu kakega strateškega foruma, ampak o prispevku finskega učiteljskega sindikata (!). CEPS (Center za študij edukacijskih strategij na Pedagoški fakulteti v Ljubljani) in SVIZ (Sindikat vzgoje, izobraževanja, znanosti in kulture Slovenije) sta zelo ažurno in požrtvovalno poskrbela za to, da je sicer drobna a zgoščena knjižica pod zgoraj navedenim naslovom in natisnjena v lični ter bralcu prijazni brošuri, dostopna zainteresiranemu slovenskemu bralstvu skupaj z lucidnim kritičnim uvodom in daljnovidno raz-mišljujočo spremno besedo. Posebno zanimivost besedilu knjižice daje dejstvo, da je nastalo v učiteljskih vrstah, in da ga je tudi izdal finski učiteljski sindikat, kar je že samo na sebi »inovativno«, če se izrazimo po najnovejši evropski modi. Finska je - kot je znano - na področju izobraževanja dosegla zvezdniški status, med drugim, zahvaljujoč odličnim rezultatom v raziskavah PISA, saj je zlasti v dosedanjem 21. stoletju nenehno na ali čisto pri vrhu po izmerjenem znanju svojih učencev in dijakov. Za »češnjico na torti« pa je Finska zelo uspešna še v drugi raziskavi v okviru OECD, namreč PIAAC, ki raziskuje znanje odraslih. Ni dvoma, daje izmerjeni visoki delež znanja in sploh višina izobrazbene ravni finskega prebivalstva, integralni del tega, kar opisujemo kot »skandinavsko« politično kulturo in kot kulturo tolerantnosti ter smisla za vzajemno sodelovanje med ljudmi. Pri vsej zavidljivi uspešnosti pa se tudi začne novi finski problem, ki so ga zaznali učitelji skupaj s svojim sindikatom in so se zato odločili, da ne bodo čakali na uradne politične instance ter so svoje analize in projekcije v prihodnost, označeno z letnico 2030, sporočili javnosti; na srečo ne samo finski ampak tudi slovenski javnosti po zaslugi že omenjenih izdajateljev knjižice in tudi po zaslugi osebnih stikov med predstavniki obeh dežel. Kot lahko razberemo iz knjižice, je problematika, ki jo je odprl finski učiteljski sindikat, precej večdimenzionalna. Visoka uspešnost finskega izobraževalnega sistema v mednarodnem prostoru je postala prvi in z notranje finske strani opazen problem. Tako uvodničar kot pisci glavnega besedila namreč ugotavljajo, daje ta uspešnost botrovala pasivnosti upravljavcev in opuščanju razmišljanja o kljub vsemu potrebnih spremembah in izboljšavah z mislijo na prihodnost izobraževanja v družbenem in ekonomskem okolju. Drugi kompleksnejši vidik, ki ga finski sindikat zaznava (ga pa spričo svoje pozicije morda ne eksplicira z vso ostrino) pa zadeva mednarodna dogajanja in predvsem pritiske, ki jih generira neoliberalna »konkurenčnost«. Sahl-berg tako v svojem uvodu posebej poudarja, da poročilo vidi odvisnost prihodnosti Finske od tega, »kako dobro bo Fincem uspelo v prihodnosti zaščititi razmeroma visoko stopnjo enakosti dohodkov in pravičnost edukacije« (str. 7). Že na prvih nekaj straneh razberemo, da so se avtorji besedila - nastalega po obširnih razpravah v letih 2012 in 2013 med članstvom - dobro zavedali, da PISA pač meri nekaj gotovo zelo pomembnih učinkov izobraževanja, a treba je razmišljati tudi o vseh drugih prispevkih in delovanjih šolskega sistema v družbi, če naj ta deluje tako, da omogoča odprte sheme družbene reprodukcije. Tu bi si dovolil pripombo, daje prav to finsko besedilo po svoji »tipologiji« vzorna reakcija na uspeh, kakor ga meri PISA. Ta raziskava je sicer na sploh v svetu deležna tudi kritičnih pripomb, katerih utemeljenost pa je treba bolj kot snovalcem in izvajalcem raziskave pripisati delovanju neoliberalne ideologije in na njej zasnovani ekonomiji. Le-to je že 1. 1998 Gilles Deleuze poimenoval »ekonomija prevečne produkcije« (.surproduction), pri čemer je v ozadju - ali nemara bolj natančno, prav na brezsramen način v ospredju, logika »brezmejne« akumulacije kapitala. Finsko besedilo tako izhodiščno opredeli »temeljne vrednote« izobraževanja in vzgoje, ki merijo tako na tradicionalne razsvetljenske vidike (veselje do učenja, pravičnost, demokracija ipd.) kot novejše vrednote (ekologija, trajnostni razvoj). V uspešni preteklosti pa seje po ugotovitvah piscev poročila zgodilo tudi nekaj poslabšanj na večini področij, od predšolske vzgoje do izobraževanja odraslih. Tudi na Finskem je zadnja kriza udarila svoj pečat s tem, daje ogrozila stabilnost javnih financ in s tem tudi zadovoljivost financiranja izobraževanja, če o začasnem odpuščanju učiteljev in podobnih »varčevalnih« početjih niti ne govorimo. V nadaljevanju je očitna naklonjenost finskih sindikatov javnosti izobraževanja, ki naj bi tako ostalo tudi v prihodnosti do 1.2030. Glede vloge izobraževanja v prihodnosti finski učiteljski sindikat napoveduje vrsto sprememb tako na področju gospodarstva (omejenost rasti) in na področju družbenih dejavnikov. Tako, med drugim, vidijo kot bistveno nalogo sistema edukacije, da s pomočjo »defragmentacije in integracije« deluje kot »protisila družbeni polarizaciji« (str. 27). V enem izmed poglavij tega napetega branja se ne bo odveč poučiti o spremembah, ki se v dobrem in slabem obetajo učiteljskemu poklicu. Vsekakor pa je predvidljiva potreba po večjem ugledu in samostojnosti v opravljanju poklica. Dikcija besedila v njegovih projekcijah vprihodnost je po svoje še posebej zanimiva, saj sindikat, ki ni ključni akter pri razporejanju sredstev in ni vedno pov-prašan glede najpomembnejših družbenih odločitev, govori o tem da »bo Finska« več vlagala v edukacijo, ki tako ne bo »strošek«, ampak investicija. Trditev v prihodnjiku je mogoče razumeti kot sugestijo in zahtevo hkrati. Besedilo se nato podrobneje ukvarja z nevarnostmi, perspektivami in možnostmi na vsakem od področij izobraževanja posebej in se pri tem izkaže s podrobnejšimi predvidevanji. To je tisti del knjižice, ki je za vse akterje v edukaciji najbolj zanimiv še posebej glede na njihovo posebno področje dela. Če bodo na Finskem drugi akterji, namreč tisti v politiki in gospodarstvu, ta predvidevanja, o katerih tu ne govorim podrobneje, vzeli dovolj resno, bo Finska nemara spet lahko vzor za ostali svet. V teh predvidevanjih zbudi pozornost napoved ogroženosti »osnovnega izobraževanja iz umetnosti«, ki se bo po mnenju piscev umikalo iz šole, v kateri se bo število ur tega pouka zmanjšalo in se bo selilo v neformalno izobraževanje. Sindikat torej pesimistično predvideva posledice v obliki »le premožnim dostopnih« storitev; nadalje bodo učinki vidni tako v neformalnem izobraževanju kot v učiteljskem poklicu na tem področju. Prav ta vsebina besedila, ki sicer sem in tja »ne vidi« ali spregleduje omejitve neoliberalno profilirane ekonomije in ustrezajoče družbene konstrukcije, kaže na to, da se finski sindikat vendarle izrecno v polnosti zaveda, da je prav šolsko polje ključno »bojno polje« za prihodnost na sploh. Besedilo mestoma namigne na širši mednarodni kontekst, se pa ne izreče preveč na široko o tem, daje del rešitve uganke prihodnosti morda tudi v večji mednarodni sodelovalnosti. Eden od negativnih vidikov raziskav PISA, kakor se kažejo tistim, ki kritično opazujejo globalna družbena dogajanja, je namreč učinek tekmovanja med državami, s čim- erje tudi taka »mehka« dejavnost kot je edukacija vpotegnjena v brutalna razmerja gospodarske konkurence v obliki, v kakršni jo diktira neolib-eralna ideologija ter zainteresirana globalna kapitalska oligarhija. Finski sindikat se v razpravo o tem, kot rečeno, ne spušča, čeprav potrebo po njej nakaže. Pravo vprašanje o prihodnosti ne zadeva samo Finske ampak tudi vse druge dežele in misel na vzajemno pomoči in sodelovanje se ponuja kar sama. Darko Štrajn 5 authors/avtorji Author s/Avtor j i Darko Strajn, Educational Research Institute. Darko Strajn graduated in philosophy and sociology, Faculty of Arts - University in Ljubljana. He is currently heading a programme in educational research at the Educational Research Institute. He conducts full professor lectures on film theory at the graduate School for Studies in Humanities (ISH) in Ljubljana. His research comprises of topics such as education and social change, politics, aesthetics and media. Darko Strajn je diplomiral iz filozofije in sociologije na Filozofski fakulteti Univerze v Ljubljani. Trenutno vodi raziskovalni program na Pedagoškem inštitutu. Kot redni profesor predava o filmski teoriji na podiplomski šoli za humanistične študije (AMEU - ISH) v Ljubljani. Njegove raziskave obsegajo teme, kot so izobraževanje in družbene spremembe, politika, estetika in mediji. Urška Stremfel, Educational Research Institute. Urška Stremfel is a research fellow at the Educational Research Institute in Ljubljana and part-time research fellow at the Centre for Political Science Research at the Faculty of Social Sciences, University of Ljubljana. Her research interests include the European aspects of policy analysis, especially new modes of EU governance and cooperation in the field of education policy in the framework of the open method of coordination. Urška Stremfel, doktorica politoloških znanosti, je znanstvena sodelavka na Pedagoškem inštitutu, pri svojem raziskovalnem delu pa sodeluje tudi v Centru za politološke raziskave na Fakulteti za družbene vede Univerze v Ljubljani. Njen znanstvenoraziskovalni interes predstavlja evropsko sodelovanje na področju izobraževanja in njegov vpliv na nacionalni izobraževalni prostor. V tem okviru posebno pozornost namenja vlogi mednarodnih raziskav pri oblikovanju slovenske izobraževalne politike in izobraževalnih praks slovenskih šol ter razvoju na podatkih temelječega izobraževanja. Christine Sälzer, Technische Universität München Christine Sälzer is an educational researcher at the Technische Universität München (TUM), School of Education in Munich, Germany, and the Centre for International Student Assessment (ZIB). She has been the PISA National Project Manager of PISA 2012 and is currently managing PISA 2015. Her main research interests are large-scale student assessments, school absenteeism and student behaviour at school. Christine Sälzer je raziskovalka v izobraževanju na Technische Universität München (TUM), School of Education v Nemčiji, in pri Centre for International Student Assessment (ZIB). Bila je nacionalna kooordinatori-ca raziskave PISA 2012 in trenutno sodeluje pri vodenju raziskave PISA 2015. Glavno področje zanimanja so raziskave dosežkov v izobraževanju na velikih vzorcih, tematsko pa odsotnost učencev od pouka in vedenjske težave v šoli. Manfi'ed Prenzel, Technische Universität München Manfred Prenzel is the dean of the Technische Universität München (TUM), School of Education in Munich, Germany, and the president of the Centre for International Student Assessment (ZIB). He has been the PISA National Project Manager of PISA 2003, 2006 and 2009 and is currently managing PISA 2015. Besides his interest in Large Scale Assessments, his research deals with cognitive and motivational aspects of teaching and learning. Manfred Prenzel je dekan na Technische Universität München (TUM), School of Education v Nemčiji in predsednik ustanove Centre for International Student Assessment (ZIB). Bil je nacionalni koordinator raziskave PISA v letih 2003, 2006 in 2009 ter trenutno sodeluje pri vodenju raziskave PISA 2015. Ob delu pri raziskavah dosežkov v izobraževanju na velikih vzorcih se ukvarja s kognitivnimi in motivacijskimi vidiki poučevanja in učenja. Pierre Brochu, Council of Ministers of Education Canada Pierre Brochu is Director, Learning Assessment Programs with the Council of Ministers of Education (Canada). He has been Co-National Project Manager for PISA as well as co-representative for Canada on the PISA Governing Board since 2009. He is also pursuing doctoral studies in Hu- man Development and Applied Psychology at the Ontario Institute for Studies in Education (OISE) at the University of Toronto. Pierre Brochu je direktor Programov raziskovanja dosežkov učenja pri Svetu ministrov za izobraževanje v Kanadi. Od leta 2009 opravlja naloge nacionalnega sokoordinatorja raziskave PISA in sopredstavnika Kanade v Mednarodnem svetu PISA. Pripravlja doktorsko disertacijo na področju razvojne in aplikativne psihologije na Univerzi v Torontu, Ontario Institute for Studies in Education (OISE). Maria Stephens, American Institutes for Research Maria Stephens is a senior researcher at American Institutes for Research. In this capacity, Stephens provides analytic, writing, and other technical support for projects, with two foci in recent years: on coordinating international activities in the area of assessment and indicators and on district-level education reform. Currently, Stephens manages the school review process for the Say Yes to Education effort in the Syracuse City School District and also provides support to the National Center for Education Statistics' International Activities Program. She leads activities to improve the utilization of international data at the domestic level (e.g., developing an international research database, indicator reports, and reports comparing international and domestic assessments) and assists staff in making strategic decisions regarding the future of the Program for International Assessment. Maria Stephens je višja raziskovalka pri American Institutes for Research. V tej vlogi izvaja analitično, poročevalsko in drugi tehnično podporo projektom. V zadnjih letih dela predvsem na dveh vsebinskih področjih: na koordinaciji mednarodnih aktivnosti na področju preverjanja znanja in indikatorjev in na prenovi izobraževanja na ravni lokalne skupnosti. Trenutno vodi proces evalvacije šol za project Say Yes to Education v Syracuse City School District in izvaja podporne dejavnosti za program mednarodnih aktivnosti v National Center for Education Statistics. Vodi aktivnosti za izboljšanje uporabe mednarodnih podatkov na nacionalni ravni (npr. priprava mednarodne raziskovalne baze, poročila o indikatorjih in poročila s primerjavami rezultatov mednarodnih in nacionalnih raziskav) in sodeluje pri strateškem razvoju programa mednarodnih aktivnosti. Anindita Sen, American Institutes for Research Anindita Sen is a senior research analyst in the Education Department at the American Institutes for Research. Dr. Sen has worked on Annual Reports for Education data and international assessments. She has published findings from studies including the Program for International Student Assessment (PISA), the Progress in International Reading Literacy Study (PIRLS), and the Trends in International Mathematics and Science Study (TIMSS). She has also spent many years serving as a coordinator and presenter at data training seminars and summer conferences sponsored by the National Center for Education Statistics to train advanced graduate students, university faculty, and researchers from around the country in using longitudinal and international databases in their work. Dr. Sen is a graduate of the Economics Program at the New York University, where he received both his master's degree and Ph.D. Anindita Sen je višja analitičarka v Education Department pri American Institutes for Research. Pripravljala in analizirala je podatke za publikacije Annual Reports for Education in za mednarodne raziskave v izobraževanju. Objavila je več izsledkov iz Programa mednarodnih primerjav dosežkov učenk in učencev (PISA), Mednarodne raziskave bralne pismenosti (PIRLS) in Mednarodne raziskave trendov znanja matematike in naravoslovja (TIMSS). Vrsto let je bila koordinatorica in predavateljica na seminarjih za usposabljanje uporabnikov longitudinalnih in mednarodnih baz, ki jih je sponzoriral National Center for Education Statistics. Dr. Sen je diplomirala na Economics Program pri New York University, kjer je dosegla tudi naziva magistrica in doktorica znanosti. Ana Kozina, Educational Research Institute. Ana Kozina is a researcher, assistant professor and a head of the Centre for evaluation studies in Educational Research Institute. Her work is in the field of developmental and educational psychology. She is focused on the developmental and time related trends of aggression and anxiety (in childhood and adolescence) their interplay and the role anxiety and aggression play on individual level, on school level and on the community level (with possible prevention and intervention designs). In the field of education she is interested in the factors related to students' achievement (school climate, social and emotional learning, motivation...). She has been involved in several national and international research and evaluation projects. Currently she is working on postdoctoral project: Development of guidelines for aggression reduction on school level based on an anxiety-aggression model and trend analyses of anxiety and aggression in Slovenia primary schools from year 2007 to year 2011. Her work is presented on national and international level (e.g. conferences, journals, monographs) on regular basis. She is a member of Editorial board: Educational research Institute Press. Ana Kozina je diplomirana univerzitetna psihologinja, doktorica psiholoških ved in docentka za psihologijo. Zaposlena je na Pedagoškem in- štitutu kjer je vodja Centra za evalvacijske študije. Njeno raziskovalno delo sega na področji pedagoške in razvojne psihologije. Ukvarja se z razvojem agresivnosti in anksioznosti (obdobje otroštva in mladostništva) ter njune interakcije na ravni posameznika in na ravni širšega družbenega okolja (vključno z razvojem preventivnih in intervencijskih dejavnosti). Na področju pedagoške psihologije se ukvarja s preučevanjem dejavnikov (šolska klima, socialno in čustveno učenje, motivacija ...), ki vplivajo na učne dosežke otrok in mladostnikov. Njeno raziskovalno delo vključuje vključenost v mednarodne in nacionalne raziskovalne projekte ter evalvacijske študije. Trenutno je vodja temeljnega podoktorskega raziskovalnega projekta z naslovom: Razvoj, smernic za zmanjševanje agresivnosti na ravni šol. na podlagi modela povezanosti agresivnosti in anksioznosti ter analize trenda obeh pojavov v slovenskih osnovnih šolah od. leta 2007 do leta 2011. Izsledke predstavlja na nacionalni in mednarodni ravni (znanstvene konference, posveti, članki, poglavja, monografije). Je članica uredniškega odbora Založbe Pedagoškega inštituta. Ana Mlekuž, Educational Research Institute. Ana Mlekuž holds a B. A. in political sciences, is a researcher at the Educational Research Institute in Ljubljana. She is data manager for International Civic and Citizenship Education Study (ICCS 2009) and European Survey on Language Competences (ESLC 2011) and is a co-author of several scientific articles in the field of international large scale assessments. Ana Mlekuž, univ. dipl. pol, je zaposlena kot raziskovalka na Pedagoškem inštitutu. Je upravljavka podatkovnih baz za Mednarodno raziskavo državljanske vzgoje in izobraževanja (ICCS 2009) in Evropsko raziskavo o jezikovnih kompetencah (ESLC 2011) ter je soavtorica znanstvenih in strokovnih člankov s področja mednarodnih raziskav znanja Mojca Straus, Educational Research Institute. Mojca Straus is a researcher in international and national studies of different areas in education and serves as the national coordinator of the PISA Study as well as the leader of the programme of the international educational at the Educational Research Institute. In addition to research in education her research work focuses on statistical approaches to analyzing the data from international comparable studies. She leads the Educational Research Institute (Pedagoški institut) as its director. Mojca Straus se raziskovalno ukvarja z mednarodnimi in nacionalnimi raziskavami različnih področij v izobraževanju in je nacionalna koordi-natorica raziskave PISA ter vodja Programa mednarodnih raziskav v izo- braževanju na Pedagoškem inštitutu. Njeno raziskovalno delo je poleg raziskovanja šolskega polja usmerjeno v teoretično in praktično obravnavo različnih statističnih pristopov pri analizi podatkov mednarodnih primerjalnih raziskav. Kot direktorica vodi Pedagoški inštitut. Navodila avtorjem/-i cam člankov v reviji Šolsko polje Članek (praviloma v obsegu od 7000 do največ 10.000 besed) naj ima na začetku: 1) naslov ter ime in priimek avtorja/-ice, 2) povzetek v slovenskem in angleškem jeziku, do 300 do 350 besed, 3) ključne besede v slovenščini in angleščini (do 5), 4) kratko predstavitev avtorja/-ice (do 100 besed v slovenščini in angleščini), navedena naj bo tudi organizacija zaposlitve. Prispevki naj bodo napisani v knjižni slovenščini ob upoštevanjuveljavnegapravopisa, v nasprotnem primeru si uredništvo pridržuje pravico, da članka ne recenzira oziroma ga zavrne. Če je prispevek že bil objavljen v kaki drugi reviji ali če čaka na objavo, je treba to izrecno navesti. Prispevek naj ima dvojni medvrstični razmik, tip črk naj bo Times New Roman, velikost 12 pik (v opombah 10). Besedilo naj bo levo poravnano, strani pa zaporedno oštevilčene. Odstavki naj bodo ločeni s prazno vrstico. Uporabiti je mogoče tri hierarhične nivoje podnaslovov, ki naj bodo oštevilčeni (uporabljajte izključno navaden slog, v prelomu bodo ravni ločene tipografsko): 1. - 11 —111 Za poudarke uporabite izključno ležeči tisk (v primeru jezikoslovnih besedil, kjer so primeri praviloma v ležečem tisku, lahko za poudarke izjemoma uporabite polkrepki tisk). Ležeče pišite tudi besede v tujih jezikih Raba drugih tipografskih rezov (podčrtano, velike male črke, krepko kurzivno ...) ni dovoljena. Ne uporabljajte dvojnih presledkov, prav tako ne uporabljajte preslednice za poravnavo besedila. Edina oblika odstavka, kije dovoljena, je odstavek z levo poravnavo brez rabe ta bula torjev prve ali katerekoli druge vrstice v os tavku (ne uporabljajte sredinske, obojestranske ali desne poravnave ods tavkov). Oglate oklepaje uporabljajte izključno za fonetične zapise oz. zapise izgovarjave. Tri pike so stične le, če označujejo prekinjeno bese... Pri nedokončani misli so tri pike nestične in nedeljive Prosimo, da izključi te funkcijo deljenja besed Sprotne opombe naj bodo samo oštevilčene (številke so le vos ti čno za besedo ali ločilom - če besedi, na katero se opomba nanaša, sledi ločilo) in uvrščene na tekočo stran besedila. Citati v besedilu naj bodo označeni z dvojnimi, citati znotraj citatovpa z enojnimi narekovaji. Izpuste iz citatovin prilagoditve označite s tropičjem znotraj poševnic /.../. Daljše citate (več kot 5 vrstic) izločite v samostojne odstavke, ki jih od ostalega besedila ločite z izpustom vrstice in umikom v desno. Vir citata označite v okroglem oklepajunakoncucitata: (Benjamin, 1^74: str. 42). Če je avtor/-icanaveden/-a vso besedilu, priimek lahko izpustite V besedilu označi te najprimernejša mesta za likovno opremo (tabele, skice, grafikone itd.) po zgledu: [Tabela 1 približno tukaj]. Posamezne enote opreme priložite vsako vposebni datoteki (v eps, ai, tif ali jpg for matu, minimalna resolucija 300 dpi). Naslov tabele je nad tabelo, naslov grafa pa pod grafom. Prostor, ki ga oprema v prispevku zasede, se šteje v obseg besedila, bodisi kot 150 besed (polstrani) ali 500 besed (celastran). Na vir v besedilu se sklicujte takole: (Ducrot, 1988). Stran navedka navedite za dvopičjem: (Foucault, str. 57). Če so trije avtorji/-ice navedenega dela, navedite vse tri: Bradbury, Boyle in Morse (2.002.), pri večjem številu pa izpišite le prvo ime: (Taylor et al., 1^78). Dela enega avtorja/-ice, ki so izšla istega leta, med seboj ločite z dodajanjem malih črk (a, b, c itn ), stično ob letnici izida: (Bourdieu, 19^? 6a) Dela različnih avtorjev/-ic, ki se vsa nanašajo na isto vsebino, naš tej te po abecednem redu i nji h ločite s podpičjem: (Haraway, Oakley, 1005, Ramazanoglu,ioox). Pri večkrat zaporedoma citiranih delih uporabite tole: (ibid.). V članku uporabljena dela morajo biti po abecedi navedena na koncu, pod naslovom Literatura. Če so bili v prispevku uporabljeni viri, se seznam virov, podnaslovom Viri, uredi posebej. Če je naslovov spletnih strani več, se lahko navedejo tudi v posebnem seznamu z naslovom Spletne strani. Pri navedbi spletne strani se v oklepaju dopiše datum dostopa. Vsako enoto v teh seznamih zaključuje pika. Način navedbe enot je naslednji: Knjige: Bradbury, I., Boyle J., in Morse, A. (2002) Scientific Principlesfor Physical Geographers. Harlow: Prentice Hall. G arber, M. {ljyy) Symptoms of Culture. Harmondsworth: Penguin. Clanki: Kerr, D. (1999 b) Changing the political culture: the advisory group on education for citizenship and the teaching of democracy in schools. Oxford Review of Education 25 (4), str. 25-35. Poglavja v knjig: Walzer, M. (1992) The Civil Society Argument. V MOUFFE, Ch. (ur.). Dimensions ofRadical.De-mo era cy: Pluralism, Citizenship and Community. London: Routledge. Spletne strani: http://www.cahiers-pedagogiques.com/article.php35id_article=88i (pridobljeno 5.5.2008). O morebitnih drugih posebnostih se posvetujte z uredništvom. Naslov uredništva-. Solskopolje, Mestni trg 17,1000 Ljubljana; tel.: 014101140,fax: 014101z66, e-pošta: info@theschoolfield.com; eva.klemencic@pei.si Naročilo narevijo-. Solskopolj e,Slovensko dr uštvoraziskovalcevšolskegapolj a,Mestnitrgi7,io 00 Ljubljana, e-pošta: eva.klemencic@pei.si; tel.: 01 410 1153, fax: 01410 i z 66 Guidelines to the authors The submission of an article to the Šolsko polje journal should be between 7.000 to 10.000 words long. At the beginning it should include - the author's name and address; - a summary in both Slovene and English (from 300 to 350 words); - 5 keywords in both Slovene and English; - a short presentation of the author in both Slovene and English (each of up to 100 words) including his/ her institutional affiliation The submission should be accompanied by a statement that the submission is not being considered for publication in any other journal or book collection. The spacing of the article should be double spaced, the font Times New Roman (size 12 in the main text and size 10 in the footnotes). Paragraphs should be indicated using an empty row. There are three types of hierarchical subheadings, which should be numbered as follows: For emphasis, use italics only. Words in a foreign language should also be italicized. Use self numbered footnotes. Double quotations marks should be used for quotes in the text and single quotation marks for quotes within quotes. Longer quotations (more than 5 lines) should be extracted in separate paragraphs and separated from the rest of the text by omitting the rows and by having an indentation to the right. The source of the quotation should be in round brackets at the end of the quotation, e.g. (Benjamin, 1574, pp. 42-44). Please mark in the text the place where a graphic product (tables, diagrams, charts, etc..) should be included, e.g. [Table 1 about here]. These products should be attached in a separate file (in eps' ai' tif or jpg' format [300 dpi resolution]). The table title should be above the relevant table or the graph. The source in the text should be referred to as follows: (Ducrot, 1988). Please quote the page for a: (Foucault, 1551, p. 57). If there are three authors, please refer as (Bradbury Boyle and Morse, 2002) or (Taylor et al„ 1578) for four or more authors. For the works of an author that were published in the same year, distinguish between them by adding small letters (a, b, c, etc.), e.g. (Bourdieu, 1556a). Repeatedly cited works should use the following: (ibid). Please, use the following style for each of publication: Books: Bradbury, I,, Boyle J., and Morse, A. (2002) Scientific.Principles for Physical Geographers. Harlow: Prentice Hall. Garber, M. (1555) Symptoms of Culture. Harmondsworth: Penguin. Journal .Articles: Kerr, D. (1555b) Changing the political culture: the advisory group on education for citizenship and the teaching of democracy in schools. Oxford,Review ofEducation 25 (1-2), pp. 25-35. Book chapters: "Walzer.M. (1552) The Civil Society Argument. In: Mouffe, Ch. (ed). Dimensions of Radical'.Democracy: Pluralism, Citizenship and, Community. London: Routledge. Websites: http:/Avww.cahiers-pedagogiques.com/article.php3 ?id_article=88i (5.5.2008). Šolsko polje. Mestni trg 17,1000 Ljubljana; tel.: 014201240, fax: 014201266, e-pošta: info@theschoolfield.com; eva.klemencic@pei.si Šolsko polje. Slovensko društvo raziskovalcev šolskega polja. Mestni trg 17,1000 Ljubljana, e-pošta: eva.klemencic@pei.si; tel .: 0142012 53, fax: 01420 12 66 Šolsko polje Revija za teorijo in raziskave vzgoje in izobraževanja Letnik XXV, številka 5-6, 2014 EDITORIAL/UVODNIK Mojca Straus - The Timeless Questions About Educational Quality PAPERS/RAZPRAVE Darko Strajn - The PISA Syndrome: Can we Imagine Education without Comparative Testing? Urska Stremfel * Slovenia on its Own Way Towards Improving PISA Results Christine Sàlzer and Manfred Prenzel - Looking Back at Five Rounds of PISA: Impacts on Teaching and Learning in Germany Pierre Brochu - The Influence of PISA on Educational Policy in Canada: Take a Deep Breath Maria Stephens andAnindita Sen - Comparing U.S. States' Mathematics Results in PISA and Other International and National Student Assessments Ana Kozina and Ana Mlekuz * The Predictive Power of Attribution Styles for PISA 2012 Achievement: International and National Perspective Mojca Straus - (Inequalities in PISA 2012 mathematics achievement, socio-economic gradient and mathematics-related attitudes of students in Slovenia, Canada, Germany and the United States CENA: 10 EUR ISSN 1581-6036 <"1581 "6030" >