Organizacija, Volume 40 Professional papers Number 6, November-December 2007 Project CALIBRATE -Calibrating eLearning in Schools Vladimir Batagelj, Iztok Kavkler, Matija Lokar Univerza v Ljubljani, Fakulteta za matematiko in fiziko, Jadranska 19, Ljubljana, Slovenija Vladimir.Batagelj@fmf.uni-lj.si, Iztok.Kavkler@fmf.uni-lj.si, Matija.Lokar@fmf.uni-lj.si One of the main issues that the European Union supports through the IST Programme is the exchange and collaborative use of learning resources. CALIBRATE (Calibrating eLearning in Schools) brings together eight European countries to carry out a multi-level project designed to support the collaborative use and exchange of learning resources in schools. Its main aim is to provide a brokerage system among national repositories of educational materials. This paper reports on the main goals of this project, which include developing an open-source technical architecture to support content exchange/collaboration between ministries of education and other owners of educational repositories, to develop a teaching toolbox that supports the collaborative use of learning resources, research and testing new approaches that can improve semantic interoperability related to the discovery and evaluation of learning resources. One relatively important issue developed through the project concerns the guidelines with which the metadata resources in the repositories should be equipped. We will report on two major guidelines the resources in CALIBRATE should follow. A number of practical examples of preliminary versions of tools will also be outlined. Key words: e-learning, metadata, repositories, learning resources Projekt CALIBRATE Izmenjava in sodelovalna uporaba u~nih vsebin sta eni temeljnih vodil, ki ju podpira Evropska Unija skozi programe IST Projekt CALIBRATE (Calibrating eLearning in Schools) povezuje osem evropvskih dr`av v ve~nivojskem projektu, katerega na~rt je podpreti sodelovalno uporabo in izmenjavo u~nih gradiv v šolah.Njegov glavni cilj je pripraviti izmenjevalni sistem med nacionalnimi skladiš~i izobra`evalnih gradiv. ^lanek poro~a o glavnih ciljih projekta, med katerimi so razvoj odprtokodne arhitekture, ki bo podpriala izmenjavo/sodelovanje med ministrstvi za šolstvo in drugimi lastniki skladiš~ izobra`evalnih gradiv, razvoj u~ne orodjarne, ki bo podpirala sodelovalno uporabo u~nih gradiv, raziskave in preizkupanje novih postopkov na podro~ju odkrivanja in ocenjevanja u~nih vsebin. Eden od pomembnih rezultatov dose`enih med projektom so smernice o tem, katere metapodatke naj imajo gradiva v skladiš~ih. Poro~ali bomo o dveh glavnih smernicah, ki naj jim gradiva v projektu CALIBRATE sledijo. Predstavili bomo tudi nekaj razvojnih razli~ic orodij. Klju~ne besede: e-u~enje, metapodatki, skladiš~a gradiv, u~na gradiva 1 Introduction Despite the fact that there are numerous learning resources available on the World Wide Web (WWW), there is a constant problem of finding appropriate learning resources to use in the classroom. The CALIBRATE (Calibrating eLearning in Schools) project is aimed to help with this challenge. It will provide solutions (tools and services) that support the interoperability and adaptability of learning resources on the European scale. The project brings together eight European countries to carry out a multi-level project designed to support the collaborative use and exchange of learning resources in schools. Its main aim is to provide a brokerage system among the national repositories of educational materials. In this paper, we will present the main goals of the CALIBRATE project, discuss some problems in adapting educational resources for international use and present a number of illustrative examples. 2 About the project CALIBRATE is a 6th Framework project led by European Schoolnet (EUN) that brings together eight countries that are interested in supporting the exchange and collaborative use of learning resources in schools. With 279 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 support from the European Commission’s Information Society Technologies (IST) Programme, the project is expected to develop a European Learning Resource Exchange for teachers and pupils. The work builds on the results of three previous IST projects (CELEBRATE, ITCOLE and VALNET). Alongside eight ministries of education (or their representatives), the project also involves 17 partners from Austria, Belgium, Czech Republic, Estonia, Hungary, Lithuania, Poland and Slovenia. As the project proposal states, the project will ¦ develop an open-source technical architecture to support content exchange/collaboration between ministries of education and other owners of educational repositories; ¦ develop an open-source, toolbox of learning that supports the collaborative use of learning resources; ¦ research and test new approaches that can improve semantic interoperability relating to the creation and evaluation of learning resources; and ¦ validate CALIBRATE project results in up to 100 schools using an advanced validation methodology. The CELEBRATE (Context eLearning with Broadband Technologies) pilot project has demonstrated that, although teachers are willing to use learning materials from the WWW, this usage is seriously limited by the fact that such materials are insufficiently available; in particular, not enough areas are covered. The main goal of CALIBRATE was to alleviate this shortcoming: to make the resources sufficiently available and easily accessible to teachers. For this purpose, the project brought together the ministries of education from eight EU member states, each with their own repository of learning materials. Six of the participating countries are new EU member states, the choice of which was a conscious decision with the goal of promoting educational uses of ICT (Information and Communication Technology) in the enlarged Europe. One partner that took part in the project, representing its Ministry of Education, was the University of Ljubljana (through its Department of Mathematics, Faculty of Mathematics and Physics, Laboratory for Telecommunications, and Faculty of Electrical Engineering). The CALIBRATE project started in October 2005 and will run until March 2008. The idea of this project is relatively simple, although the underlying technology is quite advanced. Essentially, the project will create a network or federation of learning-content repositories. The network will allow teachers and pupils to freely access both learning resources and learning assets (images, simple text files, audio clips, etc.) that can be used in the classroom. The search for these assets should be performed in a way teachers are already familiar with and in their own language. Technology behind it will take care of producing and federating appropriate searches through participating repositories. 3 Finding educational teaching resources A relatively substantial amount of teaching and learning resources is obtainable through the Internet. If we take mathematics as an example, sites such as http://www. mathcentre.ac.uk, http://www.e-um.si/, http://www.mathe-online.at, http://www.matheprisma.de, http://www.math. com and http://planetmath.org are just a few instances of various portals where educational maths teaching resources can be found. A simple search on Google with the keywords maths, teaching, resources provides more than 6.5 million hits. However, a survey conducted in 2005 within Slovene teachers’ groups has shown that teachers do not use enough of the resources that are available. The main reason they claim for not using teaching resources is that they need suitable teaching resources in order to be able to incorporate information technology into their teaching processes. The observation of the active usage of e-materials in teachers’ work has shown that access to a large number of e-materials (mostly due to the organisation of the Slovenian portal (SIO) with links to various educational resources in 1995) after the (expected) initial boom actually led to the reduction of their use in teaching. Interviews with teachers showed several reasons that led to the observed decline in usage. The quality of electronic teaching materials (Dinevski et al, 2006), problems with distribution of the materials, difficulties in modifying them and the lack of proper classification are only some of those most often mentioned as being the most important reasons for giving up or stopping such usage (Jakon~i~-Faganel and Lokar, 2006). On the other hand, the use and preparation of suitable teaching materials in Slovenia is relatively challenging. A small population (two million) and the obligation to use only native language materials in schools mean that the market is small and the number of potential authors limited. We therefore have an overwhelming amount of teaching resources in other languages and a lack of suitable ones in the Slovene language (Lokar, 2006). The same problems were also observed in other countries. Problems with quality suggest that the use of national repositories with quality evaluation and/or control may be necessary. But this would mean a significant drop in available resources all over Europe. Thus, an obvious remedy of this situation is to connect repositories and search through all of them. As soon as we try to reduce the number of possible hits, we encounter problems with proper keywords, different age settings in different school systems across Europe, etc., and therefore, the reasons for initiating the CALIBRATE became clear. 4 Main goals of CALIBRATE CALIBRATE has been set several goals according to the most pressing needs established by the CELEBRATE project. Based on these goals, five work packages were formed. Classification of teaching resources. School programmes (curricula) in European countries differ significantly, but they mostly cover the same set of topics. For learning resources to be easily accessible, there has to be a way to 280 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 search for them based on the curriculum in each teacher’s own country. That is, teachers cannot be expected to be familiar with the large variety of school systems across Europe. The task of the first work package is therefore to find the mappings between curricula in several states and create a taxonomy - a set of keywords - with unique mapping to each curriculum. Learning resource exchange (LRE). As mentioned before, the initial availability of resources is crucial for the success of the project. Since most countries already have their own repositories of learning materials, the natural step is to bring them all together. Most learning resources are accompanied by metadata, which contain a description and data intended to simplify searching. The goal of the LRE is to enable the exchange of this metadata. For this purpose, an international standard format, LOM (learning object metadata), is used; this has been adapted in several minor points to better fit LRE needs. The metadata in the repositories are generally not stored in LOM format. Therefore, every participating country has to provide an interface for searching and converting its metadata to LOM. The EUN office provides the central brokerage system, the necessary federated search system through all the national repositories, as well as basic software support to simplify the connection to the brokerage system. The initial goal was to have all the metadata available in English. Unfortunately, this is generally not the case. Other projects (e.g. Metadata Ecology for Learning and Teaching (MELT)) have the specific goal of enriching the metadata and providing English translations. LeMill: a toolbox for adapting and producing teaching resources. To help teachers adapt the learning material to the specific needs of the class and share the adapted content with others, a web-based authoring system named LeMill is being developed. The design and adaptation of learning objects in LeMill is based on the cooperative model of work: the materials are stored on the web server, and every update is instantly available to the whole community. All the resources in LeMill are covered by the Creative Commons share-alike licence (Creative-Commons (CC-SA licence)), which fits such purposes very well. One of the problems that arose is the import of resources from national repositories to LeMill. This is an important part of the CALIBRATE integration, but it requires the imported resources to carry the same CC licence, which is often not the case in the national repositories (they are mostly older than the CC licence). Despite this, the cooperative model has several advantages, especially considering the international aspects of CALIBRATE. A suggested common scenario is the following: a Slovene teacher finds a resource in the Hungarian repository using the federated search. Based on the description (hopefully in English) in the metadata, she decides that she could use it for her classes. If the resource carries the CC-SA licence, she can import it into LeMill. She connects with a Hungarian teacher from the LeMill community with a request to translate the resource from Hungarian to English (or another language with which they are both familiar). As the last step, she translates it to Slovene and adapts it for the Slovene curriculum. The next time, the situation is reversed - and therefore everybody wins. As a side effect, the quantity of content available in mediatory languages (e.g. English) grows, so no work needs to be duplicated the next time. Evaluation. The evaluation of the project by a test group of teachers is important, partly to understand the strong and the weak points of our approach, but mainly to create the strategy of how to disseminate the finished product to as wide a group of teachers as possible. The evaluation is currently running in 20 schools in four states. A set of expertly prepared questionnaires will be used to judge the results. Dissemination. The goal of the project is not only to produce the tools, but also to educate the teachers on how to use them and how to create new resources. We hope to reach the “critical mass” of teachers, which will enable further use to spread on its own. 5 Classification and metadata The main goal of the CALIBRATE project is to help teachers to find and use teaching materials not only from their own country’s repositories but from all over Europe. The first question that usually arises is: why should they use CALIBRATE in place of Google? Almost all resources available are already indexed by Google, but it is difficult for a text-based search engine to differentiate between a teaching resource about “linear function” and an article that just happens to mention “linear function” in the context of, say, the economy. Then there is the issue of language - standard search engines find many English resources provided that the query is in English. But what about finding an Estonian resource with a Slovene query? In educational repositories, however, metadata is augmented with educational information, with quality evaluations, with copyright information, etc., and therefore, the search can be much more “educationally oriented”. CALIBRATE helps with these issues by using metadata along with the resource. Metadata contains a lot of information that helps the search: a simple description, language-neutral keywords (defined by an international thesaurus), the approximate age at which the subject is taught, etc. A great deal of metadata already exists in the national repositories, and missing pieces are currently being added to them by other projects such as MELT. An idea that seems to have even more potential is that of curriculum mapping. This is still far from finished, but when operational, a teacher can simply click on the day’s lecture in the school programme and obtain all related resources throughout Europe. With this approach, a teacher can search through repositories in his own language, but the results he obtains are often in a language he does not speak. We hope the community process as described in the paragraph about LeMill will help here. In the future also, solutions based on controlled languages (see Mitamura, 1999, for exam- 281 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 ple) could find a place in the development of learning materials. 6 Adapting educational teaching resources One of the important findings during the evaluation is that resources that are mostly text-based do not fit very well with the needs of CALIBRATE. There are several reasons for this: ¦ These resources do not “travel well” - they have to be translated into every language to be of any use, and the amount of work needed for translation is in many cases nearly equal to the amount of work needed to create the resource from scratch. ¦ Teachers have pointed out that the most important benefit they expect from the use of ICT is a way of motivating their pupils - and a text-filled page is just as boring as a schoolbook. ¦ Almost every school topic is already covered well by Wikipedia, so most teachers will first look there for textual content. So what are the resources that best fit the philosophy of CALIBRATE? Well-known examples are short movies, simulated experiments and interactive exercises -these contain little text to translate and produce a good motivational effect. Moreover, interactive exercises provide instant feedback, providing the pupil with a better opportunity to learn from his own mistakes while relieving the teacher of the tedious task of correcting. In our opinion, the heart of the problem is the poor ability to exchange content, as CALIBRATE is all about exchange. The materials in CALIBRATE must work on different computers in different countries and different languages. The teacher has to be able to cut them, translate them and adapt them to this year’s curriculum and today’s lecture. 7 The validation framework Evidence about the usefulness of the products is gathered from the schools in two broad areas: the performance of the product itself; and its general value and potential for supporting new models of pedagogy and schooling, content management and organisational change. A common framework for assessing the impact of interventions in school has been made. As can be seen (Calibrate Newsletter 1), it has several dimensions: Systemic dimension. This concerns aspects of the national schooling system. These aspects, for example, education policy, teacher education, the legal context, the curriculum and external examinations, are largely outside the control of the individual school. Nevertheless, they affect what happens in schools. The project will study results from the other five dimensions and highlight implications resulting from the project that should be communicated to policy-makers. Institutional dimension. Here we look at the changes brought about by the CALIBRATE project in the school as a whole: changes in teaching, collaboration and sharing good practice, as well as changes in the school’s ICT culture and its e-maturity Pedagogical dimension. This aspect covers teaching and learning at the individual teacher, student and class levels. The main observation regarding teachers will focus on the impact digital learning resources have on the practice of classroom teaching, mostly on teaching strategies and teaching styles. We will be looking at the methods teachers use to author, adapt, modify and share learning resources. In the area of learning, we will examine four disciplines: Mathematics, Science, Environmental Science and English as a Foreign Language. Technology dimension. This part mainly deals with how the project results work in practice.This concerns the technical performance of project outputs in real situations. Implications for ICT resourcing in the school, localisation issues, technical support, integration with existing ICT technology and adaptability to the school’s needs are examined. Economic dimension. This component brings together financial issues, e.g. purchase, set-up and usage costs, wider costs incurred and scalability. Cultural dimension. This is an important but often neglected aspect. It covers issues related to different political and educational cultures across European countries and specific linguistic concerns. Schools will report on cultural aspects of localisation, support for multiple languages and intercultural and trans-national collaboration. 8 Some practical examples In this section, we present some preliminary snapshots of CALIBRATE and LeMill portals, illustrating principles and features mentioned above. The main method of searching and conducting the CALIBRATE LRE will be through national portals. However, as interfaces are still in the development stage and there has been a need to evaluate federated searches, a language-dependent entry point for searching the resources through all connected repositories has been made (Picture 1). When a user tries to search for a certain resource, a search is performed through all interconnected resources. Results are joined and displayed regardless of the language in which the keyword, topics, age group, etc. has been entered in a particular repository. At the present moment, searches can be made on the basis of keywords, subject, age group and language of the resource, but there are no obstacles to developing much more elaborate search interfaces. For each resource found, the user can be shown relatively elaborate metadata, as can be seen in Picture 2. As mentioned before, an important part of the CALIBRATE project is the learning toolbox LeMill (Picture 3). LeMill is a web community for finding, authoring and 282 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 Picture 1: CALIBRATE search entry point sharing learning resources. First of all, the user can find a variety of learning resources. She can then use the resources she finds in her own teaching or learning. She can also add her own learning content to LeMill. She may edit her content and combine larger chunks of learning resources from individual media pieces. If she wishes, she may also join available groups producing or editing learning resources. One of the most important features of LeMill is the idea of the community-performed translation of educational materials. Let's look at a simple scenario. A teacher performs a search through the CALIBRATE LRE (Picture 4) and finds an interesting resource about linear function that has been already uploaded to the LeMill repository (Picture 5). Unfortunately, it is in an Estonian language with which he is not familiar. As a result, he searches through the LeMill community to find a member who speaks Estonian and is willing to translate the resource into a language with which they are both familiar (Picture 6). The benefits of this approach are not only in the number of appropriate resources in native languages but also in the establishment of a community of EU teachers. 9 Conclusion It is expected that the outcomes of the CALIBRATE project could help EU teachers and pupils in their search for suitable electronic teaching resources as well as creating a solid foundation of open-source tools for future repositories. We should not forget, however, that CALIBRATE and similar projects provide only a framework, which each country and partner involved should make the most 283 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 Picture 2: CALIBRATE metadata cz I en I es I et I fi I fr I hu I ka I It I pi I ru I se Log out LeMill Media Computer ¦ From Math to... Study Project / Project Learning Fle3 Web community far finding, authoring and sharing Learning resources What's going on? New to LeMiLL? Take a tour or read FAQ Teaching and (earning story / collection: Brief History of New Media New Media is a multidisciplinary and multiperspective field. The course is designed for University students coming with different backgrounds. The course intendes to generate a share understanding between different professionals of some of the phenomena related to New Media from a historical Tread more...] Find Learning content, method, tooL or people... Go Picture 3: LeMill 284 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 Picture 4: An interesting resource has been found of through systematic work on the production of electronic teaching resources and on encouraging the appropriate use of it - something which we in Slovenia in the last few years have seriously lacked. If the repositories are not professionally maintained with sufficient support staff, and if there is no systematic action on ensuring quality, even the best tools and guidelines will not help it to move forward. 10 Acknowledgments The work presented in this paper is partially supported by the European Commission under the Information Society Technologies (IST) programme of the 6th FP for RTD as part of the CALIBRATE project, contract IST-28025. The authors are solely responsible for the content of this paper. It does not represent the opinion of the European Commission, and the European Commission is not responsible for any use that might be made of data appearing herein. 11 References Batagelj V. & Kavkler I. (2007). Calibrate and the new Trubar. In: Bohte, U., Vre~a., M. (eds.), Proceedings of the International Conference Enabling Education and Research with ICT. Kranjska gora, April 19th - 21st, 2007. Ljubljana, Ar- 285 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 Picture 5: Tool for translation nes. Available also from http://www.arnes.si/sirikt/pred-stavitve/sobota-9-20-kavkler.pdf (accessed 21.6.2007). Calibrate Newsletter 1, 2007, Calibrate Newsletter, Issue No.l, February 2007, available from http://calibrate.eun. org/shared/data/calibrate/Calibrate newsletter 1 A4.pdf, (accessed 21.6.2007). Dinevski D., Jakončič-Faganel J, Lokar M. & Žnidaršič B. (2006). Quality assessment of electronic learning material, KMO'2006 : knowledge management in organizations. In: Heričko, M., Živkovič A. (eck), Proceedings of the First International Conference for New Trends in Knowledge Management. Edited by Maribor, June 13th - 14th, 2006. Maribor, Faculty of Electrical Engineering and Computer Science, Institute of Informatics. Gerlič I. (2003). Stanje in trendi uporabe računalnika v slovenskih osnovnih in srednjih šolah - raziskovalno poročilo 2003 (in Slovenian), available from http://www.ris.org/ main/baza/baza.php?bid=295 (accessed 21.6.2007). Jakončič-Faganel J. & Lokar M. (2006). Quality of electronic materials for math teaching. In: Quinney, D. (ed.), Proceedings of the 3rd International Conference on the Teaching of Mat- hematics at the Undergraduate Level. Istanbul, June 30th -July 5th, 2006: Istanbul, John Wiley & Sons. Lokar M. (2006). Electronic teaching and learning resources in math teaching in Slovenia. In: Quinney, D. (ed.), Proceedings of the 3rd International Conference on the Teaching of Mathematics at the Undergraduate Level. Istanbul, June 30th - July 5th, 2006: Istanbul, John Wiley & Sons. Mitamura T. (1999). Controlled language for multilingual machine translation. http://www.lti.cs.cmu.edu/Research/Kant/ PDF/MTSummit99.pdf (accessed 21.6.2007). Sources Celebrate, EU Project Context eLearning with Broadband Technologies (CELEBRATE), accessed at http://cele-brate.eun.org/eun.org2/eun/en/index celebrate.cfm (accessed 21.6.2007). Calibrate, EU Project Calibrating eLearning in Schools (CALI-BRATEVhttp://calibrate.eun.org/ww/en/pub/calibrate proj ect/home page.htm (accessed 21.6.2007). CreativeCommons, http://creativecommons.org/ (accessed 21.6.2007). 286 Organizacija, Volume 40 Professional papers Number 6, November-December 2007 Join LeMill chat. -^ New group My groups History of New Media My contacts Has no contacts Featured people Thieme Hennis VASIKO MARDALEISHVILI Cerane Koväcs Ildikó Browse_ people, By language: English, Estonians German, ,,, ¦ By name cloud ¦ By location: Estonia, Hungary, No country specified, ,,, ¦ By skills: teaching, matematika, research, ... ¦ By interests: music, ovas, travelling, ,,, ¦ By subject area: Informatics or ICT, Foreign languages. Mathematics, ,,, Browse groups ¦ By language: English, Hungarian, Estonian, ,,, ¦ By name cloud ¦ By tags: calibrate, school, hungary, ,,, ¦ By subject area: Informatics or ICT, Cross-curricular education. Culture, Recent forum threads in my groups Picture 6: Searching for translators LeMill, http://lemill.org/trac/ and http://lemill.net/content/ overview-of-lemill (accessed 21.6.2007). MELT, EU Project Metadata Ecology for Learning and Teaching, http://info.melt-project.eu/ww/en/pub/melt project/ about.htm (accessed 21.6.2007). SIO, Slovensko izobra`evalno omre`je - Slovenian Educational Network, http://sio.edus.si (accessed 21.6.2007). Vladimir Batagelj is Professor of Discrete and Computational Mathematics at the Faculty of Mathematics and Physics, University of Ljubljana. His main research interests are in mathematics and computer science, combinatorics with emphasis on graph theory, algorithms on graphs and networks, combinatorial optimisation, algorithms and data structures, cluster analysis and applications of information technology in education. Iztok Kavkler is a researcher at the Faculty of Mathematics and Physics, University of Ljubljana. His main research inte- rests are in programming language theory, programming and operator theory. Matija Lokar is employed at the Faculty of Mathematics and Physics, University of Ljubljana, as one of the heads of its computer centre and as a senior lecturer. He is author of several books and papers in the field of computer science and on introducing information communication technology into teaching processes. 287