Lifelong Learning Programme as a Mechanism of Change at the National Level: The Case of Slovenia
Klemen Širok Katarina Košmrlj
European Commission (ec) funding programmes in the field of education and training present supranational policy instruments bringing change to various levels of social reality at the national level. In an attempt to present the holistic view of their effect, this paper presents the results of the Lifelong Learning Programme (llp) impact evaluation, which consisted of a mixed method research approach combining a focus group discussion method, interviews, content analysis, and survey research. Results show that ec education and training funding programmes do bring change to the national (system), mezzo (organisational), and micro (individual) levels. Yet, the impact seems to be different at the observed levels as well as in the different target domains, and is weakest at the system level. This prompts the question about whether it makes much (economic) sense to exploit ec programmes as mechanisms of national policy goal implementation.
Key Words: social change, ec programmes, impact, evaluation,
education jel Classification: 125, D04
Introduction
The European education and training funding programmes have been present in the Slovenia education system since 1999. When the Erasmus, Comenius, Socrates, and other actions within Leonardo da Vinci programmes were launched between 2000 and 2006, more than 3600 teachers and mentors, 4000 students, 3000 pupils and 500 organisations participated in Slovenia. These figures are increasing with the Lifelong Learning Programme (llp), which is replacing the aforementioned programmes. In 2006, the European Commission (ec) proposed to the Parliament to integrate its various educational and training initiatives under
Klemen Širok is a Senior Lecturer at the Faculty of Management
Koper, University of Primorska, Slovenia.
Katarina Košmrlj is a Research Assistant at the Faculty of
Management Koper, University of Primorska, Slovenia.
Managing Global Transitions 10 (4): 379-398
a single umbrella, known as the Lifelong Learning Programme, 'with a significant budget of nearly € 7 billion for 2007 to 2013; the new program replaces previous education, vocational training and e-Learning programs, which ended in 2006' (European Commission 2009,1). Considering the scope of these programmes, the question immediately arises: What type of change (impact) would these supranational policy instruments bring to various levels of social reality at the national level? Due to the LLP sub-programmes' and actions' nature we assume, that the change will predominantly affect individual llp participants. Lesser impact is expected at institutional and national level.
We address this problem by presenting information on the impact made by the llp, which represents the next generation of ec action programmes in the field of education and training in Slovenia. We present two aspects of the change that the llp has brought to Slovenia. The first aspect is the scope of llp implementation, and the second aspect is the change itself the llp has brought to Slovenia as shown through the results of the impact evaluation which was carried out between December 2009 and February 2010. The evaluation covered the llp implementation during the period from 1 January 2007 through 31 December 2009.
We consider change to be a programme impact manifesting itself at various levels of social reality: macro level (i. e., education system), mezzo level (i. e., institutional), and micro level (i. e., individual). Since it is impossible to observe change in all its manifestations, we observe the llp impact across different pre-selected programme goal areas (i. e., substantive areas) determined by LLp programme goals and evaluation stakeholders. This article also mitigates deficiencies in empirical research in this field. There are many available evaluation studies (e. g. Bracht et al. 2008; McChosan et al. 2008; Association for Empirical Studies in Centre for Research into Schools and Education at the Martin Luther University 2007; Širok et al. 2007; Ernst & Young 2006; Centre for Strategy and Evaluation Services 2004; European Commission 2004; Deloitte & Touche 2001), but they predominately deal with either specific subprogrammes (for example, Erasmus), or are plagued with considerable deficiencies that hinder proper conclusions on the impact of the programme (Širok and Petric, forthcoming). Most of these evaluations seek to determine the quality of sub-programme/actions processes and procedures, including the effectiveness and/or efficiency of implementation. Their evaluation focus is thus mostly on programme outcomes, while programme impact is very rarely the objective of analysed evaluation
studies and rarely quantitatively measured (see Bracht et al. 2008; Association for Empirical Studies in Centre for Research into Schools and Education at the Martin Luther University 2007).
The analysis shows that llp represents a supranational policy mechanism, bringing important change to different levels of social reality at the national level, predominantly at the individual level. First, data indicate high levels and high quality of llp implementation in Slovenia. Second, llp objectives are found to be relevant for, and influential within, the national policy priorities. However, considering the relationship between the llp and national education policy, the relevance of the llp objectives with regard to national priorities is evident, not as an unadulterated llp impact, but rather as the extended impact of eu education policies at the national (system) level. Third, the comprehension of where and how national education policy documents overlap and relate to the eu goals and priorities are often being relegated to the implementation level. At the institutional and individual levels, llp importantly contributes to the development of numerous competencies as well as to initiation of interpersonal cooperation, but predominantly at the lower cooperation intensity levels. The recognition of common cooperation goals and intentions among individual llp end-users is also only weakly present, or not present at all. No significant impact in the dimension of personal growth in individual end-users was possible to observe, while the same respondents viewed llp as contributing to an individual's employability.
The text is organised as follows. The next section provides a short overview of the scope of llp implementation in Slovenia. The third section presents the research approach used to measure and explain the llp impact in Slovenia, as well as the evaluation data sources we utilised. The forth section presents empirical evidence on the llp impact in Slovenia. We conclude the paper by debating the role of a supranational policy mechanism at the national level.
ec Funding Programme as a Mechanism of Change and llp Implementation in Slovenia
ec programmes are financial mechanisms introduced to trigger change in the eu member states and broader, and, therefore, on the national level, as well. In this section, we present the main characteristics of these programmes and their implementation at the national level in Slovenia.
The European Communities' Funding Programmes are promoting changes, agreed upon from the supranational level to the national level,
by supporting the development of common policies and activities in the field of education and training. The Lifelong Learning Programme (European Parliament and the Council 2006) is the largest programme for community action in the field of lifelong learning. Its general objective is 'to contribute [...] through lifelong learning [...] to the development of the Community as an advanced knowledge-based society' (art. 1, sec. 2). Historically, there are several reasons why the European programmes were created and developed and, consequently, why they set education and training as the key element to unite Europe and its people. These programmes sought to unite the European region, to improve labour market mobility, to introduce various novelties into the education systems, mainly tools for transparency and raising quality, to improve the continent's competitiveness in relation to other continents, and to enable interconnection, comprehension, and understanding (Pepin 2006).
In the second half of the 1980s, the first programmes from the field of education and training were implemented (Comet, Erasmus, Petra, Lingua, Eurotecnet, etc.) and were designed with the political aim of narrowing the gap between the eu and the us, in certain key areas. With the development of a common European market, focus shifted to the recognition of professional diplomas and, later, away from emphasising harmonisation to emphasising mutual trust and comparison in the field of vocational education and training and tertiary education. Here, the Socrates and Leonardo da Vinci programmes, and more or less all actions under their umbrellas, played an important role. In the 1990s, the concepts of a knowledge-based society, lifelong learning, and quality became increasingly well-known. Since 2000, these concepts have become the pillars of further development of the European Union (European Commission 2011). During the same period, lifelong learning has gradually emerged as a principle. Since March 2000, when the European Union adopted a new economic, social, and environmental strategy, known as the Lisbon strategy, education and training have surpassed labour as the most important principle, in order to build knowledge throughout Europe. During this period, with the aid of the Socrates 11 and Leonardo da Vinci 11 programmes, politicians attached great importance to the eu education and training system. In 2002, the European Parliament stressed that the content of education systems should not be determined exclusively by references to the economy and the labour market; rather, it should be to develop awareness of one's citizenship, one's communication capability, and one's intercultural awareness and
social skills, yielding to the enriched focus and role of the recent llp programme.
Although the ec funding programmes, in the area of education and training, are considered to be supranational mechanisms used to introduce change at the national level, it has to be noted that they are, and have been, predominately decentralised, meaning that implementation has been left to the state and national levels. The European Commission and Council did not wish to harmonise Europe, but rather it tried to establish mutual trust and identification of national systems. That is why certain actions remained in the domain of a particular country, which has the possibility to implement an action in accordance with the needs of its education system, because the European Commission plans and sets those actions only in certain basic aspects (common priorities, use of funds, etc.). In the new programming period (2007-2013), llp programmes are also divided into decentralised and centralised activities. The difference between these is that the implementation of decentralised activities (mobility, partnerships, projects for transfer of innovation, and study visits) is carried out by national agencies (na). In Slovenia this na is known as cmepius, whereas in Brussels, the executing agency implements all centralised activities. In contrast to prior periods, today, as much as 80% of the programme's resources are earmarked for decentralised activities.
The ec education programmes have been extensively present in Slovenia for over a decade and their presence has gradually grown, both in terms of programme end-user numbers as well as in financial terms. From 2000 to 2006, more than 3600 teachers and mentors, 4000 students, 3000 pupils, and 500 organisations participated in the following European education and training funding programmes in Slovenia: Erasmus, Comenius, Grundtvig and other actions of Socrates and Leonardo da Vinci programmes. For these purposes, the European Commission granted Slovenia € 22,754 million (Center Republike Slovenije za mobilnost in evropske programme izobraževanja in usposabljanja 2007), before the previously mentioned programmes were consolidated in 2007 under the llp (European Parliament and the Council 2006). The programme is continuing to grow in terms of financial viability. Funds for llp implementation in Slovenia rose from 6201 (thousand Euros) in 2007 to over 6744 in 2008 and 7533 in 2009 (Flander 2010, 12). From 2007 onward, the level of financial realisation has remained high at approximately 95% (Flander 2010, 12), making it dif-
ficult to improve. The number of llp participants has grown as well. Table 1 presents the number of submitted and approved projects and the yearly increase in new applicants (end-user growth). The observed participation dynamic can be explained by several factors; however, according to cMEPius management, the most evident reason for any applicant numbers decrease in certain actions or subprogrammes lies, in changes to the national rules of application that have been introduced within the llp.
Methodology
We consider change to be a programme impact manifesting itself at various levels of social reality: the macro level (i. e., education system), the mezzo level (i.e., institutional), and the micro level (i.e., individual). This evaluation study treats impact as long-term and sustainable changes introduced by a given intervention in the lives of beneficiaries, related to the specific objectives, an intervention, or unanticipated changes caused by an intervention (Blankenburg 1995; Weinwright 2003), and classifies it as an ex-ante impact assessment evaluation (Rossi, Freeman, and Lipsey 2004). The major difficulty with this evaluation type is its inability to validly assess programme impact (ibid.), which also plagues the majority of other ec education programme evaluations (Širok and Petric, forthcoming). In order to minimise this deficiency within the constriction of our evaluation context, three research strategies were applied: a mixed method research approach, a quasi-experimental design in survey research, and data source triangulation.
The first research strategy, a mixed method research approach, was selected in order to draw valid evaluation conclusions of the programme's impact at the macro level, combining a focus group discussion method (Krueger 1994), interviews (Foddy 1994), content analysis (Babbie 2007), and survey research. The envisaged use of various research methods is a necessity originating from the need to combine different data sources and the perspectives of llp stakeholders.
The second research strategy addressing the limitations of evaluation context was addressed by the quasi-experimental design in survey research, which was applied in order to validly measure change on the mezzo and micro levels (the micro-impact of the macro level policy programmes). Impact was operationalized as a latent variable, partitioned in areas where latent, sustainable programme consequences are anticipated or reasonably expected. This partitioning resulted in a list of theoretical
table 1 llp Implementation in Slovenia: Applicants and End-Users
na yearly reports data	2007			2008			2009	
	(1)	(2)	(1)	(2)	(3)	(1)	(2)	(3)
Comenius multilat. partnerships	184	76	134	47	-27	113	56	-16
Comenius bilateral partnerships	24	3	24	7	0	15	4	-38
Comenius in-service training	86	69	118	65	37	200	56	69
Comenius assistants	30	9	12	12	-60	51	10	325
Comenius host schools	52	11	42	9	-19	30	11	-29
Comenius preparatory activities	38	37	41	38	8	23	22	-44
Comenius Regio partnerships						9	4	
Erasmus mobility - euc	28	28	29	29	4	8	8	-72
Erasmus mob. - stud. and hei staff	1910	1469	2312	2041	21	2454	2130	6
Erasmus mobility - others						35	34	
Erasmus Preparatory Visits						5	3	
Erasmus intensive programmes	6	5	13	5	117	13	8	0
Erasmus Intensive Language Courses						6	4	
LdV Mobility	83	49	81	56	-2	76	42	-6
LdV Partnerships			42	13		35	13	-17
LdV Transfer of innovation	33	7	38	7	15	23	6	-39
LdV Preparatory visits						20	15	
Grundtvig learning partnerships	36	9	48	10	33	33	14	-31
Grundtvig in-service training	29	11	45	19	55	27	9	-40
Grundtvig preparatory activities	3	3	6	4	100	10	7	67
Grundtvig assistants						3	1	
Grundtvig visits and exchanges						15	6	
Grundtvig workshops						6	1	
Grundtvig senior volunt. project						1	1	
Study visits - visits	52	18	57	47	10	49	32	-14
Study visits - organizing						5	5	
Total		1804		2409	33.54		2502	3.86
notes Column headings are as follows: (1) applications, (2) signed, (3) yearly applications increase rate (%).
concepts (Meehan 1994) that grasp the impact domain (i. e., cooperation, tolerance, employability), assuming that the action has an impact. Impact scope was determined through ex-post quasi-experimental design,
where the difference between y1 in time after the action (y1t1) and y1 in time before the action (y1t0) was attributed to the participation of beneficiaries in action (x), controlling other factors (x1, x2,...,xn). In our approach, y1t1 - y1t0 was measured by the respondent's subjective evaluation of this difference. Additionally, a measuring instrument was developed in such way that the programme effects cannot be attributed to the (non)participation in action (x), yet x can be treated as the intensity of involvement/participation in an action. Impact was measured as a mean value of a variable, measuring a specific impact domain, either as a difference between two states or as the subjective evaluation of the difference. In the survey, the five-level bipolar rating scale was utilised. The average scores below 3 were interpreted as a negative impact, values around 4, or greater, were interpreted as a positive impact, and values around 3 were interpreted as having no impact (i. e., impact absence) (Širok in Petric, forthcoming).
Since the political programme ambitions outweigh the research abilities and restrictions of evaluation context, this evaluation applied additional strategies to isolate/narrow the llp impact domains of evaluation interest. Taken together, the llp, and all its sub-programmes, follow more than 30 programme goals. Since it is impossible to observe change in all its manifesting forms, we observed the llp impact across different pre-selected programme goal areas (i. e., substantive areas), as determined by llp programme goals and evaluation stakeholders, following the principle of utilisation-focused evaluation (Patton 1996) in the survey phase. The primary evaluation users were, thus, actively integrated into the development and testing phases of the survey questionnaire. Toward this end, the na established a working group consisting of members of the contracting authority and evaluators in order to determine the fundamental premises of the ongoing evaluation. First, the structure of programme goals was determined based upon the llp decision (European Parliament and the Council, 2006). Next, the na coordinators were asked to list up to five goals that captured the essence of every evaluated sub-programme/action within the llp programme. In the next stage, selected goals were examined and anchored to appropriate sociological concepts. Selected concepts were then reviewed by the na coordinators, who added points and areas of interest, such as customer satisfaction and additional contextual variables. Lastly, the 'action-impact domain-measurement level' grid was formed, integrating the appropriate llp impact domain (i. e., employability) with the corresponding pro-
gramme action (i. e., Leonardo da Vinci mobility projects), and measurement level (individual or organisational). The actual impact of the llp on programme end-users has been measured both at the organisational (mezzo) level and at the individual (micro) level within the following llp goal areas: competence, cooperation, networking, and the European dimension. At the individual level, the impact in the area of employabil-ity and personal growth has been additionally assessed, as has tolerance at the institutional level. Since each study should take into account as many relevant control variables as possible, a careful selection of relevant control variables was guided by a strong theoretical understanding of a particular impact domain in cooperation with primary evaluation users.
The use of various data sources and associated research methods enabled both data and method triangulation and, thus, the coverage of relevant stakeholder perspectives, which also provided validity for the evaluation findings. The following data sources were used to achieve data triangulation: documentary sources including reports, accounting data, focus group discussion, interview transcripts, and raw survey data. Two focus groups were organised: one with relevant policy makers in the field of education (nine participants) and one with na management (three participants). Additionally, two interviews were conducted with one representative of each of the aforementioned target groups. Data sources and evaluation findings thus reflect perspectives of the following stakeholder groups: llp end-users at individual and institutional levels, relevant policy makers in the fields of education and training, na management, and llp sub-programme/action coordinators.
The survey was conducted as a web survey. The survey was pilot-tested in May 2009 and then officially conducted between December 2009 and February 2010. All final beneficiaries of evaluated actions, occurring between 2006 and 2009, were targeted and invited to participate in the survey. Their e-mail addresses were drawn from the na records. The target respondents in organisations were programme coordinators. Managers/leaders were not chosen in order to avoid a pro-organisational bias. In total, 123 questionnaires were returned for eight sub-programmes based on organisational participation, representing a 36.5% response rate. Target respondents for the individual surveys were all individuals who participated in llp programmes related to individual mobility. In total, 658 individual questionnaires were returned for nine sub-programmes, representing a 17.4% response rate.
Results
Our empirical analysis is presented below. As mentioned, this analysis consists of two parts; the first part presents the llp impact at the macro level on the basis of qualitative data and the second part presents the llp impact at the mezzo and micro levels by presenting the survey data. The macro-system impact of the llp in Slovenia can be characterised as the transfer of eu guidelines to the national level, missing out on complementarity and coherency, and an instructive moment. Thus, the implementation is primarily guided by a bottom-up approach, as well as by the llp impact. The findings related to the mezzo level and micro level impact confirms and complements the macro level findings, which show the llp impact in observed impact domains, with the exception of personal growth.
llp impact on the national education policy
AND system
Considering the relationship between the llp and national education policy, the relevance of the llp objectives with regard to national priorities is evident, not as an unadulterated llp impact, but rather as the extended impact of eu education policies at the national (system) level. Relevant policy makers within the Slovenian education system expressed relatively coherent perceptions that llp objectives are relevant for and influential within the national policy priorities. However, more than the unadulterated llp impact, the primary policy stakeholders understand the 'llp - national education policy' relationship as the extended impact of the eu education policies. The mechanisms and results of the internationalisation and Europeanization processes imply the context of various national priorities, thus implementing it through the same process. When Slovenia joined the eu, the establishment of national priorities was predominantly stimulated and shaped by eu goals and priorities, despite the principle of subsidiarity. This subsequently led to greater awareness and recognition of the importance of having national priorities, which in turn caused the gradual interweavement of national and programme goals and priorities (Socrates, Leonardo da Vinci, llp) throughout the Slovenian education system. Due to the Bologna process, Erasmus goals and priorities are, for example, fairly close to the goals followed by the Ministry of Higher Education, Science and Technology (Resolucija o nacionalnem programu visokega šolstva Republike 2007-2010).
The relationship between the LLp and programmes in education and training demonstrates a high degree of overlapping, but it misses out on complementarity. Relevant policy makers and the na focus group participants, as well as documentary sources, show that the LLp overlaps but does not complement other national and international programmes, although the straightforward and unequivocal added value is difficult to demonstrate, due to the previously described characteristics of the LLp goals. The LLp does complement national master programmes (Ministrstvo za šolstvo in šport Republike Slovenije 2007; Resolucija o nacionalnem programu visokega šolstva Republike 2007-2010). These policy documents are broad enough to either completely overlap with the llp (Ministrstvo za šolstvo in šport Republike Slovenije 2007) or leave it completely open to interpretation. When considering programmes as financial mechanisms, programmes tend to run separately, although certain stakeholders tend to be aware of opportunities of subject complementarity. This complementarity is hindered by administrative obstacles, which became evident when trying to complement the LLp with national rules on public spending and rules on the European Cohesion Fund. Therefore, complementarity among programmes, as a financial mechanism, is rarely visible (for example the Erasmus grant and national scholarship scheme).
Both aspects of llp complementarity to the national education system show a lack of systematic and coherent policy approach, which in turn leads to situations where primary stakeholders are left on their own to search for opportunities and to achieve their own goals. llp goals indirectly support national goals, yet primary LLp stakeholders face difficulties when trying to follow these goals as guidelines, since they are too broad, too general, and are not prioritized, making them nearly impossible to attain. llp goals indirectly support national goals (Ministrstvo za šolstvo in šport Republike Slovenije 2007; Resolucija o nacionalnem programu visokega šolstva Republike 2007-2010). Despite goal coherence at the eu and national levels, the empirical results showed that all LLp primary stakeholders face the same difficulty when trying to follow llp aims as guidelines. As one focus group member said: 'Everything we found to be important for us to achieve, we also soon found within (eu) priorities.' Consequently, lifelong learning is considered to be a commonly accepted and often misused term, being uncritically transferred from eu to national priorities and is actually not reflected or implemented in reality.
llpimpact on end-users
We observe four different competencies dimensions: individual general competencies (such as research abilities, team work, communication, etc.), independent use of knowledge, general understanding, and developmental orientation. At the institutional level, on average, the subprogrammes and actions exhibit a comparably high positive impact (average score ranging from 4.1 to 4.3) on development of (1) cultural diversity and multiculturalism; (2) understanding of other cultures and their customs; (3) ability to work in an international environment; and (4) project leadership. The competencies on which llp has the weakest impact (average score ranging from 3.2 to 3.3) are (1) learning how to learn; (2) competencies to work with socially disadvantaged groups (lower social classes) or different ethnic backgrounds; (3) cooperation with other stakeholders in the educational process; and (4) ability to read texts in other European languages. On the other hand, individuals perceive high positive impact (average score ranging from 3.9 to 4.3) on what we can call communication and internationalization competencies: (1) speaking European languages; (2) listening to European languages; and (3) understanding other cultures and their customs. At the individual level, the weakest impact (average scores 2.4 and 2.7) is found at (1) entrepreneur-ship competencies; and (2) project management. On average, positive impact on competencies is higher at the institutional level, and weak impact is lower. Individuals, it seems, tend to report fewer differences in impact on competencies. The measured impact of llp on competencies can be partially compared to the impact of preceding programmes, such as the Leonardo da Vinci ii and Socrates ii programmes (Sirok et al. 2007).
When observing the impact on the llp goal of networking, the concept of social capital (Putnam 2000) has been applied. At the institutional level, the following impact aspects of sub-programme have been measured. Bridging social capital was measured through the observation of institutional utilisation of information sources and the tightness of its affiliation to the community. Expansion of social network size and structure was also assessed. At the institutional level, we observed llp having an impact on institutional bridging social capital. The llp also contributed to expansion of the end-user's social network within Slovenia. On average, the organisations that participate in llp have increased their network by a little less than 16 organisations within Slovenia and a little
less than 15 organisations abroad. Numbers vary more in the national area; however, it is important to note that all participating institutions have established contact with at least one organisation, nationally, and at least two organisations, internationally. The increase in network size is significantly lower when the membership in international (transnational) organisations is considered. On average, the surveyed organisations entered only two new organisations in the year after mobility. On the other hand, one observes the absence of llp impact on networking at the local level (average score 2.98). Here again, caution should be taken when interpreting the results. High variability in social network increases presumably relates to differences in the size of the organisation. At the individual level, the relatively weak impact of llp on bridging social capital is also evident (average score 3.27). As a result of llp participation, individuals, on average, increase their social network by 2.7 people. Statistically significant differences are observed between end-users of different sub-programmes/actions for both social capital and social network increase. This indicates that sub-programmes/actions differ in their impact in networking domain. Individual end-users do not report a significant impact of llp on networking.
llp impact on the domain of European dimension has been observed for the following aspects: presence of supranational (European) identity; recognition of common European values; frequency of behaviours and activities that indicate presence of European identity; and recognition and consciousness of opportunities and challenges within/for eu. Analysis shows that llp significantly contributes to the end-users' European dimension. This is reflected in a direct perception of a European identity, as well as in the recognition of common European values. These behaviours indicate the existence of supranational identity and awareness of opportunities and challenges within the eu (average score 3.5). For 65-70% of individual end-users, llp contributed to the development of a European identity. Taking part in llp enabled almost 85% of individual end-users to apprehend common European values and they started, on average, more than two new activities that indicated supranational, i. e., European, identity. For now, there are no indications that different subprogrammes/actions exert different impacts on the European dimension domain. The current incidence of supranational organisational identity (Puusa 2006) is rather low and is present in less than 10% of participants included in the llp projects that focused on institutions. However, 80% of those organisations perceive a positive impact of llp on change
in their outer organisational identity in the direction of supranational identity. Considering llp as a whole, it can be argued that organisations do change their internal European identity (average score 3.67). Similar to the llps effects on external identity, taking part in llp also exerts an impact on organisational inner identity (average score 3.55). An identical average value is found by the dimension 'recognition of common values of European education area.'
The impact of llp tolerance goals within organisations has been measured on the following tolerance dimensions: tolerance to disagreement; tolerance to nonconformists; tolerance as academic freedom; and lifestyle tolerance (McClosky and Brill 1983). Results show a positive llp impact on increased tolerance to disagreement and on the increase of tolerance as academic freedom. On the other hand, no firm evidence was found for an llp impact on social tolerance (i. e., tolerance to nonconformists and lifestyle tolerance). Statistically significant differences in impact of different sub-programmes were found; Leonardo da Vinci partnerships and Leonardo da Vinci innovation transfer reported the weakest impact.
The llp impact on the employability domain was measured at the level of individual end-users with a focus on: (1) an individual's capability to gain first/new employment; (2) the capability of finding a fulfilling job (i. e., one that enables realisation of an individual's potentials); and (3) employability competence (Hillage and Pollard 1998). Considering llp as a whole, the data can be interpreted as showing a weak positive impact on increased employability. llp contributes to increased employability for all three employability aspects. More considerable impact has been detected in the domain of employability competences. Statistically significant differences exist between impacts of different sub-programmes with Erasmus, with individual mobility reporting the strongest and Study visits reporting the weakest impact.
We measured llp impact on personal growth considering aspects of personal growth according to Jones and Crandall (1986): (1) autonomy; (2) self-acceptance and self-esteem; (3) acceptance of emotions and freedom of expression of emotions; (4) trust and responsibility in interpersonal relationships; and (5) purpose in life. Since no statistically significant differences were found between sub-programmes, and since the average scale value is near 3, we can conclude that, at the moment, we have no firm evidence for llp having any impact in the personal growth domain.
The lpp goal of cooperation has been evaluated on the following sub dimensions: (1) recognition of common goals and intentions that can be accomplished through cooperation; (2) a cooperative stand, as a willingness of individuals to cooperate with other social actors; and (3) actual cooperation, monitored through following cooperation development stages, including the initial phase of establishment of communication channels, the cooperation phase of different intensity levels, and the phase of common cooperation goals establishment (Tuomela 2000).
At the individual end-users level, analysis established that llp has an important impact on initiation of interpersonal cooperation, but predominantly at the lower, less intensive cooperation levels. We also found that the recognition of common cooperation goals and intentions among individual llp end-users is only weakly present or is not present at all. Statistically significant differences on impact to competence development can be found within a group of competences on which llp has the weakest impact. An average score of 2.61 shows the absence of impact or even a negative impact of llp on recognition of common cooperation goals and intentions. This does not pose a solid foundation for cooperation since it represents the constitutive first step in cooperation. On the other hand, one recognises high willingness of individual end-users to cooperate (high average cooperation willingness of 4.24). Regarding the aspect of cooperation development (i. e., cooperation intensity and depth), less developed cooperation forms prevail (establishment of first communication contacts, information exchange without common cooperation goals). More developed forms of cooperation are otherwise present, but to a much lesser extent. This is also expressed throughout the relatively weak attainment and accomplishment of common cooperation goals (average score 3.5). A considerable number of cooperation attempts have not been successful. The results are similar on the institutional level. The number of cooperation cases decreases with the increasing quality and intensity of the cooperation relationship. From an average of 6.75 established contacts, only four cooperation cases developed to regular information exchange. In 3.6 cases, on average, the cooperation evolved to the level of preparing a new project, and only in 1.2 cases the cooperation reached sustainable cooperation beyond the existing project. In order to confirm the positive llp impact, we have to point to the fact that all organisations established at least one contact on all observed cooperation levels. At the cross-border cooperation, a considerable number of unsuccessful cooperation attempts exist. The
scope of impact within all observed cooperation levels (contact establishment, information exchange, preparation of new projects, and cooperation beyond existing project) statistically significantly varies among llp sub-programmes and actions. A high dispersion of survey results can be observed, which might be assigned to rather substantial differences in cooperation between large and small organisations (detailed analysis of variance still needs to be conducted).
According to opinions of relevant policy makers and na focus group participants, the llp does promote cooperation between participating countries. This takes place both at the secondary and tertiary education levels, although the intensity is much greater in higher education due to the corresponding goals of the llp and the Bologna process. All changes in higher education follow the framework of the Bologna process and the lpp programme is - directly or indirectly - an ec tool for implementing the Bologna process, creating common guidelines, activities, and space to foster cooperation. The llp thus intensified cooperation to a greater extent at other education levels and among other target groups (for instance, adult learners). As such, the llp is a cooperation mechanism and promotes cooperation at the same time. Thus, one can consider the cooperation between participating countries as the key added-value of the llp. Comparing the llp to similar (in its function) financial mechanisms, the llp is considered by relevant stakeholders as being less complicated and more efficient. On the other hand, the llp still faces the problem of particularism as an excessive focus on benefits to individual programme end-users.
Conclusions
In this paper we have described different aspects of changes brought by ec programmes on the case of Slovenia. Impact evaluation of llp funding programmes in education and training, shows that these programmes introduce change to all levels of social reality: national (system) level, mezzo (organisational) level, and micro (individual) level. Yet, the change seems to be different at the observed levels as well as in the different observed (target) domains.
The weakest impact seems to be present at the system level, while the highest impact is felt on the individual level. This is in accordance with our initial assumption that llp is one of the few ec initiatives that actually induce change in a bottom-up way. These findings are to some extent similar to the few other publicly accessible evaluation studies (see
McChosan et al. 2008; Bracht et al. 2006; Ecosfera 2004; Centre for Strategy and Evaluation Services 2004; Barbier Firnault et Associés 2003; De-loitte & Touche 2001). These evaluation studies show impact mainly on younger participants' (European) language competencies, while the main impact on institutional level is the creation of networks (groups of interest) that foster cooperation and exchange of good practice. Institutional impact is detected primarily in smaller schools. An increased level of contact with foreign colleagues strengthens the European dimension, and international cooperation is in fact recognised as the top motive for participation in ec programmes.
To some extent, the results prove to be consistent with our initial expectations; however, we cannot validate the measured impact in terms of its quality or extent, for we have no study to compare our measured values with. In this regard, our research is unique and its added value will be assessable when additional or similar research is available. At the time being, its main value is to raise questions and trigger further research.
Although many questions arose throughout the process of this investigation, we chose one focus on whether or not these and/or similar instruments can be sufficiently utilised as the change agents within a national education system. How much (economic) sense does it make to exploit the llp, and similar ec programmes, as a mechanism of national policy goals implementation? Because of the established influences and impacts that funding programmes in education and training have, one will soon have to consider a tighter integration of these mechanisms into national policies in the field of education and training. This seems reasonable because, in the case of decentralised actions, the players at the national level can search for and use the synergetic impacts of funding programmes in education and training. However, to do so, clear national policy priorities seem to be a necessary precondition.
References
Association for Empirical Studies and Centre for Research into Schools and Education at the Martin Luther University. 2007. 'Impact of the Comenius School Partnerships on the Participant Schools: Final Report.' Association for Empirical Studies and Centre for Research into Schools and Education at the Martin Luther University, Kassel. http:// ec.europa.eu/education/doc/reports/doc/comeniusreport_en.pdf
Babbie, E. 2007. The Practice of Social Research. 11th ed. Belmont, ca: Thomson Wadsworth.
Blankenburg, F. 1995. Methods of Impact Assessment Research Programme:
Resource Pack and Discussion. The Hague: Oxfam Novib. Bracht, O., C. Engel, K. Janson, A. Over, H. Schomburg, and U. Teichler. 2006. 'The Professional Value of Erasmus Mobility.' International Centre for Higher Education Research (iNcHER-Kassel) at University of Kassel, Kassel. ec.europa.eu/education/erasmus/doc/publ/ evalcareersum_en.pdf. Centre for Strategy and Evaluation Services. 2004. 'Impact of School Partnerships (Socrates ii/Comenius 1 2000-2006): Mid-Term Evaluation Final Report.' Oxford: Centre for Strategy and Evaluation Service. http://ec.europa.eu/dgs/education_culture/evalreports/education/ 2004/comisocii/comIsocIIsum_en.pdf. Center Republike Slovenije za mobilnost in evropske programme izobraževanja in usposabljanja. 2007. 'Poslovno poročilo Centra Republike Slovenije za mobilnost in evropske programe izobraževanja in usposabljanja (cmepius) za obdobje: 1.1. 2006-31.12. 2006.' Center Republike Slovenije za mobilnost in evropske programme izobraževanja in usposabljanja, Ljubljana. http://www.cmepius.si/files/cmepius/ userfiles/Cmepius/docs/Poslovno_porocilo_2006.pdf Deloitte & Touche. 2001. 'Final Evaluation of the Leonardo da Vinci Programme.' http://ec.europa.eu/education/leonardo-da-vinci/doc/ evaluation/leo1rep_en.pdf Ecosfera. 2004. 'Interim Evaluation of the Comenius Networks, Period Covered by the action: 2000-2006, Period Covered by the Evaluation: 2001 and 2002 Supported Networks: Final report.' Ecosfera, Rome. http://ec.europa.eu/dgs/education_culture/evalreports/education/ 2004/comiiisocii/comIIIsocIIsum_en.pdf Barbier Firnault et Associes. 2003. 'Education and Culture Interim Evaluation of the Comenius 2 Action Under the Socrates ii Programme: Executive Summary.' http://ec.europa.eu/dgs/education_culture/.../ education/.../comIIsocIIintsum_en.pdf European Commission. 2004. 'Interim Evaluation Report on the Results Achieved and on the Qualitative and Quantitative Aspects of the Implementation of the Second Phase of the Community Action Programme in the Field of Education "Socrates."' http://ec.europa.eu/ education/programmes/evaluation/intsocrates_en.pdf.
-. 2009. 'A Single Umbrella for Education and Training Programmes.'
http://ec.europa.eu/education/lifelong-learningprogramme/doc7S_en .htm
-. 2011. 'History of European Co-operation in Education.' http://ec
.europa.eu/education/more-information/moreinformation294_en .htm
European Parliament and the Council. 2006. 'Decision No. 1720/2006/Ec of the European Parliament and of the Council of 15 November 2006: Establishing an Action Programme in the Field of Lifelong Learning.' Official Journal of the European Union 327:45-68.
Ernst and Young. 2006. 'Evaluation of Action 6.1.2-6.2 of the Socrates Programme.' http://ec.europa.eu/dgs/education_culture/evalreports/ evaluation_calls_en.htm
Flander, A. 2010. Poročilo o izvajanju dejavnosti Centra rs za mobilnost in evropske programe izobraževanja in usposabljanja cmepius v letu 2009. Ljubljana: Center Republike Slovenije za mobilnost in evropske programme izobraževanja in usposabljanja.
Foddy, W. 1994. Constructing Questions for Interviews and Questionnaires: Theory and Practice in Social Research. Cambridge: Cambridge University Press.
Hillage, J., and E. Pollard. 1998. 'Employability: Developing a Framework for Policy Analysis.' Research Report 85, Department for Education and Employment, Sherwood Park.
Jones, A., and R. Crandall. 1986. 'Validation of a Short Index of Self-Actualization.' Personality and Social Psychology Bulletin 12 (1): 63-73.
Krueger, R. A. 1994. Focus Groups: A Practical Guide for Applied Research. 2nd ed. Thousand Oaks, ca: Sage.
McCloskey, H., and A. Brill. 1983. Dimensions of Tolerance: What Americans Believe About Civil Liberties? New York: Russell Sage Foundation.
McCoshan, A., N. McDonald, R. Santos, and N. Hall. 2008. 'Joint Report on the Final Evaluation of Socrates ii, Leonardo da Vinci ii and eLearning, A Final Report to the Directorate General for Education and Culture of the European Commission.' ecotec Research and Consulting and ecorys Research and Consulting, Birmingham. http://ec.europa.eu/dgs/education_culture/evalreports/training/2007 /joint/joint_en.pdf
Meehan, E. J. 1994. Social Inquiry: Needs, Possibilities, Limits. Chatham, nj: Chatham House.
Ministrstvo za šolstvo in šport Republike Slovenije. 2007. Strategija vseživ-ljenjskosti ucenja v Sloveniji. Ljubljana: Ministrstvo za šolstvo in šport Republike Slovenije.
Patton, M. Q. 1996. Utilization - Focused Evaluation: The New Century Text. 3rd ed. London: Sage.
Pepin, L. 2006. The History of European Cooperation in Education and Training: Europe in the Making - an Example. Luxembourg: Office for Official Publications of the European Communities.
Putnam, R. D. 2000. Bowling Alone: The Collapse and Revival of American Community. New York: Simon and Schuster.
Puusa, A. 2006. 'Conducting Research on Organizational Identity.' Electronic Journal of Business Ethics and Organization Studies 11 (2): 24-8.
Resolucija o nacionalnem programu visokega šolstva Republike 20072010. Uradni list Republike Slovenije, št. 94/2007.
Rossi, P. H., H. E. Freeman, and M. W. Lipsey. 2004. Evaluation: A Systematic Approach. 7th ed. Thousand Oaks, ca: Sage.
Širok, K., V. Dermol, A. Juric, D. Marjetic, and N. Trunk Širca. 2007. Socrates and Leonardo da Vinci in Slovenia: Impact of Socrates ii and Leonardo da Vinci ii Programmes in Slovenia, 2000-2006. Koper: Fakulteta za managment.
Širok, K., and G. Petric. Forthcoming. 'Quality of Impact Measurement Revised: A Case of Evaluating the Impact of Lifelong Learning Programme in Slovenia.' International Journal of Management in Education.
Tuomela, R. 2000. Cooperation. Dordrecht: Kluwer.
Weinwright, S. 2003. Measuring Impact: A Guide to Resources. London: ncvo Publications.