Metodološki zvezki, Vol. 1, No. 2, 2004, 441-454 Evaluating Policy Networks-Abilities and Constraints of Social Network Analyses Gerd Beidernikl1 Abstract In the late 1990s the European Commission launched an initiative called “Territorial Employment Pacts” (TEPs). TEPs are contractual alliances between protagonists from various sectors on local level in order to develop innovative measures for job creation and job protection in their area. This initiative therefore leads to the emergence of regional policy networks. These policy networks can be characterized as stable but only on a low level formalized communication and cooperation networks between governmental and non-governmental actors, involved in a political process. In the province of Styria (Austria) there were built up six TEPs on regional level and a research project was set up to evaluate their performance and to ensure their future development. This evaluation strongly focussed on the analysis of the policy networks themselves by using the method of Social Network Analysis (SNA). This paper sums up the results of this research project. On the one hand we are going to discuss our experiences in evaluating policy networks in labor market policy using the instrument of SNA in the field of the Styrian TEPs. We are going to unfold our research design and its features developed for initiating policy learning among the pact protagonists. On the other hand we are going to focus on a more methodologic question: Is SNA an appropriate tool for evaluating policy networks in general? We will try to outline a possible answer by deriving abilities as well as constraints of the method based upon experiences from the TEP evaluation. Our aim is to offer general thoughts on this topic and not a detailed universal statistical or explanatory model of policy networks. 1 Territorial employment pacts (TEPs) For over 20 years Europe has had to face certain labour-market problems that seem to be intractable toward national measures which had been set up to defeat them. According to this situation and the resulting growing demands of the heads of 1 CEE - Center for Education and Economy, Research and Consulting, Graz/Austria; gerd.beidernikl@zbw.at 442 Gerd Beidernikl State and Government, who stressed the needs for stronger Community action in favour of employment (ECOTEC, 2002: 2), the European Union started to reconsider its labour market policy and to reshape it towards a coordinated European labour-market policy. Member states are still responsible for their own action, but they have agreed to fit their national policy into a European policy framework. The Contract of Amsterdam finally gave birth to this concern and fixed employment as a shared interest and duty of all EU member states (Larson 2000: 23). It was in this context when in 1996 the European Commission announced an initiative to improve the constantly bad labour market situation in the member states under the title “Territorial Employment Pacts” (TEPs). TEPs are – per definition – contractual alliances between protagonists from various sectors on local level (e.g. industry, non profit organisations, local politicians) in order to develop innovative measures for job creation and job protection in their area (ZSI, 2003). The underlying presumption of this initiative is that policy activities that are developed autonomously, collectively and locally through regional partners are more likely to contribute to improving policy delivery (Huber 2002 : 2). Thru this initiative both the effectiveness and the relevance of the measures are intended to be improved by enriching national policy with regional partnership. Core principle of the TEPs (as well as other mid-1990s initiatives) is therefore the idea of basing initiatives on a multi-stakeholder partnership at a sub-national level. This – in connection with other programmes of the EU – leads to a so called “bottom-up change” in the former “top-down sector” of employment policy. Consequence of this alteration is the emergence of horizontal and vertical policy-networks that extend existing policy structures and widen the range of actors involved. These policy networks are the cause for as well as the outcome of this bottom-up change. They can be defined as mechanisms of political resource mobilization in situations where the capacity for decision making, program formulation and implementation is widely distributed among private and public actors (Kenis/Schneider, 1991: 41). Moreover they can be characterized as stable but only on a low level formalized communication and cooperation networks between heterogeneous groups of protagonists, involved in a political process (Ast, 1999: 44). This change leads overall towards the concept of the enabling state ( Evers/Legwie, 1999: 333), directing national policy without acting exclusively but by stimulating a process of self-government and an empowerment of regions. Policy turns into a permanent interaction process between political and non-political actors in a multi-level governance setting (Ast, 1999 : 31). 2 TEPs in Austria One year after the announcement the European Commission called for the submission of projects under the TEP- programme. Austria has got a long standing Evaluating Policy Networks-Abilities… 443 tradition in cooperative labour market policy and therefore the idea of the TEPs was well received and four pilot-pacts were established. Due to their success the process quite quickly led to a full-coverage establishment of TEPs on provincial level. By 2001 in all of the nine Austrian regions TEPs had been installed. In the province of Styria six pacts below the level of province (Figure 1) were established in addition to deepen the principle of bottom-up policy. The provincial pact therefore acts as an umbrella organisation, synchronizing the regional activities to the provincial strategy and financing them. The actual degree of self-determination of the regional pacts is therefore limited in some concerns from the start. Figure 1: The six Styrian Territorial Employment Pacts. The ideal size or level of TEPs is a topic widely discussed. The issue is whether pacts should be organized at the provincial level or a smaller, regional level (Campbell, 2000: 22). Organizing pacts at the level of the provinces would surely provide the necessary resources and the involvement of actors with decision powers to the pacts. On the other hand the province might be a too high level for ensuring the involvement of the communities and local companies or might miss the variances and needs of small or cross-border regions. A smaller, regional level in turn runs the risk of losing the overall aim of national policy interests or end up without actors capacitated to act in labour market policy. Austria is therefore an interesting case study as it provides the possibility to compare these two concepts. After implementing the six pacts in Styria, there was set up a research project to evaluate their performance up to the present and to ensure their future development by providing scientific expertise. This project is currently being carried out by the Center for Education and Economy (CEE) in co-operation with the Insitute of Technology and Regional Policy (InTeReg – JR) on commission of the Styrian Labour Market Service and the Provincial Government of Styria, 444 Gerd Beidernikl Department of Economy. This paper is based upon results from this research project (Paier/Beidernikl, 2002; Beidernikl/Paier, 2003b; Paier et al., 2003)2. 3 Setting the scene Territorial Employment Pacts are still a quite young figuration in labour-market policy. Therefore they offer a very specific setting for evaluation and one has to take the initial position of the TEPs into account. First of all they are still in a continuous process of development. Due to this fact, evaluation has to concentrate on these dynamic development processes themselves, instead of doing a static input-output-analysis of e.g. jobs created by the measure compared to the money invested. The situation in Styria is getting even more complicated by the fact that not a single job was created by the TEPs so far. The six regional pacts are more or less still in a design phase for projects. Secondly, the creation of jobs is not the only aim of the TEP-initiative. It aims as well at the establishment of local partnership and at initiating a process of local policy learning. The social objectives of the TEP-programme therefore have to be taken into account as an outcome as well. The process of policy adoption and policy implementation on the local level itself has to be analyzed. But this means to leave the field of classic labour-market issues and enter the field of social relations. How to evaluate the “quality of partnership”? A third speciality of TEPs is that there are only few administrative data which they generate (Huber, 2001: 5). There are a lot of (widely distributed) documents they produce, but these protocols, working programmes, contracts etc. are as heterogeneous as the pacts are themselves and vary broadly concerning the profundity of the information and their availability. They can be considered as important for getting insight into both the current states and the development over time of the partnership. But they have to be supplemented by alternative data such as interviews or surveys among the pact members to get a complete picture of the pact constitution. In this – for evaluation – quite challenging context we decided to design a multi-method research approach strongly focussed on the emerged policy networks as an outcome of the TEP initiative and the policy processes themselves; and we have chosen Social Network Analysis (SNA) as an appropriate tool for doing so. Aim of this approach is to analyse policy formulation, implementation and uptake as well as the effects of the policy in order to identify the connection between these elements and encompass therefore a complete policy cycle. Second aim is not only to judge the pacts performance, but to stimulate a regional learning process to improve the performance of the pacts. 2 The final report of the evaluation will be published in 2004. Evaluating Policy Networks-Abilities… 445 4 Evaluation design The whole research project can be characterized as a formative process evaluation (Wottawa/Thierau, 1998: 64), accompanying an initiative and ensuring its development by giving periodic inputs and providing scientific expertise. This procedure is kind of a balancing act between being a neutral evaluator and ensuring meaningful development as an actor of the TEPs. Formative evaluation therefore might run the risk of being involved too much. To avoid this, we decided to design evaluation with a strong focus on policy learning to provide an up-take of our inputs without taking active part as a protagonist. To ensure this “learning circle” of generating and imparting knowledge relevant to the local actors we have installed different mechanisms of feedback and designed the whole project as transparent as possible. This includes: Periodical reports of recent research results. Reflection meetings with members of the steering group on provincial level. Reflection meetings on regional level in all pact regions. Thematic workshops with local protagonists. A closing function open to everyone interested in the research results. Aim of the reflection meetings with stakeholders on local and on provincial level is to reflect on the results of the evaluation and transfer them to their social and contextual environment. “What are the results? What do these results indicate for us and for policy? How can we act/react to improve policy?” This procedure of working on the results together with the actors is intended to lead to local empowerment and learning through evaluation. It is a process of policy learning for all actors involved. Some authors characterized this approach as “social policy learning” (May, 1998 : 8). As was mentioned before, we used a multi-method approach, combining different instruments of data collection and analysis to get a complete picture of the TEPs. This line of proceeding included the following research fields: Structural analysis of the change in labour market policy and employment policy through the establishment of the TEPs in Styria. System analysis of the given setting in labour-market policy. Thematic analysis of the so far developed pact projects (especially the coherency with regional strengthnesses as well as other regional initiatives and the anticipated effects and impacts). Analysis of the pact development and forms of local governance. A comparative network analysis of the cooperation and communication patterns of the six regional pacts. 446 Gerd Beidernikl The SNA of the regional pacts – which the following results are based on -was organized as a standardized questioning of the Styrian TEP-members by direct mail methods. The survey was carried out between October and November 2002. All in all 476 pact members were questioned about their communication and cooperation axes in their regions as well as about general attitudes toward the TEP-concept. Documents and interviews with local experts were used in addition to complete the picture. The network calculations were computed with the software package UCINET 6.0 (Borgatti/Everett/Freeman, 2002). 5 Some results Due to the limited length of this article we will concentrate in the following on just one specific question concerning the TEPs and check it against measures provided by SNA: Has the TEP-initiative led to new forms of regional partnership? Asking this question, we have to define dimensions for analyzing “new partnership”. The three main dimensions are: · The structure of partnership: Have the TEPs been able to integrate a broad basis of actors to build up a comprehensive partnership? Have they been able to integrate new, formerly missing players in active labor market policy such as private enterprises, communal actors, politicians etc.? How do they participate? How can the structure of the network be described? · Communication patterns: Is there regular communication among the partners? Are there subgroups better or worse integrated than others? Is there a process of knowledge sharing? · Cooperation patterns: Have new co-operations emerged? Are the partners equally integrated into the pacts? Have any cliques emerged? Let us first of all look at the partners integrated into the TEPs. The membership structure of the pacts is dominated by organizations located in the field of vocational training, counseling and welfare. Together they reach an amount of over 50% of the protagonists. In some pacts they even touch the 75% line. These organizations have mostly been involved in labor market policy even before the pacts. Unemployed people are their “clients” and being involved in the pacts means keeping their own business going. Participation is therefore of high interest. The regional offices of the Styrian Labor Market Service play an important role in the pacts. They are - as regional branches of the official responsible authority – key players of the pacts and often take a kind of chairmanship. Other important groups of actors must be counted as the regular players of employment policy in Austria as well – the social partners. The employers as well as the employees’ union take part in the TEPs, but mainly on Evaluating Policy Networks-Abilities… 447 level of the steering group. Finally there must be mentioned politicians of both regional and provincial level. Private firms as potential employers are integrated just marginally. This overall view only indicates the presence, not the kind of involvement of the actors. But nevertheless it turns out at this point that some groups of protagonists envisaged to integrate into the pacts according to the guidelines are missing: departments of the provincial government, financial institutions, institutions of higher education and research centers and – most of all – industry. Other groups are present but their participation should be improved. Especially the representatives of the employers’ association should take more initiative for recruiting local firms. The position of politicians as multipliers has to be improved too. Local politicians could be key players for mobilizing local potentials, but for this they must be strengthened and integrated better. All in all 80% of the protagonists have worked on the topic of employment policy even before the TEPs and therefore the amount of “new protagonists” can be numbered 20%. The communication structure of the pacts can be described as a 3-layered model varying significantly in its degree of linkage. Layer one – the inner circle of the TEPs – assembles out of the steering group and big regional players in employment policy such as vocational training institutions. They know each other very well and the degree of linkage is nearly perfect. Layer two follows up with mainly operative protagonists. Layer three contains actors involved just marginally. The ratio of these three layers could be a quality criterion for pact assessment, describing the structural constitution of a pact. Especially layer three must be paid attention to: A high amount of actors in this layer – as we did find in one of the pacts – indicates high fragility. Focussing the groups of involved protagonists one has to say that above all local politicians and other “new actors” are integrated badly and tend to be in layer 3. These results describe the overall structure and the position of groups within the whole system. But they do not provide information about the linkage between certain kinds of groups. To figure this out SNA offers the possibility of block-modeling. Using this routine one looks at a higher level of abstraction, analyzing systematical differences in the linkage of defined blocks. These blocks are the result of the group membership of the protagonists. The communication axes between the various groups of TEP-members appear to be not equally established and vary largely. This suggests the thesis of systematical mechanisms or reasons for appearing so. The best-established axes in all of the regional pacts are those between the three main groups of traditional labor market policy: the labor market service as responsible authority, the employees’ union and vocational training institutions commissioned with carrying out qualification measures. The best-established axes are those that have existed even before the TEP. Without going further into details one can state: The linkage between certain kinds of groups has to be improved. One opportunity for improving communication and the distribution of knowledge could be the establishment of working groups crossing 448 Gerd Beidernikl these boundaries. These workshops as well could improve social learning in the regional pacts. Figure 2: Example of a cooperation network in one of the Styrian pacts. So far, the TEPs seem to be dominated by traditional players of labor market policy. But does this automatically mean that there have emerged no or just a few new partnerships? Just as well there could be brought up new partnerships between “old actors”. In the network survey there have been asked questions dealing with this aspect as well. We have tried to find out with whom the actors cooperate inside the pact, with whom they cooperate outside the pact. On an average 65% of the pact-internal co-operations exist outside as well. 35% of them are inside only and therefore a kind of new co-operations. The protagonists themselves are aware of that. Being asked : “In what way do you benefit by participating in the pact?”, over 30% state an increase of co-operation partners. But there must be mentioned as well that there are fields of lacking co-operation, alike the communication patterns of the pacts. By using search-algorithms we tried to find out whether there are any cliques in the networks cooperating among each other but not with the pact. We have not been able to prove this. But there is a tendency of building up co-operations within their own thematic interest, without participating in other work groups or themed workshops. This might be a hint for a symptomatic result: The protagonists are still keeping their eyes on their own organizational aims and consider the TEPs as an instrument for accomplishing them. To sum up the results explicated in this section we can state that the TEP-initiative has not led to a significant change in regional partnership. This statement can be proved by different arguments: Evaluating Policy Networks-Abilities… 449 · The TEPs have not succeeded in their intention to establish a broadly-based partnership on regional level. Important actors are still missing. The TEPs are dominated by the traditional players of labor market policy. · The “new protagonists” are not quite well integrated into these traditional communication and cooperation axes. They tend to be kind of outsiders. · Most of the co-operations are long-established ones. But there have emerged some new co-operations among traditional protagonists. 6 Assessment of the method In this chapter I would like to mention some of the problems we were facing while doing the analysis of the regional networks and our way of solving or at least moderating them. The first field of complication is trust, the other is non-response and both are heavily linked to each other. 1) Trust: Doing a network analysis means to analyze the relations between social actors. These relations can be communication axes, co-operations, emotions, kinship – more or less every kind of “social relation” that is thinkable. But the network analyst has got to fix these relations in some way, quantify them, put them down in form as a social matrix. But where to get the data from? Some relations are put down in documents and are therefore easily accessible (e.g. contracts). We tried to look on these officially anchored relations first but the data situation concerning the TEPs was too weak to base a proper network analysis on it. Therefore, in the analysis to come, we only used information about the formal or functional division of the pacts out of documents: “Who is in the steering group?”, “Who is working in which thematic workgroup?” and so on. The second way to get to data about social relations is to ask the related persons themselves. But collecting these social network data means asking the respondents a considerable amount of burdensome questions that seem to be private or sensitive. Respondents often find these social network questions threatening. This combination therefore is often responsible for high unit or item non-response - both of them are serious problems in every network analysis. But we had to take this risk as it was the only way of getting ahead. The size of the regional pacts - 476 protagonists over all - complicated the situation even more by making the list of network relations to ask in our survey considerably longer than in other SNA-studies. The only way to ensure high participation in the survey is therefore to take the respondents this fear and to put the importance of their answers across to them. To minimize the number of non-respondents we made high efforts in information work about the reason and the aim of the SNA-survey, about the privacy of the data and about the utilisation of the results. All in all we tried to establish an atmosphere of trust, integrating the local protagonists more as partners in the 450 Gerd Beidernikl survey, than as experimentees. We tried to use local opinion leaders in the pacts for promoting our survey and we integrated the pacts even in the phase of constructing our questionnaire. The survey itself was carried out by providing accompanying letters signed by members of the steering groups; accessory emails and postcards reminded the respondents to send back their questionnaires. We think this procedure worked and the high efforts paid off. Up to 80% of the questionnaires were returned and properly filled out. The SNA-survey was widely accepted by the pact protagonists. 2) Non-response: Nevertheless up to 20% of the forms were missing and missing data in a complete network analysis is kind of a worst case scenario because – per definition – the complete network is not recorded. A common practice among social network researchers is either to leave the non respondents out of the analysis or to replace the missing data by means of particular imputation procedures. We did kind of both. The structure of the non-respondents seemed to be at random without considerable regularities. For some calculations we therefore simply took out the missing respondents an analyzed e.g. the communication between different blocks of the networks. In other cases we tried to fill the gap of missing values as good as possible. Missing value imputation in the tradition of accuracy research was not possible, because of the fact that the questionnaire would have gotten too long if we had asked every protagonist on every relation in the network. But we realized that the reciprocity of the answers in the network was very high (accordingly the perception of the network relations by the protagonists is very accurate). Up to 87% of the answers about one relationship between two actors was the same between the two. So we decided to substitute the missing data by symmetrizing the whole matrix. This caused another bias for sure and still did not substitute the missing values between missing actors, but we accepted this handmade bias and charged at least some gaps in the network information. The question of how to treat missing values in a network survey – as non response is inherent to survey research in general – is from our point of view still the most important problem to solve to use SNA in wider fields of social research. But how to validate our modus operandi? We decided to use the same procedure as often used in qualitative social research to validate results: communicative validation (Flick et al., 1995: 168). This simply means to work on the results together with the respondents to find out wether they consider the results accurate or not. We therefore worked on the results of the SNA together with the protagonists in regional workshops to get their point of view. They generally stated that we had truly put their perception of the networks on record. This procedure not only validated our results, it as well helped to transfer the findings into the social and contextual environment of the actors and initiated the already mentioned “social policy learning”. Evaluating Policy Networks-Abilities… 451 7 Conclusion In case of the TEPs, the application of SNA was a great success in the six regional pacts as well as on the level of province. Based upon results provided by the evaluation siginificant changes could be initiated (e.g. regarding networkmanagement, pact-strategy, financing, re-dimensioning of the pact-structures). Using SNA therefore proved to be an appropriate tool for analyzing the TEPs in Styria. It was possible to show up the inner structure of the regional policy networks, working out both their restraints and capabilities. Above all it was possible to point out that a change of the policy framework does not automatically lead to a change in policy delivery. The main abilities of SNA in the context of the TEPs showed up to be: · SNA allowed to look inside the pacts, unfolding some important aspects of the communication and cooperation behavior of the protagonists. These insights showed up to be the key for solving the problems. · These “X-rays” have brought to light results that specify and partly disprove results of other TEP evaluations. · By using SNA it was possible to take a bird’s-eye view on the TEPs, explaining problems to the protagonists in a neutral, clear and comprehensible way. By working on these results in workshops SNA has led to a change in the self-perception of the actors, pointing out the context and consequences of their own acting – in the regions as well as on provincial level. · SNA has made different policy networks comparable, showing up similarities, dissimilarities and regional particularities. There have been lots of positive feedback from the protagonists themselves, referring mostly to the bird’s-eye view of the TEPs delivered by the network plots. Although SNA-measures are quite complex, the SNA-plots were able to guarantee easy reception of the results. Furthermore the tactics of working on the results together with the protagonists turned out to be of high value for increasing the acceptance and adoption of the findings. Because of its success in explaining, describing and visualizing cooperation and communication structures of networks, SNA would be a very interesting instrument to apply in a lot of different fields. A field of application could be the evaluation of different kinds of policy networks or other organizational networks of non-political interest. Next to employment policy especially networks in the area of research and development, qualification, education and social welfare are thinkable research areas. Such evaluations could provide meaningful data for e.g. the government for planning strategies of accurate sponsorship and expanding measures. The network-organizations and the networks themselves would benefit 452 Gerd Beidernikl from SNA-based analysis by getting shown up their status-quo and getting derived recommendations for future development. SNA can therefore be an instrument for the self-evaluation of partnerships, providing opportunities for improving the management, development, controlling and monitoring of network activities. At the moment, the Center for Education and Economy (CEE) is developing an online tool for evaluating the cooperation and communication patterns of transnational networks in the EU. A pilot-version of this instrument will be implemented in an EQUAL-partnership next year. Another big challenge for SNA could be the field of company networks and regional clusters. There have been carried out very promising studies (e.g. Krätke/Scheuplein 2001) demonstrating a new kind of analysis for economic interdependencies. Economic networks are a field of activity mostly for practitioners. But there is a lack of theory explaining and unfolding the benefits of networking in detail. SNA could light up this darkness and therefore provide information for economic fundamental research as well as for fieldworkers in network and cluster-management. References [1] Ast, S. (1999): Koordination und Kooperation im europäischen Mehrebenensystem. Köln: Omnia. [2] Beidernikl, G. and Paier, D. (2003a): Network analysis as a tool for assessing employment policy. In Proceedings of the Evidence-Based Policies and Indicator Systems Conference 03. London (forthcoming). [3] Beidernikl, G. and Paier, D. (2003b): Evaluierung der Kooperativen Arbeitsmarktpolitik: Netzwerkanalyse. Graz: CEE-Report. [4] Borgatti, S.P., Everett, M.G., and Freeman, L.C. (2002): Ucinet 6 for Windows. Software for Social Network Analysis. Harvard: Analytic Technologies. [5] Campbell, M. (2000): Local Partnerships in Austria. OECD/LEED-Report. [6] ECOTEC (2002): Thematic Evaluation of the Territorial Employment Pacts. Final Report. Brussels: ECOTEC. [7] Evers, A. and Leggewie, C. (1999): Der ermunternde Staat. Vom aktiven Staat zur aktivierenden Politik. In Gewerkschaftliche Monatshefte. 6, 331-340. [8] Fallend, F., Aigner, D., and Mühlböck, A. (2002): Europäisierung der lokalen Politik? Der Einfluss der europäischen Union auf Politiknetzwerke und Entscheidungsprozesse in der lokalen Wirtschafts-, Beschäftigungs- und Arbeitsmarktpolitik in Österreich. In Europäisches Zentrum für FöderalismusForschung (Ed.): Jahrbuch des Föderalismus 2002: Föderalismus, Subsidarität und Regionen in Europa. Baden-Baden. 331-340. Evaluating Policy Networks-Abilities… 453 [9] Flick, U. et al. (1995): Handbuch Qualitative Sozialforschung. Weinheim: Beltz. [10] Huber, P. (2002): Evaluating Employment Pacts, Methodological and Practical Issues – The Experience Of Austria. Paper/Presentation at the OECD/LEED Conference on Evaluating Local and Economic Development. Vienna: WIFO. [11] Huber, P. (2001): Territorial Employment Pacts in Austria. Discussion Paper. Vienna: WIFO. [12] Kenis, P. and Schneider, V. (1991): Policy-networks and oolicy-analysis: Scrutinizing a new analytical toolbox. In Marin, B. and Mayntz, R. (Eds): Policy Networks. Empirical Evidence and Theoretical Considerations. Campus. [13] Koch, P. and Hauknes, J. (2000): On policy learning and the interaction between policy makers and researchers. Workpackage Synthesis Report, WP4. Oslo: CENTRIM. [14] Krätke, S. and Scheuplein, C. (2001): Produktionscluster in Ostdeutschland, Hamburg: VSA. [15] Larson, A. (2000): Die europäische Beschäftigungsstrategie und der europäische Beschäftigungspakt. In E. Piehl and H.J. Timmann (Eds.): Der Europäische Beschäftigungspakt. Entstehungsprozeß und Perspektiven. Baden-Baden: Nomos. 23-35. [16] May, P. (1998): Fostering policy learning: A challenge for public administration. Working Paper. Seattle: University of Washington. [17] Paier, D. and Beidernikl, G. (2002): Evaluating Cooperative Labour Market Policy – System Analysis. Graz: CEE-Report. [18] Paier, D. (2003): Network analysis: A tool for analysing and monitoring knowledge processes. In K. Tochtermann and H. Maurer (Eds.): Proceedings of I-KNOW 03. 3rd International Conference on Knowledge Management. Graz: J.UCS. 583-592. [19] Paier, D. and Beidernikl, G. (2003): Evaluating Cooperative Labour Market Policy – Network Analysis. Graz: CEE-Report. [20] Paier, D., Beidernikl, G., Pohn-Weidinger, S., Gruber, M., and Woitech, B. (2003): Evaluating Cooperative Labour Market Policy. CEE in co-operation with Joanneum Research. Graz: CEE-Report. [21] Schmid, J. (2000): Lokale Beschäftigungspakte im Mehrebenensystem der EU: Weiche Steuerungsinstrumente auf lokaler Ebene. http://www.uni-tuebingen.de/uni/spi/beschaeftigungspakte.pdf. 16.06.2003 [22] Wottawa, H. and Thierau, H. (1998): Lehrbuch Evaluation. Bern: Hans Huber Verlag. 454 Gerd Beidernikl [23] ZSI (2003): Guide to Territorial Employment Pacts in Austria 2000-2006. On Commission of the Austrian Ministry of Labour and the ESF. http://www.pakte.at/rahmendokumente.html. 16.06.2003.