Slovenian Evaluation Society – Working Papers 8/1(June 2015) V o l 8 n o 1 y e a r 2 0 1 5 Measuring Smartness of Innovation Policy Bojan RADEJ, Karin ŽVOKELJ JAZBINŠEK, Metod DOLINŠEK Working Paper Creative Commons, 2.5 1 Radej, Žvokelj Jazbinšek, Dolinšek Ljubljana, june 2015, Ver. 1 S l o v e n i a n E v a l u a t i o n S o c i e t y Tabor 7, Ljubljana, Slovenia info@sdeval.si, http://www.sdeval.si Slovenian Evaluation Society – Working Papers 8/1(June 2015) CIP - Kataložni zapis o publikaciji Narodna in univerzitetna knjižnica, Ljubljana 316.4(0.034.2) 303:316.4(0.034.2) RADEJ, Bojan Measuring smartness of innovation policy [Elektronski vir] / Bojan Radej, Karin Žvokelj Jazbinšek, Metod Dolinšek. - 1. izd. = 1st ed. - El. knjiga. - Ljubljana : Slovensko društvo evalvatorjev = Slovenian Evaluation Society, 2015. - (Delovni zvezki SDE = Working paper SES ; vol. 8, no. 1) Način dostopa (URL): http://www.sdeval.si/komisija-za- vrednotenje/publikacije/589-measuring-smartness-of-innovation-policy ISBN 978-961-93348-6-7 (html) 1. Žvokelj Jazbinšek, Karin 2. Dolinšek, Metod 280056320 2 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) »Measuring Smartness of Innovation Policy« Bojan RADEJ*, Karin ŽVOKELJ JAZBINŠEK**, Metod DOLINŠEK*** * Slovenian Evaluation Society, bojan.radej@siol.net, corresponding author ** MK Projekt, Šmarje pri Jelšah, and Slovenian Evaluation Society *** Development Centre, Maribor, and Slovenian Evaluation Society Acknowledgment: This paper has been written for submission to Asia‐Pacific Tech Monitor, July – September, 2015. Presented at Third Asia‐Pacific NIS Forum “Diagnosis of NIS and Development of STI Strategies in the Open Innovation Framework”, 8‐9 April 2015, Bangkok, Thailand, organized by Asian and Pacific Centre for Transfer of Technology (APCTT) of United Nations Economic and Social Commission for Asia and the Pacific (ESCAP). Authors are grateful to Professor Osvaldo Feinstein from Universidad Complutense de Madrid for comments on previous version of this text. Proposal for citation: Radej B. K. Žvokelj Jazbinšek, M. Dolinšek. Measuring Smartness of Innovation Policy. Ljubljana: Slovenian Evaluation Society, Working paper, vol. 8, no. 1 (June 2015), 25 pp., http://www.sdeval.si/komisija‐za-vrednotenje/publikacije/589‐measuring‐smartness‐of‐innovation‐policy Not language edited. Ljubljana, June 2015 3 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) Measuring Smartness of Innovation Policy Abstract: The ‘smart specialization’ is a concept introduced for enhancing innovation in EU. Smartness lies in entrepreneurial discovery of areas of specialisations that best fit innovation potential of the territory. Smartness is studied at meso level as an area of horizontal overlaps between three domains of knowledge triangle: education, research, and innovation. Overlap is measured with correlation of evaluated policy instruments’ impacts on three evaluation domains. Case study suggests that vertical and horizontal, or ‘dumb’ and ‘smart’ aspects of innovation policy are both important for policy success. This suggests new policy concept of country’s specialization in innovation that is not merely smart but fully mesoscopic. Key words: smart specialization, knowledge triangle, impact evaluation, social complexity, meso level. 1 Challenge The ‘smart growth’ and the ‘smart specialization’ are concepts in new strategic approach (Midtkandal, Sörvik, 2012) introduced by European Commission for enhancing innovation as leading driver for progress of welfare in EU (COM(2006)‐604). The new concept replaces the traditional vertical silos approach (Degani, 2014) with only one way flow (Sjoer et al, 2012), from single sectoral challenge to single sectoral solution, neatly organized each in its own ministry or department, favouring some technologies, fields, and companies. Sector‐based specialization implies top‐down and approach in which country identifies a limited number of priority areas for knowledge‐based investments and concentrates existing capabilities, assets, competences, and comparative advantages with the aim to enhance innovative capacities. These materialize through linear progression from basic research to education and laboratory work, innovation and commercialization. As a result, an innovator, scientist and researcher many times even excluded each other from the use of the innovation in order to appropriate larger fraction of the benefits (Foray, Goenega, 2013). The new concept therefore shifts deep understanding and changes mind‐set (Markkula in Lappalainen, Markkula, 2013) from silos to cross organization model by a both ways circular and horizontal connectedness (Degani, 2014) between sectors of innovation policy. Smartness is alternative strategy to old style sector‐based specialization. If you are small, you are not in a good position to benefit from concentration and returns to size and so you have to be 4 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) smarter (Foray, Goenega, 2013). Smartness refers to learning process, where stakeholders play a key role in discovering areas of future specialisation from the bottom‐up approach (COM(2006)‐ 604). Smartness lies in innovative ‘entrepreneurial discovery’ (Foray, 2013) of the specialisation that best fits specific potential of the territory, based on local assets and capabilities, regardless of whether the concerned territory is traditionally strong in innovation or weak, high‐tech or low-tech (Midtkandal, Sörvik, 2012). Smartness emphasizes horizontal logic (Foray, Goenega, 2013) of specialization. It seeks for synergies between independent drivers of innovation and emphasises that its various sectors largely support each other – only indirectly but strategically. Smartness of specialization in innovation can be formalized with a concept of ‘knowledge triangle’ (KT) as proposed by European Institute for Technology (EIT) in 2008 (see COM(2006)‐604). The Triangle underlines the interaction between research (R), education (E) and innovation (I) as three main sectors, domains, pillars, drivers or capitals of a knowledge‐based society (Schuch, 2013). Each sector brings forward essential concerns for innovation policy, that are specific for one sector individually and these concerns remain mostly ignored in other sectors. For instance, companies are primarily concerned with innovation because of higher profit and income; education sector in its core is constituted on autonomy; research sector is devoted to enhance predictive powers of knowledge (Markkula in Lappalainen, Markkula, 2013). Separate missions of each knowledge domain justify vertical and sector‐based organization of innovation policy. Therefore, smart innovation policy needs to be understood and governed along two ‘axes’: the vertical one is illuminating sectoral concerns (E, R, or I), and the horizontal one which is presenting inter‐sectoral overlaps between I, E and R as areas of policy ‘orchestration’ (Sjoer et al, 2012). Horizontal perspective is relevant because, despite contradictions in their core, E, R and I tend to stimulate and cross‐fertilise each other (Carvalho, 2010). Smart specialization and KT are two concepts that both highlight the importance of jointly fostering innovation in many independent but overlapping sectors, which also calls for paying due attention to the linkages between them (Markkula in Lappalainen, Markkula, 2013). By horizontal overlapping I, E and R, the companies will be for instance given opportunities to commercialise the most up‐to‐date research findings. In return, research organisations will benefit from additional incomes from commercialization of their intellectual rights and further enhance their capacities for implementing basic research. Education will take advantage of linking learning with doing, providing students with better employment opportunities and furthering their professional competencies. 5 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) Pursuing smart (specialization of) innovation is paramount challenge along both axes in EU, compared to its main competitors. In vertical direction, there is insufficient concentration of knowledge resources in poles of excellence (COM(2006)‐604). The policy aim is specialization but also avoiding the government failures associated with the top‐down bureaucratic technology choices (Foray, Goenega, 2013). Furthermore, in KT there is not only one vertical perspective but many that shall be coordinated with their incompatible demands. Barriers to horizontal smartness are also profound in EU. They stand in the way of spreading new knowledge (Mulgan, 2007) between three sectors of KT, between public and private stakeholders as well as between theory and practice. There is insufficient commercial exploitation of (publicly funded) research; insufficient trans‐ and inter‐disciplinary research with insufficient focus on medium‐ and long‐term social challenges; lack of innovative governance; cultural differences between science and private companies, legal barriers, as well as fragmented knowledge and technology markets. When measuring smartness, we are not, of course, aspiring to find out how innovative the outputs, results (outcomes) and impacts of innovation policy really are but only how they, as they are, overlap and support each other. Traditionally, evaluation of innovation policy’s impacts adopt a simplistic model of results based assessment, essentially trying to understand what happened as a result of interventions and then connecting this back to programme goals (Reid, 2010). Simplistic models that assume a direct‐cause effect relationship, such as a return on investment of R&D funding, many times fail to represent the innovation appropriately (Reid, 2010). Linear theory of change in impact evaluation raises number of methodological issues when faced with complex social challenges. Insightful example is the attribution problem: how to assess a change in a policy variable caused by the intervention when change emerges from overlap between different independent causes. Causality cannot explain much in evaluation when asking complex questions. All one can usually find out is correlation between independent evaluation domains and this does not allow for strong and definite conclusions but only weak and contextually valid. Aggregation and integration problems in evaluation are another examples of basically the same difficulty linked to non‐linearity (Radej, 2013, 2014a). In innovation policy, linear thinking must be replaced with complex one which is elaborated at meso level, since it allows for logical consistency between partly independent and partly interwoven structures of generating and applying knowledge (Zenker, Muller, 2008) that involve interactive, collaborative and thus non‐linear thinking (RISS, 2011; Hirvikoski in Lappalainen, Markkula, 2013; Reid, 2010). We accordingly hypothesise that innovation policy shall be 6 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) implemented and its impacts evaluated in mesoscopic way that is partly in sectoral perspective (‘dumb’, vertical) and partly in intra‐sectoral (smart, horizontal) perspective. The second chapter presents ‘The three capital model’ (3C; Radej, 2014a) as the mesoscopic approach to measuring smartness of innovation policy. 3C model is abbreviated version of ‘The four capital model’ of sustainable development (4C: economic, social, environmental, human; Ekins, 1992; Ekins, Medhurst, 2006). The difference between 3C and 4C approach is not essential in evaluation methodology. What is important is distinction between one (usually economic) and many, and thus between simple and complex approach. Third chapter introduces experimental policy impact evaluation case study on which newly proposed methodology is tested. Fourth chapter presents evaluation results, fifth discusses them. Paper concludes with recapitulation. 7 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) 2 Model Innovation policy were traditionally evaluated with simplistic models such as when its challenges are presented with parallel vertical pillars of independent evaluation domains, in our case E, R, and I. The simplistic model is operationalized with three sets of independent evaluation criteria for each innovation sector separately – such as with innovation scorecards (IUS, 2011). This is effective approach for emphasising selected key aspects of innovation policy for each sector, such as patent activity, scientific papers’ citation or number of PhD students in natural sciences. Yet this kind of evaluation cannot tell anything about policy smartness since the model is lacking the slightest horizontal overlap between innovation sectors. EIT originally conceptualized KT in a systemic way, in which three pillars are connected with lines into triangle. Markkula (in Lappalainen, Markkula, 2013) went step further and modelled KT with Sierpiński triangle, with smaller triangles embedded into larger triangle. This is, formally speaking, chaotic presentation (see Radej, 2014b) where KT is modelled on lower level with three smaller triangles applied as fractals – a geometric figure that does not become simpler when you analyse it into smaller and smaller detail (Baranger, 2001). On the top of presentation he placed a triangle of horizontal ‘orchestration’ that is connecting three domains of KT, not with lines or overlaps between them, but with scale invariant replication of the same, triangular form on all levels of evaluation. What connects two models is that they are not complex, but simple (vertical, linear) or chaotic (horizontal, non‐linear; Radej, 2014b) schematizations. Simple and chaotic approaches are nevertheless relevant since complex approach lies precisely between them. Complex understanding is modelled in hybrid way – as partly simple and partly chaotic system (Stacey, 2002), because it shares characteristics of both: smart specialization is partly ordered, linear (sector based), and partly disordered, non‐linear (inter‐sectoral) phenomenon. Radej (2014b) proposed to present hybrid concept of complexity with Venn diagram (1880) and its three partly overlapping circles. Non‐overlapping areas of Venn diagram present three pillars or integral domains of KT that are equally important for innovation policy, but vertically incommensurable. One cannot for instance aggregate detailed E, R and I impacts into indicator of summary impact because they are not commensurable, they are expressed in different common denominators, like money, number of patents and employment, so they can be aggregated only partially and separately each in its own domain. On the other side, overlapping areas refer to inter‐sectoral 8 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) impacts that are hybrid in nature so they can be aggregated with correlation into summary indicator of impact. Presentation of complex structure with Venn diagram is appropriate since it combines opposite perspectives (sectoral vs. inter‐sectoral) in coexistence without logical contradiction (Flores‐Camacho et al, 2007) In selected case study we start evaluation of innovation policy’s smartness with first constructing conventional Leopold (1971) impact matrix which presents detailed impacts of each policy instrument on each evaluation criteria. In the second step, detailed impacts are partially aggregated by source (policy instruments) and area of impact (evaluation criteria) into Leontief (1970) square input‐output matrix. It displays how three domains of KT impact each other in sectoral and in inter‐sectoral way. In the next step, non‐diagonally located inter‐sectoral impacts are correlated in two phases: first as an overlap between two sectors (circles) and then in triadic overlap between three binary overlaps. Non‐overlapping areas in Venn diagram evaluate sectoral aspect of innovation policy, while overlapping areas are explaining smartness of innovation policy. 9 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) 3 Data Mesoscopic approach has been tested in mid‐term evaluation of selected innovation policy instruments that comprise many of the most transforming segments of KT in Slovenia from the aspect of their relevance, efficiency, effectiveness, and nine horizontal criteria (Table 1). Ministry for higher education, science and technology allocated almost 220 mil € in 2007‐2011 while companies contributed additional 57 mil € in own financing, which together accounted to 0,8% of annual GDP (while R&R expenditure in 2010 reached 2,11% GDP; MK Projekt et al, 2012). Eight policy measures or instruments have been implemented as parts of two national Operational Programs ‐ for Regional Development (RD) and for Human Resources Development (HRD): ‐ »Strategic research in companies« (SR; RD) co‐finances developing knowledge, prototype or essential improvement on technological platform that enhance access to global market. ‐ »Centres of Excellence« (CE; RD) concentrate knowledge and strengthen partnership by financing establishment and management of Centres, their research, costs of demonstration projects and investment in R&D equipment. ‐ »Competence Centres« (CC; RD) co‐finances management and development of Centres for accomplishing joint R&D, industrial research and experimental development. ‐ »Young researchers« (YR; HRD) finances R&D costs during study at master and PhD level. ‐ »Innovative Scheme« (IS; HRD) finances PhD students for costs of a tuition fee and attendance at international conferences. ‐ »Career Centres« (CA; HRD) development of service at the Universities to facilitate connections with R&D institutions and companies; aim is to improve students’ access to labour market. ‐ »Bologna Process« (BP; HRD) aims at creating comparable University programs in EU (COM(2006)‐604). Instrument finances reform of higher education programs. ‐ »Foreign professors and External experts« (FP; HRD) finances international mobility between Universities, R&D institutions and companies to stimulate transfer of knowledge, cooperation and exchange. Evaluation drew from two data sources beside official statistics. The first was provided by extensive governmental monitoring system of input data, output and also for result indicators (only incompletely) for each operation (project, scholarship, visit…) within each instrument financed. In addition to this a set of differentiated questionnaires have been prepared for each 10 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) group of beneficiaries – students, professors, researchers, businessmen, project managers. Each instrument was among others assessed against prescribed set of nine horizontal evaluation criteria (on five‐level scale, prevailingly negative impact = 1; poor positive = 2; positive = 3; strong positive = 4; excellent = 5): ‐ C1: Cost efficiency – beneficiaries were questioned about diverse aspects of administrative management of operations. ‐ C2: Instruments’ impacts on natural environment (questionnaire). ‐ C3: Leverage effect ‐ how much private investment is attracted per euro of public investment (monitoring data). ‐ C4: Regional balance of impact on 12 Slovenian regions, assessed with comparison of allocated funds per capita (monitoring data, statistical data). ‐ C5: Gender equality, as representation of women in financed operations (questionnaire). ‐ C6: Employment criterion asks if the operation increases employment opportunities on short‐term and long‐term (questionnaire). ‐ C7: Sustainability criterion asks if achievements of the project can be maintained after completion of the operation (questionnaire). ‐ C8: Impact on business environment regarding multidisciplinary knowledge, new opportunities, and organizational change in companies (questionnaire). ‐ C9: Impact on wider society ‐ local and family needs, SMEs, professional associations, research institutions, University (questionnaire). All assessed impacts of eight instruments by nine criteria were organized into Leopold evaluation matrix. In the next step they were grouped by rows and columns to obtain Leontief matrix presenting inter‐sectoral impacts between three domains of KT: ‐ Instruments grouped into R (first row of Leontief matrix): YR, IS; Criteria grouped into R (first column): C1, C4, C6, and C8. ‐ Instruments grouped into E (second row): CA, BP, FP; Criteria grouped into E (second column): G5 and G7. ‐ Instruments grouped into I (third row): SR, CE, CC; Criteria grouped into I (third column): C2, C3, and C9. Grouping is not optimal, since logical links between domains, instruments and criteria is in some cases weak. Two reasons stand behind this. Horizontal evaluation criteria (Table 1) have not been selected by evaluators. Besides, the policy instruments have not been designed explicitly by the 11 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) concept of KT. Matching between policy design and evaluation design is not optimal. For this reason the evaluation of smartness can serve as a methodological experiment, while its policy findings in this respect remain cautious and rather indicative. 12 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) 4 Results According to output indicators, Slovenian innovation policy has been very successful in mid‐term achievements (2007‐2011): 2036 projects proposals were received, 71,5% approved and 7,4% already completed. Some 800 students have started their PhD studies. 100 foreign professors and experts were involved in University programs. Almost 370 young researchers have been employed in companies or 85% more than planned for the whole period (to 2013), 47 innovations and 22 patents registered, both exceeding goals. Planned outputs for the entire period were achieved also in number of R&D projects in SME (100%), number of research hours in full time equivalent (900%!), in private co‐financing in supported projects (153%). These achievements are correlated with strong improvement in international statistical comparisons of main innovation indicators (IUS, 2011) where Slovenia is recognized as one of the fastest growing countries in the group of innovation followers. Next, evaluators accomplished cross‐sectional assessment of instruments’ impacts on evaluation criteria. Results are presented in Leopold impact matrix (Table 1). TABLE 1: Leopold impact matrix for Slovenian Innovation policy, on scale 1‐5 Horizontal Evaluation C1: G4: C6: C8: G5: C7: C2: C3: C9: Criteria Cost Regional Employ‐ Business Gender Sustain‐ Natural Leve‐ Wider efficiency balance ment environ‐ equality ability environ‐ rage society Policy ment ment Instruments KT* R R R R E E I I I YR R 2,7 3,0 3,5 3,3 3,0 5,0 2,8 n.r. 3,4 IS R 2,6 n.r. 3,2 3,5 5,0 5,0 3,3 n.r. 3,4 CA E 3,4 n.r. 3,8 2,9 2,5 5,0 3,8 1,0 3,4 BP E 3,8 2,8 3,5 3,7 3,0 4,0 2,6 n.r. 3,9 FP E 3,0 2,8 2,2 3,2 3,0 4,0 2,6 n.r. 2,9 SR, CE, CC I 3,2 2,6 3,1 3,1 3,0 3,7 4,2 3,0 3,1 Source of data: MK Projekt et al, 2012. Note: n.r. ‐ ‘Not relevant’. * ‐ ‘Grupping columns and rows on 3 domains of KT’. Leverage effect appears as poor horizontal indicator since not all of the instruments demanded private financing in implementation of operations. However, evaluation found that private financing is involved in all instruments at least in costs for preparation of project proposals that broadly accounted to 2% of allocated funds (or 7% of all private financing); private funding was also needed to finance costs not eligible for public funding, but linked to project implementation (VAT, social charges, some reimbursements). These should be more systematically taken into 13 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) account, but with opposite rationale. Excellent innovation policy impacts are mostly evident only in sustainability criteria, with some reservation for SR, CE, and CC (3,7). Results in majority of other evaluated criteria show only moderately favourable impacts. Impacts on regional balance are especially poor. This is problematic in light of aspirations for achieving territorial smartness of innovation policy. E, I and R are to a large extent concentrated in smaller number of innovative regions, so that they increased differences between ‘innovative’ and ‘non‐innovative’ regions (but not between innovative regions). Regional concentration is strongly linked to prevailing technological character of innovation policy; ‘non‐innovative’ regions many times innovate in non-technological ways such as in new models of eco‐businesses, in social economy and in cultural production, which are absent from instruments evaluated here. Thus to strengthen territorial smartness, evaluation pleaded both for more innovative design of regional policy, as well as for broader focus of innovation policy. Instruments’ vertical impacts – in sector based perspective – are the most favourable for sector E (4,4; Table 2), while sector I stays observably behind (2,9), similarly also for R (3,1). Horizontal overlaps between domains of KT are described in correlation matrix. E and R are correlated in strongest overlap (3,6).1 E impacts R (3,2) regionally asymmetrically with poor contribution to improved business environment (Table 1), while R impacts E very favourably (4,0). Extent of inter‐sectoral orchestration between E and I is assessed with 3,5. E is too weakly linked to private sector, while I is not providing sufficient guarantees to E for sustainable use of new research infrastructure (Table 1). The weakest overlap is between I and R (3,1). Impacts on sector I maintain lower regional balance, lower employment and not optimal cost efficiency. Impacts on R on the other side do not excel in efficiency and also suffer from weak employment effect (Table 1). Overall smartness of innovation policy is assessed as good with 3,4. This summary indicator of overlap is obtained in Venn diagram as an average assessment of three binary overlaps. 1 Correlation coefficient in statistics ranges from ‐1 (negative), absent (0) to +1 (positive). Correlation in Table 2 is expressed qualitatively from absent or negative (1), weakly positive to strongly positive (2‐5). We are working with horizontal evaluation criteria which are by definition equally relevant for all policy instruments. In such case, absence or negative correlation are characterized as strategic problems because they wreck integrity of evaluated issue. 14 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) TABLE 2: Venn diagram of innovation policy’s impacts, on scale 1‐5 Leontief matrix R R E I Good R 3,1 4,0 3,1 (3,1) E 3,2 4,4 3,5 Good Good I 3,1 3,5 2,9 (3,6) (3,1) Good Correlation matrix (3,4) R E I Very Poor Good Good (2,9) R 3,1 3,6 3,1 E (4,4) (3,5) I E ‐ 4,4 3,5 I ‐ ‐ 2,9 Source of data: TABLE 1. 15 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) 5 Discussion Despite excellence in KT has not been achieved in general yet, summary indicator of overlap between three domains is rather favourable pointing to smartness of (specialization in)2 Slovenian innovation policy from 2007 to 2011. Evaluation found that instruments have strongly enhanced cooperation between domains of KT, in particularly by RD’s Instruments, and especially CE. Institutions have also introduced new models of cooperation which changed stakeholders’ behaviour, a clear sign for evaluation to recognise impact of policy interventions. Researchers have also changed their behaviour with initiating much stronger cooperation with companies. Achieved smartness (3,4) is evaluated as favourable at least relative to non‐overlapping, sector-based achievements (3,5, obtained as an average assessment for three pillars on diagonal of correlation matrix). Following theoretical elaboration we would expect different situation with observably stronger sectoral performance (‘dumb’) than inter‐sectoral (‘smart’). Overlaps are harder to achieve since they require new approaches to management, additional effort in coordination and developing partnership, consensus and synergies. Achieved moderate smartness of Slovenian innovation policy is not really entirely surprising if we take into account rather specific context in which instruments were implemented – deep economic crisis with close to 7% contraction in national GDP (2009‐2011). Large public deficits linked to stabilisation of financial sector imposed austerity policy that significantly cut public budgets of educational and research institutions. On the other side it was increasingly hard for companies to assure funds needed to exploit new market opportunities linked to new technologies (POR 2011; Bešter, Murovec, 2010), mainly due to the lack of loans from banking sector. This all led to enormous increase in number of projects that could not be implemented without public financial support. In this way Ministry has obtained strong leverage for overcoming sectoral barriers between three domains of KT and for decisive deepening inter-sectoral cooperation. 2 Extent and direction of innovation policy’s specialization is not measured here and remains addressed only indirectly through assessment of outcomes and impacts of policy instruments that enhance it, particularly in the case of RD instruments, YR and IS. Extent of specialization is reflected also in the assessment of sectoral achievements which are specialized by definition. More explicit address of specialization would be achieved with inclusion of indicators of specialization in Leopold matrix. 16 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) Yet, smartness of innovation policy needs to be read with caution amid observable weaknesses on the side of disappointing sectoral impacts, particularly for R and I. Very successful sectoral outputs have not been translated into very successful social‐wide impacts (partly understandable because impacts emerge gradually over longer period of time). Overlaps between sectors are thus instituted on weak sectoral fundamentals and therefore vulnerable. Exception is sector E. Favourable achievements in E are confirmed for Slovenia also in Competitiveness index (WEF, in EO, V/2011) and in OECD (2012). Sector R continues to lag behind in openness, in social responsibility in meeting societal challenges and in commercial exploitation of opportunities. Innovative companies still perform on lower level of productivity than average company nationally; share of the highest technology export remains at disappointing 5% (EO, May 2011); income from intellectual property rights remains very low (IUS, 2011). This among others suggested that innovation policy’s impacts in companies could be strengthened in fundamentals by the means of conventional industrial and competitiveness policies. Evaluation suggests that sectoral weaknesses are linked to poor learning capacity by policy-makers about how mechanisms of innovation policy function in practice (weak theory of change), poor needs assessment of beneficiaries and market opportunities, not ambitious planning of goals and in general overshooting specific unit costs for goals achievement. One of the most systematically recurring criticism expressed by beneficiaries is that administrative management of instruments is too formalistic and meeting formal demands many times seems more important than progress in innovation (for RD in general, for YR, and FP). Administrators at the Ministry are many times unwilling to adopt changes in operation even when it is obvious that improvements are feasible and justified. Beneficiaries opined that administrative management is many times unfriendly such as when calls for proposals were not announced, short application period and very demanding procedures, sometimes with weak support to applicants, sometimes with large share of justified objections, practicing long periods for funds reimbursement, and absence of pre‐financing. Beneficiaries were sometimes forced to accept role of passive followers of policy administrators and their understanding of innovation policy mission. This invoked opportunist behaviours in part of beneficiaries. Sadly it has been evidenced in some cases that public funds are misused by individual CEs, resulting in negative perceptions about the Instrument implementation. This sort of ‘dumbness’ in innovation policy arises superficially, as a result of overextended bureaucracy, it is not caused by narrow but nevertheless justified sector-based linear rationales in each KT domain. 17 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) In horizontal perspective it is noticeable (MK Projekt et al, 2012) that overlap between instruments of RD and instruments of HRD is weaker than overlap between instruments of the same Operational program. Evaluation pointed out to persisting barriers to horizontal synergies. The public research sector many times still poorly provides knowledge resources to companies in adequate quantity and quality (Foray, van Ark, 2008). WEF has observed that University programs also poorly serve needs of companies (in EO, IX/2011). Excessive disciplinary specialization proceeds at the expense of diminishing trans‐disciplinary approaches in research and training education. OECD (2012) outlined problematic fragmentation of research field on small groups which cover broad spectre of activities and dispersed financing of research in Slovenia. By opinion of POR (2011), research continues to be systematically neglected at Universities and is usually understood as only supplementary activity. Universities and research institutes sometimes still consider companies as a separate, perhaps even an undesirable world, and similarly also many companies do not consider interaction with universities or other research organisations as a strategic input into their future (COM(2006)‐604). Transfer of knowledge from E and R into I is still weak (SVREZ, 2014). Achieved increased employment of researchers in companies is to a large extent linked to subsidies and could perish together with diminished public financing (POR, 2011). Flow of knowledge also needs to feedback from companies to E and R. In this regard, evaluation emphasised unused potential for involvement of experts from SR, CE and CC into E and FP for transferring their innovative experiences back to institutions of knowledge. According to evaluation results, strengthening link between E and I may require the companies to be more involved into search for appropriate topic for PhD dissertations prepared under IS. Imperative for strengthening horizontal overlap between domains of KT implies that innovation policy needs to be innovated with hybrid solutions. In our view relevant proposal in this regard is the concept of integrated education at a ‘research universities’ (Schuch, 2013). It makes the research‐based learning the standard; it educates graduate students as apprentice teachers and cultivates a sense of community of learners (Roumen, Ilieva, 2007). University involves students as co‐creators of knowledge and as part of the innovation system (Markkula in Lappalainen, Markkula, 2013). The students are equal partners, developing and creating new professional knowledge and skills whilst growing towards their own fullest potential as human beings (Hirvikoski in Lappalainen, Markkula, 2013). Analogously, research organizations shall be strengthened especially in their intermediary function for enhancing their capacity to link new knowledge with societal challenges. They have 18 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) access to academic, mostly fundamental knowledge that they use and translate for the needs of their users (Zenker, Muller, 2008) in profit as well as in non‐profit projects. In knowledge‐based society companies also need to enhance profitability in increasingly hybrid way – integrally with improving their social responsibility, environmental sustainability and ethical standards. Finally, for smarter innovation policy, public sector innovation shall be pursued in administration and organization, in policy design and regulations, in service and goods delivery, in financial support and concepts (Windrum in Hollanders et al, 2013; Foray, van Ark, 2008). In EU, on average, two thirds of government institutions introduced innovation in their operations during preceding three years (UNU‐MERIT, 2011) – ranging from improved services to improved legislation – the latter being the strongest area of innovation in government. EC has introduced specific recommendations for simplification in administration, financing and implementation of Cohesion policy instruments.3 Hollanders et al (2013) estimated that companies that perceive an increase (improvement) of 1 unit in the index of public administration are 13.4% more likely to use services for innovation. And companies that use services for innovation are 27% more likely to innovate. 3 http://ec.europa.eu/regional_policy/sources/docgener/informat/2014/simplification_sl.pdf 19 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) 6 Conclusion Policy smartness does not need to become foggy immeasurable concept, useful only for decorative political talk. But measuring it may require innovative approach. Old style output and result based methodology in linear bottom‐up or top‐down approaches are only appropriate for assessment of specific and isolated concerns of sectors in achieving their fragmented goals, but they fail in evaluation of policy challenges that are complex in character and integrative in scope. The case study confirms initially stated hypothesis. Measuring smartness of innovation is mesoscopic challenge since it comprises two orthogonal explanatory axes: vertical, in a sectoral perspective and horizontal between overlapping sectors of innovation policy. All sectors are equally important even though leading innovation processes in independent directions. The contradiction can be resolved at meso level of evaluation. Foray (2013) explained that a too high level policy approach transforms policy into sectoral concern, but a too fine grained level transforms it into policy through which all projects of some merits will be funded where no specialization can take place. The smartness in innovation policy shall be addressed at middle level and with mid‐grained granularity (Foray, Goenega, 2013) just as it is suggested by triadic concept of KT. However, smartness is only one aspect of innovation challenge which suggests broadening the concept of specialization in innovation, that is not merely smart but fully mesoscopic. 20 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) Bibliography Baranger M. 2001. Chaos, Complexity, and Entropy: A physics talk for non‐physicists. Cambridge: New England Complex Systems Institute. Bešter J., N. Murovec N. 2010. Spremljanje inovativnosti slovenskih podjetij. Ljubljana, Inštitut za ekonomska raziskovanje. Carvalho M. 2010. Green Knowledge Triangle. http://www.gracacarvalho.eu/ Degani M. 2014. New “Horizons” of the Knowledge Triangle. Krakow. EC. Proposal for a Regulation of the European Parliament and of the Council on the European Institute of Technology. IMPACT ASSESSMENT ‐ integrating ex ante evaluation requirements. Commission Staff Working Document, COM(2006)‐604 final Ekins P. 1992. A Four‐Capital Model of Wealth Creation in Real‐Life Economics: Understanding Wealth Creation (in Ekins P., M. Max‐Neef, eds.). London, New York: Routledge, p. 147-55. Ekins P., Medhurst J. The European Structural Funds and Sustainable Development: A Methodology and Indicator Framework for Evaluation. Sage: Evaluation, 12/4(October 2006):474‐95. EO ‐ Ekonomsko ogledalo UMAR (Slovenian Economic Mirror, IMAD). Ljubljana: UMAR, September 2011. EO ‐ Ekonomsko ogledalo UMAR (Slovenian Economic Mirror, IMAD). Ljubljana: UMAR, May 2011. Flores‐Camacho F., Gallegos‐Cazares L., Garritz A., Garciá‐Franco A. Incommensurability and Multiple Models: Representations of the Structure of Matter in Undergraduate Chemistry Students. Springer, Science & Education 16(2007):775–800. Foray D. 2013. Measuring smart specialisation: entrepreneurial discovery, new activities and inclusiveness. Provinciehuis Groningen: S3 Thematic Workshop. Foray D., X. Goenega. The goals of smart specialisation. JRC‐IPTS ‐ S3 Platform, Policy Brief Series n° 1/2013 (May). Foray D., B. van Ark. 2008. Overview on Knowledge for Growth: European issues and policy challenges in EC. Knowledge for Growth. European Issues and Policy Challenges. European Commission, Directorate‐General for Research. Hirvikoski T. 2013. The Knowledge Triangle. Promoting Innovation and Multidimensional Learning, in Lappalainen, Markkula. Hollanders H., A. Arundel, B. Buligescu, V. Peter, L. Roman, P. Simmonds, N. Es‐Sadki. 2013. European Public Sector Innovation Scoreboard 2013. EC ‐ Directorate‐General for Enterprise and Industry IMD World Competitiveness Yearbook 2011. Lausanne: IMD World Competitiveness Center. IUS. 2011. The Innovation Union's performance scoreboard for Research and Innovation. European Commission, EC‐ Directorate‐General for Internal Market, Industry, Entrepreneurship and SMEs. Lappalainen P., M. Markkula (eds). 2013. The Knowledge Triangle– Re‐Inventing the Future. 21 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) European Society for Engineering Education SEFI. Leontief W. Environmental Repercussion and the Economic Structure: An Input‐Output Approach. Cambridge Mass.: The MIT Press, Review of Economics and Statistics, 52/3(1970):262‐71. Leopold L. B., F. E. Clarke, B. B. Hanshaw, J. R. Balsley. 1971. A procedure for evaluating environmental impact. Washington: Geological Survey Circular 645. Markkula M. 2013. The Knowledge Triangle Renewing the University Culture in Lappalainen, Markkula. Midtkandal I., J. Sörvik What is Smart Specialisation? Nordregio News Issue 5/2012. MK Projekt, Greta Associati, Metis, B. Radej, M. Dolinšek. 2012. Vrednotenje ukrepov za spodbujanje raziskovalno razvojnih aktivnosti v gospodarstvu in institucijah znanja (Mid-term evaluation of measures for stimulation of research and development activities in companies and in institutions of knowledge). Ljubljana, Final report. Mulgan G. 2007. Ready or not? Taking innovation in the public sector seriously. Nesta ‐ National Endowment for Science, Technology and the Arts. OECD. 2012. OECD Reviews of Innovation Policy: Slovenia. Paris: OECD Publishing. POR. 2011. Poročilo o razvoju 2011 (Development Report IMAD). Ljubljana: Urad za makroekonomske analize in razvoj. Radej B. Apples and Oranges: Synthesis without a common denominator. Ljubljana: Slovenian Evaluation Society, Working Papers, 7/1(February 2014a). Radej B., M. Golobič. Divided we stand: Social integration in the middle, Ljubljana: Slovenian Evaluation Society, Working Papers, 6/1(June 2013). Radej B. Social Complexity: Operational definition. Slovenian Evaluation Society: Working papers, 7/2(June 2014b). Reid A. 2010. Measuring up: evaluating the effects of innovation measures in the Structural Funds. Technopolis. RISS. 2011. Resolucija o raziskovalni in inovacijski strategiji Slovenije 2011–2020 (Resolution on research and innovation strategy of Slovenia). Ljubljana: Ministry for Science. Roumen N., S. Ilieva. 2007. Building a Research University Ecosystem: the Case of Software Engineering Education at Sofia University. Track on "Widened Software Engineering" (WISE 2007). Dubrovnik, Croatia, September 3‐7. Schuch K. 2013. Knowledge Triangle. Centre for Social Innovation. Vienna: OEAD. Sjoer E., B. Nørgaard, M. Goossens. 2012. Opportunities and Challenges in the Implementation of the Knowledge Triangle. SEFI, 40th annual conference, Thessaloniki, 23‐26 September. Stacey R.D. 2002. Strategic management and organisational dynamics: the challenge of complexity. 3rd ed. Harlow: Prentice Hall. SVREZ. 2014 Strategija pametne specializacije Republike Slovenije (Strategy of Smart Specialization for Slovenia). Ljubljana, Služba Vlade RS za razvoj in evropske zadeve. UNU‐MERIT. 2011. Innovation union scoreboard 2010. The Innovation Union's performance scoreboard for Research and Innovation. Maastricht Economic and social Research and training centre on Innovation and Technology, Inno Metrics. 22 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) Venn J. On the Diagrammatic and Mechanical Representation of Propositions and Reasonings. Philosophical Magazine and Journal of Science (July 1880). Zenker A., E. Muller. 2008. European Research Area and Applied Research: Context and Opportunities. Draft version. 23 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) _______________________________________About the SES's Working papers series___________ Publishes scientific and technical papers on evaluation of public policies and from related disciplines. Freely accessible on the internet. Already published: Volume 1 (2008), no: Excercises in Aggregation (In Slovenian; B. Radej, 23 pp) 1 Synthesis of Territorial Impact Assessment for Slovene Energy Programme (In Slovenian; B. Radej, 43 pp) 2 Meso-Matrical Synthesis of the Incommensurable (B. Radej, 21 pp) 3 Volume 2 (2009), no: Anti-systemic movement in unity and diversity (B. Radej, 12 pp) 1 Meso-matrical Impact Assessment - peer to peer discussion of the working paper 3/2008(B. Radej, 30 pp) 2 Turistic regionalisation in Slovenia (in Slovenian, J. Kos Grabar, 29 pp) 3 Impact assessment of government proposals (In Slovenian; B. Radej, 18 pp) 4 Performance Based Budgeting (In Slovenian; B. Radej, 33 pp) 5 Volume 3 (2010), no: Beyond »New Public Management« doctrine in policy impact evaluation (B.Radej, M.Golobič, M.Istenič) 1 Basics of impact evaluation for occassional and random users (In Slovenian; B. Radej) 2 Volume 4 (2011), no: Intersectional prioritisation of social needs (In Slovenian; B. Radej, Z. Kovač, L. Jurančič Šribar, 45 pp) 1 Primary and secondary perspective in policy evaluation (In Slovenian; B. Radej, 30 pp) 2 Aggregation problem in evaluations of complex matters (In Slovenian; B. Radej, 41 pp) 3 Movement 99%: With exclusion to the community (In Slovenian; B.Radej, 42 pp) 4 Volume 5 (2012), no: Excellence squared: self-assessment in public administration with CAF (In Slovenian; B.Radej, M.Macur) 1 Partial whole: example of territorial cohesion (In Slovenian; B.Radej, M.Golobič, 31 pp) 2 Volume 6 (2013), no: Divided we stand: Social integration in the middle (B.Radej, M.Golobič, 26 pp) 1 With Exclusion to the Community (B.Radej, July 2013) 2 Volume 7 (2014), no: Apples and Oranges: Synthesis without a common denominator (B.Radej, February 2014, 40 pp) 1 Volume 8 (2015), no: Measuring Smartness of Innovation Policy (B.Radej, K. Ž. Jazbinšek, M. Dolinšek, June 2015, 25 pp) 1 SDE operation is nonfinacially supported by IER – Institut for economic research, Ljubljana, http://www.ier.si/ 24 Radej, Žvokelj Jazbinšek, Dolinšek Slovenian Evaluation Society – Working Papers 8/1(June 2015) Naslov »Measuring Smartness of Innovation Policy« Podatki o avtorjih Radej Bojan, Karin Žvokelj Jazbinšek, Metod Dolinšek Podatki o izdaji ali natisu 1. izdaja/ 1st edition Kraj in založba Ljubljana: Slovensko društvo evalvatorjev / Slovenian Evaluation Society Leto izida 2015 (Vol. VIII, No. 1) Naslov knjižne zbirke Delovni zvezki SDE/ Working papers SES Podatki o nosilcu avtorskih pravic SDE, Ustvarjalna gmajna 2.5/ Creative Commons 2.5, Slovenija Podatek o nakladi (število Elektronska publikacija, http://www.sdeval.si/komisija‐ natisnjenih izvodov) za‐vrednotenje/publikacije/589‐measuring‐smartness‐of‐ innovation‐policy Mednarodne identifikatorje (ISBN, ISBN 978-961-92453 ISMN, ISSN) Maloprodajno ceno publikacije Publikacija je brezplačna / Free 25 Radej, Žvokelj Jazbinšek, Dolinšek