DOI: 10.17573/ipar.2015.3-4.10 1.01 Original scientific article How to Assess Whether Qualified Evaluations of e-Government Projects Are Conducted? The Case of Slovenia Jože Benčina University of Ljubljana, Faculty of Public Administration, Slovenia joze.bendna@fu.uni-lj.si Tina Jukic University of Ljubljana, Faculty of Public Administration, Slovenia tinajukic@fu.uni-lj.si ABSTRACT Countries are investing significant resources in the development of e-government, but evaluations of e-government projects are rarely conducted in a comprehensive way. In order to take the right measures and to be effective in fostering the use and raising the quality of evaluations of e-government projects it is important to reveal the state of the practice. The paper presents the results of the research based on identification of parameters required to be measured in order to conduct qualified evaluations of e-government projects. Thirteen such paramaters were identified via the focus group, with participants comprising (Slovenian) e-government decision-makers and researchers. Using these 13 parameters and 7 additional questions about the knowledge and use of evalution methods a questionnaire was designed and applied in a survey on Slovenian public administration. Employing a binomial test and Fisher's combined probability test it was established that in Slovenian public administration qualified evaluations of e-government projects were not being conducted. The method has demonstrated an eligible usability both for analysing the general situation and for exposing opportunities for improvement in various specific situations. Keywords: e-government, e-government project, evaluation, methodolgy JEL: K40 1 Introduction Over the past 15 years, there has been put an enormous effort into the development of e-government. This is due to the fact that e-government has become one of the priority fields within the national (and international) development strategies (e.g. EU Action plan, Slovenian Strategy). Therefore, Bencina, J., & Jukic, T. (2015). How to Assess Whether Qualified Evaluations of 235 e-Government Projects Are Conducted? The Case of Slovenia. International Public Administration Review, 73(3-4), 235-255. Jože Benčina, Tina Jukic it is not surprising that countries have invested significant resources in the development of e-government services, Nevertheless, despite significant resources being invested in the development of e-government (castelnovo & Simonetta, 2007; eGEP, 2006a; Heeks, 2012), decisions on how these resources should be spent or which services should have priority in the development and implementation thereof have not (always) been well-grounded. A number of projects in e-government were the result of lobbying (Kertesz, 2003; Yildiz, 2007) or of "copying" other organizations; in some cases, such decisions were simply made ad hoc (Datar, 2010) - even though the literature suggests that proper planning is one of the key success factors in successful e-government implementation (Rana, Dwivedi, & Williams, 2013). Therefore, some countries have already developed their own methodologies for evaluating e-government projects. Also, this topic has attracted a lot of attention in research (Jukic & Bencina, 2015; Stanimirovic, Jukic, Nograsek, & Vintar, 2014; Todorovski, Jukic, Leben, & Vintar, 2014; Nograsek, Stanimirovic, Jukic, & Vintar, 2012). Some of these approaches encompass over a hundred parameters of evaluation (different categories of costs, benefits, risks etc.). Methodologies with so many parameters possibly make sense in countries with a highly-developed evaluation culture. But the same does not hold true for countries where evaluation has not yet become an equally important phase of the project cycle, nor for the countries where project management mechanisms have not beent fully in place yet. These countries may use a narrower set of parameters in their initial evaluations of e-government projects - as much as necessary for a qualified evaluation, Besides that, the raising of awarness must be supported by a clear insight into the state in the field of evaluation of e-government projects. For this end a simple and efficient tool for the assessment of the general situation and for the exposition of the opportunities for improvment in different subfields or specific e-government projects is needed. The research question addressed in this paper is: Are qualified evaluations of e-government projects carried out in Slovenian public administration? In this regard, four objectives have been pursued: 1. to present the existing approaches to the evaluation of e-government projects; 2. to define the parameters required for qualified evaluation of e-government projects; 3. to define the questionnaire for the assessment of the status quo regarding qualified evaluation of e-government projects, 4. to conduct a survey on the state in the field of qualified evaluations of e-governemnt projects in the Slovenian public administration. 236 International Public Administration Review, Vol. 13, No. 3-4/2015 How to Assess Whether Qualified Evaluations of e-Government Projects Are Conducted? The Case of Slovenia In the next chapter, a theoretical background of evaluation is given, followed by a chapter which presents the research methodology. In the fourth chapter, the results of the research are presented. In the last chapter, the final conclusions are summarized. 2 About Evaluation Based on Owen's (Thomas, Seddon, & Fernandez, 2008, p. 79) definition of evaluation, we similarly define evaluation of e-government projects as the process of providing information in order to support decision-making on the realization/implementation of the services. The literature provides a range of typologies of evaluation, e.g. according to time, driver, and subject of evaluation, as well as level of its aggregation. For the purpose of this paper, typology according to the time in which an evaluation is to be conducted is of crucial importance. From this point of view, we draw a distinction between ex-ante and ex-post evaluation. When conducting ex-ante evaluation, we tend to evaluate all potential alternative solutions and their effects (Kustec Lipicer, 2002). Thus, the purpose of ex-ante evaluation is to make a forecast (Remenyi, 1999), mainly on the basis of the anticipated costs, benefits, effects, opportunities, and challenges (Homburg, 2008). Ex-ante evaluation is of a much more theoretical nature than ex-post evaluation, which follows the phase of implementation (Pollitt, 2008). The general purpose of ex-post evaluation is to gather and evaluate effects (Kustec Lipicer, 2002), e.g. economic, social or whichever outcomes (Homburg, 2008). Ex-post evaluation may also serve as an important source of feedback for further ex-ante evaluation, ultimately making it more realistic (Cracknell, 2000). 3 Methodology The methodology was designed in accordance with the research objectives in three segments: • definition of the parameters required for the qualified evaluation of e-government projects, • definition of the tool for the assessment of the status quo in the evaluation of e-government projects, • design of the survey on the status quo in Slovenian public administration. To identify the parameters needing to be measured in order to conduct qualified evaluations, the following steps were undertaken: first, on the basis of literature regarding e-government evaluation and current approaches to evaluating e-government projects, an extended list of potential parameters was prepared. In order to identify the parameters required for qualified ex-ante evaluation of e-government projects, the focus group combined with Mednarodna revija za javno upravo, letnik 13, št. 3-4/2015 237 Jože Benčina, Tina Jukic the Delphi method1 was conducted with two representatives of the (former) Ministry of Public Administration and two representatives of the Slovenian e-government research area, Once the parameters of qualified evaluation were identified, a web questionnaire was constructed. The questions were, with the exception of those intended for identifying the respondents' profiles, designed in the form of statements; respondents were asked to express their level of agreement with those statements using a seven-point scale, where 1 represented the lowest level of agreement, and 7 the highest level of agreement with a particular statement, The statments measure respondents' level of agreement with statements related to the measurement of those of the above-mentioned evaluation parameters that were defined as crucial for qualified evaluation, how familiar they are with the methods for evaluating projects, and how common the use of the suitable methods is in practice, To answer the research question "Are qualified evaluations of e-government projects conducted in Slovenian public administration?" a survey was designed, This way the appropriateness of the survey as the supporting tool for raising awarness and fostering improvement in the qualified evaluation of e-government projects was justified, The previously mentioned web questionnaire was employed as the data-gathering tool, A web survey designed with The Survey System 9,0 was conducted, The sample covered the Slovenian public administration; more precisely, it included 266 organizations, namely: • 15 ministries; • 40 bodies affiliated with these ministries; and • 211 municipalities, The survey was first tested with seven respondents and modified based on the feedback gathered. Then, invitations were sent via email to the CIOs2 in the case of ministries and their affiliated bodies, and to the directors of the municipal administrations in the case of municipalities, The email included a request to forward the invitation to the organisation's most knowledgeable person on making decisions related to the development and implementation of e-government services offered to citizens and businesses by their organization, Suitable descriptive statisitics were used for data analysis, The research question was answered by testing the hypothesis, Due to small sample numerous a nonparametric binomial test was employed and to combine 1 The focus group followed consensus building Delphi procedure in four rounds, In the first round, process began with an open-ended questionnaire, In the second step, each Delphi participant rated the parameters, In the third round, each member of the focus group received a questionnaire with parameters and ratings, In the final round, to provide the final opportunity for the revision of judgments, the list of remaining parameters with ratings was distributed to participants, 2 Chief Information Officers, 238 International Public Administration Review, Vol. 13, No. 3-4/2015 How to Assess Whether Qualified Evaluations of e-Government Projects Are Conducted? The Case of Slovenia the results from several independent tests bearing upon the same overall hypothesis (H0) the Fisher test (i.e. Fisher's combined probability test -Equation 1) was used. Statistical analyses were carried out with the help of The test comprises two steps. First, based on the Fisher form, the chi-square value X2 is calculated as the sum of the natural logarithms of k p-vaiues multiplied by -2. Then from the probability distribution of the chi-square value, considering 2k degrees of freedom, the combined p is derived. The three methodological segments form an open research framework, where the results of the first step allow for a variety of possible continuations. This could be the evaluation of e-government projects, benchmarking the analysis of chosen projects, analysing the efficiency of organisational units, sectors, communities, regions, or states, or analysing the attitude and practice of the responsible professionals towards qualified evaluations of e-government projects. The later issue is the main challenge of our research covered by the latter two methodological segments. 4 Presentation of the Results The study was a part of a broader research project which considered the whole e-government project life cycle. From the three phases of the project-controlling cycle (ex-ante evaluation, operational control, and ex-post evaluation) the reserach question under consideration addresses ex-ante evalution. This put the focus of the study on this specific project phase and encouraged the development of a tailored solution supporting development and improvement in ex ante evalutaion of e-government projects. 4.1 Definition of the Parameters As mentioned before, the starting point in the study was the extended list of potential parameters. A review of the literature reveals at least ten approaches to the ex-ante (and ex-post) evaluation of e-government projects and services (Table 1). These are developed either at the conceptual or applicative ieveis. The majority of approaches are designed around the parameters of costs, benefits, and risks. On the basis of these approaches and the related literature, the focus group was conducted with two representatives of decisionmakers from the Ministry of Public Administration and two representatives of academia (the field of e-government). The aim of the focus group was to define the main parameters required for qualified ex-ante evaluation of e-government services. 13 such parameters were selected by the participants of the focus group. These parameters are the indicators used in order to measure the construct of ex-ante evaluation in the research model proposed in this paper. They are: SPSS. (1) i=1 Mednarodna revija za javno upravo, letnik 13, št. 3-4/2015 239 Jože Benčina, Tina Jukic • the costs of planning (labour costs related to an analysis of the current situation, planning, preparation, and execution of a public tender); • the costs of development; • the costs of implementation (equipment costs and costs related to organizational change - e.g. the costs of re-organization, management, research, and development); • the costs of operation (e.g. the costs of the organization's own staff when the service is in operation, training costs, operational material costs, maintenance costs); • source of project financing (the possibility of grants); • the benefits for external users - citizens, businesses, NGOs3 (time and financial savings, easier decision-making); • the benefits for internal users - public administration employees/ organizations (time and financial savings, an increase in solved cases, an increase in employee satisfaction, a decrease in their workload, better decision-making, improved communication/collaboration, fewer user complaints, fewer mistakes at work); • the strategic and political benefits (e.g. alignment with the strategic plans of the organization, the Slovenian public administration and/or of the European Union, normative commitment to the implementation, etc.); • general and other benefits (a lower level of corruption, a higher level of transparency, a more accountable public administration, the development of socially important areas); • realization risk (due to the technological complexity of development and implementation); • political risk (a lack of political support); • organizational risk (due to the organizational complexity of development - e.g. the participation of multiple organizations); • usability of the services (i.e. the results of the projects) in terms of their integrity, design following users' life events, multi-channel supply etc, 3 Non Governmental Organisations. 240 International Public Administration Review, Vol. 13, No. 3-4/2015 How to Assess Whether Qualified Evaluations of e-Government Projects Are Conducted? The Case of Slovenia Table 1: Review of the selected approaches to evaluating e-government projects (first-level parameters and number of all parameters) Approaches First-level parameters Number of parameters MAREVA (ADAE, 2007) - State financial value - Direct customer value - Risk - Public services social & operational value - Project necessity > 90 WiBe (Rothig, 2004) - Economic efficiency in the monetary sense (profitability) - External effects - Qualitative and strategic importance - Urgency 88 DAM & VAM (AGIMO, 2004) - Costs - Benefits - Risks - Demand assessment 174 eGEP (eGEP, 2006b) - Costs - Benefits (Effects) - Risks 149 EU-VAST (European Commission, 2001) - Costs - Benefits for the EU - Benefits for the European Commission - Financial costs and benefits - Risks - Urgency 168 VMM (OGC, 2003) - Costs - Benefits - Risks 64 Kertesz (Kertesz, 2003) - Costs - Benefits - Risks 59 Datar (Datar, 2010) - Citizens' perspective - Organizations' perspective - Stage wise costing parameters 33 Source: own, based on sources listed in the table 4.2 Questionnaire The second research step was the definition and the development of the questionnaire. Its core content is based on the results of the first research step and consists of 13 statements in 4 groups which follow the 13 parameters of qualified ex-ante e-government project evaluations. Survey participants were asked to assess their level of agreement with statements related to the measurement of the above-listed evaluation parameters, which were defined as crucial for qualified evaluation. The statements are presented in Table 2, The second part of the questionnaire deals with the question of how well the public sector professionals are familiar with the methods for the evaluation of projects and how common the use of the methods is in practice. It consists of 3 questions about the knowledge of methods for evaluating the projects and 4 questions about the use of suitable methods. The statements are presented in Table 3. Mednarodna revija za javno upravo, letnik 13, št. 3-4/2015 241 Jože Benčina, Tina Jukic Participants respond to the statements with their level of agreement on the 7-point scale: completely disagree, mostly disagree, somewhat disagree, neither agree nor disagree, somewhat agree, mostly agree, completely agree, The last part of the questionnaire gathers facts about the type of organisation (municipalities, ministries, affiliated bodies), the age of respondents, the duration of employment in public administration, the level of the respondents' influence on decisions, and the level of knowledge of the decision-making process, This makes it possible to analyse the quality of data regarding the role and structure of the pool of respondents. 4.3 Survey Research Results The survey was conducted in January 2011, and the data were collected for one month, It was answered by 51 respondents, representing a 19% response rate. Most respondents (68%) were from municipalities, followed by those from ministries (18%) and their affiliated bodies (14%), The sample is biased to municipalities, however 19.6% of municipalities have more than 10,000 inhabitants, so over 50% of units are ministries, affiliated bodies and bigger municipalities, Beside that, the Mann Whitney test for two independent samples confirmed statistical signifant differences between two groups of organisation (ministries and affiliates, municipalities) only for one out of 20 variables of survey. 86% of the respondents were older than 34, and most of them (88%) had been employed in public administration for 5 years or more, which indicates good knowledge of the way public administration functions, The respondents have a relatively significant impact on decisions regarding the realization of e-government projects in their organizations (with an average score of 5 on a 1-7 scale), and the same holds true for their knowledge of the decisionmaking process in the realization of such projects in their bodies (average score 5). Therefore, it can be deemed that, despite the low response rate (19%), the responses gathered are of relatively high quality, The presentation of the survey results is based on the descriptive statistics. The statistical significance of the differences between groups of parameters and between knowledge and use of suitable evalutation methods was tested by the paired t-test (McDonald, 2009, p. 180). The aggregated results of the level of ex-ante evaluation of e-government projects in the Slovenian public sector (Graph 1) show a very high level of the frequency of the use of ex-ante evaluation amongst respondents, 242 International Public Administration Review, Vol. 13, No. 3-4/2015 How to Assess Whether Qualified Evaluations of e-Government Projects Are Conducted? The Case of Slovenia Graph 1: Ex-ante evaluation of (aggregated) four groups of parameters in Slovenian public administration 6.0 - 5.8 ■ Costs Benefits Risk Usability Source: own Somewhat suprisingly the scores for usability and benefits are on the top of the standings, where the differences between the two are statistically insignificant (paired t-test, p = 0.098). Both are an important factor driving the extent of the use of e-government services, although the researchers report that in general the use of e-government services does not meet expectations (Kumar, Mukerji, Butt, & Persaud, 2007). The other 2 factors of exante evalutions costs and risks are valued significantly lower (paired t-tests, p < 0.005), with statistically insignificant differences between them (paired t-test, p = 0.085). The results express the quite common characteristic of public servants' attitudes, where the consideration of risks and costs is often overlooked. The respondents value most aspects of the ex-ante evaluation of benefits nearly as highly as usability, while the consideration of strategic and political benefits takes a slighltly lower range (Table 2). Along with the lowest-scored aspect of the risks, consideration of the political risk (political support for the project), a lack of strategical and political consideration could be concluded. This phenomenon is characteristic for less-developed environments and diminishes the effectiveness and efficiency of e-government. Considering the evaluation of costs, the interresting fact is that the aspect of evaluating financing sources takes the top score, way above the ex-ante evaluations of various kinds of costs. The lowest score for risk evaluation supports the assumption that in the public sector the importance of considering projects' cost and risk issues is neglected. 5.8 5.6 c 5.4 ro u ^ 5.2 5.0 4.8 4.6 Mednarodna revija za javno upravo, letnik 13, št. 3-4/2015 243 Jože Benčina, Tina Jukic Table 2: Measurement of parameters required for qualified evaluation of e-government projects Please express your level of agreement with the following statements about the evaluation of various cost categories when deciding about the realization of e-government projects in your organization. Average When we conduct ex-ante evaluation, we always consider the costs of planning (the costs of employees related to an analysis of the current situation, planning, preparation, and execution of a public tender). 5.G6 When we conduct ex-ante evaluation we always consider the costs of development. 5.22 When we conduct ex-ante evaluation we always consider the costs of implementation (the costs of equipment and costs related to organizational change - e.g. the costs of re-organization, management, research, and development). 5.18 When we conduct ex-ante evaluation we always consider the costs of operation (e.g. the costs of the organization's own staff when the service is in operation, training costs, operational material costs, maintenance costs). 5.24 When we conduct ex-ante evaluation we always consider the source of project financing (the possibility of obtaining grants). 5.63 Please express your level of agreement with the following statements about evaluation of different benefits categories when deciding about realization of e-government projects in your organization. When we conduct ex-ante evaluation we always consider benefits for external users - citizens, businesses, NGOs (time and financial savings, easier decisionmaking). 5.78 When we conduct ex-ante evaluation we always consider benefits for internal users - public administration employees/organizations (time and financial savings, an increase in solved cases, an increase in employee satisfaction, a decrease in their workload, better decision-making, improved communication/ collaboration, fewer user complaints, fewer mistakes at work). 5.75 When we conduct ex-ante evaluation we always consider strategic and political benefits (e.g. alignment with the strategic plans of the organization, the Slovenian public administration and/or of the European Union, normative commitment to the implementation, etc.). 5.24 When we conduct ex-ante evaluation we always consider general and other benefits (a lower level of corruption, a higher level of transparency, a more accountable public administration, the development of socially important areas). 5.69 Please express your level of agreement with the following statements about evaluation of different risk categories when deciding about the realization of e-government projects in your organization. When ex-ante evaluation of e-government projects in our organization is conducted, we ... always consider the realization risk (technological complexity of development and implementation). 5.24 always consider the political risk (political support for the project). 4.57 always consider the organizational risk (organizational complexity of development - e.g. the participation of multiple organizations). 5.31 Please, assess your agreement with the following statement. When deciding about the realization of e-government projects, our organization always considers the usability of the services (i.e. the results of the projects) in terms of their integrity, design following users' life events, multi-channel supply, etc. 5.8G Source: own The answers to the second part of the questionnaire show considerably different facts. The aggregated scores of knowledge and use of suitable methods (Graph 2) are significantly lower than the scores of ex-ante 244 International Public Administration Review, Vol. 13, No. 3-4/2015 How to Assess Whether Qualified Evaluations of e-Government Projects Are Conducted? The Case of Slovenia evaluation parameters. A closer look at the individual questions reveals that the respondents feel a lack of knowledge of information about projects and investment project evaluation methods. Unsurprisingly the same holds for the use of the methods, where the result is even slightly worse, but the difference is not statistically significant (t-test, p = 0.219). Graph 2: Knowledge and use of suitable methods for project evaluation 4.2 - 4.1 4.1 4 c ro