Efficiency Measurement of Digitalization on EU Countries: A Study Based on Data Envelopment Analysis Esin Cumhur Yalçın Kırklareli University, Turkey esincumhuryalcin@gmail.com Propose: The current research aims to identify the countries that have transformed digitalisation into their growth and employment rates in the European Union countries between 2015 and 2019 and measure the relevant countries' performances (the effectiveness). Study design/methodology/approach: In the study, the five dimensions of the DESI index, Connectivity, Human Capital, Use of Internet, Integration of Digital Technology and Digital Public Services were used as input, while GDP Growth Rate and Employment Rate belonging to EU-28 countries between the years 2015-2019 were used as output. The VRS-INPUT Oriented DEA method was used to calculate the maximum output with the available inputs in the analysis. Findings: In countries such as Bulgaria, Cyprus, Hungary and Romania, it can be stated that the effective use of DESI indicators, which are considered a measure of digitalisation, has a one-on-one effect on the GDP and ER values included in the model as output. DESI indicators are not used effectively in the GDP and ER values of countries like Denmark, Finland, and Spain. In general, in terms of digitalisation, it has been revealed that developing countries use digitalisation more effectively in terms of economic growth and job creation. Originality/value: This article is a pioneering study exploring the identification of EU countries by data envelopment, translating digitalisation into growth and employment rates. In this context, current research is valuable in terms of closing a significant gap in the literature. Keywords: digitalization, DESI, economic growth, employment rate, Data Envelopment Analysis (DEA). Introduction Almost every country in the world is taking initiatives to grow economically and open up new employment areas for citizens. Countries can support economic growth and create new employment opportunities with different strategies. In this context, some countries may aim to create an agriculture-based strategy, some to support industrial production, and some to achieve this through the services industry. Regardless of the engine of countries’ development goals, each country inevitably becomes involved in this process with technological development and digitalisation. Digitisation, defined as the digitisation of analogue data for easy access and sharing, closely affects every area from economy to health, tourism to marketing. One of the interesting points is that digitalisation affects the areas of economic growth and employment. The subject in question is one of the important questions of both economics and sociology. Some authors claim that digitalisation will boost growth but may have a suppressive effect on employment. In contrast, others suggest that digitalisation will support economic growth and employment by providing more jobs than those lost. Current research also aims to answer this complex question with an application performed under certain restrictions. Accordingly, the current research aims to identify the countries that have transformed digitalisation into their growth and employment rates in the European Union countries between 2015 and 2019 and to measure the performances, or in other words, the effectiveness of the relevant countries. International Journal of Management, Knowledge and Learning | ISSN 2232-5697 Volume 10 (2021) 323–333 | https://www.doi.org/10.53615/2232-5697.10.323-333 In this context, one of the critical discussions in literature is the measurement of digitalisation. Because it is also worth noting that there is no unity of meaning in literature in the measurement of digitalisation; however, some of the popular tools used in the measurement of digitalisation are; "Enabling Digitalization Index - EDI" developed by Euler Hermes (Hermes, 2020); "Digital Adoption Index - DAI" developed by The World Bank (World Bank, 2020); "The Digitalization Index – DiGiX" (BBVA, 2020) developed by BBVA Research and "Digital Economy and Society Index – DESI" developed by European Commission (European Commission, 2020). In this research, DESI was preferred for the measurement of digitalisation as this index not only presents the data collected in the last five years in an organised manner but also measures digitalisation in terms of economic (for example, the share of sales made by businesses over the internet) and social (for example, the level of health services provided by the state through electronic applications). Current research is valuable in terms of closing a significant gap in the literature. Accordingly, literature surveys conducted by the author have shown that economic growth and employment issues are often studied in European Union countries. However, the determination of countries that convert digitalisation to growth and employment rates by data enveloping is not the specific subject of any research. In this respect, the study aims to make a significant difference in the context of the countries of the European Union in terms of determining the winning countries of digitalisation. The research carried out for this purpose consists of four parts. First, the conceptual framework is given in the first and second parts of the research. Then, in the third part, the method, in the fourth part, the data, and in the final part, the results are discussed. Digital Economy and Society Index (DESI) DESI is an index that presents digital performance in European countries as a summary of certain indicators. This index also reveals the evolution of digital competitiveness in European countries (European Commission, 2020a: 10). The DESI index is calculated by combining five main components in specified proportions. These five main components consist of 37 sub- components. The five main components that makeup DESI are also considered digital competition areas of EU countries. Accordingly, DESI consists of (i) Connectivity, (ii) Human Capital, (iii) Use of the Internet, (iv) Integration of Digital Technology and (v) Digital Public Services. For the calculation of the DESI index, connectivity is evaluated as (25%), Human Capital (25%), Use of the Internet (15%), Integration of Digital Technology (20%) and Digital Public Services (15%). The score obtained from these calculations ranges between 0 and 100. The decrease in the score of the countries indicates that the level of digital competitiveness decreases in terms of both components and the main index. In contrast, the increase in the score indicates that the level of digital competitiveness increases in terms of both components and the main index. Examining these five components is essential to reveal DESI's identity and main purpose (European Commission, 2020b). Connectivity: DESI's connectivity component consists of two sub-dimensions. These are both the demand and supply sides of fixed and mobile broadband internet. Under fixed broadband, the availability of general and ultrafast broadband, fast broadband and fixed, very high capacity networks are considered. Mobile broadband includes 4G coverage, mobile broadband usage and 5G readiness. These dimensions also consider the prices offered to consumers, and connectivity is recognised as a social right in the EU. Denmark has the highest score when the score is examined on a country-by-country basis for 2020, followed by Sweden and Luxembourg. The weakest performance for this component of DESI is Greece, Cyprus and Bulgaria (European Commission, 2020c: 4). Human Capital: DESI's human capital component has sub-components of "internet user skills" and "advanced skills and development". The first component is based on the European Yalçın | Efficiency Measurement of Digitalization on EU Countries 324 Commission's Digital Skills Indicator, which is calculated based on the number and complexity of activities involving the use of digital devices and the internet. The second component includes indicators related to ICT experts and ICT graduates. According to the latest data, Finland is the leader in both sub-components of Human Capital, followed by Sweden and Estonia in terms of overall performance. Italy, Romania and Bulgaria are at the bottom of the Human Capital component (European Commission, 2020d: 3). Use of the Internet: DESI's use of the internet component measures how many people use the internet and what activities they carry out online. The activities here are divided into three main groups. The first group includes modern communication activities (e.g., participating in video calls), the second group includes the use of online content (e.g. music, movies, TV social interaction), and the third group includes the percentages of those who use online shopping and banking transactions. According to the latest data, Finland, Sweden, and the Netherlands are the countries with the most active use of internet users. At the same time, Romania, Bulgaria and Italy are the least active countries (European Commission (2020e: 3). Integration of Digital Technology: DESI's integration of digital technology components measures the digitisation of businesses and e-commerce. This component includes electronic information sharing rates of enterprises, social media usage rates, big data and cloud computing usage rates, and evaluates indicators such as the percentage of enterprises' domestic and international online sales. By bringing digital technologies into the business, businesses can gain a competitive advantage and develop their products and services. According to the latest data, the best performing countries are Ireland, Finland and Belgium, while Bulgaria, Romania and Hungary are the lowest-scoring countries (European Commission, 2020f: 4). Digital Public Services: DESI's digital public services component measures the supply and demand side of digital public services. The expansion of digital technologies brings new responsibilities and expectations to the public sector. Effective e-government can deliver a wide range of benefits for governments and businesses, including transparency, openness, efficiency and savings (European Commission, 2020g: 3). Finally, when the DESI index is evaluated as a whole; on the one hand, both the demand and supply side of fixed and mobile broadband internet; on the other hand, the number of internet usage activities and ICT specialists and ICT graduates; and in addition, how many people use the internet and what activities they carry out online, the digitalisation of businesses and e- commerce, and finally both the supply and demand side of digital public services are measured. (European Commission, 2020a; Başol & Yalçın, 2021). According to the latest data shown in Figure 1, Finland, Sweden, and Denmark are the countries with the highest DESI score. In contrast, the countries with the lowest score are Bulgaria, Greece and Romania. 2015 DESI Overall Index 2016 DESI Overall Index Yalçın | Efficiency Measurement of Digitalization on EU Countries 325 2017 DESI Overall Index 2018 DESI Overall Index 2019 DESI Overall Index Figure 1. 2015-2019 DESI Overall Index (Source: https://digital-agenda-data.eu) Digitalisation, Economic Growth and Employment Economic growth represents an increase in the number of final goods and services produced in a country over a certain time. The concept of economic growth, which is also an expression of increases in real income per capita, mainly defines long-term increases in production capacity, which are related to the supply direction of the economy. These increases in production capacity are closely related to the development process in advanced technological and institutional structures. One of the crucial elements that ensure economic growth is increased physical capital and human capital levels and productivity per capita through investments and technological development. Technological developments gained further momentum in the 20th century as the research process became increasingly institutionalised. On the other hand, employment measures the extent of utilisation of available labour resources (OECD, 2013: 2). Employment comprises all working-age persons who during a specified brief period (ILO, 2005: 1). The process of economic growth begins mainly with the function of production. Accordingly, it is possible to consider economic growth as an increasing function of the labour force (L), natural resources (N), capital (C) and technology (T). The resulting increase in resources of growth: is expressed as Y = f{ΔL, ΔC, ΔN, ΔT}. As can be seen from the function, due to the limited change in natural resources, the development of economic growth depends on improvements in labour, technology and capital components. It is possible to summarise the literature's technological change-based growth theories in two groups. Solow (1957) first included the technological development in the growth equation, which he defined as the "Solow Yalçın | Efficiency Measurement of Digitalization on EU Countries 326 residual" and accepted as an extrinsic Cobb-Douglas production function, and that 70 per cent of the growth is due to technological transformation. Another growth theory was put forward by Romer (1986), who stated that the growth model has a driving effect on market forces by accepting technological development internally. Different perspectives emerge when the impact of digitalisation and technology on employment is examined in the literature. According to some views, it is claimed that technological advances will reduce unemployment rates by creating new employment areas. In contrast, an opposing view claims that due to technological advances, business owners state that they will adopt the situation where the labour and capital ratio is the cheapest, and this will cause a decrease in the demand for labour over time. In this context, it is an important research topic in literature to reveal the effect of digitalisation on economic growth and the labour force and the relationships between them from another point of view of technology. Method Technical efficiency and productivity indices are widely used to determine the production performance of an enterprise, a sector, a country, or, in other words, a unit of production (Yanga, Choib & Leec, 2021; Ibrahim, Alola & Ferreira, 2021; Li & McNeil, 2021; Puyenbroeck, Montalto & Saisana, 2021). Efficiency is the ratio of the maximum output obtained by the best production techniques to the actual output or the capacity of decision- making units (DMUs) to produce the maximum output possible with existing technology. The change in productivity, in general, is an indicator of how much the national economy has internalised global technology and how this is reflected in total factor productivity (Coelli et al., 1998, p:134). Different methods can be used to measure total factor productivity and changes in its components. The two most commonly used methods are Stochastic Production Boundary Analysis and Data Envelopment Analysis-DEA. The DEA is based on the assumption that resources are being used effectively. In other words, some economic units produce below the level of production that is not efficient or defined by the best techniques. The stochastic boundary approach applies parametric econometric methods, while the DEA applies nonparametric mathematical methods (Charnes et al., 1978). DEA, a linear programming-based method, is used in the present study. Data Envelopment Analysis Data Envelopment Analysis is a linear programming-based technique developed to measure decision-making units' performance (relative effectiveness) that use the same inputs and produce the same outputs. DEA is a linear programming-based method developed to measure the relative effectiveness of assets that convert economic inputs that produce similar goods or services into outputs (Banker ve Thrall, 1992). Data Envelopment Analysis (DEA) is an activity measurement technique first developed by Charnes, Cooper and Rhodes in 1978 to calculate the relative performance of similar units and based on the linear programming model by structure (Charnes et al., 1978; Charnes et al., 1981; Charnes et al., 1989). After applying DEA, units can be classified as active and inactive, and also the amount and sources of inactivity can be defined for inactive units. In this way, the DEA method can calculate how much input reduction and/or output increase in inactive decision-making units. Another advantage of DEA is that it also allows measuring efficiency in a system with inputs and outputs in different units (Lábaj et al., 2014, p:410). DEA is divided into two basic models: CCR and BCC. The first proposed DEA model is the Charnes Cooper Rhodes (CCR) model proposed in 1978. The Banker Charnes Cooper (BCC) Yalçın | Efficiency Measurement of Digitalization on EU Countries 327 model was later proposed in 1984. The main difference between these two models is based on the assumptions that their returns relative to the scale are constant/not. In the current study, the Input Oriented CCR Dual model is proposed. Input Oriented CCR Dual Model An input-oriented CCR model is a model that tries to determine the degree to which inactive decision-makers should reduce their input for any level of output, that is, reduce the amount of input while outputs are kept constant. The dual-state models of this model are as follows. * minθ θ= 0 1 1, 2,..., n ij j i j X x i mλ θ = ≤ = (1) 0 1 1, 2,..., n rj j r j y y r sλ = ≥ = 0 1, 2,...,j j nλ ≥ = Since θ=1 is a feasible solution to (1), θ*≤1. If θ*=1, then the current input levels cannot be reduced, indicating that DMU is on the frontier. If θ*<1, then DMU is dominated by the frontier. θ* represents the input-oriented efficiency score of DMU (Cook and Zho, 2005,p:3). In these models, s represents the number of outputs, m the number of inputs, and n the number of decision-making units. The dual model aims to make the maximum weighted average of the outputs of the decision-making unit, whose effectiveness is wanted to be calculated. In the constraints, the weighted average of the inputs of the decision-making unit whose efficiency is desired to be calculated is 1. In addition, the weighted average of outputs for all decision- making units should be less than the weighted average of inputs. In this way, the output/input ratio is no more than 1 for each decision-making unit (Banker,1984, p: 1084-1085). Data In the study, the five dimensions of the DESI index, Connectivity, Human Capital, Use of Internet, Integration of Digital Technology and Digital Public Services were used as input, while GDP Growth Rate and Employment Rate belonging to EU-28 countries between the years 2015-2019 were used in the analysis. Information about the input variables and output variables used is given in Table 1. Table 1: Variables Variables Description Source Inputs Digital Economy and Society Index (DESI) Connectivity (C) DESI Connectivity Dimension calculated as the weighted average of the five sub-dimensions: 1a Fixed Broadband take-up (25%), 1b Fixed broadband coverage (25%), 1c Mobile broadband (35%) and 1d Broadband price index (15%) European Commission Human Capital (HC) DESI Human Capital Dimension calculated as the weighted average of the two sub-dimensions: 2a Internet User Skills (50%) and 2b Advanced Skills and Development (50%) European Commission Use of Internet (UI) DESI Use of Internet Dimension calculated as the weighted average of the three sub-dimensions: 3a Internet Use (25%), 3b Activities Online (50%), 3c Transactions (25%) European Commission Integration of Digital Technology DESI Integration of Digital Technology Dimension calculated as the weighted average of European Commission Yalçın | Efficiency Measurement of Digitalization on EU Countries 328 (IDT) the two sub-dimensions: 4a Business digitisation (60%) and 4b e-Commerce (40%) Digital Public Services (DPS) DESI Digital Public Services Dimension calculated by taking the score for 5a e- Government European Commission Outputs GDP Growth (Annual %) (GDP) The annual percentage growth rate of GDP at market prices is based on constant local currency. Aggregates are based on constant 2010 U.S. dollars. GDP is the sum of gross value added by all resident producers in the economy plus any product taxes and minus any subsidies not included in the value of the products. It is calculated without deductions for depreciation of fabricated assets or depletion and degradation of natural resources. The World Bank Employment Rate (ER) Employment to population ratio, 15+, total (%) (modelled ILO estimate) The World Bank Table 2 contains descriptive statistics for 2015-2019 for input/output variables. Table 2: Descriptive Statistics of Input/Output Variables Statistics Inputs Outputs C HC UI IDT DPS GDP ER 2019 Mean 1166.29 1207.33 825.17 824.79 1011.55 2.57 55.15 Std Dev. 192.15 313.93 161.70 277.48 180.66 1.35 4.81 Min 737.38 712.81 524.48 337.85 674.88 0.30 42.84 Max 1501.47 1938.48 1128.71 1382.37 1274.46 5.55 61.59 2018 Mean 1045.39 1194.51 784.73 781.25 940.50 3.28 54.99 Std Dev. 179.20 297.54 167.50 245.54 190.54 1.75 4.94 Min 649.20 786.59 472.60 366.55 616.33 0.80 42.04 Max 1381.93 1903.64 1105.03 1268.99 1244.71 8.17 61.39 2017 Mean 974.29 1147.95 743.78 732.90 875.25 3.61 54.14 Std Dev. 176.75 291.16 168.12 242.01 208.03 1.70 4.96 Min 586.35 681.63 393.67 331.40 505.76 1.51 40.95 Max 1327.61 1866.76 1067.49 1207.78 1210.75 8.15 60.51 2016 Mean 899.17 1126.53 706.22 672.75 826.91 2.79 53.21 Std Dev. 184.17 286.17 162.47 221.33 231.09 1.44 5.15 Min 528.29 688.90 390.60 319.32 404.60 -0.19 39.95 Max 1243.98 1835.85 998.13 1134.85 1206.76 6.75 59.89 2015 Mean 817.50 1109.60 681.30 605.12 763.83 3.74 52.60 Std Dev. 190.26 282.10 166.72 194.89 239.40 4.69 5.15 Min 436.10 686.71 325.60 305.67 309.32 -0.44 39.20 Max 1137.94 1805.19 979.62 959.30 1159.39 25.16 59.67 According to Table 2, there is an increase in the average of C, HC, UI, IDT and DPS, which are input variables based on years in EU countries. The GDP average, the output variable, was recorded as the highest on a percentage basis in 2015 and the lowest in 2019. ER average can be said to have increased in percentage terms based on years. When the data set was examined, the country with the highest GDP was Ireland in 2019, 2018, while the lowest was Italy, the highest GDP in 2017 and 2015 was Ireland, the lowest Greece, and in 2016 the highest GDP was Cyprus, the lowest Greece. Likewise, in 2015-2019 ER was the highest in the Netherlands and the lowest in Greece. Results and Discussion Data Envelopment Analysis can be done with the original model CCR, which accepts fixed return on a scale, or the modified BCC model, which accepts variable return on a scale. According to the scale, Constant Returns to Scale–CRS means that the same rate of increase in Yalçın | Efficiency Measurement of Digitalization on EU Countries 329 inputs is observed. On the other hand, variable Returns to Scale (VRS) are a case of less or more increases in output compared to the amount of increase in inputs. In the study, the CCR model was considered based on VRS. When the findings in Table 3 are evaluated yearly, Bulgaria, Cyprus, Hungary and Romania are among the EU countries subject to the study and the countries where DEA was most effective in 2015-2019. On the other hand, Denmark, Finland and Spain are the countries with the lowest event score in the relevant periods. Table 3: VRS- INPUT Oriented DEA Efficiency Results EFFICIENCY SCORES RANK COUNTRY 2019 2018 2017 2016 2015 2019 2018 2017 2016 2015 1 Austria 0,859 0,883 0,912 0,896 0,852 14 16 10 12 14 2 Belgium 0,802 0,776 0,755 0,724 0,705 19 22 23 22 23 3 Bulgaria 1,000 1,000 1,000 1,000 1,000 1 1 1 1 1 4 Croatia 0,915 0,958 0,908 0,892 0,963 11 11 11 13 11 5 Cyprus 1,000 1,000 1,000 1,000 1,000 1 1 1 1 1 6 Czechia 0,960 1,000 1,000 0,988 1,000 8 1 1 8 1 7 Denmark 0,689 0,697 0,665 0,654 0,599 26 27 27 27 27 8 Estonia 0,847 0,771 0,809 0,761 0,770 17 23 20 21 20 9 Finland 0,672 0,724 0,674 0,638 0,576 27 26 26 28 28 10 France 0,747 0,793 0,794 0,803 0,782 24 20 21 20 18 11 Germany 0,900 0,943 0,906 0,864 0,852 12 12 12 16 15 12 Greece 0,938 0,977 1,000 1,000 1,000 9 10 1 1 1 13 Hungary 1,000 1,000 1,000 1,000 1,000 1 1 1 1 1 14 Ireland 1,000 1,000 1,000 0,945 1,000 1 1 1 10 1 15 Italy 0,757 0,850 0,823 0,881 0,833 23 19 19 14 16 16 Latvia 0,764 0,869 0,904 0,951 1,000 22 17 13 9 1 17 Lithuania 0,857 0,866 0,859 0,913 0,882 15 18 16 11 13 18 Luxembourg 0,781 0,785 0,759 0,814 0,757 21 21 22 19 21 19 Malta 0,880 1,000 0,856 0,712 0,753 13 1 18 23 22 20 Netherlands 0,787 0,769 0,738 0,711 0,686 20 24 24 24 24 21 Poland 0,969 1,000 1,000 1,000 0,981 7 1 1 1 10 22 Portugal 0,843 0,914 0,856 0,880 0,923 18 14 17 15 12 23 Romania 1,000 1,000 1,000 1,000 1,000 1 1 1 1 1 24 Slovakia 1,000 1,000 0,998 1,000 1,000 1 1 9 1 1 25 Slovenia 0,852 0,885 0,860 0,816 0,781 16 15 15 18 19 26 Spain 0,638 0,666 0,664 0,678 0,640 28 28 28 26 26 27 Sweden 0,712 0,729 0,709 0,679 0,641 25 25 25 25 25 28 United Kingdom 0,926 0,936 0,886 0,828 0,804 10 13 14 17 17 In Table 3, the average efficiency score of countries in 2015 is 84.9%. The number of efficient countries is 9 (Bulgaria, Cyprus, Czechia, Greece, Hungary, Ireland, Latvia, Romania, Slovakia). While the average efficiency score of the countries was 85.8% in 2016, the number of efficient countries decreased to 7 (Bulgaria, Cyprus, Greece, Hungary, Poland, Romania, Slovakia). While the average efficiency score of the countries was 86.9% in 2017, the number of efficient countries was 8. (Bulgaria, Cyprus, Czechia, Greece, Hungary, Ireland, Poland, Romania). In 2018, while the average efficiency score of the countries was 88.5%, the number Yalçın | Efficiency Measurement of Digitalization on EU Countries 330 of efficient countries increased to 9 (Bulgaria, Cyprus, Czechia, Hungary, Ireland, Malta, Poland, Romania, Slovakia). Finally, according to the findings of 2019, while the average efficiency score of 28 EU countries was 86.1%, the number of efficient countries decreased to 6 (Bulgaria, Cyprus, Hungary, Ireland, Romania, Slovakia). In light of this result, it is possible to say that input variables C, HC, UI, IDT and DPS, which are active decision-making units, have a significant impact on countries' GDP and ER rates. Therefore, it can be stated that the effective use of DESI indicators, which are considered a measure of digitalisation, has a one-on-one effect on GDP and ER values included in the model as output in countries such as Bulgaria, Cyprus, Hungary and Romania. On the other hand, DESI indicators are not used effectively in GDP and ER values of countries such as Denmark, Finland and Spain. In this context, the fact that countries such as Denmark, Finland, Spain, Sweden (although GDP and ER rates are high) completed their digitalisation processes earlier in 2015 - 2019 can be expressed as one of the reasons why the efficiency scores are low. From the same point of view, Bulgaria, Greece, Cyprus, Hungary and Romania are the countries with the lowest GDP and ER ratio, especially as seen from the 2020 data. Despite this, each step taken in digitalisation seems to result in a hundred per cent effectiveness on the relevant outputs. Conclusion The VRS-INPUT Oriented DEA method was used in this study to determine the countries that have transformed digitalisation into growth and employment rates in EU countries for the years 2015-2019, to calculate the maximum output with the available inputs. At the moment, one of the crucial questions is the choice of countries that are the subject of analysis. This situation can be evaluated from several aspects. First of all, the study aimed to research countries with certain standards on digitalisation and where there is a consensus with data. The second is the idea that digitalisation can make a difference even for countries that have reached a certain level of economic maturity. The third is that decision-making units (DMUs) are expected to carry homogeneous properties in data envelopment analysis. In light of these reasons, it was decided that the countries subject to analysis should be EU countries. The input data used for the analysis were determined as sub-components of DESI, and the output data was determined as economic growth (GDP %) and employment rate. In addition, it has been decided that the clear reflection of digitalisation will be seen in these components based on economic theory (Solow, 1957; Romer, 1986). As a result, it is generally observed that countries that have reached a certain saturation in the digitalisation process have a low-efficiency rate on GDP and ER. On the other hand, in countries that are continuing the digitisation process, the effectiveness of each step taken in the name of digitisation on economic growth and employment is reflected the highest degree. Research results show that (i) in developed countries (Denmark, Sweden, Finland, etc.), the efficiency of digitalisation in the transformation of economic growth and employment rate is low. The most important reason for this is that these countries have completed the digitalisation process, and the steps taken in the context of DESI have reached a certain saturation. Another important finding is that in developing countries concerning digitalisation, such as Bulgaria, Greece, Romania and Cyprus, improvements in digitalisation indicators have created an effect on income and employment. When all findings are brought together, in developed countries in terms of digitalisation, improving digitalisation does not affect the country's income and employment. On the contrary, in developing countries, improving digitalisation affects the country's income and employment. Finally, it should be noted that these results are valid in EU countries for the years between 2015-2019. Yalçın | Efficiency Measurement of Digitalization on EU Countries 331 In future studies, which variables should be selected as inputs (economic growth and employment as output) in addition to digitalisation for countries that have reached a certain point in digitalisation can be studied. In addition, how digitalisation affects other outcomes, such as the competition index and the country's economy's structure, can also be an important study topic. In addition, making analyses by differentiating country groups (OECD, BRICS, etc.) can also contribute to the field. References Banker, R. D. & Thrall, R. M. (1992). Estimation of returns to scale using data envelopment analysis, European Journal of Operational Research, 62(1): 74 – 84. Banker, R.D., Charnes, A. & Cooper, W.W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30, 1078-1092. Başol, O. & Yalçın, E. C. (2021). How does the digital economy and society index (DESI) affect labor market indicators in EU countries? Human Systems Management, 40(4): 503-512. BBVA (2020). DiGiX: The Digitization Index. https://www.bbvaresearch.com/en/publicaciones/digix-the- digitization-index/ (13.12.2020). Charnes, A., Cooper, W. W. & Li, S. (1989). Using data envelopment analysis to evaluate efficiency in the economic performance of Chinese cities, Socio-Economic Planning Sciences, 23(6): 325-344, Charnes, A., Cooper, W.W. & Rhodes, E. (1978) Measuring the efficiency of decision-making units. European Journal of Operations Research, 2(6): 429-444. Charnes, A., Cooper, W.W. & Rhodes, E. (1981) Evaluating program and managerial efficiency: An application of data envelopment analysis to program follow through. Management Science, 27(6):668-697. Coelli T., Rao D.S.P. & Battese G.E. (1998). Efficiency measurement using data envelopment analysis (DEA). In: An introduction to efficiency and productivity analysis. Springer, Boston, MA. https://doi.org/10.1007/978- 1-4615-5493-6_6. Cook, W. D. & Zhu, J. (2005). Modeling performance measurement applications and implementation issues in DEA. Springer Science + Business Media, Inc, Boston. DESI. (2020). DESI by components. https://digital-agenda-data.eu/charts/desi- components#chart={%22indicator%22:%22desi%22,%22breakdown-group%22:%22desi%22,%22unit- measure%22:%22egov_score%22,%22time-period%22:%222019%22}. European Comission (2020). The Digital Economy and Society Index. https://ec.europa.eu/digital-single- market/en/digital-economy-and-society-index-desi (13.12.2020) European Commission (2020a). Digital Economy and Society Index (DESI) 2020: Thematic Chapters. European Commission Report. European Commission (2020b). DESI by Components. 03.04.2021 https://digital-agenda-data.eu/charts/desi- components#chart={"indicator":"desi","breakdown-group":"desi","unit-measure":"egov_score","time- period":"2020"}. European Commission (2020c). Digital Economy and Society Index (DESI) 2020: Connectivity. European Commission Report. European Commission (2020d). Digital Economy and Society Index (DESI) 2020: Human Capital. European Commission Report. European Commission (2020e). Digital Economy and Society Index (DESI) 2020: Use of Internet Services. European Commission Report. European Commission (2020f). Digital Economy and Society Index (DESI) 2020: Integration of Digital Technology. European Commission Report. European Commission (2020g). Digital Economy and Society Index (DESI) 2020: Digital Public Services. European Commission Report. Hermes, A. (2020). Enabling Digitalization Index 2018: Measuring digitality. https://www.eulerhermes.com/en_global/news-insights/news/enabling-digitalization-index-2018- measuring-digitagility.html (13.12.2020). Ibrahim, M. Alola, A.A. & Ferreira, D.C. (2021). A two-stage data envelopment analysis of efficiency of social- ecological systems: Inference from the sub-Saharan African countries. Ecological Indicators, 123: 107381. ILO. (2005). Employment by status in employment. ILO Publishing. Lábaj, M., Luptáčik, M. & Nežinský, E. (2014). Data envelopment analysis for measuring economic growth in terms of welfare beyond GDP. Empirica, 41, 407–424. Li, J. Q. & McNeil, S. (2021). Data envelopment analysis for highway asset investment assessment journal of traffic and transportation engineering. 8(1): 117-128. OECD. (2013). OECD employment outlook. OECD Publishing. Yalçın | Efficiency Measurement of Digitalization on EU Countries 332 Puyenbroeck, T.V., Montalto, V. & Saisana, M. (2021). Benchmarking culture in Europe: A data envelopment analysis approach to identify city-specific strengths. European Journal of Operational Research 288: 584- 597. Romer, P. M. (1986). Increasing returns and long-run growth. The Journal of Political Economy, 1002-1037. Solow, R. M. (1957). Technical change and the aggregate production function, Review of Economics and Statistics, No: 39-3, (August-1957), s.312-320. World Bank. (2020). Digital Adoption Index. https://www.worldbank.org/en/publication/wdr2016/Digital- Adoption-Index (13.12.2020). World Bank. (2020). Employment to Population Ratio. https://data.worldbank.org/indicator/SL.EMP.TOTL.SP.ZS. World Bank. (2020). GDP Growth. https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG?locations=XC. Yanga, F., Choib, Y. & Leec, H. (2021). Life-cycle data envelopment analysis to measure efficiency and cost- effectiveness of environmental regulation in China’s transport sector. Ecological Indicators, 126: 107717. Yalçın | Efficiency Measurement of Digitalization on EU Countries 333