Metodoloski zvezki, Vol. 16, No. 2, 2019, 1-19
Delphi Method: Strengths and Weaknesses
Danica Fink-Hafner1 Tamara Dagen2 May Dousak3 Meta Novak4 Mitja Hafner-Fink5
Abstract
The paper presents the Delphi method and tests its usefulness when searching for a consensus on definitions, especially in a particular social science field. Based on an overview of the characteristics and uses of the Delphi method, a special Delphi design for searching for minimal common definitions of globalisation, Europeani-sation and internationalisation in higher education and their mutual relationships is presented in detail. While the method proved valuable, its strengths and weaknesses are also discussed. Finally, ideas for adjusting the Delphi method are proposed.
1 Introduction
When browsing through any basic social science methodology textbook one does not often encounter mention of the Delphi method.6 However, if it is mentioned at all, it more often than not appears in the category "other methods". Why is this so? This is probably down to its key characteristics: while using the Delphi method, researchers seek to portray the 'social reality' based on experts' judgements, not on primary data concerning this reality. It is many times not even about truly presenting reality, but merely to help develop a theory by seeking a consensus among experts (e.g., Okoli and Pawlowski, 2004; Paivarinta et al. 2011; Brady 2015). It is also acknowledged that seeking consensus by use of the Delphi method is common in the natural sciences and various medical sub-fields, applied studies aimed at predicting future developments in various areas, finding practical solutions, in policymaking and so on (e.g., Weaver 1971; Land and Schneide, 1987; Patton and Sawicki, 1993; Rayens and Hahn 2000; Donohoe and Needham 2009; Brady 2015). It has been applied in "various fields of study such as program planning, needs assessment, policy determination, and resource utilization to develop a full range of alternatives, explore or expose underlying assumptions, as well as correlate judgments on a topic spanning a wide range of disciplines" (Hsu and Sandford, 2007). Participatory
1 Faculty of Social Sciences, University of Ljubljana, Ljubljana, Slovenia; danica.fink-hafner@fdv.uni-
ljsi
2University of Zagreb, Zagreb, Croatia; tamara.dagen@unizg.hr
3Faculty of Social Sciences, University of Ljubljana, Ljubljana, Slovenia; may.dousak@fdv.uni-lj.si
4Faculty of Social Sciences, University of Ljubljana, Ljubljana, Slovenia; meta.novak@fdv.uni-lj.si
5Faculty of Social Sciences, University of Ljubljana, Ljubljana, Slovenia; mitja.hafner-fink@fdv.uni-
ljsi
6Both Delphi method and technique are used in the literature. For this article, we decided to use Delphi method.
2
Fink-Hafner et al.
action research also relies on it (cf. Fletcher and Marchildon 2014; Brady 2015). In the last few decades, application of the Delphi method has been growing due to the spread of the Internet generally and Internet-based research tools that support Delphi in particular. The e-Delphi method maximises the advantages and limits the disadvantages of the traditional version of the method (Donohoe, Stellefson and Tennant, 2012).
This paper presents: (a) various aspects of the Delphi method as one of several social research designs; and (b) its application while searching for agreed definitions in certain academic fields where a consensus regarding some key terms is missing. More specifically, an empirical example is presented of the Delphi method being used to develop definitions of three (theoretical) concepts in the area of higher education: globalisation, internationalisation, and Europeanisation. This example is referred to while discussing various epistemological, methodological and procedural aspects of the method, and also underpins a discussion of the method's strong and weak points, as well as alternatives that combine the Delphi method with other methods or research approaches.
2 The Delphi Method in Social Research - Epistemologi-cal and Procedural Aspects
In essence, Delphi is a vehicle and a method for informed consensus-building within a group with respect to a complex problem by using a series of questionnaires delivered in multiple iterations to collect data from a panel of selected participants - "panellists" (Dalkey and Helmer, 1963; Donohoe, Stellefson and Tennant, 2012).
First documented use of the Delphi method goes back to the 1950s, being discussed in a methodological sense in the early 1960s (Dalkey and Helmer, 1963). From the outset - starting with an Air Force-sponsored Rand Corporation study conducted in the early 1950s (as reported by Dalkey and Helmer, 1963) - it was developed as a method for finding the most reliable consensus among a selected group of experts. It has often been understood as a 'forecasting method' (Weaver, 1971; Moniz, 2005) and used in various applied studies and policy development processes (Patton and Sawicki, 1993; Rayens and Hahn, 2000). Despite the originally largely applied nature of the research that relied on the Delphi method, researchers later began to apply the Delphi method in 'basic' research. More recently, researchers have also begun using the Delphi method in the theory-building process (e.g. Paivarinta et al. 2011; Brady 2015). Thus, the search for a consensus by use of the Delphi method may appear in different phases of the research process and for solving various research problems in basic (social) research, such as: identification of the research topic, specification of research question(s), selection of variables of interest, preliminary identification of causal relationships, definition of constructs (Okoli and Pawlowski 2004: 27).
Quite a few problems emerge if one tries to place the Delphi method in the context of social science paradigms. It is a method (and technique) that rises above the paradigmatic divide. Namely, it includes elements of qualitative and quantitative approaches, and of constructivism and positivism (Amos and Pearse, 2008). The method is frequently used in studies with a mixed methods design, often with quantitative data (data collection, measurement). The Delphi method may be characterised as some kind of paradigmatic
Delphi method
3
pragmatism. However, data analysis methods for quantitative Delphi studies have seen greater development than for qualitative ones (Brady 2015).
Since the Delphi method aims to obtain a highly reliable consensus of opinion of a group of experts (Dalkey and Helmer, 1963), the Delphi process' characteristics (including a series of detailed questionnaires and controlled feedback on the opinion) are very important. Indeed, Delphi goes beyond simple intuitive expert estimations as it applies relatively strict control over the methods of interviewing and re-interviewing panellists and the summarising of the results (Linstone and Turoff, 1975; Sackman, 1975).
2.1 The Process Entailed by Delphi
The main idea underlying the Delphi process is controlled indirect interaction among experts (participants with knowledge of the topic that is the subject of Delphi) with a tendency for the experts' judgements to converge as the experiment goes on. To identify the experts, a minimum qualification must be defined. According to Melnyk et al. (2009), participants in the group need to be recognised and validated as domain experts, while researchers still attempt to obtain a broad range of individual perspectives concerning those criteria. Although there is no standard number of participants, panels of fewer than 10 participants and over 1000 are rare, with 10- to 100-member panels being the most common (Avella, 2016).
Control over the Delphi process is in the hands of the researchers who designed and are implementing the Delphi. This includes researchers' preparation of structured feedback to be given to the Delphi participants, including statistical analysis of (previous-round) expert judgements for the panel experts' next estimation.
At first, Delphi communication included face-to-face meetings of a researcher with the interviewed expert, but soon even this practice gained an extra mode - communication via online interactive computer links (Pal, 1987), sometimes also named the e-Delphi (Donohoe, Stellefson and Tennant, 2012). From the outset, no face-to-face communication among experts has been part of Delphi. Indeed, experts are also anonymous with respect to each other.
In contrast to a focus group, the Delphi method consists of repeated individual interviews with each expert to avoid any direct confrontation and all the biases that may then arise like confirmation of the most dominant view and the opportunity for more creativity and expression of individual opinions.7 Even when quite a small number of experts is included in Delphi (initially it was a small group of experts, around 10, called panellists). Delphi communication includes a mix of (elements of) qualitative interviews and quantitative survey interviews which can straddle the divide between these two methods and provide a more complete picture (Iqbal and Pippon-Young, 2009).
In a narrow sense (obtaining data and searching for consensus), the Delphi method is conducted over two or more, typically three, rounds of interviews called iterations, where each has a different goal. The number of iterations depends on the level of consensus among the participating experts and the additional 'improvement' to the consensus the
7The original authors emphasise the need to avoid "hasty formulations of preconceived notions", "inclinations to close one's mind to novel ideas", "tendencies to defend stance once taken" and promoting the "gradual formation of a considered opinion" (Dalkey and Helmer, 1963).
4
Fink-Hafner et al.
last round brings. The questionnaire for the following iteration builds on the result of the previous iteration. The information obtained in one round influences the progress of the Delphi method.
The research begins with good knowledge of the subject, which requires a literature review. One difference among versions of Delphi method research is seen in the varying ways of preparing the questionnaire for the first round of the research. While some researchers rely entirely on experts to identify specific elements, indicators and issues and prepare an open questionnaire, others regard the literature analysis as a zero step in order to prepare a set of elements and indicators for the first-round questionnaire to be sent to the experts (Quyen, 2014).
Researchers who argue the first round should be open and allow participants to express their own views on the issue believe that one can thereby gather information beyond what is available in the literature (Iqbal and Pippon-Young, 2009). Some researchers build from the literature and construct the survey instrument for the first round accordingly. This allows fewer rounds of the Delphi method to be conducted and saves time and expenses, e.g. a two-round Delphi method (Iqbal and Pippon-Young, 2009). Researchers deciding to use a quantitative survey instrument in the first round usually start by identifying indicators or elements in the first step, ensuring their validation and ranking by importance in the second step, and the search for a consensus and validated results in the third step (Hsu and Sandford, 2007; for more details, see e.g., Boulkedid et al., 2011; Schmidt, 1997; Uphoff et al., 2012; Wilson et al., 2012).
A Delphi questionnaire changes from iteration to iteration: (1) in the first round, the questionnaire can be more qualitative with the goal to identify various possible elements relevant to the research problem and discernible beyond the literature review (possible future events, definitions or elements of a definition, indicators etc.); (2) in the second round, the questionnaire is more quantitative and standardised - different assessment and ranking scales are used; (3) in the third step, the questionnaire is prepared on the basis of the results of the second round (ranking or validation of elements, exclusion of irrelevant elements), and allows the participants to evaluate the outcome of the second round and, if necessary, make further revisions. The number of rounds/iterations depends on when the panellists reach a consensus or the researchers are happy with the result and do not see possibilities of making much progress with an additional round.
Literature in the Delphi method shows that research relies on various variants of the classic Delphi method (e.g., Dunn, 1994; Huisman, de Boer and Botas, 2012). They differ in the number of iterations (circles, waves or rounds), data collection mechanisms, the administrative process that accompanies the research, additional methods used alongside Delphi, etc. In recent decades, a Fuzzy Delphi was also developed, which some researchers claim saves time and eliminates the need for additional evaluation of Delphi results by utilising triangulation-based statistics to determine the distance between the levels of consensus on the expert panel (Ishikawa et al., 1993).
2.2 The Strengths of Delphi
Authors have pointed out the many strengths of the Delphi method. First, it allows for communication among experts with the help of mediator (a researcher), thereby eliminating several obstacles to a rational academic debate. For example, Delphi:
Delphi method
5
•	enables direct confrontations of the experts with each other to be avoided (Dalkey and Helmer, 1963);
•	links together existing knowledge and areas of agreement/disagreement (Iqbal and Pipon-Young 2009);
•	does not demand proximity or a face-to-face meeting and thus allows for experts' independent thought (Dalkey and Helmer, 1963); there is less opportunity to (be forced to) conform with the dominant view (Donohoe and Needham 2009);
•	enables anonymity, which encourages creativity, honesty and a balanced consideration of ideas while reducing the risk of group dynamics negatively influencing outcomes (Donohoe and Needham 2009; Iqbal and Pipon-Young 2009); certain downsides associated with group dynamics such as manipulation or coercion to conform with or adopt a certain viewpoint can be minimised as Delphi gives less of an opportunity to conform with the dominant view (Iqbal and Pipon-Young 2009);
•	reduces the effect of noise - that communication which occurs in a group process which both distorts the data and deals with group and/or individual interests rather than focusing on problem-solving (Hsu and Sandford, 2007);
•	through the feedback given in Delphi, an individual expert may enrich his/her insight into empirical factors or theoretical assumptions allowing them to correct any misconceptions (Dalkey and Helmer, 1963);
•	due to all of the above, it allows for the results to be accepted (Bleijenbergh, Ko-rzilius and Verschuren, 2011);
•	experts contribute to the understanding and resolution of important problems (Donohoe and Needham, 2009);
•	enables group communication free from geographical constraints (Donohoe and Needham, 2009); and
•	saves money and time; it also reduces travel costs (Donohoe and Needham, 2009).
The following methodological qualities of Delphi are mentioned in particular:
•	verifiability, comprehensibility and holism (holism includes knowledge about both the phenomenon and its context) (Bleijenbergh, Korzilius and Verschuren, 2011). In line with these authors, verifiability in Delphi is primarily about the openness and transparency of knowledge production in practice-oriented research;
•	by straddling the divide between the qualitative and quantitative methods it is able to provide a more complete picture (Iqbal and Pipon-Young, 2009);
•	the controlled feedback process, and the suitability of several techniques of statistical analysis for interpreting the data (Hsu and Sandford, 2007);
6
Fink-Hafner et al.
•	the flexible methodology accommodates many variations and applications (Dono-hoe and Needham, 2009); flexibility and reflexivity allows researchers to adapt the technique to the research context (Donohoe and Needham, 2009);
•	the respondents can think about the problem over several rounds, which enhances the validity of the data (Donohoe and Needham, 2009); the feedback process permits and encourages the Delphi participants to reassess their initial judgements on the information provided in previous iterations (Hsu and Sandford, 2007); and
•	the e-Delphi computerises the Delphi process to optimise the method's ability to organise the thinking of a diverse group located in different areas organise, which saves time and cost; it also offers advantages for data management and analysis as well as connects the opinions of experts and practitioners from around the world quite conveniently for researchers and participants while taking care of security and accessing the benefits of the data collected (Donohoe, Stellefson and Tennant 2012).
The Delphi method is particularly suitable for the following cases:
•	Delphi has legitimacy and suitability for solving highly complex problems (Dono-hoe and Needham 2009);
•	Delphi is flexible and well suited when there is incomplete knowledge about phenomena; especially when the goal is to improve the understanding of problems, opportunities or solutions, or to develop forecasts (Skulmoski, Hartman and Krahn, 2007);
•	Delphi is appropriate for exploring areas in which controversy, debate or a lack of clarity exist (Iqbal and Pipon-Young, 2009); and
•	it is also an acceptable substitute for direct empirical evidence when the latter is unavailable (Dalkey and Helmer, 1963).
2.3 The Weaknesses of Delphi
There is a lack of guidance and agreed standards on how to interpret and analyse the results, universally agreed definitions of consensus, and how to select the participants.
Delphi is quite time-consuming and laborious for both researchers and participants, explaining why it is vulnerable to drop-outs. Participants might also drop out due to the long temporal commitment, distraction between rounds, or disappointment with the process (Donohoe and Needham, 2009). The same authors also note that use of monetary payments or moral persuasion to convince the participants may introduce bias into the results.
Although anonymity as previously mentioned is an aspect of Delphi's strengths, it also produces some downsides - less ownership of ideas (Iqbal and Pipon-Young, 2009).
Methodological limitations include the difficulty of generalising the results to a wider population due to the sample size (Hartman and Jugdev, 1998; Schmidt et al., 2001),
Delphi method
7
the response rate among the invited experts, limited views, the uneven spread of expertise among the participants or the participants' specific agenda and/or geographic/cultural location (Nambisan et al., 1999; Niederman et al., 1991; Hsu and Sandford, 2007).
At least a potential weakness of Delphi is the considerable discretion given to researchers while applying the method. As the procedure depends on the quality of the feedback provided, the careful analysis of the responses is a big responsibility of the researcher. Besides that, it is difficult to determine what constitutes a sufficient consensus in the Delphi procedure (Donohoe and Needham, 2009).
The experts' responses in the Delphi process might not be truly independent, especially when the experts involved are in contact with each other (Dalkey and Helmer, 1963).
While in the case of e-Delphi, entering data into computer-based data screens is an advantage, it might also be an inconvenience for some experts (Donohoe, Stellefson and Tennant, 2012). Technical problems related to (unreliable) access may arise in an international e-Delphi.
2.4 Overcoming the Weaknesses
Two main suggestions are found in the literature for the overcoming the Delphi method's weaknesses. The first is to validate the Delphi results by means of triangulation - complementing the Delphi with another research approach (Skulmoski, Hartman and Krahn, 2007). The second is to apply the e-Delphi technique that maximises the advantages and limits the disadvantages of the traditional Delphi (Donohoe, Stellefson and Tennant, 2012).
3 Example of using the Delphi Method - a Consensus on the Definitions of Globalisation, Internationalisation and Europeanisation in Higher Education
3.1 The Delphi Design
Our Delphi has a theoretical goal - to develop common definitions in the HE field - as already described. The design includes qualitative and quantitative characteristics. Qualitative aspects are incorporated in the preparatory phase, namely, in the literature review (as a substitute for the first, explorative round of interviews). In this sense, we relied on a modified Delphi design. Besides that, the experts were able to add and otherwise modify the text sent to them during the Delphi process and add their comments. Quantitative aspects of our Delphi are seen in measuring the response rate, using rating scales for the elements of definitions, and measuring levels of agreement among the respondents with regard to their acceptance of particular elements of definitions and later on also of the definitions that were offered.
The Delphi research design was carried out in two phases: in phase one, we prepared the starting points while in phase two we conducted surveys among experts using two questionnaires over two rounds.
8
Fink-Hafner et al.
More precisely, the steps in the first phase were: (1) an extensive literature analysis; (2) interviews with experts; (3) preparation of a list of experts for the Delphi method research; (4) selecting scientific journals and a handbook for the analysis; (5) preparing a selection of articles from the scientific journals and the handbook based on the specific methodological approach; (6) analysis of articles/chapters in order to isolate elements of definitions and indicators; (7) creating the initial online questionnaire for the first round of the research (creating a list of elements for definitions on an abstract level and a list of indicators for the first round).
The second phase entailed the following steps: (a) implementation of the first round of the Delphi method research, including: (1) sending the initial online questionnaire to the experts; (2) analysis of the results obtained in the first round of the Delphi method research; (3) preparation of the second questionnaire; (b) implementation of the second round of the Delphi method research, which included: (1) sending the second online questionnaire to those experts who participated in the first round; (2) analysis of the results obtained in the second round of the research); (c) synthesis of the consolidated definitions and the final list of indicators, followed by preparation of minimal definitions of the three terms and a final list of indicators. Figure 1 illustrates the individual phases and steps of the Delphi method survey: Defining globalisation, internationalisation and Europeanisation in the field of higher education.
The Delphi process (two rounds of the survey) took place by way of Internet communication (the 1KA e-mail and Internet survey tool). The administrator was able to adapt 1KA to make completion of the e-questionnaire interactive. Technical help for the participants was offered by the administrator, when needed. He also had an insight into the amount of ongoing data gathering. Only a few of the invited experts contacted the administrator due to technical problems they had experienced. To ensure that only the invited experts participated in the survey, each was given a unique link to the survey. The panellists' anonymity was ensured and no names were disclosed to the other participants. Personal data were protected by using identity codes, not names.
3.2 First Phase: Preparing the Delphi
The preparatory phase for the Delphi method consisted of a literature review. Between January and June 2016, a broad literature corpus concerning globalisation, internationalisation and Europeanisation in HE was analysed. This extensive analysis led to insights for the researchers and experts regarding understanding and interpreting these three concepts in the field of HE, which in later stages of the Delphi method research proved crucial while preparing the questionnaires and analysing the results.
Based on the interviews with several experts8 and considering the impact of international scientific journals, it was decided to prepare the first questionnaire based on analysis of selected articles and on two scientific journals by following a specific methodological process (Higher Education and Journal of Studies in International Education) and two chapters from The SAGE Handbook of International Higher Education (Deardorff, de
8Two types of experts: (1) distinguished professors dealing with research in the higher education field holding insights into recent research and literature in the field; (2) experts involved in the work of several international university networks and organisations and their specialised bodies (e.g. EUA, UNICA etc.).
Delphi method
9
Phase 1
Figure 1: Delphi method design - phases and steps
10
Fink-Hafner et al.
Wit, Heyl and Adams, 2012).
Analysis of articles from the two scientific journals was limited to the 1999 to June 2016 period since internationalisation in the higher education field in Europe significantly came to researchers' attention when the Bologna Declaration was signed in 1999 and the Bologna Process then started.
The selection of articles from the two journals relied on a methodological approach based on keyword analysis. The first analysis used different combinations of the terms globalisation, internationalisation and Europeanisation and showed that a large number of articles using various combinations of these terms was published in the selected period. The majority of articles used just one keyword (term), most frequently internationalisation (702 articles), then globalisation (676 articles). Analysis of the combination of two concepts (two key words) revealed that in the majority of the articles the authors concentrated on combining the concepts and gaining insights into the two-way relationship between globalisation and internationalisation in the area of higher education (402 articles), followed by the considerably less well-represented combination of the keywords internationalisation and Europeanisation (only 37 articles). Finally, the analysis showed that the combination of all three concepts (globalisation, internationalisation, Europeani-sation) was found in just 28 articles (8 in Higher Education; 20 in Journal of Studies in International Education). These 28 articles together with the first two chapters of The SAGE Handbook of International Higher Education (Internationalization Within the Higher Education Context by Rumbley, Altbach and Reisberg; and Concepts, Rationales, and Interpretive Frameworks in the Internationalization of Higher Education by Knight, 2012) provided the framework for preparing the first questionnaire.
After a technical analysis of the articles (by year, topics, authors) and an analysis using the keywords globalisation, internationalisation and Europeanisation for the period 1999 to June 2016 (content analysis), general definitions of globalisation, internationalisation and Europeanisation were isolated along with their definitions in HE or description given in the selected articles. The next step in the procedure for preparing the first online questionnaire was to single out the elements from these definitions and, finally, to create a list of elements on the abstract level. Based on the extensive literature analysis, for this initial questionnaire a list of indicators was prepared of globalisation and internationalisation in higher education on different levels.
Simultaneously with preparing the first questionnaire, a list of experts around the world who would be invited to participate in the Delphi method survey was formed. The selection of experts was based on the following three main sources:
1.	the comprehensive analysis of literature in order to find researchers and experts across the globe, where the focus was on literature dealing with globalisation, internationalisation and Europeanisation in the higher education field;
2.	the analysis of articles from two scientific journals (Higher Education and Journal of Studies in International Education) and chapters from The SAGE Handbook of International Higher Education (Deardorff et al., 2012); and
3.	an interview with Professor Pavel Zgaga from the University of Ljubljana (an expert on the area of higher education).
Delphi method
11
Following the above steps, a total of 150 experts around the world was identified: (1) authors of internationally recognised papers in the area of the internationalisation of higher education and in the HE field in general; (2) authors of 28 analysed articles from two scientific journals (Higher Education and Journal of Studies in International Education) from 1999 to mid-2016 (authors of 28 analysed articles); and (3) authors of all chapters in The SAGE Handbook of International Higher Education (Deardorff et al., 2012).
Although researchers and experts involved in research into internationalisation in the HE field are very mobile and frequently change their affiliation, the following data show the world regions/states the researchers and experts so identified for the Delphi method research come from: 1) Europe (Austria, Belgium, Czech Republic, Denmark, Estonia, Finland, France, Croatia, Italy, Hungary, Netherlands, Norway, Germany, Poland, Portugal, Russia, Slovenia, Serbia, Sweden, Switzerland, Spain, United Kingdom); 2) North America (Canada, Mexico, USA); 3) South America (Brazil, Chile); 4) Australia and New Zealand; 5) Asia (United Arab Emirates, India, Israel, Japan, Kazakhstan, China, Hong Kong, Taiwan, Malaysia, Singapore, Vietnam).
By applying the snowball method (Atkinson and Flont, 2001; Noy, 2008), with the help of experts who participated in the first round of the research a further 25 experts from the countries listed below were identified: Australia, Belgium, France, Hong Kong, Luxembourg, Hungary, Mexico, Netherlands, Norway, Germany, Portugal, United Kingdom.
3.3 Second Phase: Waves of Data Gathering and Analysis
The Delphi method survey "Defining globalisation, internationalisation and Europeanisa-tion in the field of higher education" was conducted in two rounds, with each entailing several specific steps.
There are only a few world-leading experts in the field of globalisation and internalisation in higher education which meant obtaining a high level of cooperation was our top priority. Hence, a mixed mode system combining a personalised postal mail preno-tification letter sent by post and a convenient web survey was chosen as being the most appropriate and efficient way for the collecting data (de Leeuw, 2005; Biemer and Lyberg, 2003).
The Delphi method research started in early December 2016 by sending a carefully composed and designed personalised prenotification official letter sent by post on 6 December 2016 to the postal addresses of the 150 identified experts. The letter informed them of the aim and purpose of the research and asked them to accept the invitation and participate in the survey by completing an electronic questionnaire that would be sent to their e-mail addresses within the next 20 days.
To personalise the communication with the experts, all further electronic communication, including carefully written personalised invitation e-mails, was conducted using a single personal email address of one research team member who also regularly and promptly replied to any additional questions and comments from the experts (most related to methodological issues pertaining to the questionnaire). An online questionnaire was created using the 1KA survey software and tested on all major operating systems and device form factors.
12
Fink-Hafner et al.
The first round of the Delphi method survey included three steps: (1) sending a link to the experts to the first questionnaire; (2) analysing the results; and (3) preparing the second online questionnaire.
After extensive pilot testing, an email with a link to the first online questionnaire was sent to the email addresses of 150 experts around the world on 25 December 2016. Two more carefully written reminders were sent out: the first after a generous two-week time on 9 January 2017 and the second after one extra week on 16 January 2017. Based on expert recommendations gathered during the first round of the survey, an additional 25 experts were invited to participate in the first round. Thus, altogether 175 experts were invited to participate. The survey in the first round of our Delphi design was completed on 23 January 2017.
The questionnaire for the first round of the Delphi method research included elements for building consolidated general definitions of globalisation, internationalisation and Eu-ropeanisation and a definition of these three terms in the HE area. Further, based on the extensive literature analysis, the questionnaire offered a list of indicators of globalisation and of internationalisation in HE on different levels.
The participants (experts) were asked to rate on a scale from 0 to 10 (where 0 is not at all important, and 10 is extremely important) and evaluate the following elements and indicators: (1) the importance of the elements for the creation of consolidated general definitions of globalisation, internationalisation and Europeanisation; (2) the importance of the elements for the creation of consolidated definitions of globalisation, internationalisation and Europeanisation in the higher education field; (3) the importance of the offered indicators of globalisation in higher education at the global/regional/national/university levels; (4) the importance of the offered indicators of internationalisation in higher education at the European/national/university levels. At the end of each section of the questionnaire, the researchers had an opportunity to comment on the elements offered and to propose new elements of definitions not included in the questionnaire. The same procedure was applied in the sections concerned with the indicators of globalisation and internationalisation in the higher education field. By combining qualitative and quantitative approaches, we allowed the participants to include additional information and opinions not found in the literature.
We received 58 valid answers from the experts in the first round, represent 33.14% of the whole population of invited researchers. Most experts who participated in the first round come from Europe, North America, Australia and Asia.
First, we prepared a summary report containing the various comments and new elements arising from the open-ended questions as well as ratings of all the elements in terms of mean values, standard deviation, variance and Coefficient of Quartile Variation (CQV)9. The elements and indicators with the highest mean values and lowest CQV were used in the definitions proposed in the second round of the questionnaire.
Based on the 58 experts' answers, 28 elements from a total of 86 proposed for all definitions in the first questionnaire, achieved a high level of consensus among the respondents. Accordingly, these 28 elements provided the main outline for the working versions of definitions of globalisation, internationalisation and Europeanisation generally and in higher education field specifically that were later formulated for the questionnaire in the
9Calculated as (CQV) = (Q3 - Qi)/(Qs + Qi) (Queyen, 2014).
Delphi method
13
second round. Further, a total of 35 elements achieved a relative level of consensus among the respondents and were therefore additionally discussed within our research group (the "Delphi Team") while preparing the second questionnaire. Conversely, 23 elements were rejected due to the experts' low ratings.
From a total of 30 proposed indicators of globalisation in the higher education field on the global/European/national/university levels, and 30 indicators of internationalisation in HE at the European/national/university levels in the first questionnaire, only 8 indicators of globalisation and 20 indicators of internationalisation achieved a strong consensus among the experts (see Table 1). According to the results, these 28 indicators were immediately included in the second questionnaire. Moreover, based on the experts' answers, 11 indicators of globalisation and 8 indicators of internationalisation achieved a relative level of consensus and the Delphi. These indicators were considered when the second-round questionnaire was being drafted. However, due to their low ratings, 11 indicators of globalisation and 2 indicators of internationalisation were rejected from further analysis. In addition, the Delphi Team included 3 indicators of globalisation and 8 indicators of internationalisation which had been proposed by experts in the first round of the survey.
Table 1: Number of indicators from the first-round questionnaire through
to the final selection
	Global.	Intern.	Total
First round	30	30	60
Second round			
High consensus in the 1st round	8	20	28
Relative consensus in the 1st round	11	8	19
New - proposals in the 1st round	3	8	11
Final selection after the 2nd round	9	15	24
Note: Global. = Globalisation, Intern. = Internationalisation
Building on the results of the first round of the Delphi method research and a broad analysis, the Delphi Team worked on preparing the second questionnaire during March and April 2017.
The Delphi Team relied on the rating results and experts' comments on the proposed elements and indicators, as well as their additional suggestions for new elements and indicators, to create: (1) working versions of broad, consolidated general definitions of globalisation, internationalisation and Europeanisation; (2) working versions of broad, consolidated definitions of globalisation, internationalisation and Europeanisation in the higher education field; and (3) a consolidated list of indicators of globalisation and internationalisation in higher education on different levels.
Broad working versions of consolidated definitions of the three terms were assembled primarily on the basis of elements that had attracted a strong consensus among the experts in the first round and, additionally, based on the Delphi Team's discussion of individual elements that achieved a relatively strong consensus among the experts. On top of this, while preparing the second questionnaire the Delphi Team considered all of the comments and suggestions given by the experts in the first round.
14
Fink-Hafner et al.
The second round of our Delphi method survey involved three steps: (1) sending a link to the experts to the second questionnaire; (2) analysing the results; and (3) the formulating of minimal (thin) definitions and a final list of indicators.
Implementation of the second-round survey started on 10 April 2017 by sending emails with a link to the second online questionnaire to the email addresses of the 58 researchers who had participated in the first round of research.
The questionnaire for the second round contained proposed broad, consolidated general definitions of globalisation, internationalisation and Europeanisation and definitions of these three terms in the HE field. The questionnaire also contained a consolidated list of indicators of globalisation and internationalisation in HE on different levels.
The first section of the questionnaire asked the experts to decide whether they agree with the proposed definitions or wished to change them and give comments. The 1KA web survey software enabled the team to utilise techniques unfeasible with a paper-based questionnaire or in a personal interview: the experts were presented with the proposed definitions. They could interactively alter them and then comment on their changes or write down further suggestions. In the second section of the questionnaire, the experts were also asked to rank the indicators and to decide on their importance on a scale from 1 (least important) to 5 (most important). A technical solution made the ranking easier by allowing the experts to interactively drag and drop the indicators into rows according to the importance they attributed them with.
While conducting the second round of the Delphi method research, the Delphi Team sent out two reminder emails to the experts (26 April 2017 and 8 May 2017). The second round of the Delphi method research was completed on 15 May 2017.
In the Delphi method survey's second round, a total of 36 valid expert responses was received, representing 62.06 % of all the researchers who participated in the first wave (58 experts). Most researchers who participated in the second round come from Europe, North America, Australia and Asia.
The strongest consensus among the experts (more than 50 %) was achieved by the following working versions of the broad, consolidated definitions: the general definitions of globalisation (58.33 %), the general definition of internationalisation (55.55 %) and the definition of internationalisation in the higher education field (55.55 %). A relatively high level of consensus among experts, just below 50 %, was achieved by: the general definition of Europeanisation (47.22 %) and the definition of Europeanisation in the higher education field (47.22 %). The weakest consensus among the experts was seen for the definition of globalisation in HE (38.88 %). This definition also attracted the biggest share of the experts' comments and suggestions (direct suggestions for a change in the elements and specific terms) and more general comments (in total 41.66 %).
Detailed analysis showed that most of the suggested interventions in the definitions may be related to the conceptual differences existing between the researchers. These differences may, inter alia, point to the different national/regional contexts of the researchers (e.g., differences between US researchers and those from Central European countries, etc.) and various perceptions of certain phenomena that either influence the concepts of globalisation, internationalisation and Europeanisation or not, in both the field of higher education and generally. These perceptions may also vary with respect to scientific areas.
Based on analysis of the results of the two rounds of the Delphi method survey, we attempted to construct the thin consolidated definitions presented in the following section.
Delphi method
15
3.4 Delphi Results
Our Delphi results have a clear theoretical focus - consensual thin definitions of globalisation, internationalisation and Europeanisation. In order to ensure consistency in the general definitions of these three terms and the definitions of these three terms in the HE field, the experts provided feedback concerning both the general and HE-specific definitions.
The following consensual thin definitions of globalisation, internationalisation and Europeanisation in general and in the HE field emerged:
Globalisation in general is a process of worldwide social connecting over and across borders (where social means societal, economic, cultural and political).
Globalisation in the higher education field is a process of increasing interconnected-ness in HE, a part of the globalisation process in general, which includes transnational HE connections disembedded from the national context, depending on the rise of global competition and global use of new technologies.
Internationalisation in general is a steerable process of greater cooperation and cross-border formal relations between states, institutions and organisations.
Internationalisation in the higher education field is a steerable process of greater cooperation within HE which goes beyond national borders and includes an international and/or global dimension in the teaching, research, service functions, purpose and delivery of HE.
Europeanisation in general (in broader terms) is part of globalisation.
Europeanisation in general (in a narrow sense) is a process of internationalisation based on policymaking at the EU level (and not limited to the EU).
Europeanisation in higher education (in broader terms) is part of globalisation.
Europeanisation in higher education (in a narrow sense) is a process of internationalisation in higher education based on policymaking and implementation in the EU framework.
As noted in the previous section, these thin definitions reflect the responses of experts who actually took part in both Delphi rounds despite the intention to include all of the revealed experts from around the world (for this purpose we used snowball sampling). Further, the 'measuring' of consensus and the overall judgements of the researchers in designing and implementing the Delphi process impacted the final outcome. However, as we were searching for the broadest consensus possible, this allowed definitions to be developed at a level of abstraction able to 'travel' across time and space. Conversely, they do not provide an insight into variations regarding conceptual differences between researchers, variations in researchers' perceptions of their scientific field and the different national/regional milieus of the researchers in both the field of higher education and generally.
16
Fink-Hafner et al.
4 Discussion, Summary and Conclusions
Our research allows us to confirm some of the previously recognised advantages and disadvantages of the Delphi method (as mentioned in the literature review). Advantages included the following: the method's flexibility; avoidance of experts directly confronting each other; the feedback process permits and encourages the selected Delphi participants to reassess their initial judgements on the information provided in previous iterations; the ability to provide anonymity to respondents; the controlled feedback process; the suitability of several statistical analysis techniques for interpreting the data; the issue of confidentiality is facilitated by the geographic dispersion of the subjects and the use of electronic communication to solicit and exchange information; it reduces the potential impact of group dynamics (such as manipulation or coercion to conform with or adopt a certain viewpoint). We also agree with previous comments made concerning the Delphi method's disadvantages: part of the procedure depends on the careful analysis of the responses - which is a matter for researchers; we cannot be 100 % sure the experts' responses are truly independent (we are unable to monitor possible communication among the surveyed experts); Delphi is time-consuming and laborious; the criteria for defining and determining a consensus - although again a matter for researchers - hold the potential for the moulding of opinions; the expertise possessed by the Delphi panellists might be unevenly distributed; the potential for low response rates.
Based on our research, we may add that Delphi is helpful for finding a minimal common denominator among a considerable number of experts while also avoiding the experts coming into direct contact to overcome the burden of possible personal animosities. In the circumstances of a large number of experts (like in our case), it is also useful that the Delphi procedure combines quantitative and qualitative approaches in order to ensure good quality feedback for the Delphi rounds. Yet, its negative sides are: rather low response rate and additional drop-outs as the Delphi process continues; requirement of fluency in English; uneven response of experts from various geographical/cultural milieus; the search for common grounds in the framework of a relatively heterogeneous expert community; the experts' feedback beyond the overall common grounds is lost if it is not further processed to reveal clusters of the experts' responses in terms of their similarities.
To conclude, the Delphi method holds the potential to help when searching for an academic consensus on particular definitions. Nevertheless, the questions of how to better implement and adjust Delphi call for answers. Fuzzy Delphi may be helpful here yet on certain occasions, such as for theoretically relevant issues (e.g. searching for a consensus on some basic definitions), additional ways of expert community consensus and legitimacy building are still needed. This explains why we propose several ways for adjusting the Delphi by evaluating its results.
Possible forms of evaluation (based on our empirical study) may include: (1) obtaining feedback on the particular Delphi results by enlisting two focus groups - one including academics taking part in the Delphi and the other including academics who did not take part; (2) conducting a survey on the Delphi produced thin definitions with an aim to achieve a response from the global expert community; and (3) further analysis of the 'lost' data (revealing clusters of similar expert judgements according to the values and ideas they expressed).
Delphi method
17
Acknowledgement
The authors acknowledge the financial support from the Slovenian Research Agency (research core funding No. P5-0136 and No. P5-0151).
References
[1]	Amos, T. and Pearse, N. (2008): Pragmatic research design: An illustration of the use of the Delphi technique. The Electronic Journal of Business Research Methods, 6(2), 95-102.
[2]	Atkinson, R. and Flint, J. (2001): Accessing hidden and hard-to-reach populations: Snowball research strategies. Social Research Update, 33(1), 1-4.
[3]	Avella, J. R. (2016): Delphi panels: Research design, procedures, advantages, and challenges. International Journal of Doctoral Studies, 11(1), 305-321.
[4]	Biemer, P. P. and Lyberg, L. E. (2003): Introduction to survey quality. New York, NY: John Wiley & Sons.
[5]	Bleijenbergh, I., Korzilius, H., and Verschuren, P. (2011): Methodological criteria for the internal validity and utility of practice oriented research. Quality & Quantity, 45(1), 145-156.
[6]	Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., and Alberti, C. (2011): Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLOS One, 6(6), e20476.
[7]	Brady, S. R. (2015): Utilizing and adapting the Delphi method for use in qualitative research. International Journal of Qualitative Methods, 14(5). 1-6.
[8]	Dalkey, N. and Helmer, O. (1963): An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458-467.
[9]	De Leeuw, E. D. (2005): To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(5), 233-255.
[10]	Deardorff, D. K., de Wit, H., Heyl, J. D., and Adams, T. (Eds.). (2012): The SAGE Handbook of International Higher Education. London: Sage.
[11]	Donohoe, H. M. and Needham, R. D. (2009): Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. International Journal of Tourism Research, 11(5), 415-437.
[12]	Donohoe, H., Stellefson, M., and Tennant, B. (2012): Advantages and limitations of the e-Delphi technique: Implications for health education researchers. American Journal of Health Education, 43(1), 38-46.
[13]	Dunn, N. W. (1994): Public Policy Analysis: An Introduction. NJ: Prentice Hall.
18
Fink-Hafner et al.
[14]	Fletcher, A. J. and Marchildon, G. P. (2014): Using the Delphi method for qualitative, participatory action research in health leadership. International Journal of Qualitative Methods, 13(1), 1-18.
[15]	Hartman, F. and Jugdev, K. (1998): Leadership undertow: Project manager fears and frustrations. Proceedings of the 29th Annual Project Management Institute Seminars and Symposium, Long Beach, USA.
[16]	Hsu, C. C. and Sandford, B. A. (2007): The Delphi technique: Making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8.
[17]	Huisman, J., Boer, H., and Botas, P. C. P. (2012): Where do we go from here? The future of English higher education. Higher Education Quarterly, 66(4), 341-362.
[18]	Iqbal, S. and Pippon-Young, L. (2009): The Delphi method. Nursing Research, 46(2), 116-118.
[19]	Ishikawa, A., Amagasa, M., Shiga, T., Tomizawa, G., Tatsuta, R., and Mieno, H. (1993): The max-min Delphi method and fuzzy Delphi method via fuzzy integration. Fuzzy Sets and Systems, 55(3), 241-253.
[20]	Knight, J. (2012): Concepts, Rationales, and Interpretive Frameworks in the Internationalization of Higher Education. The SAGE Handbook of International Higher Education. London: SAGE, 27-42.
[21]	Land, K. C. and Schneide, S. H. (Eds.) (1987): Forecasting in the Social and Natural Sciences. Dordrecht: D. Reidel Publishing Company.
[22]	Linstone, H. A. and Turoff, M. (Eds.). (1975): The Delphi Method. Reading, MA: Addison-Wesley.
[23]	Melnyk, S. A., Lummus, R. R., Vokurka, R. J., Burns, L. J., and Sandor, J. (2009): Mapping the future of supply chain management: A Delphi study. International Journal of Production Research, 47(16), 4629-4653.
[24]	Moniz, A. (2005): Methods for scenario-building: It's importance for policy analysis. Published in: Proceedings of Workshop on "Innovative Comparative Methods for Policy Analysis" (September 2005): pp. 1-18. MPRA Paper No. 8094. Available at: https://mpra.ub.uni-muenchen.de/8094/ (4.3.2019.)
[25]	Nambisan, S., Agarwal, R., and Tanniru, M. (1999): Organisational mechanisms for enhancing user innovation in information technology. MIS Quarterly, 23(8), 365395.
[26]	Niederman, F., Brancheau, J. C., and Wetherbe, J. C. (1991): Information systems management issues for the 1990s. MIS Quarterly, 15(4), 475-500.
[27]	Noy, C. (2008): Sampling knowledge: The hermeneutics of snowball sampling in qualitative research. International Journal of Social Research Methodology, 11(4), 327-344.
Delphi method
19
[28]	Okoli, C. and Pawlowski, S. D. (2004): The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42(1): 15-29.
[29]	Quyen, D. T. N. (2014): Developing university governance indicators and their weighting system using a modified Delphi method. Procedia - Social and Behavioral Sciences, 141, 828-833.
[30]	Pal, L. (1987): Public Policy Analysis. An Introduction. Toronto: Methuen.
[31]	Paivarinta, T., Pekkola, S., and Moe, C. (2011): Grounding Theory from Delphi Studies. Research paper available at https://pdfs.semanticscholar.org/3b8a/b6a35b4317fef82b76446fee9c0e4ea5f400.pdf (3.7.2019).
[32]	Patton, C. V. and Sawicki, D. S. (1993): Basic Methods of Policy Analysis and Planning. Englewood Cliffs, NJ: Prentice-Hall.
[33]	Rayens, M. K. and Hahn, E. J. (2000): Building consensus using the policy Delphi method. Policy, Politics, & Nursing Practice, 1(4), 308-315.
[34]	Rumbley, L. E., Altbach, P. G., and Reisberg, L. (2012): Internationalization within the Higher Education Context. The SAGE Handbook of International Higher Education, 3-26.
[35]	Sackman, H. (1975): Delphi Critiques. Lexington, MA: DC Heath.
[36]	Schmidt, R. C. (1997): Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences, 28(3), 763-774.
[37]	Schmidt, R., Lyytinen, K., Keil, M., and Cule, P. (2001): Identifying Software Project Risks: An International Delphi Study. Journal of Management Information Systems, 17(4), 5-36.
[38]	Skulmoski, G. J., Hartman, F. T., and Krahn, J. (2007): The Delphi method for graduate research. Journal of Information Technology Education, 6, 1-21.
[39]	Uphoff, E. P., Wennekes, L., Punt, C. J., Grol, R. P., Wollersheim, H. C., Hermens, R. P., and Ottevanger, P. B. (2012): Development of generic quality indicators for patient-centered cancer care by using a RAND modified Delphi method. Cancer nursing, 35(1), 29-37.
[40]	Weaver, W. T. (1971): The Delphi forecasting method. The Phi Delta Kappan, 52(5), 267-271.
[41]	Wilson, S., Hauck, Y., Bremner, A., and Finn, J. (2012): Quality nursing care in Australian paediatric hospitals: A Delphi approach to identifying indicators. Journal of Clinical Nursing, 21(11-12), 1594-1605.