THE DEVELOPMENT OF A CONSENSUS DEFINITION FOR HEALTHCARE 
IMPROVEMENT SCIENCE (HIS) IN SEVEN EUROPEAN COUNTRIES: 
A CONSENSUS METHODS APPROACH 
Brigita SKELA-SAVIČ1*, Rhoda MacRAE2, Manuel LILLO-CRESPO3, Kevin D ROONEY2,4
1Faculty of Health Care Jesenice, Spodnji Plavž 3, 4270 Jesenice, Slovenia
2Institute of Care and Practice Improvement, School of Health, Nursing and Midwifery, 
University of the West of Scotland, Hamilton, ML3OBA, Scotland
3Faculty of Health Sciences, University of Alicante, Spain, Carretera de San Vicente del Raspeig s/n 03690 
San Vicente del Raspeig, Alicante, Spain
4Anaesthesia & Intensive Care Medicine, Royal Alexandra Hospital, Corsebar Road, Paisley, PA2 9PN, Scotland
Received: Feb 28, 2016
Accepted: Aug 29, 2016
Original scientific article
Introduction. There is a limited body of research in the field of healthcare improvement science (HIS). Quality 
improvement and ‘change making’ should become an intrinsic part of everyone’s job, every day in all parts 
of the healthcare system. The lack of theoretical grounding may partly explain the minimal transfer of health 
research into health policy.
Methods. This article seeks to present the development of the definition for healthcare improvement science. 
A consensus method approach was adopted with a two-stage Delphi process, expert panel and consensus 
group techniques. A total of 18 participants were involved in the expert panel and consensus group, and 153 
answers were analysed as a part of the Delphi survey. Participants were researchers, educators and healthcare 
professionals from Scotland, Slovenia, Spain, Italy, England, Poland, and Romania. 
Results. A high level of consensus was achieved for the broad definition in the 2nd Delphi iteration (86%). The 
final definition was agreed on by the consensus group: ‘Healthcare improvement science is the generation of 
knowledge to cultivate change and deliver person-centred care that is safe, effective, efficient, equitable and 
timely. It improves patient outcomes, health system performance and population health.’
Conclusions. The process of developing a consensus definition revealed different understandings of 
healthcare improvement science between the participants. Having a shared consensus definition of healthcare 
improvement science is an important step forward, bringing about a common understanding in order to advance 
the professional education and practice of healthcare improvement science. 
Uvod. Skupno število raziskav na področju znanosti o uvajanju izboljšav v zdravstvu (Healthcare Improvement 
Science - HIS) je majhno. Vendar pa bi morala izboljševanje kakovosti in vpeljava sprememb postati neločljivo 
povezana z vsakodnevnim delom zdravstvenih delavcev na vseh področjih zdravstvenega sistema. Pomanjkanje 
teoretičnih izhodišč bi lahko delno pojasnilo nizko stopnjo prenosa rezultatov raziskav v zdravstveno politiko.
Metode. Namen članka je predstaviti postopek oblikovanja definicije za pojem znanost o uvajanju izboljšav 
v zdravstvu. Pri pristopu z metodami konsenza smo uporabili metodo Delfi, izvedeno v dveh krogih, panelne 
skupine strokovnjakov in tehnike konsenzne skupine. V skupini strokovnjakov in konsenzni skupini je sodelovalo 
18 oseb, v okviru metode Delfi smo analizirali 153 odgovorov. Sodelujoče osebe so bile raziskovalci, izobraževalci 
in zdravstveni delavci iz Škotske, Slovenije, Španije, Italije, Anglije, Poljske in Romunije.
Rezultati. V drugem krogu metode Delfi je bila dosežena visoka stopnja konsenza (86-odstotna) za široko 
definicijo. Konsenzna skupina je oblikovala naslednjo končno definicijo: »Znanost o uvajanju izboljšav omogoča 
ustvarjanje znanj za izvajanje sprememb ter zagotavljanje v pacienta usmerjene zdravstvene obravnave, ki 
je varna, zmogljiva, učinkovita, pravična in pravočasna. Izboljšuje pacientove izide zdravstvene obravnave, 
sistem nudenja zdravstvenih storitev in zdravje populacije.« 
Razprava. Postopek oblikovanja dogovorne definicije je razkril različno razumevanje pojma znanost o uvajanju 
izboljšav v zdravstvu med sodelujočimi. Oblikovanje skupne dogovorne definicije je pomemben korak naprej, 
saj pripomore k splošnemu razumevanju tega pojma in s tem tudi k spodbujanju strokovnega izobraževanja in 
izvajanju znanosti o uvajanju izboljšav v praksi.
ABSTRACT
Keywords: 
healthcare, 
improvements,  
science, consensus 
methods, definition, 
Europe
IZVLEČEK
Ključne besede: 
zdravstvo, izboljšave, 
znanost, metode 
konsenza, definicija, 
Evropa
*Corresponding author: Tel: ++ 386 4 5869 360; E-mail: bskelasavic@fzj.si
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
82
OBLIKOVANJE DOGOVORNE DEFINICIJE ZA POJEM ZNANOST O UVAJANJU 
IZBOLJŠAV V ZDRAVSTVU (HIS) V SEDMIH EVROPSKIH DRŽAVAH: 
PRISTOP Z METODAMI KONSENZA
Skela-Savič B, MacRae R, Lillo-Crespo M, Rooney KD. The development of a consensus definition for Healthcare Improvement Science (HIS) in seven European 
countries: a consensus methods approach. Zdr Varst 2017; 56(2): 82-90
© Nacionalni inštitut za javno zdravje, Slovenija. 
1 INTRODUCTION
Improving healthcare quality has become a priority over 
the past years (1). While much of quality improvement 
work is unscientific (2), the adoption of a more scientific 
approach to improvement could enhance the ability of 
health systems to provide high-quality care and use their 
resources optimally (3). Healthcare improvement science 
(HIS) represents ‘the combined and unceasing efforts of 
everyone – healthcare professionals, patients and their 
families, researchers, payers, planners and educators 
– to make the changes that will lead to better patient 
outcomes (health), better system performance (care) 
and better professional development (learning)’ (4, 
p. 2). In this respect, quality improvement and change 
making should become an essential part of healthcare 
professionals’ work at all levels of the healthcare system 
(5). Implementation science and quality improvement 
efforts share the ultimate goal of improving healthcare 
quality. However, there are differences. Quality begins 
with a specific problem in a given healthcare system, 
while implementation science begins with evidence-
based practice (EBP) that is not used effectively, then 
identifies and addresses quality issues at the level of 
providers, clinics, or the healthcare system (6). Topic 
reviewers reported a diversity of outcomes used for 
quality improvement evaluations, flaws in study design, 
and incomplete reporting (7).
The Health Foundation (8) developed an explanation of 
what improvement science should be. They defined it as 
‘building a knowledge base for improving health services 
and translating this knowledge into practice to deliver the 
best possible patient care’ (8, p. 7). Moreover, it has been 
stressed that the implementation science community 
should provide guidelines for reporting the use of 
theoretical frameworks within implementation studies 
and their efficacy (9). This is important because few 
hospital-based interventions are essentially theoretical 
(10). The Health Foundation definition (8, p. 7) states 
clearly that improvement science ‘requires a systematic, 
scientifically rigorous approach to close the gap between 
current and best practice.’
In the case of quality improvement, theory needs 
development, which draws from a better understanding 
of practice (11). Many quality improvement practitioners 
believe that quality improvement is implemented 
in complex adaptive systems. They sometimes feel 
limited by research findings that fail to recognize 
the complexity and the need for adaptation. Quality 
improvement practitioners could help researchers to 
see the larger patterns specific to complex systems 
and draw upon possible solutions (12). Organizational 
learning framework could be used to explore the factors 
influencing improvement feasibility (13). Organizations 
worldwide have adopted interprofessional education 
and interprofessional collaborative practice in an effort 
to improve healthcare delivery systems (14), which fits 
well with the Health Foundation definition (8, p. 7) that 
‘improvement science embraces disciplines across health 
services research, from sociology to statistics, psychology 
to health economics.’ The next generation of healthcare 
professionals will require new skills to make sure that 
quality improvement in health care is successful (15). 
For this reason, it is important how different healthcare 
professionals understand HIS.
1.1 The Aim
The article describes a part of the project ‘Improvement 
Science Training for European Healthcare Workers 
(ISTEW),’ which aimed to develop shared academic and 
practice-based programmes that would enable European 
institutions to build improvement capacity within their 
healthcare workforce. This article discusses a keystone 
component of the project, namely the development of a 
consensus definition for HIS, based on the interprofessional 
and multicultural consensus of healthcare professionals 
considered as experts from seven European countries.
2 METHODS
2.1 Research Design and Data Collection 
Research design comprised of consensus methods, 
including the nominal group technique, the Delphi 
process, and the consensus group technique. The aim of 
consensus methods is to determine the extent to which 
experts or lay people agree about a given issue. The 
term ‘agreement’ takes two forms, which need to be 
distinguished: firstly, the extent to which each respondent 
agrees with the issue under consideration (typically rated 
on a numerical or categorical scale) and, secondly, the 
extent to which respondents agree with each other (16). 
2.1.1 The Nominal Group Technique
The nominal group technique uses a highly structured 
meeting to gather information from relevant experts 
(usually 9-12 in number) about a given issue. It consists of 
two rounds, in which panellists rate, discuss, and then re-
rate a series of items or questions (16). The nominal group 
technique was used twice. First, we implemented two 
structured Skype meetings with research group members 
from seven European countries. The goal was to develop 
a working definition of HIS, to increase consensus amongst 
participants about the meaning and content of HIS, and to 
establish the level of development and understanding of 
HIS in each country. Expert-based knowledge, literature 
and comprehension across a core group of seven experts 
was used. After two meetings, a working definition was 
developed and tested in the Delphi process. Second, the 
nominal group technique was used on the same team in 
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
83
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
84
group technique by using qualitative and quantitative 
data from the first Delphi iteration. These reformulated 
items were presented in the second Delphi iteration. 
2.1.3 The Consensus Group Technique
Following two Delphi iterations and two nominal group 
techniques, a consensus group technique (a combination 
of a focus group and a public meeting) was employed. 
The main difference between consensus groups and focus 
groups is that in the latter, the moderator or analyst 
decides, whereas in consensus groups, the participants 
themselves negotiate and decide the findings (18). We 
used this technique at a three-day meeting of all ISTEW 
project members held in Bled (June 2014), Slovenia. The 
consensus group included 18 participants from seven 
European countries. We wanted to provide everyone 
in the group the opportunity to work collaboratively, 
to be involved in the decision-making processes, to 
be valued as experts, and for the group to become a 
forum for change (19). Consensus group members were 
discussing quantitative and qualitative data of the 2nd 
Delphi iteration and the suitability of the developed HIS 
definition. The aim was to arrive at a final definition of 
HIS.
2. 2 The Sample 
A non-random convenience sample was used for the Delphi 
process. Each project team invited 10–15 people they 
knew to be involved in HIS in their respective settings. 
‘Participants’ came from seven European countries 
(England, Italy, Poland, Romania, Scotland, Slovenia, and 
Spain) and from different healthcare areas. The total 
number of respondents in the 1st iteration was 87; of 
these, 66 chose to participate in the 2nd iteration. HIS 
fields in which the respondents were involved are shown 
in Table 1.
order to develop an extended questionnaire for the 2nd 
Delphi iteration. 
2.1.2 The Delphi Process
A Delphi study is a robust method that draws on expert 
opinions, and compares them with the combined opinions 
of other participating experts over several rounds, 
until a consensus on specified criteria is reached (16). 
A two-iteration Delphi technique was developed with 
participants from seven European countries. The Delphi 
technique enabled us to collect data from several 
domains related to healthcare in education, research, 
management and practice. Moreover, two questionnaires 
were designed for each iteration with quantitative and 
qualitative elements allowing us to achieve a merging 
of opinions from 87 respondents. The respondents were 
consulted in two anonymous Delphi rounds. A 5-point 
Likert scale (1–strongly disagree, 2–disagree, 3–neither 
agree nor disagree, 4–agree, 5–strongly agree) was used 
in both questionnaires to rate the importance of each 
HIS definition element and of the definition as a whole. 
Respondents could also offer suggestions for improving 
separate parts of the definition by replying to open-ended 
questions and providing HIS descriptions used in their 
countries. Furthermore, we were interested in obtaining 
some demographic data (the area of work, respondent 
connection with HIS and involvement in HIS) and data 
about the description of HIS in their countries. Reliability 
test results were good. Cronbach’s Alpha for statements 
(N=6) for 1st Delphi iteration was 0.867 and for statements 
(N=7) for 2nd Delphi iteration 0.843. 
Over several rounds, a specific definition element was 
included in the next version of the definition if >80% of 
respondents judged it as important or very important, and 
excluded if >50% judged it as not important or moderately 
important (17). The suggested reformulations of the 
second HIS definition were analysed with the nominal 
Table 1. Participants’ fields of work in the Delphi study.
1st iteration 
(n=87)
2nd iteration
(n=66)
37% (32)
27% (18)
29% (25)
39% (26)
23% (20)
23% (15)
7% (6)
5% (3)
4% (4)
6% (4)
Patient 
organisations
Public policyClinical practiceManagementEducation,
research
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
85
Ten members of the research group participated in 
the nominal group technique, and 18 members of the 
ISTEW project group participated in the consensus group 
technique. Members of both groups were experts in HIS 
from different professions.
2.3 Research Conduction and Ethical Approval
The research was conducted from November 2013 to 
September 2014. We started with the nominal group 
technique, first conducted in November 2013 and then 
in March 2014, followed by the Delphi process (the 
first and second iterations in December 2013 and April 
2014, respectively), and finally, the consensus group 
technique in June 2014.The research protocol was 
approved by the scientific ethical committee of the 
research coordinator, the Faculty of Health Care Jesenice 
(Slovenia). Partners also adhered to their ethical approval 
processes for research. Participants were assured about 
the anonymity and confidentiality of their collaboration. 
The questionnaires were available either in English or the 
language of the expert. Partners were responsible for 
translating the questionnaires, as required.
2.4 Data Analysis
Quantitative data analysis was carried out with descriptive 
statistics (Mean, Standard Deviation, Frequency, 
Percentage) and bivariate statistics (Chi-Square (p) 
– Kruskal-Wallis H nonparametric test, Independent-
samples t-test, Paired samples t-test, Pearson 
correlation, Spearman correlation) in the statistical 
software programme SPSS, v. 20.0. The level of statistical 
significance was set at p<0.05. 
Qualitative data was obtained from the open-ended 
questions of the first and second Delphi iterations. 
First, each partner team extracted the themes from the 
qualitative data from both Delphi iterations and exposed 
these themes in the nominal group technique and the 
consensus group technique. The software programme 
NVivo, v. 10.0 and content analysis were used. 
3 RESULTS 
3.1 Characteristics of Respondents in the Delphi Survey 
We asked respondents to classify themselves in three 
ways: those who worked in the field of HIS daily (this 
group we called ‘expert’), those who worked in the field 
of HIS occasionally (this group we called ‘occasional 
worker’), and those who were informed about HIS (this 
group we called ‘informed layman’). In the 1st Delphi 
iteration, significant differences were established 
between countries in identifying respondents as experts of 
Table 2. Respondents’ involvement in HIS and differences 
between countries.
‘Expert’
‘Occasional 
worker’
‘Informed 
layman’
21 
(28.0)
44 
(62.9)
32 
(48.5)
30
(45.5)
31
(47.0)
24
(26.7)
24.411
(<0.001)
13.687
(0.033)
2.614
(0.856)
21.181
(0.002)
11.172
(0.083)
15.048
(0.020)
1st Delphi iterationGroups of 
respondents
Yes
f (%)
Yes
f (%)
X2 (p) X2 (p) 
2st Delphi iteration
HIS (p<0.001), occasional workers (p=0.033) or informed 
laymen (p<0.001). The number of answers ‘Yes’ was 97; 
10 respondents recognized themselves in more than one 
way. In the 2nd Delphi iteration, significant differences 
were established between respondents in identifying HIS 
‘expert’ (p=0.002) and in classification as an ‘informed 
layman’ (p=0.020). The number of answers ‘Yes’ was 
85; 19 respondents recognized themselves in more than 
one way. Results are in Table 2. We had more experts 
and occasional workers in the second iteration, and less 
informed laymen.  
Comment: X2 (p) is answer ‘Yes’ between countries. 
A total of 50% of respondents agreed that their national 
bodies, commissions or government which include HIS 
play an important part in healthcare and social policy in 
their country (n=43, x2=37.502, p<0.001). A total of 25 
respondents (35.2%) agreed that HIS has an accepted 
status in healthcare or social organizations, and 29 
respondents (35.4%) reported that national scientific 
research in HIS has been conducted in their country. We 
did not collect this data in the 2nd iteration.
3.2 Results of the 1st Delphi Iteration
In the 1st iteration, we tested the definition of HIS 
which was developed with the nominal group technique. 
The definition is shown in Table 3. Results reveal a high 
level of agreement with the definition as a whole on a 
5-point scale. Respondents from Scotland expressed a 
significantly lower agreement with the definition as a 
whole, but with the highest level of standard deviation 
of their answers (M=2.83, SD=1.27, p<0.001) and also 
with its separate parts (a) (M=2.25, SD=1.22, p<0.001), 
(c) (M=2.67, SD=0.99, p<0.001), (d) (M=2.57, SD=1.78, 
p=0.029), except for parts (b) and (e). 
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
The agreement with the definition increased with the level 
of respondents’ expertise. Correlation analysis showed us 
that respondents who identified themselves as ‘experts’ 
were in positive correlation with the agreement with the 
definition as a whole (r=0.264, p=0.023). Respondents’ 
fields of work were not in correlation with the level of 
agreement with the definition as a whole. 
Qualitative responses of respondents (n=21) who rated 
themselves as experts were analysed. A review of 
responses to the question ‘Which areas are missing from 
the proposed definition?’ yielded 120 codes and 8 main 
categories, namely the content of the definition, the 
purpose of the definition, multidisciplinary approach, 
definition structure, healthcare professionals, negative 
consequences, innovative emergent field, and references. 
Word frequency of codes revealed that, in the opinions of 
HIS experts, the most frequent words to be included in 
Table 3. Results for the level of agreement with the definition as a whole and with its separate parts (a – e), and differences between 
countries in the 1st Delphi iteration (n=87).
Agreement with the definition as a whole, from the part a to the part e.
Healthcare Improvement Science (HIS) is the framework for achieving efficiency, efficacy and quality in 
health and social care (a).
HIS depends on knowledge from a wide range of sources, not just research (b).
HIS is a link (connection) between numerous variables in clinical work and management/leadership work, 
such as: multi-disciplinary approach, patient involvement, patient safety, total quality management, change 
management, personal involvement of healthcare/social workers, personal development and responsibility, 
team work/group work, connections between practice-theory-problems-research, etc. (c).
HIS is thus an umbrella term for all actions (practice, education, science and policy) that can lead to better 
health treatment outcomes (health), better system performance (care), better professional development 
(learning) and healthier communities (d).
HIS also demands flexibility and responsibility in our understanding, in our theories of knowledge, and in our 
use of research evidence in health care improvement strategy (e).
3.93
(0.861)
3.85
(1.070)
4.15
(0.958)
4.15
(0.907)
4.01
(1.101)
4.26
(0.767)
21.533 
(<0.001)
30.278 
(<0.001)
7.921
(0.244)
27.436 
(<0.001)
14.017 
(0.029)
7.319
(0.292)
Chi-Square 
(p)
M 
(SD)
Legend: M – central value of a discrete set of numbers calculated by the sum of the values divided by the number of values; SD – 
Standard deviation; Chi-Square (p) – Kruskal-Wallis H nonparametric test for several independent samples; a, b, c, d, e – parts of the 
whole definition.
Comment: X2 (p) is mean value between countries. 
the proposed definition are the following: Improvement 
(n=19), social (n=14), science (n=13), health (n=10), 
methods (n=10), healthcare (n=9), care (n=8), patient 
(n=8) and work (n=8).
3.3 Results of the 2nd Delphi Iteration
In the second iteration, we tested the definition of HIS 
developed with the nominal group technique, where 
the results from the first Delphi iteration were used. 
The definition is worded in Table 4. The average level of 
agreement with the second proposed definition of HIS was 
higher (M=4.30) compared to the first iteration (M=3.93) 
(t=-3.006, p=0.004). The average portion of agreement in 
the first iteration was 78.5% and in the second 86%. For 
separate parts of the definition, the greatest agreement 
was expressed with the part (b). 
86
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
87
Expertise in the field of HIS and respondents’ area of work 
did not significantly correlate with the agreement with 
HIS definition. Significant differences in the definition as 
a whole were established between partners (p<0.001). 
For example, respondents from Scotland expressed a 
significantly lower agreement with the definition as 
Table 4.
Table 5.
Results for the level of agreement with the definition as a whole and with its separate parts (a – f), and differences between 
countries in the 2nd Delphi iteration (N= 66).
Results of countries look for the level of agreement with the definition as a whole and with its separate parts (a – f) in the 
2nd Delphi iteration.
Agreement with the definition as a whole, from the part a to the part f
Healthcare Improvement Science (HIS) is focused on safety, effectiveness, efficiency, timeliness, equity and 
continuous patient-centred improvement in healthcare (a).
HIS depends on valid and reliable knowledge from a wide range of sources, in both the academic sector and 
service sector in healthcare (b).
HIS is multidisciplinary in its approach, drawing both on biomedical and on social sciences (c).
Different factors, such as methodological frameworks for improvement, knowledge of healthcare 
professionals, patient involvement and innovation, support organizational culture, are factors that interplay 
with the approach (d).
HIS is thus an umbrella term for any action (practice, education, science and policy) that can lead 
to improved healthcare outcomes for people, better health system performance and also healthier 
communities (e).
HIS also demands flexibility and responsibility in understanding, in theories of knowledge, and in the use of 
research evidence in health care improvement strategy (f).
Whole definition
Part a
Part b
Part c
Part d
Part e
Part f
4.30
(0.61)
64
65
65
66
66
66
66
4.20 
(0.45)
4.40 
(0.55)
4.20 
(0.84)
3.60 
(0.89)
3.20 
(0.84)
4.20 
(0.45)
4.20 
(0.45)
4.56 
(0.53)
4.56 
(0.53)
4.50  
(0.76)
4.44 
(0.53)
4.56 
(0.53)
4.44 
(0.53)
4.44 
(0.73)
4.33
(0.50)
4.67 
(0.50)
4.56  
(1.01)
4.67 
(0.50)
4.22 
(0.83)
4.11 
(0.93)
4.33 
(0.50)
4.71 
(0.49)
4.57 
(0.54)
4.86  
(0.38)
4.86 
(0.38)
4.57 
(0.54)
4.71 
(0.76)
4.86 
(0.38)
4.47 
(0.52)
4.20 
(0.86)
4.40 
(1.06)
4.40 
(0.91)
4.47 
(0.74)
4.53 
(0.52)
4.40 
(0.83)
3.50 
(0.53)
3.70 
(1.06)
4.09 
(1.04)
4.09 
(0.94)
3.36  
(1.43)
3.55 
(1.29)
3.45 
(0.93)
5.613 
(<0.001)
1.456 
(0.210)
0.676 
(0.669)
1.923 
(0.092)
3.473 
(0.005)
2.389 
(0.039)
3.648 
(0.004)
4.30
0.61
4.29
0.86
4.40
0.86
4.33
0.81
4.14
0.96
4.26
0.83
4.24
0.77
4.33
(0.50)
4.20
(1.23)
4.30
(0.48)
4.10
(0.88)
4.20
(0.63)
4.30
(0.48)
4.20
(0.42)
4.29
(0.86)
4.40
(0.86)
4.33
(0.81)
4.14
(0.96)
4.26
(0.83)
4.24
(0.77)
21.227 
(0.002)
7.727 
(0.259)
7.611
(0.268)
11.468 
(0.075)
14.904 
(0.021)
11.226 
(0.082)
16.761 
(0.010)
Chi-Square 
(p)
N PolandEngland Slovenia ScotlandM (SD) RomaniaItaly Spain Chi-
Square (p)
M 
(SD)
Legend: M – central value of a discrete set of numbers calculated by the sum of the values divided by the number of values; SD – 
Standard deviation; Chi-Square (p) – Kruskal-Wallis H nonparametric test for several independent samples; a, b, c, d, e – parts of the 
whole definition.
Comment: X2 (p) is mean value between countries. 
a whole (M=3.50, SD=0.53) and also with the part (f) 
(M=3.45, SD=0.93). Moreover, respondents from Scotland 
(M=3.36, SD=1.43) and England (M=3.20, SD=0.84) 
expressed a significantly lower agreement with the part 
(d). Detail results are in Table 5. 
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
88
3.4 Results of the Consensus Group Technique
The consensus group technique started with a discussion 
of 18 experts from institutions involved in the ISTEW 
project. It was moderated by the principal investigator 
and research fellow for the project, with notes being 
taken. First, the aim was to increase the understanding of 
the first and second Delphi iteration results and to present 
partners’ qualitative analysis results. The consensus group 
discussion yielded 14 key themes of the HIS definition, 
which were merged into 3 content categories, namely 
‘country differences knowledge’, ‘science and HIS,’ 
and ‘new dimension of HIS for practice’. The strongest 
extracted elements were knowledge, change, outcome, 
population, safety, efficiency, person-centred. 
All partners agreed that the HIS definition had to be short 
and concise. The emphasis of each definition part and 
the theme from qualitative data was discussed until a 
consensus was reached. Based on this, the final consensus 
on the definition of Healthcare Improvement Science is 
as follows:
‘Healthcare improvement science is the generation 
of knowledge to cultivate change and deliver person-
centred care that is safe, effective, efficient, equitable 
and timely. It improves patient outcomes, health system 
performance and population health.’
4 DISCUSSION 
HIS is incorporated into every phase of healthcare 
proceedings and is, consequently, a complex concept to 
define. Coining a definition that encompasses the multi-
faceted nature of HIS with all of its variables is very 
challenging, particularly because the use of the term 
HIS and its synonyms in the existing literature and other 
educational documents (grey literature) varies depending 
on the country and language. 
By applying consensus methods, the definition was 
developed gradually. According to the results of two 
Delphi iterations, a high consensus among participants 
was achieved. Consensus increased from the first iteration 
to the second, until 86% agreement was reached. It is 
very important that the definition was developed through 
interprofessional interactions. Only a third of respondents 
in the Delphi survey perceived themselves as experts in 
HIS, although the Delphi study coordinators identified 
them as such. This, we suggest, reflects the extent to 
which HIS is developed in the partner countries involved, 
as we found that HIS was not equally developed in all 
partner countries. Indeed, we found that the term itself 
is not used at all in Spain, and that the concept is not well 
developed in Romania, where the term implementation 
science with its different connotations tends to be 
more common. Respondents coming from countries 
where HIS was not broadly developed may have found it 
challenging to respond in an informed way. Respondents 
from Scotland, where HIS is a political priority, expressed 
a significantly lower level of consensus with the entire 
definition compared to non-English speaking respondents, 
but the level of agreement was still high at 70%. These 
differences may be related to a number of issues. Firstly, 
most respondents from Scotland defined themselves as 
experts in HIS. Secondly, in Scotland, HIS is a more crucial 
driver of healthcare institution policies compared to the 
other countries. Finally, research and accessibility of 
scientific and professional HIS literature is greater in the 
United Kingdom compared to other non-English native 
speaking countries (20).
The definition formed in two Delphi iterations was a 
broadened one. The consensus group technique greatly 
contributed to reaching a compromise between experts, 
so that a short and concise version of the definition was 
developed. The definition includes a wide range of key 
themes, such as knowledge for cultivating change, person-
centred care, improvement of patient outcomes, health 
system performance and population health. These themes 
are also included in definitions proposed by other authors 
(4, 8). The employed method for developing a definition 
leads to a better understanding of practice (11).
Only half of respondents agreed that HIS-related national 
bodies play an important role in health and social care 
policies in their own country. However, a closer look at 
the data reveals that the situation is not comparable 
across countries. Only one third of respondents felt that 
HIS has an appropriate status in their national healthcare 
system. As a result, HIS indicators are not developed well 
and HIS is an uncommon term in some countries. This 
is a major issue because, as Booth et al. (21) stated, 
changing population demographics, disease patterns and 
new research findings demand change in healthcare and 
require new ways for ensuring continuous improvement. 
Robson (22) believes that in making healthcare safer, 
one should join all levels of the profession in an effort to 
clearly work with their working environment and patient 
safety.
For a continuous improvement, an understanding and 
recognition of the need for HIS are crucial. Our definition 
underpins the need for developing interprofessional 
study programmes and other materials. Hall and Zierler 
(23) stressed that, with the growth of interprofessional 
education and practice in healthcare schools, faculty 
members have to assume new roles in designing and 
delivering interprofessional curricula. Senior leadership 
support is also vital in ensuring sustainable change, and it 
will be a key requirement in healthcare organisations and 
universities to make healthcare improvement a priority. 
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
89
5 LIMITATIONS 
The sample was not equally weighted for the nationalities 
of the HIS experts, and this may have impacted the Delphi 
process. Knowledge of the English language and familiarity 
with it may have also played a role in providing answers, 
in questionnaire translation, and in discussions within the 
nominal and consensus groups. In countries where English 
is not spoken as the official language, most partners had 
to first translate the questionnaire, then obtain answers 
from experts in their languages, and translate the 
responses back into English. This may have resulted in 
the loss of understanding and different interpretations. 
This study has only included seven European countries; 
consequently, it would be interesting to explore the ways 
in which HIS is understood, practised and taught across 
Europe as a whole.
6 CONCLUSION
A new consensus definition of HIS was developed in a 
European context. The consensus method approach was 
pertinent to unify the new concept from multidisciplinary, 
interprofessional and multicultural perspectives. Having a 
shared definition of HIS is important to boost healthcare 
culture and professional training processes. The definition 
enables higher education institutions in health and social 
care to address this gap in their curricula, and ensure that 
improvement science becomes a core competency for all 
healthcare graduates.
ACKNOWLEDGEMENTS
This article was written on behalf of the Improvement 
Science Training for European Healthcare Workers 
(ISTEW) project team. We would like to acknowledge the 
contributions of Professor Julita Sansoni, Dr Elisabetta 
Corvo, Dr Alan Taylor, Dr Katrina Ritters, Magdalena 
Glowacka, Dr Traian Mihaescu, Dr Radu Crisan, Dr Maria 
Josefa Cabanero Martinez, Barbara O’Donnell, Dr Joca 
Zurc and Miram Sanchez San Segundo.
CONFLICTS OF INTEREST
The authors report no conflicts of interest.
FUNDING 
The research was financed by the European Union funded 
ERASMUS Lifelong Learning Project, ISTEW: Improvement 
Science Training for European Healthcare Workers (Project 
No. 539194-LLP-1-2013-1-UK-ERASMUS-EQR). 
ETHICAL APPROVAL
The research protocol was approved by the Senate 
Committee for Science, Research and Development at 
the Faculty of Health Care Jesenice (FHCJ) in December 
2013. The serial number of Ethics Approval is 12/5 in 
the academic year 2013/2014. Partners also adhered to 
their ethical approval processes for research. Participants 
received written information about different aspects of 
the study; explained to them were their rights on voluntary 
participation and withdrawal from the study at any time, 
as well as their privacy and confidentiality rights. 
REFERENCES
1. Wandersman A, Alia KA, Cook B, Ramaswamy R. Integrating 
empowerment evaluation and quality improvement to achieve 
healthcare improvement outcomes. BMJ Qual Saf 2015; 24: 645–52. 
2. Auerbach AD, Landefeld CS, Shojania KG. The tension between 
needing to improve care and knowing how to do it. N Engl J Med 
2007; 357: 608–13.
3. Marshall M, Pronovost P, Dixon-Woods M. Promotion of improvement 
as a science.  Lancet 2013; 381: 419–21.    
4. Batalden PB, Davidoff F. What is ‘‘quality improvement’’ and how can 
it transform healthcare? Qual Saf Health Care 2007; 16: 2–3. 
5. Jones A, Williams A, Carson-Stevens A. Integrating quality 
improvement into pre-registration education. Nurs Stand 2013; 27: 
44-8.
6. Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An 
introduction to implementation science for the non-specialist. BMC 
Psychol 2015; 16: 32. 
7. McDonald KM, Schultz EM, Chang C. Evaluating the state of quality-
improvement science through evidence synthesis: insights from the 
closing the quality gap series. Perm J 2013; 17: 52-61. 
8. The Health Foundation, Improvement Science Fellowships. 
London: The Health Foundation, 2011. Available October 15, 2013 
from: http://www.renal.org/docs/default-source/what-we do/
Improvement_Science_Fellowships.pdf?sfvrsn=0 
9. Helfrich CD, Damschroder LJ, Hagedorn HJ, Daggett GS, Sahay A, 
Ritchie M, et al. A critical synthesis of literature on the promoting 
action on research implementation in health services (PARIHS) 
framework. Implement Sci 2010; 5: 82-101.
10. Conry MC, Humphries N, Morgan K, McGowan Y, Montgomery A, 
Vedhara K, Panagopoulou E, Mc Gee H. A 10 year (2000–2010) 
systematic review of interventions to improve quality of care in 
hospitals. BMC Health Serv Res 2012; 12: 1-16.
11. Øvretveit J, Leviton L, Parry G. Increasing the generalisability of 
improvement research with an improvement replication programme. 
BMJ Qual Saf 2011; 20: i87-i91. 
12. Leviton L. Reconciling complexity and classification in quality 
improvement research. BMJ Qual Saf 2011; 20: i28-29. 
13. Hovlid E, Bukve O, Haug K, Aslaksen AB, von Plessen C. Sustainability 
of healthcare improvement: what can we learn from learning theory? 
BMC Health Serv Res 2012; 12: 235. 
14. Dominguez DG, Fike DS, MacLaughlin EJ, Zorek JA. A comparison of 
the validity of two instruments assessing health professional student 
perceptions of interprofessional education and practice. J Interprof 
Care 2015; 29: 144–9.
15. Estrada CA, Dolansky MA, Singh MK, Oliver BJ, Callaway-Lane C, 
Splaine M, et al. Mastering improvement science skills in the new era 
of quality and safety: the Veterans Affairs National Quality Scholars 
Program. J Eval Clin Pract 2012; 18: 508–14. 
10.1515/sjph-2017-0011 Zdr Varst 2017; 56(2): 82-90
90
16. Jones J, Hunter D. Qualitative research: consensus methods for 
medical and health service research. BMJ 1995; 311: 376–80.
17. Minkman M, Ahans K, Fabbricotti I, Nabitz U, Huijsman R. A quality 
management model for integrated care: results of a Delphi and 
Concept Mapping study. Int J Qual Health Care 2009; 21: 66–75. 
18. List D. The Consensus group technique: a users’ manual. Wellington: 
Original Books, 2005.  
19. Race KE, Hotch DF. Rehabilitation program evaluation: use of focus 
groups to empower clients. Eval Rev 1994; 18: 730-40.
20. The Scottish Government. The healthcare quality strategy for NHS 
Scotland. Edinburg: The Scottish Government, 2010.
21. Booth BJ, Zwar N, Harris MF. Healthcare improvement as planned 
system change or complex responsive processes: a longitudinal case 
study in general practice. BMC Fam Pract 2013; 14: 51. 
22. Robson W. Eliminating avoidable harm: time for patient safety to 
play a bigger part in professional education and practice. Nurse Educ 
Today 2014; 34: e1–2. 
23. Hall WL, Zierler BK. Interprofessional education and practice guide 
No. 1: developing faculty to effectively facilitate interprofessional 
education. J Interprof Care 2015; 29: 3–7.