PATIENT REPORTED INDICATOR SURVEYS (PARIS): METHODOLOGICAL 
CONSIDERATIONS OF A FIELD TRIAL IN SLOVENIA
Matija AMBROŽ
 1*  
  , Candan KENDIR
 2,3
   , Wienke BOERMA
 4
   , Zalika KLEMENC-KETIŠ
 1,5,6
 
1 
Community Health Centre Ljubljana, Metelkova 9, 1000 Ljubljana, Slovenia
2 
Organisation for Economic Co-operation and Development (OECD), 2, rue André Pascal 75016 Paris, France
3 
Amsterdam UMC location Vrije Universiteit Amsterdam, Department of 
Public and Occupational Health, Amsterdam, Netherlands
4 
NIVEL Netherlands Institute for Health Services Research, Otterstraat 118, 3513 CR Utrecht, The Netherlands
5 
University of Maribor, Faculty of Medicine, Department of Family Medicine, Taborska 8, 2000 Maribor, Slovenia
6 
University of Ljubljana, Faculty of Medicine, Department of Family Medicine, Poljanski nasip 58, 1000 Ljubljana, Slovenia
Received: Oct 11, 2023
Accepted: Nov 27, 2023
Original scientific article
*Correspondence: matija.ambroz@zd-lj.si
10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
30
OCENA IZKUŠENJ IN IZIDOV ZDRAVSTVENE OSKRBE PACIENTOV 
S KRONIČNIMI NENALEZLJIVIMI BOLEZNIMI: METODOLOŠKI 
VIDIK PILOTNE IZVEDBE RAZISKAVE V SLOVENIJI
© National Institute of Public Health, Slovenia. 
Ambrož M, Kendir C, Boerma W, Klemenc-Ketiš Z. Patient reported indicator surveys (PaRIS): Methodological considerations of a field trial in Slovenia. 
Zdr Varst. 2024;63(1):30-37. doi: 10.2478/Sjph-2024-0005.
ABSTRACT
Keywords: 
Patient Reported 
Indicator Surveys (PaRIS)
Primary care
Quality of healthcare
Patient experience
Data collection 
methodology
IZVLEČEK
Ključne besede: 
ocena izkušenj in 
izidov zdravstvene 
oskrbe
primarno zdravstvo
kakovost zdravstvene 
oskrbe
pacientove izkušnje
metodologije zbiranja 
podatkov
Introduction: Healthcare systems collect little information about the experiences and outcomes of care from 
the perspectives of patients. Patient Reported Indicator Surveys (PaRIS) is an OECD initiative to measure the 
outcomes and experiences of people living with chronic conditions, who are managed in primary care. 
Objectives: To evaluate the feasibility of the methodology employed in the Field Trial of the PaRIS survey in 
Slovenia and propose adjustments to enhance sampling in the Main Survey.
Methods: In 2022, we conducted a cross-sectional observational study in 50 family medicine practices in 
Slovenia with a target of recruiting 70 patients per practice. We used the Slovenian version of the PaRIS 
questionnaires, and evaluated sampling and data collection.
Results: The sample contained 21 providers (42.0% response rate) and 454 patients (50.7% response rate). 
The provider sample did not differ from the population characteristics, while the patient sample differed 
significantly from the patient population. All providers completed the survey online, in 20.9±11.1 minutes and 
had 1.5±1.5 restarts. Most patients (74.9%) completed the survey online and needed 36.0±22.6 minutes, and the 
mean number of restarts was 1.4±2.2.
Conclusion: Based on the results, we recommend conducting a methodology test for quality assessment studies 
before initiating the main survey. Legal issues should be addressed and considered early when developing 
the methodology. It is also necessary to be aware of the feasibility of the study in practice, to avoid a low 
participation rate.
Uvod: Zdravstveni sistemi zbirajo malo podatkov o izkušnjah in izidih oskrbe z zornega kota pacientov. Ocena 
izkušenj in izidov zdravstvene oskrbe pacientov s kroničnimi nenalezljivimi boleznimi (projekt PaRIS) je pobuda 
OECD za merjenje izidov in izkušenj ljudi s kroničnimi boleznimi, ki so obravnavani na primarnem zdravstvenem 
nivoju.
Cilj je oceniti izvedljivost in ustreznost uporabljene metodologije pilotne izvedbe raziskave PaRIS v Sloveniji in 
predlagati izboljšave vzorčenja in zbiranja podatkov v glavni raziskavi.
Metode: Leta 2022 smo v Sloveniji izvedli presečno opazovalno študijo v 50 ambulantah družinske medicine, s 
ciljem povabiti k sodelovanju po 70 pacientov iz vsake ambulante. Uporabili smo slovensko različico vprašalnikov 
PaRIS. Ocenili smo vzorčenje in zbiranje podatkov.
Rezultati: Vzorec je zajel 21 izvajalcev zdravstvenih storitev (42,0-% odzivnost) in 454 pacientov (50,7-% 
odzivnost). Vzorec izvajalcev zdravstvenih storitev se ni razlikoval od značilnosti populacije, medtem ko se 
je vzorec pacientov bistveno razlikoval od populacije pacientov. Vsi izvajalci zdravstvenih storitev so anketo 
izpolnili prek spleta v 20,9 ± 11,1 minute pri čemer so imeli povprečno 1,5 ± 1,5 ponovnih zagonov. Večina 
pacientov (74,9 %) je anketo izpolnila prek spleta. Čas izpolnjevanja spletne ankete je bil 36,0 ± 22,6 minute, 
povprečno število ponovnih zagonov pa 1,4 ± 2,2.
Zaključki: Na podlagi rezultatov lahko predlagamo, da se pri študijah ocenjevanja kakovosti, test metodologije 
izvede pred izvedbo glavne raziskave. Pravne prepreke in značilnosti je treba nasloviti in upoštevati zgodaj 
v razvoju metodologije. Prav tako je treba biti pozoren na izvedljivost študije v praksi, da se prepreči nizka 
stopnja sodelovanja.
This article was presented at the 2nd ISCPC conference, which took place in Cankarjev dom, Ljubljana, Slovenia, on 23 and 24 November, 2023. The conference was organised by the Community Health Centre 
Ljubljana and Medical Faculty, University of Ljubljana, Slovenia.

1 INTRODUCTION
Healthcare systems that place greater emphasis on the 
quality of primary care have better clinical outcomes and 
are more economically efficient and sustainable [1–5]. The 
patient’s experience with the healthcare system is being 
recognised as an important dimension of the quality of 
healthcare. It is thus an increasingly common strategy to 
monitor the quality of healthcare at all levels [6–9]. 
Tools that assess patient experience enable patients to 
provide feedback and potentially strengthen a patient-
centred health system [10].
A standardised data collection procedure is preferred 
in international surveys. Due to a wide variety of legal, 
ethical and cultural differences in primary care, a 
data collection strategy is adapted for each country in 
agreement with the consortium [11]. To date the largest 
international research on the quality of primary care 
(Quality and Costs of Primary Care in Europe (QUALICOPC)) 
was carried out in 31 countries between 2010 and 2013, 
and it included assessment of the patients’ experiences 
[12]. This study was crucial in developing the methodology 
for future large-scale international studies such as the 
Patient Reported Indicator Surveys (PaRIS). 
In 2017, the OECD launched the PaRIS initiative to 
systematically measure the outcomes and experiences of 
healthcare that matter most to people [13]. The flagship 
project within the PaRIS initiative is a survey of people 
living with chronic conditions who are managed in primary 
care, namely the PaRIS survey [11]. The PaRIS survey has 
seen three different phases: 1) design and development 
phase (2017-2020), including the study design and the 
development of PaRIS questionnaires; 2) field trial (2021-
2022), including the testing of the PaRIS survey design and 
PaRIS questionnaires in participating countries; and 3) 
main survey (2022-2023), including the full implementation 
of the PaRIS survey in participating countries and data 
collection [14]. 
In this paper, we aimed to evaluate the feasibility of the 
methodology employed in the field trial of the PaRIS 
survey in Slovenia and propose adjustments to enhance 
sampling in the main survey. While the manuscript does 
not present content-related field trial results, it provides 
a comprehensive assessment of the survey’s methodology. 
2 METHODS
2.1 Design of the study
The PaRIS survey is a cross-sectional observational study, 
and the study protocol is described in detail elsewhere 
[11]. The study presented here took place in Slovenian 
family medicine practices. Family medicine physicians 
work either as private contractors or in primary care 
10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
31
centres, and both are part of national public primary care 
system. One family medicine team consists of 1 full time 
equivalent (FTE) family physician, 1 FTE practice nurse 
and 0.5 FTE registered nurse. Practice nurses are mainly 
involved in tasks including administration, appointments, 
and clinical work, while registered nurses work as nurse 
practitioners and carry out preventive activities and care 
for stable chronic patients [15, 16]. 
2.2 Study population
2.2.1 Providers
We defined providers as primary care practices in line 
with the study design of the PaRIS survey. In Slovenia one 
family physician works in one family medicine practice, 
and the registry of family medicine practices is available 
online, with the system led by the Health Insurance 
Institute of Slovenia (ZZZS). We used this registry for 
random sampling, by assigning consecutive numbers to 
every physician in the registry and then using a random 
number generator to select them. The registry contains 
all family physicians working primarily with registered 
patients. Physicians that provide healthcare only to 
institutionalised patients or children, or who practice 
occupational healthcare services as their primary area 
of work, were excluded. There were 978 eligible family 
medicine physicians in Slovenia in February 2022. 
As this was a field trial, we aimed for 25 family medicine 
practices in a final sample. Based on previous studies, 
we estimated at least a 50% positive response rate from 
the contacted family physicians, and thus to achieve the 
target we invited 50 practices to participate.
 
2.2.2 Patients 
All patients aged 45 years old or more, living in the 
community and having had any contact (including via phone 
or email) with the practice in the six months preceding 
the study, were eligible to participate. People living in 
a long-term care facility, healthcare or other residential 
institution were excluded, as were those in the terminal 
phase of illness or with an advanced cognitive impairment.
Sampling of patients was random, and we aimed at 70 
patients per practice. Practices autonomously generated 
a list of eligible patients, utilising either a random number 
generator or systematically selecting every Nth patient. 
Each eligible patient was first contacted by the practice 
and given the necessary information about the study, and 
could then provide informed consent and contact data if 
they wanted to participate. After that, the information 
gathered by the practice was sent to the researchers. 

10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
32
2.3 Data collection
To collect data, we used the PaRIS provider and patient 
questionnaires. 
2.3.1 Provider questionnaire
The provider questionnaire includes 40 questions and 
defines the structure and characteristics of the team 
providing primary care and of the provider responsible 
– the family physician. The questionnaire was developed 
by the OECD and an international consortium (PaRIS-SUR 
consortium) in collaboration with patients and provider 
organisations, including the World Organization of Family 
Doctors (WONCA), and other stakeholders [11]. The 
questionnaire covers domains related to care delivery such 
as urbanisation, practice type, care model, information 
and administration systems, remote consultations, skill-
mix, and remuneration. The estimated time for completion 
is 10-15 minutes.
2.3.2 Patient questionnaire
The patient questionnaire includes 121 questions 
addressing individual and sociodemographic factors, 
health and healthcare capabilities, health behaviours and 
lifestyle, Patient Reported Outcome Measures (PROMs) 
such as symptoms and functioning, and Patient Reported 
Experience Measures (PREMs) such as coordination of care 
and safety. The PaRIS patient questionnaire is based on 
the PaRIS conceptual framework and consists of validated 
and established tools, already described elsewhere [11]. 
The estimated time to complete the patient questionnaire 
is 25 minutes. 
2.3.3 Translation and cognitive testing
The PaRIS questionnaires has two source questionnaires in 
English and French. The base for developing the Slovenian 
questionnaires was the original English versions developed 
and cognitively tested by the PaRIS consortium. We first 
translated the original English questionnaires into Slovene 
using the TRAP-D approach, as recommended by the 
guidelines [17]. The resulting questionnaires were then 
cognitively tested using the four-stage model of cognitive 
testing developed by Tourangeau [18]. 
Cognitive testing was only performed on questions that 
were identified as necessary for testing by the consortium. 
Changes were incorporated in the final version of the field 
trial questionnaires. 
2.3.4 Data collection process and contact strategy
After acquiring the sample of providers those who 
were selected were contacted via post in which they 
received information about the study and an invitation 
to participate. Within a week after sending the invitation 
letter, all providers were contacted again via phone by the 
research team. Those who agreed to participate received 
a link to the survey for providers and a spreadsheet to 
sample their patients. The practitioners had a month to 
sample 70 eligible patients. They were reminded weekly 
by the research team via email to complete the survey 
and provide the completed spreadsheet (Figure 1).
Symbolic incentives were used to motivate physicians to 
participate, such as one-time free registration to national 
congresses of primary care and feedback information for 
their team.
The patients in the sample were contacted via their 
preferred method. All patients who provided an email 
address received the survey link via email. Those who 
did not provide an email address received the survey 
link via SMS. Those who did not provide either an email 
or telephone number, or who opted for a paper survey, 
received this via the regular mail system. Non-respondents 
who provided an email address received a reminder via 
email for two consecutive weeks after receiving the link 
to the survey, and those who did not give an email address 
received two reminders via regular mail.
2.4 Evaluation of results
For the quality check of the samples, we identified the 
characteristics of the provider and patient population and 
performed chi-square and independent samples t-test. For 
the providers, we used the data from the aforementioned 
list of practices. We performed the sample check according 
to geographic distribution (10 health regions) and status of 
the providers (public or private with a concession). For 
the patients, we used the data from the participating 
practices with regard to the age and gender of their 
registered patients who were age 45 years or older.
For the evaluation of data collection, we used the 
following variables: 
Mode of completion, participation time for the e-survey, 
number of restarts with the e-survey, method of access to 
the e-survey, return code (partial/complete), and type of 
device used to complete the e-survey. The appropriateness 
and feasibility of the methodology were assessed using 
predefined criteria, including achieving at least 80% of the 
target sample size and a response rate exceeding 30%.
The research team systematically documented reasons for 
non-participation by directly querying both providers and 
patients. 
3 RESULTS
3.1 Response rate
On the providers’ side, 21 family medicine practices of the 
50 sampled practices participated in the study (42.0%). 

10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
33
The family medicine practices managed to recruit 896 
patients, out of which 454 responded to the patient 
questionnaire (50.7%) (Figure 1). 
3.1.1 Characteristics of the participants 
Most providers that completed the questionnaire were 
family physicians, working in the public sector, located in 
a city and in a solo practice (Table 1).
Most patients were female, 60 years of age and older, with 
secondary education, retired and living in a rural area 
(Table 2).
Figure 1. Schematic presentation of the process of recruiting of 
family physicians and patients.
Characteristics of the providers.
Characteristics of the patients.
Professional background
Family physician
Family medicine trainee
Type of practice
Public (in a primary care centre)
Private with a concession
No answer
Practice location
City
Town or suburb
Rural area
Work type
Solo practice
Group practice with own patients
Group practice with shared patients
Sex
Male
Female
I prefer not to say
Not answered
Age group
44 years old or younger
45-49 years old
50-54 years old
55-59 years old
60-64 years old
65-69 years old
70-74 years old
75-79 years old
80-84 years old
85 years or older
Not answered
Education
Primary school not completed
Primary school
Vocational school
Secondary school
University education
Master’s degree or doctorate
Not answered
Employment status
Self-employed
In paid employment
Looking for work
Looking after the home
Unable to work due to sickness or ill-health
Retired
Other
Not answered
Living area
City
Town or suburb
Rural area
Don’t know
Not answered
18
3
10
3
8
9
5
7
16
4
1
170
249
2
33
2
32
45
69
74
95
61
27
13
4
32
9
52
123
230
19
54
35
14
150
4
7
16
227
2
34
105
69
245
1
34
85.7
14.3
47.6
14.3
38.1
42.9
23.8
33.3
76.2
19.0
4.8
37.4
54.8
0.4
7.3
0.4
7.0
9.9
15.2
16.3
20.9
13.4
5.9
2.9
0.9
7.0
2.0
11.5
27.1
50.7
4.2
11.9
7.7
3.1
33.0
0.9
1.5
3.5
50.0
0.4
7.5
23.1
15.2
54.0
0.2
7.5
N
N
%
%
Characteristic
Characteristic
Table 1.
Table 2.
3.2 Sampling evaluation
The size of the provider population was 978 and the size of the 
sample was 50. There was no significant difference between 
the sample and the population regarding geographical 
distribution and status of the practice (Table 3). 

10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
34
Provider quality sample check.
Patient quality sample check.
Geographical distribution
Region 1
Region 2
Region 3
Region 4
Region 5
Region 6
Region 7
Region 8
Region 9
Region 10
Status
Public 
Private with a concession
Gender
Male
Female
Age group
45-49 years old
50-54 years old
55-59 years old
60-64 years old
65-69 years old
70-74 years old
75-79 years old
80-84 years old
85 years or older
5 vs. 100
4 vs.74
2 vs. 33
5 vs. 100
15 vs. 303
7 vs. 142
3 vs. 55
3 vs. 53
3 vs. 56
3 vs. 62
36 vs. 710
14 vs. 268
170 vs. 84944
249 vs. 95749
32 vs. 19672
45 vs. 11343
69 vs. 15712
74 vs. 14526
95 vs. 14544
61 vs. 12801
27 vs. 16070
13 vs. 8791
4 vs. 7378
9
1
1
8
0.151
0.009
6.954
98.120
1.000
0.926
0.008
<0.001
Number (sample vs. population)
Number (sample vs. population)
Degrees of freedom
Degrees of freedom
Pearson chi-square
Pearson chi-square
p value
p value
Characteristic
Characteristic
Table 3.
Table 4.
The sample and population of patients differed significantly 
regarding gender and age (Table 4).
3.3 Data collection evaluation
All the providers completed the survey online. They 
mostly used a desktop computer and accessed the survey 
via a link sent by email (Table 5). The providers completed 
the e-survey in 20.9±11.1 minutes and had 1.5±1.5 restarts.
Most patients completed the survey online (e-survey), 
others used a paper-based questionnaire. They mostly 
accessed the e-survey via a link sent by email and used a 
smartphone to do so. Only a small percentage of patients 
left the e-survey incomplete (Table 5). The completion 
time was 36.0±22.6 minutes and the mean number of 
restarts was 1.4±2.2.
Data collection evaluation – providers and patients.
Mode of completion
Online
Paper
Method of access to e-survey
Link via email
Link via SMS
Device used for e-survey
Desktop
Smartphone 
Tablet
Unknown
Return code
Complete
Partial (incomplete e-survey)
21 (100)
0
21 (100)
0
18 (85.7)
2 (9.5)
1 (4.8)
0
21 (100)
0
340 (74.9)
114 (25.1)
199 (58.5)
141 (41.5)
127 (28.0)
195 (43.0)
17 (3.7)
1 (0.2)
420 (92.5)
34 (7.5)
Providers 
N (%)
Patients 
N (%)
Characteristic
Table 5.

10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
35
3.4 Reasons for non-participation
Some of the providers refused to participate in the study 
due to the workload expected from them regarding the 
sampling and data collection. They stated that they were 
overburden with their usual work and did not have time to 
engage in the survey. 
Some of the patients refused to participate due to low 
interest in the survey, the questionnaire being seen as too 
long for them, and not having enough time to complete it.  
4 DISCUSSON
4.1 Main findings
The evaluation of the PaRIS survey field trial in Slovenia 
showed satisfactory results regarding the sampling of 
providers and data collection among them. However, the 
sampling of the patients and data collection among them 
did not yield the desired evaluation outcomes, the main 
problem being the small sample size.
The response rate of the practices was satisfactory. 
The literature shows that the response rate can vary 
significantly (from 20-80%) [19–21], although there are 
several methods that can be used to improve it [14, 20]. 
We used personal contact and reminders, which seemed 
to be effective.
The survey was mostly completed by family physicians. 
This was expected, as the physician is a team leader in 
primary care practices in Slovenia, and access to the 
patient list through the physician was required for further 
sampling. Regarding the work type of the practice, most 
providers were in a solo practice. Here, it should be 
noted that in Slovenia around 70% of the family medicine 
practices are in community health centres, and 30% work 
as private practices with a concession [22]. 
Family physicians in health centres are practicing in their 
own practice with their own registered patients, working 
with a team of their own nurses. However, a health 
centre composed of a family physician, practice nurse and 
registered nurse is not considered as a group practice in 
Slovenia. A group practice in Slovenia is described as a 
practice where two or more physicians are working in the 
same practice with other healthcare professionals [23]. 
Hence, what is considered to be a group practice in the 
international PaRIS survey was recorded as solo practice 
in Slovenia.
The sample size of the patients was much smaller than 
anticipated, which is probably a consequence of how the 
GDPR is interpreted in Slovenia. With this, practices are not 
allowed to send the contact information of their patients 
to researchers without the patients’ consent. This forced 
the researchers to adapt the international methodology, 
and to ask the practices to both sample the patients and 
acquire their informed consent. As expected, this placed 
a significant additional burden on the practices, which 
then failed to provide a sufficient patient sample size.
All the providers accessed the questionnaire online, and 
in the literature no significant differences were found 
regarding the mode of survey [19, 21]. The survey was 
completed by the providers in 20 minutes, whereas we 
anticipated 15 minutes would be needed. However, we 
still consider this feasible.
A quarter of the patients completed a paper questionnaire. 
Others completed an online survey via link sent by email 
or SMS. A meta-analysis showed that the response rate 
to online surveys is approximately 40% [23], and this 
is consistent with our results. On the other hand, the 
literature shows that using a paper questionnaire does 
not yield a lower response rate [24]. This might indicate 
that paper questionnaires can now be dropped, but this 
would be a mistake as some respondents still prefer paper 
versions [24]. Patients on average needed 36 minutes 
to complete the online survey, while there is no data 
on the time needed for the paper questionnaire. We 
expected that the online survey should be completed 
in 25 minutes, which turned out to be rather optimistic. 
The literature shows that questionnaires should include 
25 to 30 questions, and be possible to complete in 30 
minutes [24, 25]. In the case of longer questionnaires, 
we can expect a larger percentage of non-responders, 
a larger amount of missing data, and an increase in 
providing answers automatically with the last items [24, 
25]. In our study, only a small number of patients left the 
questionnaire incomplete, which indicates that although 
the questionnaire was long, it was still manageable.
4.2 Strengths and limitations
One limitation of the study was that we could not evaluate 
the same indicators on the paper-based survey. It is thus 
possible that we missed some lessons for the main survey. 
Another limitation is the missing characteristics of the 
population and sample, which would have helped with 
regard to detecting additional significant differences. 
Moreover, the patient sample did not reflect the 
characteristics of the population, and there was also a 
problem with sampling according to inclusion criteria, as 
two patients were younger than 45 years old, although 
this could be due to an error when completing the 
questionnaire. Therefore, we cannot assume that the 
patient sample was representative, and one solution might 
be to omit this age category within the questionnaire.

4.3 Implications
We evaluated the sampling method of the practices as 
satisfactory, so no adjustments will be made for the 
main survey. The same is true for data collection of the 
practices. 
For patient sampling, the evaluation indicated the need for 
revision of the methodology to achieve a better response 
rate. We are thus planning to provide the practices with 
printed survey invitations for patients containing a QR 
code with the link to the online questionnaire. This seems 
to be a good method of data collection [26]. We also plan 
to provide the practices with paper questionnaires. The 
practices will draw up a patient sample as in the field trial, 
but will not need to obtain the patients’ informed consent 
for sending the contact information. Instead, the practices 
will mail either the QR code or a paper questionnaire 
to the patients in the sample. With such methodology, 
we expect a lower burden on the practices and higher 
response rate. On the other hand, such an approach will 
make it impossible to only send the reminders to the non-
respondents.
Shortening the patient questionnaire might also contribute 
to better response rates. 
5 CONCLUSION
This study provides insights into the methodological 
challenges when conducting a cross-sectional 
observational study on patient outcomes and experiences 
in Slovenian primary care. The results suggest a need for 
pre-testing the methodology of quality assessment studies 
before the main survey, enabling researchers to refine 
their approaches for optimal results. Early consideration 
of legal issues in methodology development is essential. 
Additionally, awareness of study feasibility is crucial to 
mitigate low participation rates.
ACKNOWLEDGEMENTS
The PaRIS survey is an OECD initiative. The PaRIS-SUR 
Consortium, consisting of Nivel, Exeter University, 
Ipsos, Avis Donabedian Research Institute, and 
Optimedis assisted the OECD in the development and 
implementation. The authors thank the Working Party 
for PaRIS, the international PaRIS Patient Panel and the 
Technical Advisory Community and various experts for 
their roles throughout the study. The authors also thank 
the national project management teams, participating 
primary care professionals and patients. PaRIS is funded 
by the participating countries and receives a contribution 
from the European Commission.
The views expressed and arguments employed herein are 
solely those of the author(s) and do not necessarily reflect 
the views of the OECD or its member countries. The 
OECD cannot be held responsible for possible violations 
of copyright resulting from the posting of any written 
material on this website.
FUNDING
The authors acknowledge that the project (Assessment of 
experiences and outcomes of healthcare for patients with 
chronic non-communicable diseases in family medicine 
in Slovenia, ID V3-2102) was financially supported by 
the Slovenian Research Agency. PaRIS is funded by the 
participating countries and receives a contribution from 
the European Commission.
ETHICAL APPROVAL
The study in Slovenia was approved by the National Ethics 
Committee (No. 0120-260/2021/3).
CONFLICTS OF INTEREST
The authors report no conflicts of interest. The authors 
alone are responsible for the content and writing of the 
paper.
AVAILABILITY OF DATA AND MATERIALS 
The data that support the findings of this study are 
available from OECD. Restrictions apply to the availability 
of these data, which were used under license for this 
study. Data are available from the authors with the 
permission of OECD.
ORCID
Matija Ambrož: 
https:/ /orcid.org/0009-0008-3406-5693
Candan Kendir: 
https:/ /orcid.org/0000-0001-6877-4836
Wienke Boerma: 
https:/ /orcid.org/0000-0001-7698-0015
Zalika Klemenc Ketiš: 
https:/ /orcid.org/0000-0002-0270-1754
10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
36

REFERENCES
1. Starfield B, Shi L, Macinko J. Contribution of primary care to health 
systems and health. Milbank Q. 2005;83:457–502. doi: 10.1111/j.1468-
0009.2005.00409.x.
2. Macinko J, Starfield B, Shi L. The contribution of primary care systems 
to health outcomes within Organization for Economic Cooperation 
and Development (OECD) countries, 1970-1998. Health Serv Res. 
2003;38:831–865. doi: 10.1111/1475-6773.00149.
3. Kringos D, Boerma W, Bourgueil Y, Cartier T, Dedeu T, Hasvold T, et al. 
The strength of primary care in Europe: An international comparative 
study. Br J Gen Pract. 2013;63:e742-e750. doi: 10.3399/bjgp13X674422.
4. Klemenc Ketiš Z, Rochfort A. Sustainability for planetary health: A 
seventh domain of quality in primary care. Zdr Varst. 2022;61:198–200. 
doi: 10.2478/sjph-2022-0026. 
5. Virtič T, Mihevc M, Zavrnik Č, Mori Lukančič M, Poplas Susič A, 
Klemenc-Ketiš Z. Peer support as part of scaling-up integrated care in 
patients with type 2 diabetes and arterial hypertension at the primary 
healthcare level: A study protocol. Zdr Varst. 2023;62:93–100. doi: 
10.2478/sjph-2023-0013
6. 6Gleeson H, Calderon A, Swami V, Deighton J, Wolpert M, Edbrooke-
Childs J. Systematic review of approaches to using patient experience 
data for quality improvement in healthcare settings. BMJ Open. 
2016;6:e011907. doi: 10.1136/bmjopen-2016-011907.
7. Doyle C, Lennox L, Bell D. A systematic review of evidence on the links 
between patient experience and clinical safety and effectiveness. 
BMJ Open. 2013;3:e001570. doi: 10.1136/bmjopen-2012-001570.
8. de Boer D, Delnoij D, Rademakers J. Do patient experiences on 
priority aspects of health care predict their global rating of quality of 
care? A study in five patient groups. Health Expect. 2010;13:285–297. 
doi: 10.1111/j.1369-7625.2010.00591.x.
9. Prevolnik Rupel V, Došenović Bonča P. Is value-based health care just 
the latest fad or can it transform the Slovenian health care system? 
Zdr Varst. 2023;62:55–58. doi: 10.2478/sjph-2023-0008
10. Holt JM. Patient experience in primary care: A systematic 
review of CG-CAHPS surveys. J Patient Exp. 2019;6:93–102. doi: 
10.1177/2374373518793143.
11. de Boer D, van den Berg M, Ballester M, Bloemeke J, Boerma W, de 
Bienassis K, et al. Assessing the outcomes and experiences of care from 
the perspective of people living with chronic conditions, to support 
countries in developing people-centred policies and practices: Study 
protocol of the International Survey of People Living with Chronic 
Conditions (PaRIS survey). BMJ Open. 2022;12:e061424. doi: 10.1136/
bmjopen-2022-061424.
12. Schäfer WLA, Boerma WGW, Kringos DS, De Maeseneer J, Gress S, 
Heinemann S, et al. QUALICOPC, a multi-country study evaluating 
quality, costs and equity in primary care. BMC Fam Pract. 2011;12:115. 
doi: 10.1186/1471-2296-12-115.
13. Recommendations to OECD ministers of health from the high level 
reflection group on the future of health statistics: Strengthening 
the international comparison of health system performance 
through patient-reported indicators [Internet]. Paris: OECD; 2017 
[cited 2023 Aug 25]. Available from: https://www.oecd.org/health/
Recommendations-from-high-level-reflection-group-on-the-future-of-
health-statistics.pdf
14. 14. Kendir C, Naik R, Bloemeke J, Bienassis K de, Larrain N, Klazinga 
N, et al. All hands on deck: Co-developing the first international 
survey of people living with chronic conditions  [Internet]. 2023 [cited 
2023 Aug 25]. Available from: https://www.oecd-ilibrary.org/content/
paper/8b31022e-en
15. Poplas Susic A, Svab I, Klemenc Ketis Z. Upgrading the model of care 
in family medicine: A Slovenian example. Public Health Panorama. 
2018:550–555. 
16. Susič AP, Klemenc-Ketiš Z. Successful implementation of integrated 
care in Slovenian primary care. Zdr Varst. 2021;60:1–3. doi: 10.2478/
sjph-2021-0001.
17. Mohler P, Dorer B, De Jong J, Hu M. Guidelines for best practice 
in cross-cultural surveys. Ann Arbor, MI: Survey Research Center, 
Institute for Social Research, University of Michigan; 2016. 
18. Collins D. Cognitive interviewing practice. London; 2023. doi: 
10.4135/9781473910102
19. Taylor T, Scott A. Do physicians prefer to complete online or mail 
surveys? Findings from a national longitudinal survey. Eval Health Prof. 
2019;42:41–70. doi: 10.1177/0163278718807744.
20. Brtnikova M, Crane LA, Allison MA, Hurley LP, Beaty BL, Kempe A. A 
method for achieving high response rates in national surveys of U.S. 
primary care physicians. PLoS One. 2018;13:e0202755. doi: 10.1371/
journal.pone.0202755.
21. Weaver L, Beebe TJ, Rockwood T. The impact of survey mode on 
the response rate in a survey of the factors that influence Minnesota 
physicians’ disclosure practices. BMC Med Res Methodol. 2019;19:73. 
doi: 10.1186/s12874-019-0719-7.
22. Klančar D, Švab I. Primary care principles and community health 
centers in the countries of former Yugoslavia. Health Policy. 
2014;118:166–172. doi:  10.1016/j.healthpol.2014.08.014.
23. Muhlestein DB, Smith NJ. Physician consolidation: Rapid movement 
from small to large group practices, 2013–15. Health Aff (Millwood). 
2016;35:1638–1642. doi: 10.1377/hlthaff.2016.0130.
24. Horevoorts NJ, Vissers PA, Mols F, Thong MS, van de Poll-Franse LV. 
Response rates for patient-reported outcomes using web-based versus 
paper questionnaires: Comparison of two invitational methods in 
older colorectal cancer patients. J Med Internet Res. 2015;17:e111. 
doi: 10.2196/jmir.3741.
25. Sahlqvist S, Song Y, Bull F, Adams E, Preston J, Ogilvie D, et al. Effect of 
questionnaire length, personalisation and reminder type on response 
rate to a complex postal survey: Randomised controlled trial. BMC 
Med Res Methodol. 2011;11:62. doi: 10.1186/1471-2288-11-62.
26. Azad A, Laidlaw DAH, Orlans HO. Using QR smartphone technology 
to improve patient communication and information distribution. Eye. 
2022;36:1321–1322. doi: 10.1038/s41433-021-01757-x.
10.2478/sjph-2024-0005 Zdr Varst. 2024;63(1):30-37
37