Klinična študija / Clinical study
Psihometrične lastnosti slovenske različice lestvice MOSOPCS za ocenjevanje kulture varnosti
The psychometric properties of the Slovenian version of the Medical Office Survey on Patient Safety Culture
Avtor / Author	Zalika Klemenc-Ketiš1,2,3, Irena Makivic1, Antonija Poplas-Susič1,3
Ustanova / Institute	"'Zdravstveni dom Ljubljana, Ljubljana, Slovenija; 2Univerza v Mariboru, Medicinska fakulteta,
Katedra za družinsko medicino, Maribor, Slovenija; 3Univerza v Ljubljani, Medicinska fakulteta, Katedra za družinsko medicino, Ljubljana, Slovenija;
'Community Health Centre Ljubljana, Ljubljana, Slovenia; 2University of Maribor, Faculty of Medicine, Department of Family Medicine, Maribor, Slovenia; 3University of Ljubljana, Faculty of Medicine, Department of Family Medicine, Ljubljana, Slovenia;
Ključne besede:
varnost pacientov, kultura varnosti, validacijske študije, primarno zdravstveno varstvo, faktorska analiza
Key words:
patient safety, safety culture, validation studies, primary care, factor analysis
Članek prispel / Received
12. 11. 2019
Članek sprejet / Accepted
2. 11. 2020
Naslov za dopisovanje / Correspondence
Izr. prof. dr. Zalika Klemenc-Ketiš, dr. med., Medicinska fakulteta, Univerza v Mariboru, Taborska 8, 2000 Maribor, Slovenija
Telefon +386 41516067 Fax +386 13003911 E-pošta: zalika.klemenc@um.si
Izvleček
Namen: Varnost pacientov na primarni ravni zdravstvenega varstva se močno razlikuje od varnosti pacientov na sekundarni ali terciarni ravni. Namen študije je bil preizkusiti veljavnost in zanesljivost slovenske različice vprašalnika o kulturi varnosti pacientov (MOSOPSC) v primarnih zdravstvenih ustanovah. Metode: Opisana študija je bila presečna študija v največjem zdravstvenem domu v Sloveniji - Zdravstvenem domu Ljubljana. Vse delavce z vodilno funkcijo (N = 221) smo povabili k sodelovanju v elektronski študiji in uporabili slovensko različico MOSOPSC. Izvedli smo analizo potrditvenega faktorja (CFA) MOSOPSC. Za določitev zanesljivosti smo izračunali ICC za notranjo skladnost. Uporabili smo tudi metodologijo deljive polovice (formula Spearman-Brown) in izračunali Pearsonov koeficient za zanesljivost preizkusa.
Rezultati: Končni vzorec je vseboval
Abstract
Purpose: Considerable differences are found between the safety of patients at the primary level of healthcare and the safety of patients at the secondary or tertiary levels. This study aimed to test the validity and reliability of the Slovenian version of the Medical Office Survey on Patient Safety Culture (MOSOPSC) in primary healthcare settings. Methods: A cross-sectional study was carried out in the largest community health centre in Slovenia, the Ljubljana Community Health Centre. We invited all employees who had a leadership function (N = 221) to participate in the electronic study, and used the Slovenian version of the MOSOPSC. We conducted confirmatory factor analysis (CFA) to examine the MOSOPSC. To determine its reliability and internal consistency, the intraclass correlation coefficient (ICC) was used. We also applied split-half methodology (the Spearman-
40| acta medico-biotechnica
2020; 13 (2): 38-45
Klinična študija / Clinical study
154 udeležencev (69,7 % odzivnost), od tega 136 (88,3 %) žensk. Povprečna starost vzorca je bila 46,2 ± 10,0 let. Model CFA je pokazal dobro prileganje. Notranja konsistenca MO-SOPSC je bila dobra ali zelo dobra, Cronbachov alfa pa je znašal od 0,55 do 0,90. Pearsonov koeficient časovne stabilnosti se je gibal med 0,540 in 0,712. Zanesljivost po formuli Spearman-Brown je bila dobra.
Zaključek: Izsledki študije kažejo, da bi lahko bila slovenska različica MOSOPSC z izvirnimi faktorji zanesljivo in veljavno orodje za merjenje kulture varnosti med zaposlenimi v osnovnem zdravstvu.
Brown formula) and calculated the Pearson's coefficient to assess the testretest reliability.
Results: The final sample consisted of 154 participants (69.7% response rate), of which 136 (88.3%) were female. The mean age of the sample was 46.2 ± 10.0 years. The CFA model showed good fit indices. The internal consistency of the MOSOPSC was good or very good, and Cronbach's alpha ranged from 0.55 to 0.90. Pearson's coefficient of temporal stability ranged from 0.540 to 0.712. The reliability, based on the Spearman-Brown formula, was good. Conclusion: The results of our study suggest that the Slovenian version of the MOSOPSC, with its original factors, could be a reliable and valid tool for measuring safety culture among primary health care workers who have a leadership role.
INTRODUCTION
The main goal of healthcare is to manage the health of a large number of patients at a reasonable cost, and with a reasonable level of safety (1). Patient safety is part of patient treatment quality, which also includes effectiveness, timeliness, and equality, and it is defined as the avoidance, prevention, and correction of unwanted outcomes or injuries resulting from the healthcare process (1).
Patient safety is influenced by several different factors. The structural factors represent both the physical structures (equipment and buildings) and the basic characteristics of the organisation (number and qualifications of staff)
(2).	These properties may change, but only slowly. In addition, the link between these properties and the outcome of healthcare are not well understood, although research shows that they have an impact on patient safety. Other important factors related to staff have been identified, and these include staff motivation, safety culture, teamwork, the use of technology, working conditions, and so on (1).
Patient safety culture is part of the patient safety concept, and it is defined as a product of the attitudes, values, competencies, and patterns of behaviour of individuals and groups that determine healthcare in an organisation
(3).	Organisations that have a positive security culture
demonstrate communication based on mutual trust, common perceptions of the importance of security, and confidence in the effectiveness of error prevention. A culture of security is based on the views and values of each individual within an organisation (1). The safety of patients at the primary level of healthcare varies considerably when compared with the safety of patients at the secondary or tertiary level. Nevertheless, most research and literature are based on patient safety at the secondary and tertiary levels. Similarly, the concept of patient safety has been transferred from the secondary to the primary level, in spite of the fact that two significantly different things are involved. At the primary level, there is a very high amount patient contact, which usually involves complex interactions (4), and the uncertainty that is typical of work at the primary level is very important (5). At the secondary level, patient safety is about reducing uncertainty, exploring opportunities, and diminishing errors; at the primary level, however, it should focus on accepting uncertainty, exploring probabilities, and minimising danger. It is also important to strive for openness and transparency in the area of patient safety (6).
There are several tools for measuring patient safety. Some assess the security culture, while others involve a system of
40| acta medico-biotechnica
2020; 13 (2): 38-45
Klinična študija / Clinical study
reporting adverse health outcomes. The Medical Office Survey on Patient Safety Culture (MOSOPSC), which was developed by the Agency for Research and Quality of Health Care, aimed to become the best tool for measuring safety culture at the primary level of healthcare (6, 7). This tool allows for the measurement of patient safety culture, the detection of possible differences, an understanding of the safety of a particular organisation, and an evaluation of the impact of specific interventions for improving patient safety culture. It comprises nine areas of safety (8).
In Slovenia, quality and safety at the primary healthcare level have been the focus of studies for several years (5, 9-11). Patient safety features in primary healthcare settings have been investigated by means of a study on the Quality and Costs of Primary Care in Europe (QUALICOPC), which deals with the organisation and accessibility of primary healthcare services (12). Safety culture has also been measured using the Slovenian version of the SAQ-AV in out-of-hours primary healthcare settings (13-15). However, the tool that was used by the researchers lacked the appropriate validity and reliability measurements that are necessary to confirm its reliability. It was also determined that other factors related to safety culture could have been relevant, and further studies were recommended to examine this issue. The present study aimed to test the validity and reliability of the Slovenian version of the MOSOPSC in primary healthcare settings.
MATERIALS AND METHODS
Type of study and settings
We carried out a cross-sectional study in the largest community health centre in Slovenia, the Ljubljana Community Health Centre. This health centre provides healthcare services to the municipality of Ljubljana, which comprises approximately 280,000 people. Ljubljana Community Health Centre is divided into eight units, which are scattered across the city. It employs around 1,500 employees from different medical and non-medical backgrounds.
The study was approved by the National Ethics Committee (No. 107/07/16).
40| acta medico-biotechnica
2020; 13 (2): 38-45
Participants
We invited all of the employees who carried out a leadership function (N = 221) to participate in the study. Employees who had a leadership function were from different professional backgrounds (e.g. physicians, dentists, nurse practitioners, registered nurses, administrative staff etc.), and they were appointed as leaders of different units within the health centre, such as chiefs of nurses, chiefs of physicians, heads of whole units, director of the health centre, and so on. They worked mostly within their own professional fields, but a certain proportion of their working hours was devoted to their leadership tasks.
Data collection
We used the MOSOPSC survey (7, 8), which we translated into Slovenian according to the standard procedure (16). First, two independent experts translated the survey from the English language into the Slovenian language. Any differences that arose during the process were discussed and a common solution was reached. Then, another two independent experts back-translated the survey into the English language, and differences were addressed in the same manner as before. This MOSOPSC survey was developed by the Agency for Research and Quality of Health Care. The tool facilitates measurement of the patient safety culture, the detection of possible differences, an understanding the safety of a particular organisation, and an evaluation of the impact of specific interventions for improving patient safety culture. It contained nine domains of safety (7). Domain A presents a list of patient safety and quality issues (nine items) which were answered according to a scale which ranged from "daily" to "not in the past 12 months". Domain B describes the information exchange with other settings (5 items) and it was answered according to a scale that ranged from "daily problems" to "no problems in the past 12 months". Domain C, which consisted of four sub-domains (i.e., teamwork, work pressure and pace, staff training, and office processes and standardisation), contained 15 items related to working in a medical office, and these were answered according to a five-point Likert-type scale (1 - strongly disagree, 5 - strongly agree). Domain D, with its three sub-domains (i.e., communication openness, patient care tracking/follow-up, and communication about errors),
Klinična študija / Clinical study
measured communication and follow-up using 12 items that were answered according to a five-point Likert-type scale (1 - never, 5 - always). Domain E included four items on leadership support which were only answered by individuals who had no leadership function; such persons responded to each question by referring to a five-point Likert-type scale (1 - strongly disagree, 5 - strongly agree). Participants who had a leadership function were required to respond to the seven items of Domain F, which consisted of two sub-domains (organisational learning, and overall perceptions of patient safety and quality). Domain G describes overall ratings on quality (five items) and patient safety (one item). Participants' responses were measured using a five-point Likert-type scale (1 - poor, 5 - excellent).
The researchers of the present study were authorised by the Agency for Healthcare Research and Quality to use this questionnaire; permission was granted on June 8, 2016.
Data related to demographic characteristics were collected (e.g. gender, age, function, work experience, working hours, and location of work). The survey was completed electronically; the link was sent to the email addresses of the participants in February 2017. A reminder was sent two weeks later. Participation was anonymous, and any possible identifiers, such as e-mail and IP addresses, were removed by the administrative coordinator of the project. It was not possible for the researchers to link the participants to their responses.
One month after the first survey, the researchers resent the link to the survey, in order to determine the temporal stability of the scale.
Statistical analysis
Confirmatory Factor Analysis (CFA) of the MOSOPSC was carried out. To determine the reliability of the MOSOPSC, the researchers calculated the ICC for internal consistency. Split-half methodology (the Spearman-Brown formula) was also employed. The Pearson's correlation coefficient was calculated to assess test-retest reliability (the survey was completed by the same participants twice over a one-month period). Negatively worded items (C3, C6, C8, C10, C12, C14, D4, D7, D10, E1, E2, E4, F3, F4, and F6) were reverse coded.
RESULTS
Sample description
The final sample included 154 participants (69.7% response rate), of which 136 (88.3%) were women (Table 1). The mean age of the sample was 46.2 ± 10.0 years, the
Table 1. Demographic characteristics of the participants
Characteristic	Number (%)
Gender	
Male	18 (11.7)
Female	136 (88.3)
Profile	
Physician, dentist	54 (35.1)
Registered nurse	36 (23.4)
Managerial staff	28 (18.2)
Administrative staff	2 (1.3)
Nurse assistant	18 (11.7)
Other clinical staff	16 (10.4)
Health centre unit	
Center	24 (15.6)
Moste-Polje	19 (12.3)
Uprava	8 (5.2)
Šentvid	41 (26.6)
Vic-Rudnik	23 (14.9)
Bežigrad	19 (12.3)
Emergency care	4 (2.6)
Šiška	16 (10.4)
mean time that participants spent in their current post was 13.6 ± 10.0 years, and the mean number of weekly working hours was 36.2 ± 10.4. The mean length of time that participants had been in employment was 21.9 ± 10.1 years.
Confirmatory factor analysis
In general, the CFA model was good. However, the value of some indicators may suggest that the model did not
40| acta medico-biotechnica
2020; 13 (2): 38-45
Klinična študija / Clinical study
have a perfect fit (the p value and NFI for some indicators in Domains C and D, and almost all the indicators for Domain F). However, when we omitted the factor of communication openness from Domain D, the results improved (p = 0.233; relative chi-square = 1.216, CFI = 0.981, NFI = 0.907, RMASE = 0.038, Hoelter 0.05 =
200). Similarly, when we omitted item F7 from the CFA, Domain F significantly improved (p = 0.406; relative chi-square = 1.036, CFI = 0.995, NFI = 0.890, RMASE = 0.015, Hoelter 0.05 = 287). However, Domain C did not fit perfectly, regardless of changes in its factors (Table 2).
Table 2. Goodness-of-fit indices for MOSOPSC (Domains C, D, E, F, and G)
Domain	Factors	P	Relative chi-square	CFI	NFI RMSEA		Hoeltei 05
C	Teamwork Work Pressure and Pace Staff Training Office Process and Standardization	0.007	1.423	0.933	0.816	0.053	137
D	Communication Openness Patient Care Tracking Communication about Errors	0.010	1.520	0.933	0.836	0.058	136
E	Managerial Support	0.926	0.077	1.000	0.999	0.000	5959
F	Organizational Learning Overall Perceptions of Safety and Quality	0.007	2.216	0.795	0.726	0.089	119
G	General Quality	0.067	2.063	0.988	0.977	0.083	165
P: should exceed 0.05. Relative chi square: should be less than 2 CFI: Comparative Fit Index, should be close to 1. N							sJFI: Normed
Fit Index, should exceed 0.90. RMSEA: Root Mean Square Error of Approximation, should not exceed 0.10. Hoelter 0.05: should exceed 200.
Reliability
The internal consistency of the MOSOPSC was good or very good (ICC was above 0.6) (Table 3). Cronbach's alpha ranged from 0.55 to 0.90 (Table 3). The temporal stability of the MOSOPSC was good, as the Pearson coefficient ranged from 0.540 to 0.712 (Table 3). According to the Spearman-Brown formula, its reliability was good (Table 4).
DISCUSSION
The results of this study indicate that the Slovenian version of the Medical Office Survey on Patient Safety Culture is a reliable and valid tool for measuring patient safety in primary healthcare settings in Slovenia. The original factors could be used and, if necessary, Domains D and F could be adjusted in order to achieve higher validity.
In general, the CFA for the domains that were eligible for this method of analysis showed acceptable goodness-of-fit values. Domain F, in particular, was problematic,
40| acta medico-biotechnica
2020; 13 (2): 38-45
but the fit greatly improved when item F7 was omitted. It is therefore suggested that this item should be excluded from a Slovenian version of the MOSOPSC. In some domains, the p and NFI values were not optimal. The CFI was better because it was not as sensitive to the effect size when compared with the p and NFI model fit values, which are highly sensitive to the sample size (17)(18). Overall, the reliability of the MOSOPSC was good according to the internal consistency, temporal stability, and the Spearman-Brown formula. The Cronbach's alpha of one domain (i.e., office processes and standardisation) was below 0.6, indicating poor reliability. However, other measures of reliability were better. The original version of the MOSOPSC had Cronbach's alpha values from 0.75 (communication about errors) to 0.90 (information exchange with other settings). The values in our study were lower overall, but still acceptable. Therefore, our findings suggest that the Slovenian version of the MOSOPSC is a reliable tool. To the best of the authors' knowledge, this was the first study to use CFA to validate the MOSOPSC in
Klinična študija / Clinical study
Table 3. Internal consistency and temporal stability of the MOSOPSC
A	Patient Safety and Quality Issues	0.895	0.833-0.941	0.752	0.712
B	Information Exchange with Other Settings	0.899	0.857-00.933	0.824	0.686
C1	Teamwork	0.812	0.761-0.855	0.686	0.575*
C2	Work Pressure and Pace	0.793	0.732-0.845	0.789	0.668*
C3	Staff Training	0.824	0.776-0.865	0.647	0.677*
C4	Office Processes and Standardization	0.728	0.652-0.793	0.547	0.612*
D1	Communication Openness	0.824	0.772-0.868	0.731	0.518*
D2	Patient Care Tracking/Follow-Up	0.827	0.755-0.883	0.689	0.573*
D3	Communication about Errors	0.856	0.806-0.896	0.762	0.643*
E	Leadership Support for Patient Safety	0.869	0.821-0.908	0.790	0.695*
F1	Organizational Learning	0.786	0.548-0.921	0.784	0.651*
F2	Overall Perceptions of Patient Safety and Quality	0.603	0.185-0.852	0.782	0.637*
G1	Overall Ratings on Quality	0.924	0.905-0.941	0.899	0.652*
G2	Overall Ratings on Safety	0.699	0.586-0.781	/	0.540*
* Significant at p<0.01
Table 4. Reliability of the MOSOPSC according to the Spearman-Brown formula
Domain	Domain title	Spearman-Brown value	Domain split (first/second half)
A	Patient Safety and Quality Issues	0.645	A1-A5/A6-A9
B	Information Exchange with Other Settings	0.821	B1-B2/B3-B4
C	Teamwork Work Pressure and Pace Staff Training Office Processes and Standardization	0.773	C1-C8/C9-C15
D	Communication Openness Patient Care Tracking/Follow-Up Communication About Errors	0.880	D1-D6/D7-D12
E	Leadership Support for Patient Safety	0.802	E1-E2/E3-E4
F	Organizational Learning Overall Perceptions of Patient Safety and Quality	0.854	F1-F4/F5-F7
G	Overall Ratings on Quality and Safety	0.871	G1a-G1c/G1d-G1e
a language other than that used by the original tool. Ornelas et al. (19) used the Portuguese version of the MOSOPSC in their study, but they did not carry out CFA; they only determined the Cronbach's alpha,
which showed similar values to those in our study. Timm et al. (20) carried out a cross-cultural adaptation of the Portuguese version of the MOSOPSC and determined the content validity scores by expert analysis. Afterwards,
40| acta medico-biotechnica
2020; 13 (2): 38-45
Klinična študija / Clinical study
they slightly modified the tool, which was then reported to have satisfactory content validity and high reliability. CFA was not performed. Another validation study was conducted in Spanish without carrying out CFA, and resulted in a valid, reliable, consistent, and useful tool for assessing patient safety culture in primary healthcare settings (21).
A cross-cultural adaptation of the instrument for use in a new country, culture, or language is necessary, and for this purpose, there is a minimum of two (backward and forward) translators. The validity and reliability are tested only after this step has been carried out (22). This first step also includes the determination of the face validity. In our case, internal consistency and test-retest were used to measure the reliability, and CFA was used to test the proposed theory, from which cultural adaptation can also be assessed (22). CFA was chosen over other similar methods, such as exploratory factor analysis (EFA), as it tests whether the data fit a hypothesised measurement model. EFA is used at the early stages ofscale development. In our study, we already had the original scale, so CFA was more appropriate. Some authors suggested that there is a need to semantically and culturally adapt a scale by translating it into languages other than that used by the original tool (23). This would ensure equivalence between the translated versions of the scales and this should be achieved using the Delphi method. In our study, we used the backward and forward translations as part of the standard procedure, with two independent translators, as suggested by the literature (16). In our study, the sample was limited to employees who had a leadership role. This could produce a selection bias,
which limits the validity of our results. Although all of the participants had a leading role, some of them did not tick the appropriate box (they completed Domain E instead of F). This could also affect the validity of our results. It is also possible to argue that the number of participants invited to take part in the study was low, but we invited all employees who had a leadership function. While the response rate was high, there was no information about the characteristics of the non-respondents, so this could also be a source of bias. We compared the characteristics of our sample with the characteristics of all employees who had a leadership function, and there was only a difference in the number of employees and the number of participants from different units. In our study, male respondents accounted for just 12% of the sample, but this reflected the actual situation with respect to the gender distribution of employees (in 2019, the percentage of male employees was 11.8%. The results of our study suggested that the Slovenian version of the MOSOPSC, with its original factors, could be a reliable and valid tool for measuring the safety culture of primary healthcare workers who have a leadership role. Further studies should explore the safety culture of other primary healthcare organisations, and perhaps use alternative tools for its measurement, in order to recognise other factors that are also important. The tool should also be used in practice to improve quality.
ACKNOWLEDGMENTS
ZKK was partly supported by the Slovenian Research Agency (Research in the Field of Public Health, P3-0339).
REFERENCES
1.	Vincent C, Burnett S, Carthey J. The measurement and monitoring of safety. London: Imperial College London; 2013.	4.
2.	Richter JP, McAlearney AS, Pennell ML. The influence of organizational factors on patient safety: Examining successful handoffs in health care. Health Care Manage Rev. 2016;41(1):32-41.
3.	Deilkas ET. Patient safety culture - opportunities 5.
40| acta medico-biotechnica
2020; 13 (2): 38-45
for healthcare management. Oslo: University of Oslo, Norway; 2010.
Tusek-Bunc K, Petek Ster M, Petek D. Correlation of Coronary Heart Disease Patient Assessments of Chronic Illness Care and Quality of Care Procedures. Acta Medico-Biotechnica. 2018;11(1):45-53.
Ivetic V, Poplas-Susic T, Pasic K, Selic P. Beliefs
Klinična študija / Clinical study
and viewpoints of family medicine physicians on approaches to identify and treat medically unexplained symptoms. Acta Medico-Biotechnica. 2016;9(2):47-57.
6.	Parker D, Wensing M, Esmail A, Valderas JM. Measurement tools and process indicators of patient safety culture in primary care. A mixed methods study by the LINNEAUS collaboration on patient safety in primary care. Eur J Gen Pract. 2015;21 Suppl:26-30.
7.	Sorra JS, Franklin M, Streagle S. Medical Office Survey on Patient Safety Culture. Rockville: Agency for Healthcare Research and Quality; 2008.
8.	Hickner J, Smith SA, Yount N, Sorra J. Differing perceptions of safety culture across job roles in the ambulatory setting: analysis of the AHRQ Medical Office Survey on Patient Safety Culture. BMJ quality & safety. 2015;25(8):588-94.
9.	Klemenc-Ketis Z, Svab I, Poplas Susic A. Implementing Quality Indicators for Diabetes and Hypertension in Family Medicine in Slovenia. Zdr Varst. 2017;56(4):211-9.
10.	Petek D, Mlakar M. Quality of care for patients with diabetes mellitus type 2 in 'model practices' in slovenia - first results. Zdr Varst. 2016;55(3):179-84.
11.	Zelko E, Svab I. Overcoming cultural cleavages: results from a health promotion intervention among Roma. Acta Medico-Biotechnica. 2016;9(1):33-41.
12.	Pavlic DR, Sever M, Klemenc-Ketis Z, Svab I. Process quality indicators in family medicine: results of an international comparison. BMC Fam Pract. 2015;16(1):172.
13.	Klemenc-Ketis Z, Deilkas ET, Hofoss D, Bondevik GT. Patient Safety Culture in Slovenian out-of-hours Primary Care Clinics. Zdr Varst. 2017;56(4):203-10.
14.	Klemenc-Ketis Z, Maletic M, Stropnik V, Deilkas ET, Hoffos D, Bondevik GT. The safety attitudes questionnaire - ambulatory version: psychometric
properties of the Slovenian version for the out-of-hours primary care setting. BMC Health Serv Res. 2017;17:36.
15.	Klemenc-Ketis Z, Deilkas ET, Hofoss D, Bondevik GT. Variations in patient safety climate and perceived quality of collaboration between professions in out-of-hours care. J Multidiscip Healthc. 2017;10:417-23.
16.	Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the Process of Cross-Cultural Adaptation of Self-Report Measures. Spine. 2000;25(24):3186-91.
17.	Jackson DL, Gillaspy JA, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: an overview and some recommendations. Psychol Methods. 2009;14(1):6-23.
18.	Timothy BA. Confirmatory factor analysis. Boston University2013.
19.	Ornelas MD, Pais D, Sousa P. Patient Safety Culture in Portuguese Primary Healthcare. Qual Prim Care. 2016;24(5):214-8.
20.	Timm M, Soares Rodrigues MC. Cross-cultural adaptation of safety culture tool for Primary Health Care. Acta Paul Enfem. 2016;29(1):26-37.
21.	Silvestre-Busto C, Torijano-Casalengua ML, Oli-vera-Canadas G, Astier-Pena MP, Maderuelo-Fer-nandez JA, Rubio-Aguado EA. Adaptation of the Medical Office Survey on Patient Safety Culture (MOSPSC) tool. Revista de calidad asistencial : organo de la Sociedad Espanola de Calidad Asistencial. 2015;30(1):24-30.
22.	Arafat S. Cross cultural adaptation & psychometric validation of instruments: Step-wise description. Int J Psychiatry. 2016;1(1):1-4.
23.	Petek D, Pusnik A, Selic P, Cedilnik Gorup E, Trontelj Z, Riou M et al. Semantic and cultural equivalence of the Working Alliance Inventory Short-Revised scale for therapeutic alliance in family medicine: lessons learned in Slovenia. Zdr Varst. 2019.
40| acta medico-biotechnica
2020; 13 (2): 38-45