SELF-ASSESSMENT QUESTIONNAIRE FOR FAMILY DOCTORS' ASSESSMENT OF QUALITY IMPROVEMENT COMPETENCIES: A CROSS-CULTURAL ADAPTATION IN SLOVENIA
VPRAŠALNIK ZA SAMOOCENO KOMPETENC ZDRAVNIKA DRUŽINSKE MEDICINE NA PODROČJU IZBOLJŠEVANJA KAKOVOSTI: MEDKULTURNA PRILAGODITEV V SLOVENIJI
Zalika Klemenc-Ketiš1, Primož Kuhar2, Janko Kersnik3, Genc Burazeri4,
Katarzyna Czabanowska5
Prispelo: 22. 1. 2013 - Sprejeto: 6. 5. 2013
Original scientific article UDC 614.2(497.4)
Abstract
Aim: To perform a cross-cultural adaptation of the Quality Improvement Competency Self Assessment (QICS) questionnaire for family physicians into the Slovenian language and to validate it in a representative sample of Slovenian FPs.
Methods: This cross-sectional observational postal survey was conducted in a random sample of 398 Slovenian FPs. We used the QICS questionnaire that was developed on the basis of the new Quality Improvement Competency Framework for family medicine. The QICS questionnaire consists of 37 items included in six domains. The questions can be answered on a five-point Likert scale. The validity of the translation was provided by the backward translation from Slovenian to the English language and by the reference group consisting of experienced FPs in the consensus process. The reliability of the questionnaire was assessed by Cronbach's alpha coefficient and Spearman rho to determine the test-retest reliability (the questionnaire was sent to the physicians in the sample twice in a period of two weeks).
Results: The final sample consisted of 100 (25.1%) family physicians, out of which 71 (71.0%) were women. Mean age of the sample was 43.3 ± 9.6 years. Mean score of the QICS questionnaire was 127.0 ± 30.1 points (first round) and 127.8 ± 30.6 points (second round). Cronbach's alpha scores were 0.984 (first round) and0.988 (second round). Spearman's rho for the summary score of the whole scale was 0.829 with p < 0.001.
Conclusion: The Slovenian version of the QICS questionnaire proved to be a valid and reliable tool for self-assessment of quality improvement competencies by FPs in terms of continuous professional development.
Key words: clinical competence, family medicine, self-assessment, quality improvement
Izvirni znanstveni članek UDK 614.2(497.4)
Izvleček
Namen: Izvesti medkulturno prilagoditev vprašalnika o samoocenjevanju kompetenc zdravnika družinske medicina na področju izboljševanju kakovosti (vprašalnik QICS) in ga validirati na reprezentativnem vzorcu slovenskih zdravnikov družinske medicine.
Metode: Ta presečna opazovalna raziskava je bila izvedena v naključnem vzorcu 398slovenskih zdravnikov družinske medicine. Uporabili smo vprašalnik QICS, kije bil razvit na podlagi novega teoretičnega okvira izboljševanja kakovosti v družinski medicini. Vprašalnik QICS je sestavljen iz 37 vprašanj, vključenih v šest področij. Na vprašanja je mogoče odgovoriti po petstopenjski Likertovi lestvici. Veljavnost prevoda je bila zagotovljena z dvosmernim prevodom in s pomočjo referenčne skupine, ki so jo sestavljali izkušeni zdravniki družinske medicine. Zanesljivost vprašalnika smo ocenjevali s pomočjo koeficienta Cronbach alfa in koeficienta Spearman rho za ugotavljanje časovne stabilnosti (vprašalnik je bil poslan zdravnikom v vzorcu dvakrat v razmiku dveh tednov).
1 University of Maribor, Medical Faculty, Slomškov trg 15, 2000 Maribor, Slovenia
2Community Health Centre Lenart, Maistrova ulica 22, 2230 Lenart, Slovenia
3University of Ljubljana, Medical Faculty, Department of Family Medicine, Poljanski nasip 58, 1000 Ljubljana, Slovenia
Correspondence to: e-mail: zalika.klemenc.ketis@gmail.com
Rezultati: Končni vzorec je bil sestavljen iz 100 (25,1 %) zdravnikov družinske medicine, od katerih je bilo 71 (71,0 %) žensk. Povprečna starost vzorca je bila 43,3 ± 9,6 leta. Povprečno število točk na vprašalniku QICS je bilo 127,0 ± 30,1 (pivo pošiljanje) in 127,8 ± 30,6 (drugo pošiljanje). Cronbach alfa je bil 0,984 (pivo pošiljanje) in 0,988 (drugo pošiljanje). Spearman rho je bil 0,829 s p < 0,001.
Zaključki: Slovenska različica vprašalnika QICS je zanesljivo in veljavno orodje za samooceno kompetenc zdravnikov družinske medicine na področju izboljševanja kakovosti v sklopu stalnega podiplomskega izobraževanja oz. stalnega strokovnega dograjevanja.
Ključne besede: klinične kompetence, družinska medicina, samoocenjevanje, izboljševanje kakovosti
1 INTRODUCTION
Self-assessment is the ability of physicians to perform self-rating or use self-audit with a goal of generating summary judgments on their performance to determine their own learning needs and find resources to meet them (1). Self-assessment is an integral part of many appraisal systems and has been proposed as an essential aspect of individual behaviour by several regulatory bodies and those developing learning outcomes for clinical students (2, 3). Self-assessment can be used in various aspects of education and lifelong learning such as in achieving credits in continuous medical education (CME), in relicensing process and in assessing professional competencies (1, 4-6). Competence is viewed as an attainment of a static set of attributes rather than a dynamic process in which physicians continuously use their practice experiences to progress in a competence towards the attainment of expertise (7). Competency-based continuous professional development (CPD) emphasises self-directed learning processes and promotes the role of assessment as a professional expectation and obligation (7, 8). So far, several competency models and frameworks have been developed in order to enhance educational activities in family medicine in different ways and in different levels of education (9). However, the attempts to address the need for family physicians' (FPs) training in quality improvement (QI) have been unevenly spread across countries (10-14). So far, several tools for physicians' self-assessment have been used (1). Some of them measured basic medical knowledge (15, 16), some basic clinical skills (17, 18) and some specific clinical knowledge and skills (19, 20). None of them was specifically based on the proposed competency framework such as some of the other tools were (21). Also, the results on the physicians' self-assessment have not been externally evaluated to ensure objectivity (1). Namely, previous studies have suggested that physicians have a limited ability to accurately perform self-assessment (1). Also, self-assessment might be dependent on the ability of
physicians to determine their own learning needs and therefore can result in the failure of professionals to generate summary judgments of their performance (22). A recent study on the QI competencies for CPD for European FPs provided a QI competencies framework (QICF) (9). This framework served as the basis for the development of the QI Competencies Self-assessment (QICS) questionnaire for FPs. This questionnaire can be used by FPs, teachers of family medicine, decision makers and patients to identify gaps in competencies of FPs (23, 24).
As this questionnaire has been developed in the English language, the aim of this study was to perform a cross-cultural adaptation of the questionnaire into the Slovenian language and to validate it in a representative sample of Slovenian FPs.
2 METHODS
2.1	Type of study and settings
This was a cross-sectional observational postal survey and was conducted in the representative sample of Slovenian FPs. It was part of an international study (25). The study was approved by the National Ethics Committee (No. 96/05/21).
2.2	Study population
The study population consisted of Slovenian FPs. A representative sample of 398 FPs was drawn randomly from the membership list of Slovenian Family Doctors Society. We aimed at a total of 30% FPs in the final sample. In 2011, there were 937 working FPs in Slovenia (26), out of which 255 (27.2%) were men. Most FPs in 2011 were in the age group 50-59 (313, 33.4%) (26).
2.3	Data collection
Data was collected by a postal survey. We sent out two rounds of mail; the second one was sent for the purposes of obtaining the data for the test-retest reliability. The
mail consisted of the questionnaire (described below), the invitation letter and a pre-stamped return envelope. The first round was sent at the beginning of May 2012. Prior to the first shipment, we sent out an email invitation to all participants in order to increase the response rate. The second round was sent two weeks after the first round. Similarly, it was preceded by an email invitation. We used the QICS questionnaire that was developed on the basis of the new QICF in family medicine (9). The questionnaire was translated from the original English version into the Slovenian language using the standard procedure (27). First, the English version was translated to the Slovenian language by two independent experts. They discussed the differences and produced a common version that was translated back into the English language by two independent experts. Both versions were then checked for differences and the final Slovenian version was produced. The QICS questionnaire consists of 37 items included in the following domains: patient care and safety (8 items), effectiveness and efficiency (7 items), equity and ethical practice (8 items), methods and tools (5 items), leadership and management (4 items) and continuing professional development (5 items). The participants were asked to assess their own level of competencies on a five-point Likert scale ranging from 1 ("novice" = I have little or no knowledge/ability, or no previous experience of the competency described and need close supervision or instruction) to 5 ("expert" = I am a primary source of knowledge and information in the medical field). So, the minimum summary score of the whole questionnaire was 37 points and the maximum 185 points. The minimum/maximum summary scores of the domains were 8/40 points for patient care and safety, 7/35 points for effectiveness and efficiency, 8/40 points for equity and ethical practice, 5/25 points for methods and tools, 4/20 points for leadership and management and 5/25 points for continuing professional development. The questionnaire also
consisted of demographic factors: sex, age, working period, number of registered patients, type of practice, area of practice, working style, involvement in teaching activities, involvement in research and participation in CME activities.
The QICS questionnaire was piloted in a sample of 10 FPs and adjusted according to their suggestions. Its face validity was provided by experts in family medicine teaching and experienced FPs as the reference group in the consensus process.
2.4 Statistical analysis
Data were analysed using the SPSS 19.0 package (SPSS Inc., Chicago, IL). Descriptive statistics were computed. We calculated the reliability (Cronbach's alpha) coefficient for the composite score of the questionnaire and the Spearman rho to determine the test-retest reliability of the questionnaire. We also calculated the reliability (Cronbach's alpha) coefficients of each competency domain.
For the bivariate analyses, we used a Mann-Whitney test and a Spearman correlation test. We regarded p < 0.05 as statistically significant.
3 RESULTS
3.1 Demographic characteristics
There were 168 (42.2%) FPs in the sample, out of which 68 (40.5%) did not complete both rounds. So, the final sample consisted of 100 (25.1%) FPs, out of which 71 (71.0%) were women. Other demographic characteristics of physicians and practices are presented in Table 1. Mean age of the sample was 43.3 ± 9.6 years. Mean working period was 16.4 ± 9.8 years. Mean number of registered patients per FP was 1,650 ± 815.
Table 1. Demographic characteristics of respondents and practices. Tabela 1. Demografske lastnosti zdravnikov in ambulant.
Characteristic/Lastnost	N (%)
Sex/spol Male/moški Female/ženska	29 (29.0) 71 (71.0)
Education status/izobrazba Specialist of family medicine/specialist družinske medicine Resident of family medicine(specializant družinske medicine Other specialization/druga specializacija No specialization/brez specializacije	75 (75.0) 17 (17.0) 3 (3.0) 5 (5.0)
Practice status/status ambulante Public/javna Private contractor/zasebnik s koncesijo No answer/brez odgovora	69 (69.0) 30 (30.0) 1 (1.0)
Practice organization/organizacija dela More physicians at the same location/več zdravnikov na eni lokaciji Only physician at this location/sam na eni lokaciji No answer/brez odgovora	82 (82.0) 15 (15.0) 3 (3.0)
Practice location/lokacija ambulante Urban/urbani predel Rural/ruralni predel No answer(brez odgovora	65 (65.0) 34 (34.0) 1 (1.0)
Involvement in education/vključenost v poučevanje	68 (68.0)
Involvement in research/vključenost v raziskovanje	22 (22.0)
Continuous medical education/stalno podiplomsko izobraževanje Never/nikoli 1-2 times per year/1-2-krat na leto 3-4 times per year/3-4-krat na leto No answer/brez odgovora	12 (12.0) 53 (53.0) 22 (22.0) 13 (13.0)
3.2 Reliability of questionnaire
The mean scores of the QICS questionnaire were 127.0 ± 30.1 points (first round) and 127.8 ± 30.6 points (second round). Mean scores of the individual
domains for both rounds are presented in Table 2. The temporal stability of the whole questionnaire was good with a Spearman's rho of 0.829 with p < 0.001. Data on temporal stability of the individual domains are presented in Table 2.
Table 2. Scores of the questionnaire and temporal stability.
Tabela 2. Dobljene točke na vprašalniku in časovna stabilnost vprašalnika.
Dimension/dimenzija	Mean score ± standard deviation/ povprečje ± standardna deviacija		Spearman's rho	P
	First round/ prvo pošiljanje	Second round/ drugo pošiljanje		
Patient care & Safety/oskrba in varnost bolnika	27.9 ± 6.6	28.6 ± 6.7	0.813	< 0.001
Effectiveness & Efficiency/ucinkovitost in uspešnost	23.7 ± 6.2	23.8 ± 6.0	0.796	< 0.001
Equity & Ethical Practice/pravna in etična oskrba	29.8 ± 6.6	29.5 ± 6.5	0.884	< 0.001
Methods & Tools/metode in orodja	14.4 ± 5.3	15.1 ± 5.2	0.777	< 0.001
Leadership & Management/vodenje in upravljanje	13.7 ± 3.6	13.9 ± 3.8	0.813	< 0.001
Continuing Professional Development/ stalen poklicni razvoj	17.6 ± 4.4	17.5 ± 4.4	0.867	< 0.001
All/skupaj	127.0 ± 30.1	127.8 ± 30.6	0.829	< 0.001
Cronbach's alpha coefficients of the whole questionnaire were 0.984 (first round) and 0.988 (second round). Cronbach's alpha coefficients of the questionnaire's dimensions were 0.941 (first round) and 0.953 (second round) for Patient care & Safety, 0.941 (first round) and 0.949 (second round) for Effectiveness & Efficiency, 0.951 (first round) and 0.958 (second round) for Equity & Ethical Practice, 0.944 (first round) and 0.960 (second round) for Methods & Tools, 0.907 (first round) and 0.928 (second round) for Leadership & Management and 0.939 (first round) and 0.950 (second round) for Continuing Professional Development.
3.3 Competencies' correlations
Older FPs had higher summary scores of the questionnaire (Spearman's rho = 0.529, p < 0.001). FPs with more working experiences had higher summary scores of the questionnaire (Spearman's rho = 0.527, p < 0.001). Specialists of family medicine had higher summary scores of the questionnaire when compared to others (135.8 ± 24.7 vs. 100.8 ± 29.8, p < 0.001). FPs who reported not attending any CPD activity in the last year had lower summary scores of the questionnaire when compared to others (105.3 ± 34.2 vs. 130.1 ± 27.7,
p = 0.012). FPs involved in education had higher summary scores of the questionnaires when compared to others (131.7 ± 29.3 vs. 118.6 ± 29.2, p = 0.024).
4 DISCUSSION
4.1	Summary of main findings
The Slovenian version of the QICS questionnaire proved to be a valid and reliable tool for self-assessment of QI competencies by FPs in terms of CPD. This questionnaire can be used in terms of a whole scale as well as in terms of each separate competency scale. Older FPs, those with longer working experiences, specialists of family medicine and those involved in education assessed their level of competencies higher. On the other hand, those not attending any CME activity assessed their level of competencies lower.
4.2	Contextualisation of the findings
This was the first study in Slovenia that dealt with self-assessment of desired QI competencies in FPs. A recent Slovenian study on medical students' attitudes towards family medicine competencies showed high
validity and reliability of the questionnaire used (28). It also showed that such questionnaires can be used for evaluating changes of students' attitudes towards undergraduate curricula and for prediction of students' preferences regarding their future professional career in family medicine (28).
So far, the QICS questionnaire has been translated into the Albanian language and validated in a sample of FPs (24) and of patients (23). The original English version has not yet been validated. As in our study, the Albanian study in a sample of FPs showed that the Albanian version of the questionnaire was reliable and had high temporal stability (24). However, the temporal stability of the individual competencies in our study was different than in the Albanian one. The latter found the lowest Spearman's rho scores for the leadership and management domain and the highest for the patient care and safety domain (24). In our study, the lowest Spearman's rho scores were found in the methods and tools domain and the highest in the equity and ethical practice domain. Probably, there are organisational and cultural differences between both countries, which could explain the differences in our results (29, 30). The instrument proved to be a reliable one also in a sample of patients assessing the desired level of given competencies of their FPs (23). Mean summary score of the questionnaire (127 points) in our study was higher than in the Albanian one (96 points) (24). It seems that Albanian FPs are less confident in their competencies or actually possess less QI knowledge and skills. This might be the consequence of some differences between both countries, especially in terms of primary care organisation in the past. In Slovenia, primary care traditionally has a strong and important role in the health care system (29). Also, family medicine as a discipline and specialisation in family medicine, have existed since the 1960s (31). Trainees have to perform a QI project during a module in specialty training curriculum (14). In Albania, on the other hand, there was not any strong role of primary care and also there was not any formal education in family medicine at any level of education until 1997, when the Department of Family Medicine at the University of Tirana was established (32).
In our study, the highest level of competencies was found in the equity and ethical practice field, which is in line with previous studies in Slovenia that showed high level of awareness about ethical dilemmas and their solving (33, 34).
The finding that older FPs and those with longer work experience assessed their level of competencies higher was expected as through work experience we also gain
confidence in our knowledge, skills, and expertise. The result that specialists in family medicine rated their competencies higher is probably a consequence of the fact that specialisation in family medicine in Slovenia follows the European guidelines (35) and is based on family medicine competencies (31). It is also interesting that FPs who did not attend CPD activities rated their competencies lower. Presumably, they have other problems rather than being not interested in education, i.e. lack of time, lack of money. On the other hand, it seems that they are aware of their low competencies, which support the importance and necessity of self-assessment.
However, the question remains whether the self-rated level of competencies will be really that high when assessed by external assessors. Some studies have shown that self-assessment might not be very objective and that external review was necessary (1). But such competencies' assessment might be very time-consuming and therefore difficult to implement. Therefore, the QICS questionnaire might be very useful, as it was developed also for assessing the level of competencies of FPs by patients, teachers and policy makers (9, 23, 24). When comparing their self-perceived level of competencies to the desired ones by patients, teachers and policy makers, FPs might gain an objective view of their real level of competencies and might develop and grasp correct self-educational activities in order to improve the quality of their work. Nevertheless, self-assessment has been accepted as a part of formal assessment that is most important in formative assessment (35).
4.3 Strengths and limitations of the study
The main strength of this study is the fact that the QICS questionnaire is based on a theoretical framework (9), which justifies the content and use of this tool. Also, the process of cross-cultural adaptation of the QICS questionnaire was consistent with the recommended guidelines (27).
The main limitation of this study is the fact that the questionnaire was given only to family physicians and not also to patients and policy makers. Also, the response rate was low but still consistent with the usual response rate achieved with postal surveys (36). The sex distribution in our sample was consistent with the actual one, whereas FPs in our sample were younger when compared to average age of all Slovenian FPs (26). Therefore, the results should be interpreted with care and given further consideration when trying to generalise them to the whole FP population in Slovenia.
4.4 Recommendations for further research
Further studies should validate this tool also in a sample of patients, family medicine teachers and policy makers. Also, the original English version should be validated in a representative sample. A large international study should analyse the level of competences, spot the differences between the countries and plan appropriate educational interventions as a part of CPD in individual countries.
5 CONCLUSION
The Slovenian version of the QICS questionnaire can be used as a self-assessment tool for quality improvement by family physicians. It can also be used by family medicine teachers to assess the gap between the desired and the self-assessed competencies of quality improvement of their students or residents.
Conflict of interests
The authors declare no conflicts of interest.
Acknowledgements
The instrument for this survey was developed with the support of the European Commission Lifelong Learning Program in the framework of the Leonardo da Vinci Project "Innovative lifelong learning of European General Physicians in Quality Improvement supported by information technology" (InGPinQI): No. 2010-1-PL1-LEO05-11473. The postal survey was financially supported by Osnovno zdravstvo Gorenjske. We thank all family physicians who participated in the survey.
References
1.	Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 2006; 296: 1094-102.
2.	Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME guide no. 10. Med Teach 2008; 30: 124-45.
3.	Davis NL, Willis CE. A new metric for continuing medical education credit. J Contin Educ Health Prof 2004; 25: 183-9.
4.	Silver I, Campbell C, Marlow B, Sargeant J. Self-assessment and continuing professional development: the Canadian perspective. J Contin Educ Health Prof 2008; 28: 25-31.
5.	Duffy FD, Lynn LA, Didura H, Hess B, Caverzagie K, Grosso L et al. Self-assessment of practice performance: development of
the ABIM Practice Improvement Module (PIM). J Contin Educ Health Prof 2008; 28: 38-46.
6.	Regnier K, Kopelow M, Lane D, Alden E. Accreditation for learning and change: quality and improvement as the outcome. J Contin Educ Health Prof 2005; 25: 174-82.
7.	Campbell C, Silver I, Sherbino J, Cate OT, Holmboe ES. Competency-based continuing professional development. Med Teach 2010; 32: 657-62.
8.	Van der Lee N, Fokkema JPI, Fedde S. Generic competencies in postgraduate medical training:their importance illustrated by a doctor's narrative on competency-based practice. Zdrav Var 2012; 51: 280-4.
9.	Czabanowska K, Klemenc-Ketis Z, Potter A, Rochfort A, Tomasik T, Cziszar J et al. Development of a aompetency framework for quality improvement in family medicine: a qualitative study. J Contin Educ Health Prof 2012; 32: 174-80.
10.	Klemenc-Ketis Z, Vanden Bussche P, Rochfort A, Emaus C, Eriksson T, Kersnik J. Teaching quality improvement in family medicine. Educ Prim Care 2012; 23: 378-81.
11.	Gordon P, Tomasa L, Kerwin J. ACGME outcomes project: selling our expertise. Fam Med 2004; 36: 164-7.
12.	Tomolo AM, Lawrence RH, Watts B, Augustine S, Aron DC, Singh MK. Pilot study evaluating a practice-based learning and improvement curriculum focusing on the development of systemlevel quality improvement skills. J Grad Med Educ 2011; 3: 49-58.
13.	Edwards FD, Frey KA. The future of residency education: implementing a competency-based educational model. Fam Med 2007; 39: 116-25.
14.	Ster MP, Mihailovic I, Svab I, Kersnik J. Teaching about quality improvement in specialist training for family medicine in Slovenia. Srp Arh Celok Lek 2012; 140: 489-94.
15.	Young MJ, Glasziou P, Ward JE. General practitioners' self ratings of skills in evidence based medicine: validation study. Br Med J (Clin Res Ed) 2002; 324: 950-1.
16.	Tracey JM, Arrol B, Richmond DE, Barham PM. The validity of general practitioners' self assessment of knowledge: cross sectional study. Br Med J (Clin Res Ed) 1997; 315: 1426-8.
17.	Leopold SS, Morgan HD, Kadel NJ, Gardner GC, Schaad DC, Wolg FM. Impact of educational intervention on confidence and competence in the performance of a simple surgical task. J Bone Jt Surg (Am) 2005; 87: 1031-7.
18.	Claridge JA, Calland JF, Chandrasekhara V, Young JS, Sanfey H, Schirmer BD. Comparing resident measurements to attending surgeon self-perceptions of surgical educators. Am J Surg 2003; 185: 323-7.
19.	Fernandez A, Schillinger D, Grumbach K, Rosenthal A, Steward AL, Wang F et al. Physician language ability and cultural competence: an exploratory study of communication with Spanish-speaking patients. J Gen Intern Med 2004; 19: 167-74.
20.	Woods R, McCarthy T, Barry MA, Mahon B. Diagnosing smallpox: would you know it if you saw it? Biosecur Bioterrorism 2004; 2: 157-63.
21.	Biesma RG, Pavlova M, Vaatstra R, van Merode G, Czabanowska K, Smith et al. Generic versus specific competencies of entry-level public health graduates: employers perceptions in Poland, the UK, and the Netherlands. Adv Health Sci Educ 2008; 13: 325-43.
22.	Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med 2005; 80: S46-54.
23.	Alla A, Czabanowska K, Klemenc-Ketis Z, Roshi E, Burazeri G. Cross-cultural adaptation of an instrument measuring primary health care users' perceptions on competencies of their family physicians in Albania. Med Arh 2012; 66: 240-2.
24.	Alla A, Czabanowska K, Mone I, Kijowska V, Roshi E, Burazeri G. Validation of an instrument on self-perceived level of competency among family physicians in Albania. Eur J Public Health 2012; 22(Suppl 2): 283-4.
25.	Czabanowska K, Burazeri G, Klemenc-Ketis Z, Kijowska V, Tomasik T, Brand H. Quality improvement competency gaps in primary care in Albanian, Polish and Slovenian contexts: a study protocol. Acta Inform Med 2012; 20: 136-40.
26.	Slovenian Institute for Public Health. Available November 17, 2012 at: http://www.ivz.si/.
27.	Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol 1993; 46: 1417-32.
28.	Petek MP, Svab I, Klemenc-Ketis Z, Kersnik J. Development and validation of a questionnaire for evaluation of students' attitudes towards family medicine. Coll Antropol 2012; 13: 305-13.
29.	Oleszczyk M, Svab I, Seifert B, Krzton-Krolewiecka A, Windak A. Family medicine in post-communist Europe needs a boost: exploring the position of family medicine in healthcare systems of Central and Eastern Europe and Russia. BMC Family Practice 2012; 13: 15.
30.	Vyshka G, Kruja J. Inapplicability of advance directives in a paternalistic setting: the case of a post-communist health system. J Med Ethics 2011; 12: 12.
31.	Bulc M, Svab I, Pavlic DR, Kolsek M. Specialist training of Slovene family physicians. Eur J Gen Pract 2006; 12: 128-32.
32.	Rrumbullaku L, Theodorakis PN, Caca P, Lionis C, Trell E. Medical education in Albania: current situation and perspective, with reference to primary care. Croat Med J 2002; 43: 50-3.
33.	Klemenc-Ketis Z, Kersnik J, Ojstersek J. Perceived difficulties in managing ethical problems in family practice in Slovenia: cross-sectional study. Croat Med J 2008; 49: 799-806.
34.	Klemenc-Ketis Z, Kersnik J. Prevalence of ethical dilemmas in Slovenian family practice. Acta Medico-Biotechnica 2009; 2: 47-56.
35.	Heyrman J. EURACT Educational Agenda. Leuven: European Academy of Teachers in General Practice, 2005.
36.	Kaplowitz MD, Hadlock T, Levine R. A comparison of web and mail survey response rates. Public Opinion Quarterly 2004; 68: 94-101.