STANDARDIZATION OF QUALITY OF DIAGNOSES, INTERVENTIONS, AND OUTCOMES (Q-DIO) MEASUREMENT INSTRUMENT FOR USE IN SLOVENIA Maja KLANČNIK GRUDEN1,2*, Maria MÜLLER-STAUB3,4,5, Majda PAJNKIHAR1, Gregor ŠTIGLIC1,6 1University of Maribor, Faculty of Health Sciences, Žitna ulica 15, 2000 Maribor, Slovenia 2University Medical Centre Ljubljana, Zaloška 2, 1000 Ljubljana 3Institute of Nursing, Hanze University of Applied Sciences, Groningen, the Netherlands 4Pflege PBS, Obere Hofbergstrasse, 9500 Wil, Switzerland 5Stadtspital Waid, Tièchestrasse 99, 8037 Zürich, Switzerland 6University of Maribor, Faculty of Electrical Engineering and Computer, 2000 Maribor, Slovenia Received: Nov 29, 2020 Accepted: Oct 13, 2021 Original scientific article *Corresponding author: Tel. + 386 41 973 942; E-mail: maja.klancnikg@um.si 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 14 STANDARDIZACIJA MERSKEGA INŠTRUMENTA KAKOVOST NEGOVALNIH DIAGNOZ, INTERVENCIJ IN IZIDOV (Q-DIO) ZA SLOVENSKI PROSTOR © National Institute of Public Health, Slovenia. Klančnik Gruden M, Müller-Staub M, Pajnkihar M, Štiglic G. Standardization of quality of diagnoses, interventions, and outcomes (Q-DIO) measurement instrument for use in Slovenia. Zdr Varst. 2022;61(1):14-23. doi: 10.2478/sjph-2022-0004. ABSTRACT Keywords: translations, adaptations, reliability, validity, nursing records IZVLEČEK Ključne besede: prevodi, prirejanje, zanesljivost, veljavnost, zapisi zdravstvene nege Purpose: To describe the cross-cultural adaptation of the Quality of Diagnoses, Interventions and Outcomes (Q-DIO) Instrument into the Slovene language. Methods: Based on general international guidelines, a six-step process of localization to translate and adjust the instrument from English into the Slovene language was used. Content validity was quantified based on an agreement of eight experts. The instrument was tested using a sample of 140 nursing documentations from two Slovenian tertiary hospitals. Results: 26 of 29 items showed an excellent content validity index ranging from 0.857 to 1.000, and a modified kappa index above 0.856. The content validity indexes of the three remaining items adjusted based on experts’ comments were subsequently estimated at 1.000. Construct validity was significantly different between the two groups of documentations. The Cronbach coefficient for the whole questionnaire was 0.860. Cronbach’s alpha if item deleted reamins above 0.80 for all items. The criteria for the difficulty grades of items and discrimination validity were acceptably met for more than 75% of items. Conclusion: Based on the results of the study, it may be concluded that Q-DIO is a reliable instrument for measuring the quality of nursing documentation. The deviations in the results of some items are due to poor nursing documentation quality, and indicate that nursing classifications have not yet been fully implemented into practice in the study setting. Additional testing of the instrument is recommended. Namen: Opisati medkulturno prilagoditev inštrumenta Kakovost negovalnih diagnoz, intervencij in izidov (Q-DIO) v slovenski jezik. Metode: Za prevod in prilagoditev inštrumenta iz slovenskega v angleški jezik je bil, na osnovi mednarodnih priporočil, uporabljen šeststopenjski postopek lokalizacije. Vsebinska veljavnost je bila izračunana na osnovi strinjanja osmih strokovnjakov. Testiranje inštrumenta je bilo izvedeno na vzorcu 140 negovalnih dokumentacij iz dveh slovenskih terciarnih bolnišnic. Rezultati: Indeksi vsebinske veljavnosti posameznih postavk so bili odlični za 26 od 29 postavk in so se gibali od 0,857 do 1,000. Modificirani Cohenov indeks kapa pa se je gibal nad 0,860. Indeksi vsebinske veljavnosti drugih treh postavk, ki so bile popravljene na podlagi mnenj strokovnjakov, so bili naknadno ponovno ocenjeni. Vrednosti vseh treh so naknadno znašale 1,000. Konstruktna veljavnost se je značilno razlikovala med skupinama dokumentacij. Cronbachov koeficient za celotni vprašalnik je znašal 0,860. Koeficient Cronbach α posameznih postavk je ostal nad 0,80 tudi ob izključitvi katerekoli postavke. Analiza posameznih postavk inštrumenta glede pogostosti izbire postavke in veljavnosti diskriminacije je pokazala sprejemljive vrednosti za več kot 75 % postavk. Zaključek: Na podlagi rezultatov raziskave je mogoče sklepati, da je Q-DIO zanesljiv instrument za merjenje kakovosti negovalne dokumentacije. Odstopanja v rezultatih nekaterih postavk so posledica slabe kakovosti negovalne dokumentacije in kažejo, da klasifikacije zdravstvene nege še niso bile v celoti vpeljane v prakso v študijskem okolju. Priporočljivo je nadaljnje testiranje instrumenta. 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 15 1 INTRODUCTION A health record is the official record of all important interactions with the patient (1) in electronic or paper form (2), and an important source of data for healthcare providers, as it supports them in making clinical decisions (3, 4). Documentation also enables continuity of work and greater safety and quality of patient care (5), and is also crucial for the evaluation of patient health care (6). Regardless of the positive effects, various authors have noted that there are often gaps in patient records, repetitive records of irrelevant information, and inaccurate data (1, 7), and further that nursing documentation does not always support nurses in making clinical decisions (3). Nursing documentation was introduced in more comprehensive form with the introduction of the Slovenian Patient Classification Tool for nursing intensity evaluation (8), while the standardized language of nursing, especially for nursing diagnosis, began to be used after the translation of the Manual of Nursing Diagnosis by Marjory Gordon (9). The introduction of the electronic health record further accelerates this process. The survey, carried out at all three levels of health care (primary, secondary and tertiary) in Slovenia, revealed that all phases of the nursing process were being documented in only one-third of cases (10). The newer version of the nursing process – called the Advanced Nursing Process (11) – has not yet been implemented in Slovenia. Based on a literature review, some questions arose: what is the actual quality of nursing records in Slovenia? Do nurses identify appropriate nursing diagnoses that are consistent with the data collected during the assessment; are nursing diagnoses based on clinical judgment? Do nurses set specific goals and effective interventions for the patient that are individualized? Studies in this field are insufficient and do not provide all the necessary answers. Müller-Staub et al. (12) developed the Quality of Diagnoses, Interventions and Outcomes (Q-DIO) instrument to measure the accuracy of the nursing diagnosis and the coherence between nursing diagnoses, interventions, and nursing- sensitive patient outcomes in nursing documentation. The Q-DIO instrument was developed on the basis of a literature review. The concepts of nursing diagnosis, nursing interventions, nursing outcomes and the nursing process provided the theoretical framework for the development of the Q-DIO instrument. The development of a coherent nursing plan that will serve as the basis of nursing care quality requires choosing an accurate nursing diagnosis and selecting effective interventions and outcomes for each patient (6). The proper use of these concepts has a key impact on the quality of nursing documentation. The main purpose of the Q-DIO instrument is to assess the accuracy of nursing diagnoses, the effectiveness of nursing interventions, and the quality of patient outcomes in nursing documentation. The basis for filling in the instrument are patient healthcare records, which can be in electronic or paper form. The use of a standardized nursing language in an institution is not necessary for using the Q-DIO in quality assessments (12). The Q-DIO instrument, originally developed in English (12), has also been translated into and validated in Brazilian Portuguese (13), French (14), Italian (15, 16), German (17, 18), and Spanish (19). Psychometric testing of adapted versions of the instrument was published for the adaptation in Brazilian Portuguese and in German languages. These studies showed good psychometric properties (17, 18, 21). As in medicine and other health professions, nursing also uses rating scale questionnaires to measure characteristics for which there are no standardized physical measures. In order to use such questionnaires in the professional field, they need to be standardized and tested (validated). They also need to be tested or “calibrated” when transferred to other (cultural or linguistic) environments. The standardized transfer process is called localization and takes place in six phases: forward translation, target language version synthesis and adaptation, backward translation, source language version comparison and target adaptation, comprehensibility testing, and reliability testing (20). 1.2 Purpose and goals The purpose of our research was to localize the Q-DIO measurement instrument (Figure 1) for use in the Slovenian setting, which includes testing its content validity and internal reliability. The Q-DIO measurement instrument was developed by Maria Müller-Staub et al. (6, 12). Its name in Slovenian is: “Kakovost negovalnih diagnoz, intervencij in izidov “. 2 METHODS In validating the Slovenian Q-DIO instrument (Figure 2), we used psychometric testing and considered the general international recommendations (20, 22, 23). The 18 guidelines cover the following areas: pre-condition, test development, confirmation, administration, scoring and interpretation, and documentation (22). 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 16 Nursing diagnoses as process Information is documented about: 1. Actual situation, leading to the hospitalization 2. Anxiety and worries related to hospitalization, expectations and desires about hospitalization 3. Social situation and living environment/circumstances 4. Coping in the actual situation/with the illness 5. Beliefs and attitudes about life (related to the hospitalization) 6. Information of the patient and relatives/significant others about the situation 7. Intimacy, being female/male 8. Hobbies, activities for leisure 9. Significant others (contact persons) 10. Activities of daily living 11. Relevant nursing priorities according to the assessment 11 Items, maximum score=22, mean=2 Nursing diagnoses as product 12. Nursing problem/nursing diagnosis label is documented 13. Nursing diagnosis label is formulated according to NANDA and numbered 14. The etiology (E) is documented 15. The etiology (E) is correct, related /corresponding to the nursing diagnosis (P) 16. Signs and symptoms are formulated 17. Signs and symptoms (S) are correctly related to the nursing diagnosis (P) 18. The nursing goal relates/corresponds to the nursing diagnosis 19. The nursing goal is achievable through nursing interventions 8 Items, maximum score=32, mean=4 Nursing interventions 20. Concrete, clearly named nursing interventions according to NIC are planned (what will be done, how, how often, who does it) 21. The nursing interventions effect the etiology of the nursing diagnosis 22. Nursing interventions carried out, are documented (what was done, how, how often, who did it) 3 Items, maximum score=12, mean=4 Nursing-sensitive patient outcomes 23. Acute, changing diagnoses are assessed daily or form shift to shift/ enduring diagnoses are assessed every fourth day 24. The nursing diagnosis is reformulated 25. The nursing outcome is documented 26. The nursing outcome is observably/measurably documented according to NOC 27. The nursing outcome shows - improvement in patient’s symptoms - improvement in patient’s knowledge state - improvement in patient’s coping strategies - improved self-care abilities - improvement functional status 28. There is a relationship between nursing-sensitive patient outcomes and nursing interventions 29. Nursing outcomes and nursing diagnoses are internally related 7 Items, maximum score=28, mean=4 Total Items 29 Measurement Instrument Q-DIO Figure 1. Q-DIO measurement instrument – English version. 3-point scale 5-point scale 2 4 4 4 1 3 3 3 1 1 1 0 2 2 2 0 0 0 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 17 Negovalne diagnoze kot proces Navedene informacije so dokumentirane: 1. Aktualna situacija, ki je vodila v hospitalizacijo 2. Tesnoba in skrbi povezane s hospitalizacijo, pričakovanja in želje glede hospitalizacije 3. Socialna situacija in življenjsko okolje/pogoji 4. Spopadanje z aktualno situacijo/obolenjem 5. Prepričanja in odnos do življenja (v povezavi s hospitalizacijo) 6. Informacije pacienta in sorodnikov/pomembnih drugih o situaciji 7. Intimnost v odnosu, biti ženska/moški 8. Hobiji, prostočasne aktivnosti 9. Pomembni drugi (kontaktne osebe) 10. Aktivnosti vsakodnevnega življenja 11. Relevantne prioritete zdravstvene nege na podlagi ocenjevanja (anamneze in ocene stanja) 11 vprašanj, maksimalno število točk = 22, povprečje = 2 Negovalne diagnoze kot produkt (zapis) 12. Negovalni problem/oznaka negovalne diagnoze je dokumentiran 13. Oznaka negovalne diagnoze je formulirana skladno z NANDA-I in oštevilčena 14. Etiologija (E) je dokumentirana 15. Etiologija (E) je pravilna in povezana/ustrezna za negovalno diagnozo (P) 16. Znaki in simptomi so formulirani 17. Znaki in simptomi (S) so pravilno povezani z negovalno diagnozo (P) 18. Negovalni cilj je povezan/ustreza negovalni diagnozi 19. Negovalni cilj je dosegljiv z negovalnimi intervencijami 8 vprašanj, maksimalno število točk = 32, povprečje = 4 Nursing interventions 20. Načrtovane so konkretne, jasno poimenovane negovalne intervencije po NIC (kaj bo narejeno, kako bo narejeno, kako pogosto, kdo bo naredil) 21. Negovalne intervencije vplivajo na etiologijo negovalne diagnoze 22. Izvedene negovalne intervencije so dokumentirane (kaj je bilo narejeno, kako je bilo narejeno, kako pogosto, kdo je izvedel) 3 vprašanja, maksimalno število točk = 12, povprečje = 4 Na zdravstveno nego odzivni izidi pacientov 23. Akutne, spreminjajoče diagnoze se ocenijo dnevno ali oblikujejo v vsaki izmeni / trajne diagnoze pa se ocenijo vsak četrti dan 24. Negovalna diagnoza je spremenjena 25. Negovalni izid je dokumentiran 26. Negovalni izid je dokumentiran po NOC na podlagi opazovanja in merjenja 27. Negovalni izid kaže - izboljšanje v pacientovih simptomih - izboljšanje pacientovega znanja - izboljšanje pacientovih strategij spopadanja s situacijo - izboljšanje zmožnosti samooskrbe - izboljšanje v funkcionalnem statusu 28. Obstaja povezava med na zdravstveno nego odzivnimi izidi pacientov in negovalnimi intervencijami 29. Negovalni izidi in negovalne diagnoze so notranje povezani 7 vprašanj, maksimalno število točk = 28, povprečje = 4 Skupno število vprašanj = 29 Merski inštrument Q-DIO Figure 2. Q-DIO measurement instrument – Slovene version. 3-točkovna lestvica 5-točkovna lestvica 2 4 4 4 1 3 3 3 1 1 1 0 2 2 2 0 0 0 2.1 Description of the measurement instrument The Q-DIO instrument consists of 29 items (Q), which are divided into 4 domains: nursing diagnoses as process, nursing diagnoses as product, nursing interventions and nursing-sensitive patient outcomes. The items under Nursing diagnoses as process (Q 1 to 11) measure the comprehensiveness and documentation quality of nursing history and patient demographics. The items under Nursing diagnoses as a product (Q 12 to 19) measure the accuracy and compliance of nursing diagnoses with the PES statement (problem, etiology, symptoms) (9). The items under Nursing interventions (Q 20 to 22) measure the effectiveness of intervention planning and implementation in relation to the nursing diagnosis. The items under Nursing-sensitive patient outcomes (Q 23 to 29) measure outcomes planning and achievement in relation to nursing interventions and diagnosis (6, 12, 21). 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 18 For the first domain – Nursing diagnoses as process – a 3-point Likert scale was used to distinguish between grades: 0-missing documentation, 1-partially completed documentation, and 2-comprehensive documentation (6). For the second, third, and fourth domains, a 5-point Likert scale was used to distinguish between grades: 0-no formulation, 1-partially correct only one part of diagnosis, intervention or outcome formulation (e.g. label), 2-correct one part of formulation, 3-partially correct all parts of diagnosis (label, etiology, signs and symptoms), intervention or outcome formulation (concrete, clearly named, planned, documented and affecting the etiology), 4-correct all parts of diagnosis, intervention or outcome formulation (12). 2.2 Description of the sample 140 nursing records of children aged 1 to 9 years, who were hospitalized in 2017 and 2018 with lower respiratory tract infection within the Respiratory Diseases (J00-J99) group according to the International Classification of Diseases (ICD-10) (24), were selected in the sample for analysis from two institutions: University Medical Centre Ljubljana and University Medical Centre Maribor (70 from each institution). This group was chosen because lower respiratory tract infection is one of the most common medical diagnoses in children, and also because these children are treated in different wards, which is crucial for research. The first institution used electronic-based documentation and the standardized language for nursing diagnoses according to Marjory Gordon (9), while the second institution used paper-based documentation without a standardized language. At the end of 2017 the research setting at the first institution employed 90 registered nurses (38.8%) and 142 health technicians (61.2%) for 157 bed units (1.47 employees per bed), while the research setting at the second institution employed 24 registered nurses (32%) and 51 health technicians (68%) for 60 bed units (1.25 employees per bed). The research sample was estimated using WINPEPI v11.32 at a statistical power of 90% and a significance level of 2.5% based on the difference of averages and standard deviation found in a comparable study (25). The sample size was calculated as 69 health records per facility; that number was rounded to 70 per facility. In one ward, there were not enough medical records from the 2017 that would meet the criteria, so medical records from 2018 were added until the sample was fleshed out. Inclusion criteria for nursing records were: a) Records of patients with a hospital stay of at least 4 days (according to the Instructions for using the Q-DIO instrument (12)), b) which comprised an assessment, nursing problems/ nursing diagnoses, and nursing care planning documented in standardized care plan forms. Excluded were nursing documentation for re-hospitalized patients (that were already included in the sample) and documentation that did not contain standardized forms of nursing assessments and nursing care plans (which includes nursing problems/nursing diagnoses). 2.3 Description of the research procedure and data analysis The process of adaptation or so-called localization of the Q-DIO measurement instrument for the Slovenian setting is part of a broader research project conducted within the scope of a doctoral study. Data collection was conducted in both institutions from August 2019 to August 2020 by a single independent researcher, who coded and anonymized all the collected data. The principles of the Declaration of Helsinki (26) were taken into account. Data analysis was performed using the SPSS 27.0 software package. 2.3.1 Forward translation We first obtained the consent of the author of the Q-DIO instrument for its translation into Slovenian. The instrument was then translated into Slovenian by two independent researchers – registered nurses, one also with Bachelor of Science in Nursing and the other with Bachelor of Science in Organization and Management. 2.3.2 Target language version synthesis and adaptation Both translated versions were then compared, harmonized and unified on the basis of a discussion between the translators. 2.3.3 Backward translation A harmonized version of the Q-DIO instrument was handed over to a third independent researcher with experience in translating professional texts on nursing from English into Slovenian. The researcher performed a backward translation – translated the instrument back into the English language without the original English version of the instrument. 2.3.4 Source language versions comparison and target adaptation We compared the new translation with the original text in English, and then harmonized and adapted the Slovenian version accordingly. The obtained and presumably final version of the translation was used for further evaluation in two steps. 2.3.5 Comprehensibility testing The next step was comprehensibility testing, or, in other words, a content validity analysis of the entire Q-DIO instrument and of individual items, which we carried out 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 19 with the help of eight experts – nurses with a bachelor’s degree, clinical mentors, and teaching assistants or lecturers on nursing. Comprehensibility was assessed using a 4-point scale (1-not relevant/not comprehensible, 2-somewhat relevant/poorly comprehensible, 3-quite relevant/partially comprehensible, 4-highly relevant/ fully comprehensible). The experts judged that the Q-DIO items constitute a representative sample of the universe of items to measure what the instrument aims to measure. Based on the scores, we calculated the item content validity index (I-CVI) and the scale validity index (S-CVI). To calculate I-CVI, we used the following formula: the number of experts who rated the item with 3 or 4 divided by the total number of all experts. S-CVI was calculated as the average of all I-CVI. According to Polit and Beck (27), the acceptable values for I-CVI are >0.78 and for S-CVI≥0.90. Based on the experts’ answers, we reworded the poorly evaluated statements (taking into account any reasonable comments and suggestions) and sent them back to the same experts for re-assessment. The subsequent comprehensibility scores were given by five experts. In addition to the content validity index, we also checked the index of agreement between experts or modified kappa statistics (Κ*) to reduce the possibility of chance agreement. The formula Κ*=(I-CVI-PC)/(1-PC) was used. The probability of chance agreement between experts on the significance of the items (PC) was calculated with the following formula: PC=N!/A!x(N-A))x0.5N, where N represents the number of experts and A the number of experts agreeing on the relevance (scores 3 and 4). The modified Kappa is defined as excellent when the value of Κ*≥0.75, and as still acceptable when Κ*≥0.60 (27). Divergent or discriminant construct validity analysis gather evidence that the focal measure is not a measure of a different construct (27). It was assessed with a T-test in comparison between two groups of nursing documentations: electronic-based documentation and standardized terminology for nursing diagnoses; paper- based documentation and no standardized terminology. 2.3.6 Reliability testing This was followed by reliability testing, which can be evaluated by internal consistency and stability testing. The internal consistency or internal reliability of the instrument tells us whether its components measure the same characteristic (28). We used the Cronbach coefficient ≥[0-1], whose desired value is α≥0.80 (27). The Cronbach coefficient depends, in part, on the length of the instrument, so it is good to use additional reliability analyses (27). The analysis of the instrument’s individual items was also checked using the frequency of endorsement of items and the discrimination validity. The frequency of agreement with the statements is a function of the item’s weights, and is calculated using the weight index (<1.6 being recommended for the first 11 items, and <3.2 for the remaining 18 items of the Q-DIO instrument) (6). Item discrimination (corrected item-total correlation) is a function of internal reliability and shows us how well it is possible to replace the instrument’s score with the score of a single item. Values ≥0.3 were considered as an acceptable criterion (29). The instrument’s stability was checked on a pilot sample of 17 patient records by two independent researchers. To be prepared for using the instrument, the researchers studied all available literature on its use that was provided from its initial author. We used two calculations: interrater reliability and interrater agreement. To calculate the reliability between researchers or the consistency of estimates between two independent researchers, we used the Pearson correlation. To calculate the agreement between two independent researchers, we used Cohen’s Kappa value. 3 RESULTS In the first phase, i.e. the “forward translation” of the Q-DIO instrument, the translators did not encounter any major conceptual, linguistic, or terminological problems. In the second phase, they stopped to coordinate the translations of the concepts of “assessment” and “nursing-sensitive patient outcomes”. The first term, i.e. “ocenjevanje”, is not yet widely used in Slovenian nursing, so we added an explanation in parentheses that it refers to the patient’s history and condition assessment. We also do not yet have an established translation for the second term. The translators agreed to use the most semantically appropriate translation, i.e., “na zdravstveno nego odzivni izidi pacientov”. Other harmonisations were more in terms of greater comprehensibility, e.g. “informacije o situaciji” → “opis situacije”, “formulirano” → “oblikovano”, and the use of established terms, e.g., “aktivnosti za zabavo” → “prostočasne aktivnosti”. The third phase, i.e. “backward translation”, showed almost perfect semantic correspondence between the two English texts. In the fourth phase, i.e. “source language versions comparison and target adaptation”, we made only one change (“dolgotrajne negovalne diagnoze” → “trajne negovalne diagnoze”). In the fifth phase, i.e. “comprehensibility testing”, we calculated the content validity indexes. S-CVI of the entire instrument was 0.945, 0.877 for the first part, 1.000 for the second and third part, and 0.964 for the fourth part. The content validity indexes of individual items, with the exception of three items, were all acceptable and estimated at being between 0.857 and 1.000. The modified Kappa index also showed excellent content validity for most items (scores between 0.856 and 1.000). The exceptions were items Q7, Q11, and Q24, which were re-examined and harmonized on the basis of experts’ comments (Intimacy, being female/male: “Zasebnost/ intima, biti ženska/moški” → “Intimnost v odnosu, ženska/ moški”; The nursing diagnosis is reformulated: “Negovalna diagnoza je preoblikovana” → “Negovalna diagnoza je spremenjena”). In item Q11, a grammatical error was corrected. The revised items were sent for re-evaluation to experts (n=5). All new content validity indexes of individual items that were subsequently adjusted were estimated at 1.000. 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 20 Regarding discriminant construct validity, there was statistically significant difference between the two groups of documentations in means for the sum of all 29 items (t=39.56, p<0.001). Mean scores for group one (electronic records with standardized terminology for nursing diagnoses) were 47.8±6.2 (95% confidence interval 46.3-49.3) and for group two (paper-based records and no standardized terminology) 29.4±7.7 (95% confidence interval 27.6-31.3). In the last, sixth, phase, i.e. “validity testing”, we performed calculations of the reliability analysis (Table 1). The value of the Cronbach coefficient for the whole Q-DIO instrument 1. Actual situation leading to the hospitalization 2. Anxiety and worries related to hospitalization, expectations and desires about hospitalization 3. Social situation and living environment/circumstances 4. Coping in the actual situation / with the illness 5. Beliefs and attitudes about life (related to the hospitalization) 6. Information of the patient and relatives/significant others about the situation 7. Intimacy, being female/male 8. Hobbies, activities for leisure 9. Significant others (contact persons) 10. Activities of daily living 11. Relevant nursing priorities according to the assessment 12. Nursing problem/nursing diagnosis label is documented 13. Nursing diagnosis label is formulated according to NANDA and numbered 14. The etiology (E) is documented 15. The etiology (E) is correct, related /corresponding to the nursing diagnosis (P) 16. Signs and symptoms are formulated 17. Signs and symptoms (S) are correctly related to the nursing diagnosis (P) 18. The nursing goal relates /corresponds to the nursing diagnosis 19. The nursing goal is achievable through nursing interventions 20. Concrete, clearly named nursing interventions according to NIC are planned (what will be done, how, how often, who does it) 21. The nursing interventions affect the etiology of the nursing diagnosis 22. Nursing interventions carried out, are documented (what was done, how, how often, who did it) 23. Acute, changing diagnoses are assessed daily or form shift to shift/enduring diagnoses are assessed every fourth day 24. The nursing diagnosis is reformulated 25. The nursing outcome is documented 26. The nursing outcome is observably /measurably documented according to NOC 27. The nursing outcome shows - improvement in patient’s symptoms - improvement of patient’s knowledge state - improvement of patient’s coping strategies - improved self-care abilities - improvement functional status 28. There is a relationship between nursing-sensitive patient outcomes and nursing interventions 29. Nursing outcomes and nursing diagnoses are internally related 1.16 0.27 1.04 0.68 1.21 0.85 1.31 0.20 1.79 1.69 0.96 3.39 1.91 1.00 0.74 0.00 0.00 3.39 1.65 1.23 0.35 3.73 3.37 1.73 1.49 0.00 0.84 1.55 1.08 0.692 0.621 0.548 0.833 0.560 0.795 0.507 0.436 0.487 0.510 0.535 0.964 1.940 1.080 0,828 0.000 0.000 0.774 1.292 0.851 0.493 0.534 0.977 1.280 1.000 0.000 0.949 1.889 1.258 -0.270 0.377 0.000 0.468 0.361 0.636 0.558 0.349 0.043 -0.347 0.086 0.768 0.722 0.757 0.724 0.000 0.000 0.612 0.661 0.766 0.594 -0.213 0.769 -0.097 0.752 0.000 0.501 0.412 0.412 0.870 0.857 0.864 0.854 0.857 0.850 0.854 0.858 0.863 0.869 0.862 0.844 0.844 0.843 0.847 0.860 0.860 0.850 0.845 0.845 0.854 0.867 0.844 0.876 0.844 0.860 0.853 0.860 0.856 Table 1. Item analysis of the Quality of Nursing Diagnoses, Interventions, and Outcomes Instrument. Item mean Standard deviation Corrected Item-Total Correlation Cronbach’s Alpha if Item Deleted Subscale / Item was 0.860, which corresponds to recommendations. An analysis of Cronbach’s alpha if item deleted remains above 0.80 for all items. The average values of items Q1 to Q11, measured on a 3-point Likert scale, were less than 1.6, except for items Q9 “Significant others (contact persons)” and Q10 “Activities of daily living”, which exceeded the recommended average value. The average values of most other items measured on a 5-point scale were less than 3.2, which is in accordance with the recommendations. Minor deviations were observed in three items – Q12, Q18, and Q23, and a moderate deviation in Q22 “Nursing interventions were documented”. When reviewing the corrected item total correlations of individual items, several deviations can be observed in the first part of the instrument (items Q1 to Q11), where they are below the recommended value (r<0.3) Q1, Q3, Q9, Q10 and Q11. In other parts of the instrument (items Q12 to Q29), deviations were found only in Q22 and Q24. Pearson’s correlation test showed excellent reliability or correlation of scores between two researchers (interrater reliability) (r=0.940, p<0.01), and the Kappa index showed good agreement of scores between two researchers (interrater agreement) (κ=0.866, p<0.01), which ensures the instrument’s good stability. 4 DISCUSSION The purpose of our research was to localize the Slovenian version of the measuring instrument Quality of Nursing Diagnoses, Interventions and Outcomes (Q-DIO). Localization was performed in six phases. The first four phases (forward translation, target language version synthesis and adaptation, backward translation, source language versions comparison and target adaptation) contribute to better semantic equivalence (27) and ensure the instrument’s professional integrity (20). Through the fifth and the sixth phases of our study, the content validity and internal reliability of the Slovenian version of the instrument was confirmed, but certain research limitations need to be taken into account in the interpretation. To assess content validity, we used the content validity index, which represents the degree of agreement of a group of experts on the relevance, integrity and balance of instrument items, and is recommended in the context of the instrument’s cultural adjustment and psychometric testing (27). The content validity index of the entire instrument was appropriate, as were the indexes of individual items. Three items that did not meet the required criteria were reviewed, harmonized, and re- evaluated by experts. The new indexes were all relevant. With discriminant construct validity we assessed the ability of the Q-DIO instrument to discriminate between different types of nursing records (electronic with standardized 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 21 terminology versus paper-based without standardized terminology). The analysis showed significant differences in the means of all instrument items among the two groups of documentations. Similar results were obtained in a study by da Costa Linch et al. (21), who validated a Q-DIO instrument for use in Brazil and in the United States. The internal reliability of the entire instrument was adequate, as shown by the calculation of the Cronbach coefficient. The item mean (item difficulty criterion) exceeds the recommended values for questions Q9 and Q10, while the corrected item total correlations are lower than recommended for items Q1, Q3, Q10, and Q11 for the sub-concept “Nursing diagnoses as process”. In the section “Nursing diagnoses as product”, two items, Q12 and Q18, exceed the recommended item mean value, while the corrected item total correlations are lower than recommended for items Q16 and Q17. As part of “Nursing interventions”, only item Q22 exceeded the recommended item mean value, as well as the recommended corrected item total correlation value. In the section “Patient outcomes responsive to nursing care”, item Q23 exceeded the item mean value, while the corrected item total correlation values are lower than recommended for items Q24 and Q26. Based on these results, items Q1, Q3, Q9, Q10, Q11, Q16, Q17, Q22, Q24, and Q26 did not show favourable results and could be suggested to be left out of the instrument. When examining these items, deviations are seen as the result of poor nursing documentation quality. This also indicates that performing and documenting a comprehensive nursing assessment, accurate nursing diagnoses with related, effective interventions and measurable patient outcomes, as well as using nursing classifications, is not yet the case in the study setting. Compared to other studies, items Q3, Q9, Q11 (6), and Q22 (13) also deviated from the recommended values. However, all Q-DIO items showed good psychometric properties in other studies (17, 18, 21). The authors therefore conclude that these aspects, e.g., a more detailed nursing history involving psycho-social aspects of the individual (Q1, 3, 9, 10+11), defining the nursing diagnosis according to PES (Q16+17), observably and measurably planned nursing interventions (Q22) and observably and measurably set nursing outcomes (Q24+26), should be implemented in clinical practice and education with greater emphasis. Since all items provide an overview of the quality of nursing documentation as a whole and the key items of the instrument, i.e. Q12, Q20, and Q25 (6), showed good results, we suggest that these items remain in the instrument. The results indicate that further implementation and training of nurses in the Advanced Nursing Process and its documentation, which is based on nursing classifications, are needed (11, 30, 31). The stability of the Q-DIO instrument was measured by interrater reliability and interrater agreement, which were both relevant and provide us with stable quality assessments even when Q-DIO is used by different assessors. 5 CONCLUSION We conclude that by following the process of localization and internationally recommended phases of transferring the measuring instrument between different cultural environments, we were able to obtain a validated and professionally suitable instrument for measuring the quality of nursing documentation in Slovenia. The results of comprehensibility, internal reliability, item analysis and stability showed good psychometric properties of the Slovenian Q-DIO instrument. However, due to the poorer test results of some items, additional testing of the instrument is recommended. CONFLICTS OF INTEREST The authors declare no conflict of interest. FUNDING The study received no funding. ETHICAL CONSIDERATIONS The study was approved by the Medical Ethics Committee of Slovenia on January 04, 2018 (approval No. 0120-536 / 2017/4). REFERENCES 1. Taylor L, LeMone L. Fundamentals of nursing: the art and science of nursing care. 7th ed. Philadelphia: Walters Kluwer, Lippincott Williams & Wilkins, 2011. 2. Perry AG, Potter P, Ostendorf W. Clinical nursing skills and techniques. 9th ed. St. Luis: Elsevier, 2018. 3. Blair W, Smith B. Nursing documentation: frameworks and barriers. Contemp Nurse. 2012;41(2): 160–8. doi: 10.5172/conu.2012.41.2.160. 4. Duclos-Miller PA. Improving nursing documentation and reducing risk. Brentwood: HCPro, 2016. 5. Petkovšek-Gregorin R, Skela-Savič B. Stališča medicinskih sester o pomembnosti dokumentiranja v zdravstveni negi. Obzor Zdrav Neg. 2015;49(2):106-25. doi: 10.14528/snr.2015.49.2.50. 6. Müller-Staub M, Lunney M, Lavin MA, Needham I, Odenbreit M, van Achterberg T. Testing the Q-DIO as an instrument to measure the documented quality of nursing diagnoses, interventions, and outcomes. Int J Nurs Terminol Classif. 2008;19(1):20-7. doi: 10.1111/j.1744-618X.2007.00075.x. 7. Beach J, Oates J. Maintaining best practice in record-keeping and documentation. Nurs Stand. 2014;28(36):45–50. doi: 10.7748/ ns2014.05.28.36.45.e8835. 8. Klančnik-Gruden M, Bregar B, Peternelj A, Marinšek N. Slovenska kategorizacija zahtevnosti bolnišnične zdravstvene nege (SKZBZN). Ljubljana: Zbornica zdravstvene in babiške nege Slovenije, Zveza strokovnih društev medicinskih sester, babic in zdravstvenih tehnikov Slovenije, 2011. 9. Gordon M. Negovalne diagnoze: priročnik. Radizel: Kolaborativni center SZO za primarno zdravstveno nego, Zdravstveni dom dr. Adolfa Drolca Maribor, 2003. 10. Rajkovič U, Milavec-Kapun M, Dinevski D, Prijatelj V, Zaletel M, Šušteršič O. The status of nursing documentation in Slovenia: a survey. J Med Syst 2016;40(9):198. doi: 10.1007/s10916-016-0546-x. 11. Müller-Staub M, Abt J, Brenner A, Hofer B. Expert report on nurses’ responsibility. Bern: Swiss Nursing Science Association (ANS), 2015. 12. Müller-Staub M, Lunney M, Odenbreit M, Needham I, Lavin MA, van Achterberg T. Development of an instrument to measure the quality of documented nursing diagnoses, interventions and outcomes: the Q-DIO. J Clin Nurs. 2009;18(7):1027-37. doi: 10.1111/j.1365- 2702.2008.02603.x. 13. da Costa-Linch GF, Müller-Staub M, Moraes MA, Azzolin K, Rabelo R. Cross-cultural adaptation of the Quality of Diagnoses, Interventions and Outcomes (Q-DIO) instrument into Brazilian Portuguese. Int J Nurs Knowl. 2012;23(6):153-8. doi: 10.1111/j.2047-3095.2012.01210.x 14. Lötscher R. Final Report: Translation process of the Quality of Diagnoses, Interventions and Outcomes (Q-DIO) instrument into French (F-Q-DIO). In: 10th Biennial Conference of the Association for Common European Nursing Diagnoses, Interventions and Outcomes. - eHEALTH AND NURSING: Knowledge for Patient Care. ACENDIO Conference Brochure 2015. Bern: Association for Common European Nursing Diagnoses, Interventions and Outcomes, 2015. 15. Ongaro C. L’Apprendimento del ragionamento diagnostico nei corsi di laurea in infermieristica: lo strumento Q-DIO e la valutazione delle pianificazioni dell’assistenza a scopo didattico: doctorate. Brescia: Universitá degli studi di Brescia, 2009. 16. Baudino M, Vernassa A, DePieri C, Massano S, Contini S, Boarino G, et al. Assessment and care plans based on standardized and computerized language: a retrospective study. In: 10th Biennial Conference of the Association for Common European Nursing Diagnoses, Interventions and Outcomes. - eHEALTH AND NURSING: Knowledge for Patient Care. ACENDIO Conference Brochure 2015. Bern: Association for Common European Nursing Diagnoses, Interventions and Outcomes, 2015. 17. Zanon D, Gralher D, Müller-Staub M. Interventionen zur Pflegediagnose „Akute Schmerzen“: Evaluation der schmerzlindernden Pflege bei hospitalisierten Patientinnen und Patienten nach Hüftendoprothesenversorgung im Vergleich zur Pflegedokumentation mittels Q-DIO-Pain: eine Mixed Methods Studie. Pflege. 2017;30(3):129- 38. doi:10.1024/1012-5302/a000533. 18. Löffler C. Mangelernährung im Akutspital (Masterthesis). Winterthur: ZHAW University, 2018. 19. Mateos-García MD, Müller-Staub M, Porcel-Gálvez A, Gil-García E, Ramos-González-Serna A, Barrientos Trigo S. Instrument for the assessment of the quality of Diagnoses, Interventions and Outcomes (Q-DIO): cross-cultural adaptation to Spanish. In: 10th Biennial Conference of the ACENDIO. eHealth and nursing: knowledge for patient care. Bern: Association for Common European Nursing Diagnoses, Interventions and Outcomes, 2015. 20. Lajovic J. Localization of measurement instruments: example of the AMS questionnaire. Zdrav Vestn 2008;77:461-5. 21. da Costa-Linch GF, Rabelo-Silva ER, Keenan GM, Moraes MA, Stifter J, Müller-Staub M. Validation of the Quality of Diagnoses, Interventions, and Outcomes (Q-DIO) instrument for use in Brazil and the United States. Int J Nurs Knowl. 2015;26(1):19-25. doi: 10.1111/2047- 3095.12030. 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 22 22. International Test Commission. The ITC guidelines for translating and adapting tests. 2nd ed, 2017. Accessed September 19th, 2020 at: https://www.intestcom.org/page/14. 23. Sperber AD. Translation and validation of study instruments for cross- cultural research. Gastroenterology. 2004;126(Suppl 1):S124-8. 24. Moravec-Berger D, editor. Mednarodna klasifikacija bolezni in sorodnih zdravstvenih problemov za statistične namene: MKB-10. 10th ed. Ljubljana: Inštitut za varovanje zdravja Republike Slovenije, 1995. 25. Rabelo-Silva ER, Dantas Cavalcanti AC, Ramos Goulart Caldas MC, de Fatima Lucena A, de Abreu Almeida M, da Costa Linch GF, et al. Advanced nursing process quality: comparing the International Classification for Nursing Practice (ICNP) with the NANDA-International (NANDA-I) and Nursing Interventions Classification (NIC). J Clin Nurs. 2017;26(3-4):379-87. doi: 10.1111/jocn.13387 26. Declaration of Helsinki – ethical principles for medical research involving human subjects. Fortaleza: 64th WMA General Assembly; 2013. Accessed September 19th, 2020 at: https://www.wma.net/ policies-post/wma-declaration-of-helsinki-ethical-principles-for- medical-research-involving-human-subjects/. 27. Polit DF, Beck CT. Nursing research: generating and assessing evidence for nursing practice. 10th ed. Philadelphia: Wolters Kluwer, 2017. 28. Vrbnjak D, Pahor D, Štiglic G, Pajnkihar M. Content validity and internal reliability of Slovene version of Medication Administration Error Survey. Obzor Zdrav Neg. 2016;50(1):20-40. doi: 10.14528/ snr.2016.50.1.69. 29. Polit DF, Beck CT. Nursing research: principles and methods. 7th ed. Philadelphia: Lippincott Williams & Wilkins, 2004. 30. Bruylands M, Paans W, Hediger H, Muller-Staub M. Effects on the quality of the nursing care process through an educational program and the use of electronic nursing documentation. Int J Nurs Knowl. 2013;24(3):163-170. doi: 10.1111/j.2047-3095.2013.01248.x. 31. Leoni-Scheiber C, Mayer H, Muller-Staub M. Relationships between the Advanced Nursing Process quality and nurses’ and patient’ characteristics: a cross-sectional study. Nurs Open. 2020;7(1):419-9. doi: 10.1002/nop2.405. 10.2478/sjph-2022-0004 Zdr Varst. 2022;61(1):14-23 23