ANALYSIS OF TWO DIVERSE NURSING RECORDS APPLICATIONS:  
MIXED METHODS APPROACH
Rok DRNOVŠEK1,3*, Marija MILAVEC KAPUN2, Vladislav RAJKOVIČ3, Uroš RAJKOVIČ3
1University Medical Centre Ljubljana, Department of Neurology, Zaloška cesta 2a, 1000 Ljubljana, Slovenia
2University of Ljubljana, Faculty of Health Sciences, Zdravstvena pot 5, 1000 Ljubljana, Slovenia
3University of Maribor, Faculty of Organizational Sciences, Kidričeva cesta 55a, 4000 Kranj, Slovenia
Received: Jul 9, 2021
Accepted: Apr 4, 2022
Original scientific article
*Corresponding author: Tel. + 386 31 401 935; E-mail: rok.drnovsek@kclj.si
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144
137
ANALIZA DVEH APLIKACIJ ZA DOKUMENTIRANJE ZDRAVSTVENE NEGE:  
MEŠANI METODOLOŠKI PRISTOP
© National Institute of Public Health, Slovenia. 
Drnovšek R, Milavec Kapun M, Rajkovič V, Rajkovič U. Analysis of two diverse nursing records applications: mixed methods approach. 
Zdr Varst. 2022;61(3):137-144. doi: 10.2478/sjph-2022-0019.
ABSTRACT
Keywords: 
nursing process, 
information technology, 
documentation, 
user experience, 
quantitative/
qualitative approach
IZVLEČEK
Ključne besede: 
proces zdravstvene 
nege, informacijska 
tehnologija, 
dokumentacija, 
uporabniška izkušnja, 
kvantitativno-
kvalitativni pristop
Introduction: Poor adoption of electronic health records among healthcare workers can diminish their impact. 
Healthcare informatics solutions development should diligently acknowledge end-user needs. This study 
compares a user experience and perceived quality of the nursing process integration in two different applications 
for electronic documentation of the nursing care plan. Both applications were designed and tested in Slovenia.
Methods: In the first phase, final year undergraduate nursing students were recruited (n=73) and randomly 
assigned into two groups. Each group used one of the applications for a duration of five hours. A survey 
among students was conducted. In the second phase, additional students were recruited (n=40) and invited to 
participate in qualitative analysis of the unfavourably rated application.
Results: The modern, visually improved application was favourably rated by students in terms of all aspects of 
application usability. However, students reported a significant number of inadequacies regarding the nursing 
process methodology integration. On the contrary, the students using the poorly rated and visually outdated 
application reported no such concerns. Qualitative analysis of student reflections identified additional positive 
features of software design that were not detected in survey results analysis.
Conclusion: This study showed that a user-centred approach can be used to compare diverse electronic 
solutions. Detected discrepancies in findings using qualitative and quantitative analysis show the importance 
of integrating diverse research approaches for adequate evaluation of software solutions. Furthermore, this 
study design promotes empowerment of healthcare workers to participate in the development and critical 
evaluations of software solutions.
Uvod: Neustrezna zasnova rešitev za elektronsko dokumentiranje v zdravstvu lahko poveča odpor zdravstvenih 
delavcev do njene uporabe pri delu in posledično zmanjša njihov doprinos k izboljšanju klinične prakse. 
Implementacija novih informacijskih rešitev v zdravstvu mora zato temeljiti na identifikaciji in zadovoljevanju 
potreb končnih uporabnikov in slediti sodobnim strokovnim smernicam. Namen raziskave je bil z osredotočenjem 
na uporabnika primerjati uporabniško izkušnjo in strokovno ustreznost dveh različnih aplikacij za dokumentiranje 
zdravstvene nege.
Metode: V raziskavo so bili vključeni redni in izredni študenti zadnjega letnika dodiplomskega študija zdravstvene 
nege (n = 73), ki so bili naključno razporejeni v dve skupini, vsaka je uporabila eno izmed aplikacij. Delo je potekalo 
v računalniški učilnici Zdravstvene fakultete Univerze v Ljubljani v obsegu petih ur. Po predstavitvi osnovnih 
funkcionalnosti so študenti samostojno uporabljali aplikacijo za dokumentiranje intervencij zdravstvene nege 
in oblikovanje negovalnih diagnoz. Po zaključku uporabe so izpolnili spletni anketni vprašalnik, kjer so podali 
oceno uporabnosti. Dodatno so bili v raziskavo vključeni še študenti (n = 40), ki so si ogledali videoposnetek, na 
katerem je izkušeni uporabnik predstavil funkcionalnosti slabše ocenjene aplikacije. Po ogledu so o uporabnosti 
aplikacije v zdravstveni negi podali pisno kritično oceno in refleksijo, ki sta bili kvalitativno analizirani.
Rezultati: Študenti so novejšo in vizualno privlačnejšo aplikacijo ocenili kot boljšo v vseh sklopih uporabljenega 
vprašalnika, vendar so v odgovorih na odprta vprašanja navedli tudi nekatere njene pomembne pomanjkljivosti. 
Te so bile povezane predvsem z integracijo procesne metode dela v zasnovo aplikacije. Študenti, ki so uporabljali 
starejšo in vizualno manj privlačno aplikacijo, takšnih pomanjkljivosti niso navajali, a so kljub temu aplikacijo 
ocenili slabše v vseh vidikih ocenjevanja njene uporabnosti. V refleksijah so študenti identificirali nekatere 
pomembne pozitivne lastnosti aplikacije in izpostavili njene slabosti, ki bi lahko prispevale k slabši oceni 
aplikacije glede na predhodno izvedeno anketo.
Zaključki: Raziskava kaže, da lahko z osredotočenostjo na uporabnika uspešno primerjamo različne rešitve 
za elektronsko dokumentiranje v zdravstvu. Pomemben rezultat raziskave so tudi neskladnosti med rezultati, 
pridobljenimi s kvalitativnimi in kvantitativnimi pristopi, ki kažejo na potrebo po integraciji različnih pristopov 
raziskovanja v evalvacijo elektronskih rešitev v zdravstvu. Rezultati tudi opozarjajo, da lahko velike razlike v 
kakovosti vizualne zasnove in preprostosti uporabe vplivajo na verodostojnost uporabnikove ocene strokovne 
ustreznosti. Za doseganje čim večje uporabnosti in ustrezne prilagoditve programske opreme morajo biti 
uporabniki vključeni v vse faze razvoja informacijskih rešitev v zdravstvu. Metodologija predstavljene raziskave 
spodbuja opolnomočenje zdravstvenih delavcev za sodelovanje v razvoju in evalvaciji digitalnih rešitev.
1 INTRODUCTION
The first healthcare computer-based information systems 
were used mainly for physician order entries, test results 
reporting, financial management and pharmaceutical 
and radiological activities support (1). Lately, healthcare 
organisations have been adopting information systems 
capable of storing and managing all clinical practice data. 
The integration of modern information and communication 
technologies (ICT) in patient care and the use of electronic 
health records (EHR) for health data management have 
shown some positive impacts on healthcare quality 
and patient outcomes (2-4). Nevertheless, ICT design 
should be adequately adapted for the specific clinical 
environment. In a study of 3900 hospitals in the United 
States, ICT implementation showed no positive effects on 
mortality, cost savings, adverse drug events or readmission 
rates. The reasons for this were attributed to a lack of 
organisational changes, poor adoption and poor utilisation 
among healthcare professionals (5). Efficient strategies 
for the implementation of user-friendly ICT solutions are 
crucial and should consider healthcare workers’ needs, to 
reduce their resistance to adoption and ensure optimal 
staff support for the best clinical results (6). 
The nursing process is a fundamental method for planning 
and providing integrated nursing care, enabling nurses to 
identify patients’ needs and plan nursing care for those 
needs. The nursing process is cyclical and consists of four 
to six phases according to different sources. Our study used 
the five-phase nursing process in the theoretical background 
consisting of assessing, diagnosis, planning, implementing 
and evaluating (7). This definition of the nursing process 
establishes the nursing diagnosis as an individual phase. 
The organisation NANDA International, Inc. (NANDA-I) 
defines the nursing diagnosis as a clinical judgment about 
individual, family, or community responses to actual or 
potential health problems or life processes, forming the 
basis of nursing care planning (8). The entirety of the 
nursing process is reflected in nursing documentation as 
documented nursing health history, nursing diagnosis, 
nursing care plan, nursing-related outcomes and planned 
interventions (6, 7).
Electronic nursing record (ENR) systems are designed 
for electronic documentation of nursing-specific data 
and should enable documentation of nursing care plans 
according to the nursing process. The nursing process 
contributes significantly to nurses’ participation in 
patients’ treatment, but in Slovenia, it is often limited to 
education and theory and is not adequately represented in 
contemporary clinical nursing documentation (9). It should 
be fully implemented in contemporary patient care and 
ENR systems (10), enabling an efficient implementation 
of the nursing process in clinical practice to improve the 
quality of healthcare, improve nursing documentation and 
support holistic interdisciplinary patient care (11). 
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144
138
New technology acceptance can be enhanced if a user-
centred approach is used in product development and 
evaluation (12). A user-centred approach has been used 
before, to evaluate the perceived usability of diverse 
health-related electronic systems’ functionalities and user 
satisfaction (13, 14). It could also be employed to assess 
and improve the adequacy of nursing process methodology 
integration and usability of diverse ENR systems. An 
extensive review of software usability evaluation studies 
reported that the most commonly used methods are 
usability testing, heuristic evaluation and questionnaire 
methods, while also highlighting the importance and lack 
of studies using both qualitative and quantitative methods 
for evaluation of usability (15).  
In this study, qualitative and quantitative methods were 
used to conduct a user-centred analysis of two Slovenian 
electronic documentation applications that differed in 
basic functionalities, graphical interface and nursing 
process incorporation. Our primary objectives were to 
analyse the opinions and attitudes of end-users regarding 
the usability of two systems and the ability of those 
systems to support the nursing process.  Our secondary 
objectives were to generate a list of key system features 
to support usability and to demonstrate selected methods 
of evaluating systems.  
2 METHODS
Quantitative and qualitative methods were used to 
conduct a user-centred analysis of two diverse applications 
for documentation of nursing practice. To achieve an in-
depth analysis of results an explanatory sequential mixed 
method was used.
2.1 Research participants
All students in the last, sixth, semester of an 
undergraduate nursing programme at the University of 
Ljubljana, Faculty of Health Sciences were recruited 
(n=73) and randomly assigned into two groups by a 
coin toss. Among participating students, 61 of them 
submitted a completed survey after interacting with the 
evaluated application. Of 61 students, 53 were women 
and 8 were men, most of them using ICT every day (n=58; 
95%). On computer-related skills, 56% of students rated 
their skills as “competent”, 26% as “advanced” and 18% 
as “beginner”. Most students had never used software 
designed for documentation of healthcare-related 
data (n=50; 82%). Among the students that completed 
the survey, 26 students used application B and 35 
students used application A. No significant differences 
in demographic data were detected between the groups.
A qualitative analysis of the unfavourably rated application 
followed. For qualitative analysis, an additional 40 students, 
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144
139
who did not participate in previous study procedures, were 
invited. Among them, 26 students agreed to participate in 
the study, with 25 of the submitted student reflections 
being suitable for analysis.
2.2 Data collection
Quantitative data for the conducting of our study were 
gathered using a survey. Each group used one of the 
applications during the study for a total of five hours. First, 
the lecturer (one of the researchers) guided the students 
through the creation of a uniform nursing care plan to 
provide basic knowledge concerning the functionalities of 
the application used. Students then designed a nursing care 
plan by themselves, for one patient they had encountered 
during clinical placement in community settings, and 
prepared a semi-structured written case report.
After completion, an on-site anonymised online survey 
among the students was conducted, regarding the user 
experience and the perceived suitability for quality 
nursing care plan documentation. The questionnaire 
contained 47 statements regarding seven different aspects 
of the application use. Participants rated statements 
regarding the application’s speed, visual design, overall 
impression and learnability on a nine-point semantic 
differential table. The sets of statements and their design 
were adopted from a questionnaire previously already 
used in a national research project (16) for evaluating ICT 
user experience. The statements regarding suitability for 
patient data documentation, suitability for nursing care 
plan formulation and satisfaction with the formulated 
nursing care plan were specific for the used applications 
and were designed for this study. The students were 
asked to express their agreement with statements using 
a five-point Likert scale (strongly disagree, disagree, 
neither agree nor disagree, agree, strongly agree). The 
questionnaire also included demographic questions and 
two open-ended questions, in which students could 
express up to three best and three worst attributes of the 
application. Beforehand the questionnaire was evaluated 
by an interdisciplinary expert group and pilot tested with 
11 students. Statistical analysis of the pilot study was 
not carried out. Following the students’ interaction with 
the questionnaire, a short focus group was undertaken. 
Students reported fatigue when evaluating a large number 
of statements on a single page of the questionnaire. The 
questionnaire was restructured so that a lower number of 
statements were presented per page of the questionnaire.
For an in-depth understanding of the students’ attitudes, 
further qualitative analysis of the unfavourably rated 
application was undertaken. Students were provided with 
written key information regarding the application and a 
demonstrational video featuring an experienced user 
formulating a nursing care plan. The students were then 
asked to submit their opinions related to the application 
observed in the video in the form of a reflection. 
Participants were instructed to write a holistic evaluation 
and were encouraged to be honest and critical regarding 
the observed application. 
2.3 Data processing
The software IBM SPSS Statistics 23 was used for data 
analysis. Data were not distributed normally according to 
the Shapiro-Wilk test; therefore in the results section we 
report the median (M) and the first and third quartile values 
(Q1–Q3) for representation of data, and we used the Mann-
Whitney U test for comparison of the included applications. 
Before aggregating the data, the internal consistency of 
the individual cluster of content-related statements was 
estimated using Cronbach’s alpha coefficient. Statistical 
significance was acknowledged for p values lower than 
0.05. Responses to open-ended questions were analysed 
with a deductive qualitative approach using appropriate 
predefined code categories (17), which were defined 
according to the application usability aspects included in 
previous items of the questionnaire. For qualitative data 
analysis of students’ written reflections, thematic analysis 
of a text was used (18). All suitable student reflections were 
analysed. The saturation of data was reached with 19 units.
2.4 Evaluated applications
Two applications were compared in this study: 
e-Documentation for nursing care and Clinical Information 
System, referred to as applications A and B respectively. 
In this study, the versions of the applications used were 
designed exclusively for educational purposes and 
contained no actual clinical data, but were identical to their 
clinical practice counterparts in design and functionalities.
Application A is an ENR system developed as a joint effort 
of academic and clinical experts. Under development since 
2002, it presents an information model for organisation 
and management of patient data exclusively for nursing 
care (9). It aims to provide a professionally suitable ICT 
model to promote nursing quality (19, 20). The students 
formulated the nursing care plan by documenting the 
patient’s needs according to an adapted Henderson 
theoretical model (21), and used the problem aetiology and 
signs/symptoms principle for nursing diagnosis formulation 
(7). All subsequent elements of the nursing care plan were 
manually typed into predefined dialogue boxes. 
Application B was designed as a complete EHR system 
for healthcare and administrative data management in a 
hospital setting. Deployed for clinical use in 2011, it aims 
to minimise documentation time, make data management 
more efficient and prevent adverse drug events (22). The 
students formulated the nursing care plan by selecting 
desired statements from a list of NANDA International, 
Inc. (NANDA I) standardised nursing diagnoses organised 
140
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144
Overall impressiona 
Learnabilitya 
Operabilitya 
Visual designa 
Usability for patient 
data documentationb
Usability for nursing care 
plan formulationb
Satisfaction with formulated 
nursing care planb
Application usability aspect Application M (Q1–Q3)Cronbach’s alpha Mann-Whitney U 
A 
B
A 
B
A 
B
A 
B
A 
B
A 
B
A 
B
4.0 (3.0–5.0) 
7.0 (5.0–8.0)
5.0 (2.5–6.0) 
7.0 (5.0–8.0)
5.0 (3.0–5.5) 
6.8 (5.5–8.0)
5.0 (3.8–6.5) 
7.0 (6.0–9.0)
3.0 (2.8–4.0) 
4.0 (4.0–4.5)
3.0 (2.6–4.0) 
4.0 (4.0–5.0)
3.0 (3.0–4.0) 
4.0 (3.5–4.0)
0.942 
0.943 
0.849 
0.906 
0.938 
0.950 
0.948
p<0.001 
p=0.002 
p<0.001 
p<0.001 
p=0.002 
p=0.001 
p=0.015 
Table 1. Comparison of survey results for the observed aspects of application use.
a Statements regarding the overall impression, learnability, operability and visual design were rated on a nine-point differential table. 
b Statements regarding usability for patient data documentation, usability for nursing care plan formulation and satisfaction with the 
formulated nursing care plan were rated on a five-point Likert scale. A higher score indicates greater favourability towards that aspect 
of application use. M – Median, Q1–Q3 – interquartile range.
according to Gordon’s functional health patterns (23). 
They then selected appropriate aetiological factors, 
nursing-related outcomes and interventions from lists, 
tailored according to the selected nursing diagnosis. 
Nursing interventions, nursing-related outcomes and 
aetiological factors are preformulated but not linked with 
standardised nursing language. 
2.5 Consent to conduct research
All participating students were informed beforehand 
about the nature of the study, participated voluntarily and 
were able to withdraw from the study at any point. This 
research project involved human participants, therefore 
approval from an Ethical Committee for Research was 
obtained beforehand.
3 RESULTS
Of 61 students, 53 were women and 8 were men, most of 
them using ICT every day (n=58; 95%). On computer-related 
skills, 56% of students rated their skills as “competent”, 
26% as “advanced” and 18% as “beginner”. Most students 
had never used software designed for documentation of 
healthcare-related data (n=50; 82%). Among the students 
that completed the survey, 26 students used application B 
and 35 students used application A. No significant differences 
in demographic data were detected between the groups.
The user experience and perceived quality of the nursing 
process methodology integration were addressed in 47 
statements rated by students. The aggregated results 
for the observed aspects of each application’s use are 
presented in Table 1.
Although application B was rated favourably in all aspects 
of application usability, students’ answers to open-ended 
questions showed different results. Participants were 
asked to identify the three best and three worst attributes 
of the application used. Students reported 258 attributes: 
136 concerned application B and 122 application A. 
Figure 1 represents the number of reported positive and 
negative attributes for both applications, thematically 
categorised according to the questionnaire-based aspects 
of application usability.
The most frequently reported positive attributes of 
application B were the transparent overview of data and 
documentation speed, while the most frequent negative 
attributes were inappropriateness for older staff. Regarding 
the nursing care plan formulation, the students reported 
positive attributes that included promotion of holistic 
nursing care plan formulation, integration of preformed 
listed terms and simplicity of the formulation process. 
Reported negative attributes regarding the nursing care 
plan formulation included difficulty in navigating listed 
terms and scarce selection of listed terms. Additionally, 
participants reported some professional concerns regarding 
application B, which included inadequate individualization 
of a nursing care plan, inappropriate listings of preformed 
terms and inadequate patient data security. 
141
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144
Figure 1. Quantitative representation of reported negative and positive attributes for the observed applications.
The most frequently reported positive attributes 
of application A were nursing needs assessment 
transparency, while the most frequent negative attributes 
were documentation speed, outdated visual design and 
inability to correct documentation errors. Regarding the 
nursing care plan formulation, the students reported 
positive attributes that included integration of the nursing 
care process, promotion of holistic nursing care plan 
formulation and support of individualized formulation of 
nursing diagnoses. Only one negative attribute regarding 
the nursing care plan formulation process in application A 
was detected. A student reported difficulties formulating 
adequate nursing diagnosis using the problem, aetiology 
and signs/symptoms principle. 
Unfavourably rated application A was included in additional 
in-depth analysis. In a qualitative analysis of students’ 
reflections, we used open coding thematic analysis of text 
to formulate codes, which were then grouped into eight 
code categories and three themes, as shown in Table 2.
Professional 
suitability
Functionalities 
and design
Improvements 
recommendations
Theme Category Codes
Nursing theory 
compatibility
Flexibility
Structured data 
representation
Non-user-friendly 
functionalities
Poor visual design
Time-consuming
Functionalities 
enhancements 
Ensure user training
Follows the nursing process, Holistic, Universal, Supports nursing 
theory, Secure, Relatable to existing paper-based documentation
Flexible for enabling free text, Flexible for allowing limited data 
documentation, Diverse nursing diagnosis creation, Individually fitted 
nursing care plan, Efficient compared to paper-based documentation, 
Clear data organization, Colour coded problem documentation, Easy to use, 
Inhibits incompleteness of documented data, Supports patient progress 
tracking, Practical graphical representations, Clear patient data overview
Does not automatically save data, Does not flag logical inconsistencies, Does 
not provide pre-structured elements of a nursing care plan, Unintuitive to use, 
Poor standardized languages integration, Poor overview of medical data
Monotonous colour scheme, Outdated visual design, 
Sharp edges, Small buttons, Small font
Does not provide pre-structured elements of a nursing care 
plan, Unintuitive to use, Overuse of manual typing
Include images documentation, Improve interdisciplinary use, 
Formulate data to enable research, Include risk screening tools, Include 
notifications and warnings, Include standardized nursing languages 
Provide nursing process training, Provide user training
Table 2. Detected themes codes categories and codes in qualitative analysis of submitted student opinions.
4 DISCUSSION
The study participants comprised a relatively homogeneous 
student group that frequently uses ICT but rarely for 
healthcare-related data documentation. Although the 
students were in the final semester of a nursing science 
degree, a large percentage (82%) had never used any 
application for patient data documentation. This indicates 
the lack of implementation of nursing informatics in the 
Slovenian nursing curriculum, which should be addressed 
in response to the increasing importance of informatics 
competencies in a modern clinical environment (24). In their 
reflections students also reported the need for staff training 
as a recommendation for improved ENR implementation. 
Application B provided a better user experience and 
was perceived more suitable for nursing care plan 
documentation, since according to our survey it was 
rated favourably in all seven analysed aspects of 
application usage. According to the questionnaire results, 
application B performed better in operability, learnability, 
attractiveness and understandability, factors that in 
literature characterise greater software usability (25). 
The application is visually well designed and enables data 
input with minimal manual typing, making the application 
fast and easy to use. Its visual design features a colourful 
user interface with large letters and tabs. Some patient 
data can be documented using interactive images, and 
vital signs can, without additional effort, be presented 
in charts or tables (22). On the other hand, the user 
interface of application A, last updated in 2005, is poor in 
visual design and requires an excessive amount of manual 
typing. Answers to the open-ended questions also pointed 
out the superior visual design of application B, which 
could be associated with an improved user experience and 
would explain the preference for it in the survey results. 
Open-ended questions results show that students using 
application B were predominantly dissatisfied with the 
level of individualisation of the nursing care plan and 
reported inadequate collections of listed aetiological 
and diagnostic terms. This points to an incomplete 
implementation of nursing standardized languages, which 
are normally associated with an increase in nursing 
care plan quality (26, 27). Based on the participants’ 
answers to open-ended questions, we could assume a 
more satisfactory integration of the nursing process in 
application A. Students using this application reported 
predominantly positive attributes related to nursing care 
plan formulation. Previous research (19) on the use of 
electronic documentation application A showed that its 
design prompts the user towards consistent documentation 
of individualised aetiological and diagnostic terms related 
to nursing diagnosis statements. Although application B 
was more recently developed and represents a modern 
EHR system, we can speculate that its design focuses on 
simplicity of use, visual design and incorporation of user 
142
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144
The students were dissatisfied with the visual design and 
some functionalities of the application. Regarding visual 
design, the colour scheme seems to play an important 
role, as one of the students wrote: 
“The programme would benefit from some aesthetic 
improvements since the outlook should feature livelier 
colours as opposed to the current grey design. I would 
recommend for example light blue, white or light green. 
Something unobtrusive but friendly to the user’s eyes.”
Besides poor visual design, the reported limitations of 
the application’s functionalities were focused on time 
consumption and the amount of manual typing required 
for documenting patient data. Both those aspects can be 
summed up by a student that wrote: 
“In the majority of cases documentation of data requires 
manual input of nursing care plan elements, which takes 
the healthcare worker an enormous amount of time.”
Additionally, the qualitative analysis provided some 
information regarding the positive aspects of application 
A. In students’ opinions, the nursing process is suitably 
integrated into the inner workings of the applications, as 
two students wrote: 
“The application does a good job of guiding the user 
through the stages of nursing documentation. All the parts 
of the documentation (anamnesis, needs evaluation, goals 
and so on) are clearly designed and the user is able to 
easily switch between different parts of the application.”
“In my opinion, the application is easy to use since it is 
clearly structured, understandable and guides you through 
the nursing care process.”
Additionally, participants reported some professional 
concerns. Concerning reported inadequacies of the 
application, the students suggested a stricter use of 
standardized nursing languages. In their opinion, the use 
of preformed statements and standardized languages such 
as NANDA-I is beneficial for efficient documentation and 
promotion of quality care. As examples of this, students 
wrote that the use of preformed statements could 
decrease the time needed for documentation and improve 
the quality of documented data. 
“What I believe is even more important is the implementation 
of NANDA diagnosis into the application, which would 
additionally ease users’ workload and save time.”
“We should consider abolishing the option of nursing 
diagnosis creation according to the problem, aetiology 
and symptoms/signs principle. Manual creation of nursing 
diagnosis is not suitable for in-depth statistical analysis 
and objective patient condition evaluation, since there is 
no universal way of interpreting data.”
As seen above the students recognized standardized 
nursing languages to be important in ensuring suitable 
formulation of patient data for research purposes. 
needs while not ensuring patient individualisation and 
adequate integration of the nursing process.
Qualitative analysis of written reflections further confirms 
that visual design and time needed for documentation 
significantly influenced the perceived professional 
suitability of the evaluated applications. Students 
recognized numerous positive aspects of application A, 
predominantly regarding the organization and presentation 
of nursing-specific data and nursing care plans. This leads us 
to believe that the overwhelmingly inferior survey score of 
application A could be a consequence of poor visual design 
and excessive need for manual typing. Specifically, students 
disliked the sharp edges, small font and monotonous colour 
scheme that made the overall appearance of application 
A look outdated. Time spent documenting was another 
concern regarding the application use, with students 
urging the use of preformed statements and integration 
of standardized nursing language for faster and easier 
formulation of nursing care plans. 
A user-centred approach can be used to evaluate the user 
experience and quality of nursing process integration in 
diverse ENR systems. Furthermore, our study highlights 
the importance of using qualitative and quantitative 
approaches in software evaluation.
Firstly, diverse opinions regarding the aspects of evaluated 
software solutions are difficult to adequately include in a 
structured questionnaire. While the main aspects of the 
applications’ use were included in the structural parts of 
the questionnaire, some were not predicted. For example, 
in open-ended questions, the students using application 
B reported concerns about inadequate patient data 
safety. Students did not consistently give reasons for their 
concerns, but one of the students reported that enabled 
access to other hospitalised patients indicated a lack of 
patient data security. Such functionality is not enabled in 
application A, for which no such concerns were detected. 
This aspect of application use was not included in any 
item of the structured part of the questionnaire and could 
therefore not be subjected to quantitative analysis.
Secondly, the survey results could prove to be misleading. 
Answering open-ended questions, students using application 
B reported the inadequate scope for individualising nursing 
care plans, which was not evident in the rated statements 
of the questionnaire. Although the questionnaire included 
items regarding the applications’ usability for the creation 
of an accurate and individually appropriate nursing 
diagnosis, the students did not rate these aspects of the 
application significantly lower. The qualitative analysis 
of student reflections also confirmed that students were 
satisfied with multiple aspects of application A that include 
data organisation, holistic care promotion, nursing process 
integration and so on, however, this was not evident in 
the survey results. Again, the students did not rate the 
applications’ usability for nursing care plan formulation 
and quality of formulated nursing care plan significantly 
higher in comparison to items describing visual design in 
rated statements of the questionnaire. 
This could indicate that students might have reported 
a relatively lower level of favourability for all aspects 
of application use, in response to the visually poorly 
designed and slowly working application, while in the 
case of application B assigning relatively higher levels 
of favourability in response to the novel aesthetically 
appealing design. This is supported by the undesirably 
high internal consistency coefficient of the entire 
questionnaire (Cronbach’s alpha=0.97). These concerns 
are a significant result of our study since they indicate 
that future user-centred evaluations of software solutions 
should be carried out and interpreted cautiously. This 
study also highlights the importance of empowering end-
users for competent participation in the development and 
critical evaluations of future digitalization of the nursing 
process. Users may be unable to objectively evaluate ENR 
systems’ professional suitability if they are influenced by 
a poor or excellent software design and therefore provide 
misleading results. Open-ended questions and in-depth 
qualitative analysis should be included in future studies 
since they can provide relevant information regarding the 
professional suitability of evaluated software solutions 
that otherwise cannot be adequately addressed in a 
structured quantitative questionnaire. 
The study limitations include the low number of 
participants and its educational setting. Participating 
students lacked clinical expertise and did not use the 
evaluated applications before the study took place. This 
made them unable to provide the best possible feedback 
while evaluating the software in a strictly controlled 
environment. Each student had access to a computer, 
was familiar with the patient beforehand and had no 
time restriction while documenting patient data.  Our 
study could be improved if the students were placed in a 
simulated environment and interacted with the evaluated 
applications for a longer time. This could better mimic 
the intended use of the application in actual clinical 
practice by an experienced user. Future research should 
take place in a clinical environment among a larger 
number of experienced users. An important application 
usability aspect is usage in the actual clinical environment 
scenarios, where users may be faced with a lack of 
appropriate hardware, environmental disturbing factors 
and time constraints (28). Furthermore, participants in 
this study were limited to a single university. Nevertheless, 
this study offers important insights for future nursing 
documentation applications development. 
143
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144
5 CONCLUSION
We used a user-centred approach to compare two 
diverse applications for electronic documentation of the 
nursing process. The results of this study provide new 
information on the user-centred evaluation of nursing-
specific applications for clinical practice documentation. 
Additional findings using qualitative analysis further show 
that future user-centred evaluations of software solutions 
must consider the possible inability of users to objectively 
rate professional suitability under the influence of a 
satisfactory or dissatisfactory experience using the 
evaluated software. Such studies could lead to the 
empowerment of healthcare professionals to participate 
in the development and critical evaluations of future 
digital solutions.
CONFLICTS OF INTERESTS
The authors declare that there is no financial, personal, or 
academic conflict of interest.
FUNDING
The authors acknowledge the financial support of the 
Slovenian Research Agency (research core funding No. P5-
0018). 
ETHICAL APPROVAL 
For the conducting of this study, ethical approval from the 
Ethical Committee for Research in Organizational Sciences 
at the Faculty of Organizational Sciences, University of 
Maribor was obtained (Document ID: 514-3/2020/4/902-DJ).
REFERENCES 
1. Brixey JJ, Brixey JE, Saba VK, McCormick KA. Essentials of nursing 
informatics study guide. New York: McGraw-Hill Education, 2015.
2. Iljaž R, Brodnik A, Zrimec T, Cukljati I. E- healthcare for diabetes 
mellitus type 2 patients – a randomised controlled trial in Slovenia. Zdr 
Varst. 2017;56:150–7. doi: 10.1515/sjph-2017-0020.
3. Plantier M, Havet N, Durand T, Caquot N, Amaz C, Biron P, Philip I, et al. 
Does adoption of electronic health records improve the quality of care 
management in France? Results from the French e-SI (PREPS-SIPS) study. 
Int J Med Inform. 2017;102:156–65. doi: 10.1016/j.ijmedinf.2017.04.002.
4. Kruse CS, Stein A, Thomas H, Kaur H. The use of electronic health 
records to support population health: a systematic review of the 
literature. J Med Syst. 2018;42:214. doi: 10.1007/s10916-018-1075-6.
5. Agha L. The effects of health information technology on the costs and 
quality of medical care. J Health Econ. 2014;34:19–30. doi: 10.1016/j.
jhealeco.2013.12.005.
6. De Groot K, De Veer AJE, Paans W, Francke AL. Use of electronic health 
records and standardized terminologies: a nationwide survey of nursing 
staff experiences. Int J Nurs Stud. 2020;104:103523. doi: 10.1016/j.
ijnurstu.2020.103523. 
7. Berman A, Snyder S, Frandsen G. Kozier & Erb’s fundamentals of 
nursing: concepts, practice, and process. Boston: Pearson, 2016.
8. Herdman, T H,  Kamitsuru S. Nanda International, Inc. Nursing diagnoses: 
definitions & classification 2018-2020. New York: Thieme, 2018.
9. Rajkovič U, Milavec Kapun M, Dinevski D, Prijatelj V, Zaletel M, Šušteršič 
O. The status of nursing documentation in Slovenia: a survey. J Med 
Syst. 2016;40:198. doi: 10.1007/s10916-016-0546-x.
10. Schachner MB, González ZA, Sommer JA, Recondo FJ, Gassino FD, Luna 
DR, et al. Computerization of a nursing chart according to the nursing 
process. Stud Health Technol Inform. 2016;225:133–7. doi: 10.3233/978-
1-61499-658-3-133.
11. da Costa C, Linch GFDC. Implementation of electronic records related 
to nursing diagnoses. Int J Nurs Knowl. 2020;31:50-8. doi: 10.1111/2047-
3095.12219.
12. Moyle W, Jones C, Pu L, Chen SC. Applying user-centred research 
design and evidence to develop and guide the use of technologies, 
including robots, in aged care. Contemp Nurse. 2018;54:1-3. doi: 
10.1080/10376178.2017.1438057.
13. Alexander S, Graham HL, Nussdorfer D, Beal R. Nurse attitudes 
related to accepting electronic health records and bedside 
documentation. Comput Inform Nurs. 2018;36:515-20. doi: 10.1097/
CIN.0000000000000491.
14. Kralj D, Kern J, Tonkovic S, Koncar M. Development of the quality 
assessment model of EHR software in family medicine practices: 
research based on user satisfaction. J Innov Heal Informatics. 
2015;22:340–58. doi: 10.14236/jhi.v22i3.158.
15. Sagar K, Saha A. A systematic review of software usability studies. Int J 
Inf Technol. 2017. doi:10.1007/s41870-017-0048-1.
16. Brodnik, A. E-podpora procesa zdravstvene oskrbe. Slovenia Research 
Agency. Accessed March 13th, 2021 at: https://www.dlib.si/stream/
URN:NBN:SI:DOC QMPTOCHE/bfaa544d-94f4-4b20-aed1-83156b9ec3ce/
PDF .
17. Armat MR, Assarroudi A, Rad M, Sharifi H, Heydari A. Inductive and 
deductive: ambiguous labels in qualitative content analysis. Qual. Rep. 
2018;23:219-21.
18. Castleberry A, Nolen A. Thematic analysis of qualitative research data: 
is it as easy as it sounds? Curr Pharm Teach Learn. 2018;10:807-15. doi: 
10.1016/j.cptl.2018.03.019.
19. Rajkovič U, Šušteršič O, Zupančič J. A model for e-documenting in 
community nursing. Slov Nurs Rev. 2006;40:197–204.
20. Šušteršič O, Rajkovič V, Leskovar R, Bitenc I, Bernik M, Rajkovič U. 
An information system for community nursing. Public Health Nurs. 
2002;19:184-90. doi: 10.1046/j.0737-1209.2002.19306.x.
21. Semenenko S, Boreiyko L, Tymofyychuk I, Roman L. The history of 
the formation of nursing by the example of the Virginia Henderson 
model. Curr issues Soc Scin Hist Med. 2018;2:105–8. doi: 10.24061/2411-
6181.2.2018.37.
22. Mencigar D. Model e-dokumentiranja zdravstvene oskrbe v intenzivni 
terapiji otrok: master thesis. Ljubljana: University of Ljubljana, 2014.
23. Gordon M. Manual of nursing diagnosis, 13th Ed. Jones & Bartlett 
Publishers, 2014.
24. Choi M, Park JH, Lee HS. Assessment of the need to integrate academic 
electronic medical records into the undergraduate clinical practicum: 
a focus group interview. Comput Inform Nurs. 2016;34:259-65. doi: 
10.1097/CIN.0000000000000244.
25. Saxena S, Dubey SK. Impact of software design aspects on usability. Int 
J Comput Appl. 2013;61:48-53. 
26. Thoroddsen A, Ehnfors M, Ehrenberg A. Content and completeness of 
care plans after implementation of standardized nursing terminologies 
and computerized records. Comput Inform Nurs. 2011;29:599-607. doi: 
10.1097/NCN.0b013e3182148c31.
27. Johnson L, Edward K-L, Giandinoto J-A. A systematic literature review 
of accuracy in nursing care plans and using standardised nursing 
language. Collegian. 2018;25:355–61. doi: 10.1016/j.colegn.2017.09.006.
28. Dobnik M, Maletič M, Skela-Savič B. Work-related stress factors in 
nurses at Slovenian hospitals – a cross-sectional study. Zdr Varst. 
2018;57(4):192–200. doi: 10.2478/sjph-2018-0024.
144
10.2478/sjph-2022-0019 Zdr Varst. 2022;61(3):137-144