ZNANSTVENI PRISPEVKI
B A usability comparison of input devices for precise and intuitive interaction with 3D visualizations
Uporabniška primerjava vhodnih naprav za natančno in intuitivno interakcijo v 3D vizualizacijah
Ciril Bohak1, Jože Guna2, Peter Škrij3, Alja Isakovic4, Matija Marolt1
1University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, 1000 Ljubljana 2University of Ljubljana, Faculty of Electrical Engineering, University of Ljubljana, Tržaška cesta 25, 1000 Ljubljana 3University of Primorska, Faculty of Mathematics, Natural Sciences and Information Technologies, Glagoljaška 8, 6000 Koper 4Cofound.it One, informacijske tehnologije, d.o.o., Kersnikova ulica 10, 1000 Ljubljana
ciril.bohak@fri.uni-lj.si; joze.guna@fe.uni-lj.si; peter.skrlj92@gmail.com; alja.isakovic@gmail.com; matija.marolt@fri.uni-lj.si
Abstract
The paper presents a user study comparing the performance and usability of different input devices for precise manipulation of 3D objects: a regular mouse, a 3D mouse, and the gesture-based Leap Motion Controller. We show that the 3D mouse is well suited for the task as it yielded the lowest rotation error rate and best overall usability while tasks were completed with the 3D mouse as quickly as with the regular mouse interface. The 3D mouse earned an average System Usability Scale score of 88.7, the regular mouse 72.4, however, the Leap Motion Controller was barely suitable for the task, receiving an average System Usability Scale score of 56.5. The study showed that users needed the most time to finish the tasks with the Leap Motion Controller and that its gestures were not as easy to learn compared to the 3D mouse. This led us to conclude that the 3D mouse is currently the best input option among the tested devices for 3D tasks that require both high precision, quick completion and a fast learning curve. Keywords: User experience; usability evaluation; user study; 3D manipulation; natural user interfaces; 3D mouse.
Izvleček
Članek predstavlja uporabniško študijo, ki primerja uspešnost in uporabnost različnih naprav za natančno manipulacijo s 3D objekti: običajno računalniško miško, 3D miško in vmesnik Leap Motion, ki temelji na uporabi kretenj. Pokazali smo, da je 3D miška zelo primerna za takšna opravila, saj so z njo uporabniki dosegli najmanjše napake v rotaciji, hkrati pa je dosegla najvišjo oceno za uporabniško izkušnjo. Uporabniki so izvedli opravila enako hitro kot z običajno miško. 3D miška je dosegla v povprečju oceno 88,7 na lestvici System Usability Scale, običajna miška je dosegla 72,4, vmesnik Leap Motion pa 56,5. Med rezultati študije se izkazalo, da uporabniki največ časa za naloge porabijo z uporabo vmesnika Leap Motion, in da se njegove uporabe niso priučili tako hitro kot uporabe 3D miške. Zaključek študije je, da je 3D miška trenutno najbolj primerna vhodna naprava med testiranimi za opravila v 3D okolju, ki potrebujejo tako natančnost, hitrost in se je njihove kot hitro učno krivuljo.
Ključne besede: uporabniška izkušnja; uporabniko ovrednotenje; uporabniška študija; 3D manipulacija; naravni uporabniški vmesniki; 3D miška
94 uporabna INFORMATIKA
2018 - številka 3 - letnik XXVI
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovic, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
1 INTRODUCTION
The traditional computer mouse made desktop computers more accessible to millions of users by offering an easy-to-learn input method for the »point--and-click« graphical user interfaces (GUI) based on the WIMP (»window, icon, menu and pointing device«) paradigm (van Dam, 1997). WIMP GUIs were designed and optimized for 2D document-based applications, but create a greater »cognitive distance« when 3D objects are introduced (van Dam, 1997). And while professional 3D computer-aided- design (CAD) users have been successfully using mouse and keyboard input for 3D navigation and manipulation, it is an interaction method that requires a high level of learning, practice and abstraction.
As interactive 3D content is becoming commonplace in a wide range of fields, the mouse input method might not offer the best choice for casual users who still require a certain level of precision and ease of use, but cannot afford to spend a lot of time learning a new user interface of a 3D application. And while there are already several alternative input devices that were designed for post- WIMP style of interaction, they are mostly used in specialized fields (e.g., 3D mice used by CAD professionals) or with gaming consoles (e.g., the motion sensing Microsoft Kinect), which means that they are still highly unfamiliar to most users. Therefore, the question is whether any of the alternative input devices that were
designed for 3D interaction can be learned more easily by beginners than the already familiar (yet not optimized for 3D) mouse.
Visualization of 3D datasets is a good example of an application requiring users to interact with data in 3D space in different ways. Most of user interaction tasks can be divided into navigation, selection/manipulation and system control (Bowman et al., 2001). The 2D nature of the computer screen on which the 3D dataset is projected makes both viewpoint navigation and object manipulation essential in order for the user to fully and effectively grasp the presented data. Therefore, 3D desktop applications have to support a wide range of camera movements that navigate around 3D objects (Rotate-Pan-Dolly) and techniques for manipulating 3D objects (Rotate-Scale- Translate) (Jankowski and Hachet, 2015). All of the above-mentioned aspects are important while developing a highly usable interface for 3D interaction.
For us, the problem of choosing the best input method for 3D interaction, both in terms of precision and usability, arose when we developed NeckVeins (Bohak et al., 2013), a medical visualization platform. NeckVeins displays 3D vascular models of patients (Figure 1), captured with computed tomography (CT) or other volumetric methods (e.g., MRI or ultrasound). The application is used by medical professionals to explore the 3D data from different viewpoints.
Figure 1: The user interface of the NeckVeins application
2018 - številka 3 - leti
nik XXVI
uporabna INFORMATIKA 95
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovič, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
Since the application was developed for medical purposes, it is essential that navigation in 3D space is intuitive and easy to use, while still offering high precision. An additional requirement for the application was that it should work with existing hardware, so we explored inexpensive input devices that are already on the market and that can easily be plugged into existing systems, and excluded alternatives like multi-touch that require more expensive system upgrades. This led us to implement three different modes of interaction with the following input devices:
■	a regular mouse that users are already familiar with. The mouse is used for object manipulation. Zoom functionality is implemented in discrete steps due to the nature of the mouse wheel design;
■	a 3D mouse (3Dconnexion Space Navigator1), which builds on the familiarity and popularity of a computer mouse, but adds six-degrees of freedom (6-DOF -movements along three world axes and rotations around them) for intuitive and precise 3D navigation. 6-DOF in comparison with regular mouse which offers weo-degrees of freedom offers wider range of input actions. Users can toggle between control of the object and control of the camera by pressing one of the buttons on the 3D mouse, thus performing navigation and manipulation tasks with same device. In our application, we can adjust the sensitivity of interactions as well as toggle between using the dominant movement/rotation (rotation or movement along individual axis) or all of them;
■	a Leap Motion Controller2, which offers a touchless, gesture-based natural user interface (NUI). The Leap Motion Controller tracks the 3D position and orientation of hands and fingers in the space above the device. We linked the position and orientation of hands to object rotation and zoom. We have not implemented camera movement functionality for the Leap Motion Controller due to complexity introduced by additional gestures. Interaction with the Leap Motion Controller starts by opening the palm and ends by closing it. Scaling and rotations are bound to hand movements. In our application, we can adjust the sensitivity of movements detected by the device.
1	https://www.3dconnexion.eu
2	https://www.leapmotion.com
The rest of the paper is organized as follows: following the introduction, related work is presented in Section 2. The user study research method and hypotheses are presented in Section 3. Results of the user study are described, analyzed and discussed in Sections 4 and 5, respectively. Finally, key conclusions are drawn in Section 6.
2 RELATED WORK
A lot of Human-Computer Interaction (HCI) research that compares different input devices focuses on input speed by measuring the time it takes to complete the task and the error rate during the task, which are both easy to measure and compare directly. That was also the case in one of the first user studies on input devices conducted in 1967, which found the computer mouse to be the most accurate of all studied devices, but not the fastest (Ortega et al., 2016). Ivan Sutherland's light pen was slightly faster than the mouse, but it had a greater error rate and caused discomfort during prolonged use. This user study sets a good example of how utility, the device's feature set, is not enough without usability, which describes how easy and pleasant it is for the users to use the features of the devices (Nielsen, 2012). Other authors have also showed that interaction science is well established and needed (Pike et al., 2009).
The usability aspect is commonly measured and compared with standardized questionnaires like the System Usability Scale (SUS). The SUS was introduced in 1986 by John Brooke (1996) and is still one of the most widely used standard usability tools, espe-dally because of its versatility, reliability and simplicity aspects. The SUS questionnaire consists of 10 questions, half of them worded negatively and half positively towards the usability aspects of the system under test. For each question the participants can rate how strongly they agree with the specific question on a 5-point Likert scale. The final result of the SUS questionnaire is a score on a scale from 0 (negative) to 100 (positive) (Bangor et al., 2009), which can be used to compare results across different user studies.
In terms of user performance, a lot of research on input devices focuses on pointing tasks, based on Fitt's law. And while pointing and selection is also part of 3D interaction, especially in virtual environments (Teather and Stuerzlinger, 2010), studies on 3D interaction often include tasks that mirror real-world applications. A common task is 3D docking,
96 uporabna INFORMATIKA
2018 - številka 3 - letnik XXVI
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovic, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
in which participants aim to match the position, rotation and scale of a sample object in 3D space (Jan-kowski and Hachet, 2015).
A good example of a study based on a 3D docking task is the work by Besançon et al. (2016) that evaluated both the performance and usability aspects of 3D data manipulation with a standard mouse, touch input built into the screen, and tangible input using a hand-held cuboctahedron with markers for camera-based 6-DOF 3D tracking. Participants completed the 3D docking task faster with the tangible input, followed by touch and mouse respectively, but none of the techniques provided higher precision than the others. The participants did however feel they had the most precise control using the mouse, followed by the touch interface. And while the participants preferred the novel tangible interface overall, the authors concluded that each input method has its own advantages and limitations that have to be considered while making a choice.
Other usability studies on touch input focus mostly on 3D navigation in map applications, but Yu et al. (2010) compared a touch interface with mouse interaction on different 3D scientific visualizations. They found the touch interface to be as good as the mouse in terms of speed, easy to learn and preferred by participants for exploration and wayfinding. Bade et al. (2005) compared different mouse-based interaction techniques for predictable 3D rotations in 3D radiological visualizations. The study shows that the three input modalities provide similar levels of precision but require different interaction times.
It is more common for 3D interaction studies to include 3D mice and other modified mouse alternatives that were designed with 3D interaction in mind. Perelman et al. (2015) compared the performance of a 3D mouse with the Roly-Poly Mouse (RPM), which combines the positioning abilities of a traditional mouse with rolling and rotating abilities of 3D devices. The study found the RPM faster for 3D pointing, but both performed equally well in the 3D docking task. Similarly, Balakrishnan et al. (1997) evaluated a 4-DOF Rockin' Mouse and found it 30% faster than a standard mouse in a typical 3D interaction task. The Rockin' Mouse was preferred by the participants, especially by expert users, but it did require some practice. Hinckley et al. (1997) also showed that 3D input devices can provide faster 3D rotation than 2D input techniques without sacrificing precision.
And while touch interfaces and various mouse modifications with more degrees of freedom appear to be a viable alternative to 2D mouse-based input, gesture-based input shows a lot of potential for users with special needs or environments with specific requirements, even though it is often slower than the traditional mouse. Bhuiyan and Picking (2011) presented a usability evaluation of gesture-based navigation that showed positive results among older and disabled users in terms of ease of use and learning of the system. Coelho and Verbeek (2014) found that the gesture-based Leap Motion Controller performed better than the mouse in single target 3D pointing tasks, but was more time consuming and less precise when multiple targets were introduced. Expert users showed a bias towards the mouse in 3D tasks, but the Leap Motion Controller scored surprisingly high SUS score results despite accuracy issues. Ryu et al. (2011) found that a touchless mouse (similar to the Leap Motion Controller) was about three times slower than a regular mouse, but did not cause significantly more errors in the pointing and selection tasks. The authors concluded that the touchless mouse could be a viable alternative, despite an inferior throughput, in environments like hospital operation rooms, where direct touch can be problematic.
Natural touchless user interfaces are a good fit for sterile medical environments. Ebert et al. (2012) describe the use of a Kinect 3D sensor and additional voice commands in a touch-free navigation system for radiological images. The Kinect interaction was slower than the standard mouse input. That was partly due to the lack of familiarity with the gesture-based system and the authors concluded that more training might be needed. Another study tried to tackle the challenge of non-contact navigation with the Leap Motion Controller (Gratzel et al., 2004). They linked hand gestures from the Leap Motion Controller with application key bindings in the GameWave3 application. They obtained good results and also tested the device in a real-life situation during surgery, but their method was not tested from the usability standpoint. Similarly, the Leap Motion Controller was reportedly successfully used during dental surgery to control and consult the surgical plan during the operation, but the authors Rosa and Elizondo (2014), did not perform a usability study.
3 GameWave can be obtained in the Leap Motion Airspace store.
2018 - številka 3 - leti	nik XXVI
uporabna INFORMATIKA 97
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovič, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
Due to specific requirements of medical 3D applications, it is important to further explore the usability and precision of emerging gesture-based interfaces and compare their performance with a familiar alternative (mouse) and specialized 3D input devices (3D mouse) to make sure that precision can be preserved, while still providing an easy-to-learn option that does not cause unnecessary cognitive load during already complicated tasks such as diagnostics and surgery. That is why our presented study considers both aspects to provide recommendations for user-centered 3D input in the medical field and other fields where non-expert users have to control 3D visualizations with precision.
3 METHOD
The main goal of this study was to identify which of the tested input devices enables fast and precise interaction with a 3D medical visualization and is also easy to use. We conducted a user study to compare the performance of the devices in terms of speed by measuring completion time and in terms of precision by measuring rotation and zoom error rates. In addition to the performance, the usability of each of the three input devices was also evaluated by giving study participants 3D docking tasks that model some 3D interaction skills used by medical professionals, and by having the participants complete a SUS usability questionnaire for each of the devices.
Based on related research findings, we defined the following hypotheses:
■	H1: In terms of time required to complete individual 3D docking tasks, the Leap Motion Controller will be the slowest device, while the mouse and the 3D mouse will not differ significantly.
■	H2: Rotation error rate of all devices will not differ significantly.
■	H3: Zoom error rate of all devices will not differ significantly.
■	H4: All input devices will be suitable for use with the NeckVeins application (SUS score is equal or better than »OK«).
■	H5: The participants will favor the Leap Motion Controller due to its novelty regardless of its performance.
3.1 Participants
Even though the NeckVeins application was built for medical professionals, we decided to test its 3D inte-
raction and the three chosen input devices on a more diverse group of participants. A total of 29 participants took part in our study, of those 55% men and 45% women. 14 participants were between ages 18 and 24, 11 between ages 25 and 34, 3 between ages 35 and 44, and 1 between 45 and 54 years. Of those, 15 were students, 13 were employed and 1 unemployed. Their professional background was diverse, ranging from technical and natural sciences to medical, humanistic and social areas of expertise.
Most of the participants (70%) had no previous experience with the NeckVeins application. More than half of the participants had previous experience and understood the use of the regular computer mouse for 3D object manipulation, but 78% of the participants had no experience with manipulating 3D objects with a 3D mouse.
3.2	Apparatus
The user study was performed in a dedicated room where participants were isolated from outside factors such as noise or interruptions, so the same conditions were ensured for all participants.
The experiment was conducted on a desktop computer preinstalled with a modified NeckVeins application, which contained different 3D docking tasks that the participants had to complete for each input device modality. In order, not to distract the participants (who were not all medical professionals) with the content of visualizations, the tasks consisted of docking a neutral 3D object (teapot), so that the participants could focus on the 3D manipulation task at hand. One of the 3D docking tasks and the user interface of the testing application is shown in Figure 2. The interaction mappings for the individual devices were implemented as presented in Figure 3.
The same setup (shown in Figure 4) was used by all of the participants and each participant was tested with each of the three interaction methods. For the regular mouse test, all participants used a Logitech M90 mouse. The 3D mouse was a 3Dconnexion Space Navigator, and the first generation of the Leap Motion Controller was used.
3.3	Procedure
The moderator guided each experimental run using the same test plan for each participant to ensure that all participants performed the same tasks with all three input devices under the same conditions. An
98 uporabna INFORMATIKA
2018 - številka 3 - letnik XXVI
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovic, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
t-riaa* % - Mlaka
MtUanili pM«l«d»k n aesUk UnIIim)*

Figure 2: A screenshot of the modified NeckVeins application with a 3D docking task. The participant controls the 3D model of the red teapot on the left and tries to match its rotation and zoom level to the grey teapot model in the right side of the screen.
Figure 3: Mappings of the interaction for individual actions and individual devices.
observer was also present throughout the experiment. The experimental setup is shown in Figure 3.
At the beginning of each experimental run, the participant was asked to sign a participation agreement and an optional video recording consent. The participant was able to refuse video recording to reduce possible stress during the experiment.
Next, the participant completed a short entry questionnaire about their demographic information (age and gender), and their previous experience with the tested input devices, 3D object manipulation, and the NeckVeins application. Previous experience was rated on a 5-point Likert scale, »1« indicating no expe-
rience/knowledge and »5« indicating full experience/ knowledge.
After the questionnaire, the moderator explained the purpose of the study, introduced the test version of the NeckVeins application (Figure 2) and described the goal of the 3D docking tasks that were used in the experiment. The participant was asked to confirm whether they understood the goal of the 3D docking tasks.
After the participant confirmed their understanding of the task, the actual experiment begun. For each input device, the procedure was the same. First, the participant was given a description of the device
2018 - številka 3 - leti	nik XXVI
uporabna INFORMATIKA 99
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovič, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
they would be using in that round. The participant was given 5 minutes to get familiar with the device before starting with the 3D docking tasks. During the familiarization period, any questions regarding the application and the input device were answered by the moderator. After the familiarization period, the testing begun if the participant confirmed that they knew how to use the input device in the application, otherwise the uncertainties were clarified first. This assured a basic level of familiarity before each round of testing.
During the test for each device, the participant completed seven 3D docking tasks for each of the three input devices, for a total of 21 tasks per participant. The order of input devices and tasks was randomized for counterbalance. For three devices, this means 6 device order sequences, so we had 4 and 5 repetitions of each device order sequence. The goal of each 3D docking task was to align, in terms of rotation and zoom, a red colored teapot displayed on the left-hand side of the screen (the participant's work area) with a grey colored teapot displayed on the right-hand side of the screen (the task goal). The different colors were selected for immediate distinc-
tion between the object of manipulation object and target object. Figure 2 shows a screenshot of the test application. Individual tasks had differently oriented teapots, varying in desired orientation (rotation) and scale (zoom), but the interaction was the same for all seven tasks. The order of the presented input devices and 3D docking tasks was counterbalanced — randomly selected and balanced among the participants.
The moderator started each of the 3D docking tasks by selecting one of the predefined tests and starting the timer. The time needed to complete the task, as well as the rotation and zoom error rate of the positioned teapot with respect to the reference teapot were automatically recorded by the test application for each individual task separately from start of the task — when the new task was displayed to the participant — to task completion — when the participant was satisfied with the alignment by pressing the space key on the keyboard.
The choice of speed versus precision was left to the participant and the moderator did not interfere during the tasks unless fatigue or distracte-dness were noticed. The observer took notes of the
100 uporabna INFORMATIKA
2018 - številka 3 - letnik XXVI
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovic, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
participant's body language, actions, comments and any problems that occurred in the test application. In case of an error in the test system or when the participant felt they were unable to complete the task, the system was reset and the participant was allowed to repeat the task. In case of a misunderstanding when performing the task at hand, the experimental run was paused, additional explanation was given to the participant and then the experimental run resumed. If there was a problem with the participant's distraction or tiredness, the experiment was also paused and the participant was given some time before the experiment was resumed. If this was not possible, the participant was removed from the experiment and all the collected data was discarded. The data was continuously recorded throughout the experiment.
After the participant finished with all seven 3D docking tasks for an individual input device, they were asked to fill in the SUS questionnaire for that device. The procedure was then repeated for the other two input devices in the same manner.
After completing the experiment with all three input devices, the participant was asked additional questions regarding the comparison between individual input devices. The comments and suggestions were written down by the observer.
3.4 Design
The independent variable in the user study was the input device used for the task. Three devices were tested in the study: regular mouse, 3D mouse, and the Leap Motion Controller.
The measured dependent variables were: completion time measured in seconds, rotation error rate measured in degrees, zoom error rate measured in percentages, and the SUS usability score on a scale from 0 - 100.
Completion time was determined by measuring the time needed (in seconds, precision was one second) to complete each individual task for each input device (for each user). The average values and standard deviations were calculated alongside with the shortest and longest time needed to perform the tasks.
Rotation error rate was determined by summing the measured errors in rotation compared to the perfect position (in arc degrees (°) around each individual axis (X, Y and Z), measurement precision was 0.01°) of completing each individual task for each input device (for each user). Its minimal - best value is 0° and
its maximum - worst value is 540° (180° around each axis). The average values and standard deviations were calculated alongside with the best (lowest) and worst (highest) error rate of all cases. With such metric, it is also possible to check whether rotation error about a certain axis stands out more than others.
Zoom error rate determined by measuring the error of zooming compared to the perfect position (in %, measurement precision was 0.01%) for completing each individual task for each input device (for each user). The average values and standard deviations were calculated alongside with the smallest and largest zoom error of all cases.
The usability aspect was determined by measuring the SUS score of each input device, using the standard SUS questionnaire (Brooke, 1996; Bangor et al., 2009).
To better design the experiment, the study was conducted in two parts. The first part was a pilot study in which we tested 7 participants and evaluated the user study methodology and the test application. The participants from the pilot study were not included in the presented data analysis or in the second part of the study due to the knowledge gained during the trial run. We used the results, comments and responses from the pilot study participants to improve the test plan, questionnaire and the test application. The second part was a comprehensive study with 29 participants and was completed in the time span of one month.
4 RESULTS
4.1 Performance and usability evaluation
Evaluation results in terms of completion time, rotation and zoom error rates are shown in Tables 1-3. Average values, standard deviations, min and max values calculated over all tasks and users are given for each device, rounded to one decimal place.
A general linear model of repeated measures has been used to identify the possible statistically significant differences (p < 0.05) between input devices using Analysis of variance (ANOVA) in terms of the defined dependent variables. The results show that the Leap Motion controller yielded significantly worse completion times than both the regular and the 3D mouse (p < 0.0001). The Leap Motion controller also showed significantly higher rotation error rates than the 3D mouse (p < 0.0005). On the other hand, the regular mouse had significantly lower zoom error
2018 - številka 3 - leti	nik XXVI
uporabna INFORMATIKA 101
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovič, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
Table 1: Task completition times aggregated across all tasks for each device separately. Lower values (in bold) indicate better performance.
Input device	Average time (s)	Standard deviation (s)	Fastest time (s)	Longest time (s)
Mouse	36.6	27.1	5.0	196.0
3D Mouse	35.4	28.5	4.0	196.0
Leap Motion	64.5	46.6	11.0	285.0
Table 2: Rotation error rate in arc degrees (°) aggregated across all tasks for each device separately. Lower values (in bold) indicate better performance.			
Input device	Average err. (°)	Standard deviation (°) Lowest err. (°)	Highest err. (°)
Mouse	17.1	37.9 6.2	210.0
3D Mouse	10.6	23.3 0.1	191.0
Leap Motion	21.9	46.7	0.3	256.0
Table 3: Zoom error rate aggregated across all tasks for each device separately. Lower values (in bold) indicate better performance.	
Input device	Average err. (%) Standard deviation (%) Lowest err. (%) Highest err. (%)
Mouse	34.1 80.2 0.0 800.0
3D Mouse	52.9 60.7 0.0 396.0
Leap Motion	58.9	71.7	0.0	600.0
rates than the 3D mouse (p < 0.005) and the Leap Motion Controller (p < 0.0001).
Correlational tests showed that the participants' previous experience with various input devices and 3D manipulation did not significantly influence their performance in the 3D docking tasks.
Results of the SUS questionnaire are presented in Table 4, where the average SUS value for each user for each individual input device is calculated according to the methodology presented in (Brooke, 1996; Bangor et al., 2009; Lewis and Sauro, 2009; Brooke, 2013). Minimum and maximum achieved SUS scores are also shown for each input device. The descriptive SUS interpretation is added according to the (Bangor et al., 2009; Brooke, 2013).
4.2 User observation and interviews
While observing the participants, we noticed they had problems while trying to achieve rotational as well as zoom precision with the Leap Motion Controller.
Our follow up questions showed that the main reason for this was the lack of feedback when participants waved their hands above the device, which caused some confusion. They had problems remembering the correct gestures for individual actions (how to move the hands to rotate or zoom) and remembering to close the palm for stopping interaction and to open the palm for resuming it, thus unintentionally triggering interaction. They also reported some frustrations when trying to make very precise movements. On the other hand, some of the users felt that such touchless interaction presents a very natural way of interaction and said that the actions were intuitive.
The majority of the participants were very satisfied with the 3D mouse, which we attribute to the fact that movements of the 3D mouse directly reflected in the movements of the objects. Therefore, users did not need to remember certain gestures and motions so they could concentrate more on completing the tasks and less on how to handle the device itself.
Table 4: Results of the SUS scores questionnaires showing input device performance in terms of usability. Higher score (in bold) means better performance.
Input device	Average score	Standard deviation	Minimum score	Maximum score	SIS score interpretation
Mouse	72.4	18.4	45	97.5	Good/C
3D Mouse	88.7	11.4	50	100	Excellent/B
Leap Motion	56.5	19.0	20	95	OK
102 uporabna INFORMATIKA
2018 - številka 3 - letnik XXVI
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovic, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
From the interviews, we also extracted that most of the users performed the tasks as fast as possible and paid less attention to precise positioning.
None of the users had problems with fatigue or distractions. There were some minor problems with the experimental setup; in these cases, we disposed the invalid data and repeated the erroneous part of the experiment.
5 DISCUSSION
5.1	Completion time
Hypothesis H1 predicted that the Leap Motion Controller will be the slowest device, while the mouse and the 3D mouse will be closer together in terms of time required to complete the 3D docking tasks. Table 1 shows that the average time for completing each task was 45 seconds. Completing tasks with a mouse took 37 seconds on average, with 3D mouse 35 seconds, and with the Leap Motion Controller 65 seconds. The participants were able to complete the tasks with the mostly unfamiliar 3D mouse just as fast as with the familiar mouse, which makes the 3D mouse a good alternative to mouse in terms of speed. On the other hand, statistical tests confirmed that the Leap Motion Controller required significantly more time than both other alternatives. These results confirm hypothesis H1 and are all in line with the results of other studies presented in Section 2.
5.2	Rotation error rate
Hypothesis H2 stated that there will be no significant differences between the tested input devices. Results (Table 2) showed that the average rotation error rate for all input devices was 16.5°. The average rotation error rate achieved with the regular mouse was 17.1°, with the 3D mouse 10.6° and with the Leap Motion Controller 21.9°. There was no significant difference between the regular mouse and other two input devices, but there was a significant difference between the 3D mouse and the Leap Motion Controller, so we cannot confirm hypothesis H2.
The results also show that the users were able to rotate most accurately with the 3D mouse. By combining the completion time results, our results show that the 3D mouse is a good choice when rotational precision and speed are needed. Again, the Leap Motion performed the worst.
5.3	Zoom error rate
The hypothesis H3 also stated there will be no significant differences between the tested input devices in terms of zoom error rate. Results (Table 3) show that the average zoom error rate for all input devices was 48.6%. The average zoom error rate achieved with the regular mouse was 34.1%, with the 3D mouse 52.9% and with the Leap Motion Controller 58.9%.
From the results, we can conclude that zooming is best performed with a regular mouse, where the average zoom error is the lowest. However, standard deviation shows that users made more consistent mistakes with the 3D mouse than they did with a regular one. The difference can be attributed to the fact that zooming with a regular mouse is performed in discrete steps due to the functioning of the mouse wheel, which makes it easier to select an appropriate zoom level. The 3D mouse has a continuous zoom, which makes it harder to judge the small differences when adjusting for the appropriate zoom level.
There is no significant difference between the 3D mouse and the Leap Motion Controller, but there is a significant difference between the mouse and both other devices, so hypothesis H3 could also not be confirmed.
5.4	Usability
The user study also aimed to evaluate the usability aspect of the tested input and hypothesis H4 stated all three input devices will be suitable for the tested application with a SUS score equal or better than »OK«. Results (Table 4) show that the average SUS score for all input devices was 72.5. The average SUS score achieved with the regular mouse was 72.4, with the 3D mouse 88.7 and with the Leap Motion Controller 56.5. The participants really liked the implementation of the 3D mouse and most of them (21) would gladly recommend this device to other people.
After matching the results to an adjective rating scale, the regular mouse modality scored »Good/C«, the 3D mouse »Excellent/B«, and the Leap Motion Controller »OK«. From these results, we can conclude that the usability of a regular mouse and 3D mouse is acceptable, whereas the Leap Motion Controller has a low marginal score, but still scores an »OK« on the adjective rating scale, so we can confirm hypothesis H4, as all three devices were evaluated as suitable.
2018 - številka 3 - leti	nik XXVI
uporabna INFORMATIKA 103
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovič, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
5.5 Overall impression and user preference
The hypothesis H5 predicted that the participants will tend to favor the Leap Motion Controller due to its novelty factor. Our results actually show that the participants were more impressed by the 3D mouse, which was also an unfamiliar device for most participants, but was easier to handle.
One of the reasons why the Leap Motion performed so poorly might be in the implementation of the gestures used to manipulate the test object in 3D space. The interaction with the test object was started by opening the palm of the hand above the sensor. Hand movements to the left and right rotated the object about the vertical axis, and movement up and down rotated the object about the horizontal axis. Moving the hand closer to the screen or away from the screen resulted in zooming action. One could also tilt the palm about the horizontal axis that pierced the screen which resulted in the rotation about this axis.
When considering all of the presented factors, the 3D mouse is the most appropriate input device, and the best choice in terms of rotation precision and usability, and equivalent to the regular mouse in terms of completion time. The regular mouse also provides solid performance in all aspects. However, the Leap Motion Controller has not yet reached its full potential. On the one hand, it provides a natural an intuitive interaction, on the other hand the interaction is slower compared to the other two input devices and also has higher error rates and lower usability.
6 CONCLUSIONS AND FUTURE WORK
We presented a study that evaluated three different input devices for 3D docking tasks in a modified 3D medical visualization application that requires high precision. The devices tested were: regular mouse, a 3D mouse, and the gesture-based Leap Motion Controller. We compared the performance (completion time, rotation and zoom error rates) and usability of all three input devices on 7 different 3D docking tasks in randomized order with a total of 29 participants. Results show that the 3D mouse is the most appropriate input device in terms of rotation precision, equivalent to regular mouse in terms of completion time, and has also received favorable subjective assessments. The 3D mouse also achieved the best usability score.
Considering the poor performance of the Leap Motion Controller, we conclude that this novel input device is not ready yet to be used in everyday
environments despite its suitability in sterile environments such as hospital operation rooms. However, user feedback still leads us to believe that it is a promising natural touchless interface once its precision and interaction model is improved. Our results show that even though the gesture-based interaction felt natural to the participants, the gestures still had to be learned and are in fact not as closely matched to the 3D interaction on desktop computers as those of a 3D mouse.
The comments and opinions we gathered through the 3D docking tasks performed with the Leap Motion Controller will be used in a new release of the NeckVeins application. We have also implemented a new gesture setup for interacting with the Leap Motion Controller, which we plan to evaluate in our future work.
Overall, our study finds the 3D mouse as the most promising input device for 3D visualizations among those that are readily available on the market and are easy to add to existing 2D desktop setups. We therefore recommend further research to include this device in real-world situations to fully explore its potential when precision and ease of use are needed in manipulating 3D visualizations. Because our study included a diverse group of participants, we believe the results of our work can also apply to other types of precise 3D visualizations that are used by non-expert users, especially when long training is not an option. As part of future work, we plan to further explore similar scenarios and add support for new types of natural input interfaces, such as touch and voice input.
7	ACKNOWLEDGEMENTS
The authors would like to thank all the participants in the experiment for volunteering and taking their time helping in our study. We would especially like to thank the users of the pilot study that gave us many good suggestions on how to improve the experiment as well as how to adapt the testing application.
8	REFERENCES
[1]	Bade, R., Ritter, F., and Preim, B. (2005).
Usability Comparison of Mouse-Based Interaction Techniques for Predictable 3D Rotation. Smart Graphics, 3638, pages 138-150.
[2]	Balakrishnan, R., Baudel, T., Kurtenbach, G., and Fitzmaurice, G. (1997).
The Rockin'Mouse: Integral 3D Manipulation on a Plane. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pages 311-318. ACM.
104 uporabna INFORMATIKA
2018 - številka 3 - letnik XXVI
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovic, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
[3]	Bangor, A., Kortum, P., and Miller, J. (2009). Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale.
Journal of Usability Studies, 4:114-123.
[4]	Besançon, L., Issartel, P., Ammi, M., and Isenberg, T. (2016). Usability Comparison of Mouse, Touch and Tangible Inputs for 3D Data Manipulation. CoRR.
[5]	Bhuiyan, M. and Picking, R. (2011).
A Gesture Controlled User Interface for Inclusive Design and Evaluative Study of its Usability.
Journal of Software Engineering and Applications, pages 513-521.
[6]	Bohak, C., Žagar, S., Sodja, A., Škrlj, P., Mitrovic, U., Pernuš, F., and Marolt, M. (2013).
Neck Veins: an Interactive 3D Visualization of Head Veins. In Proceedings of the 4th International Conference World Usability Day Slovenia 2013, Ljubljana, Slovenia, 25th November 2013, pages 64-66.
[7]	Bowman, D. A., Kruijff, E., LaViola, J. J., and Poupyrev, I. (2001).
An Introduction to 3-D User Interface Design. Presence: Teleoperators and Virtual Environments, 10(1), pages 96-108.
[8]	Brooke, J. (1996).
SUS - a Quick and Dirty Usability Scale. Usability Evaluation in Industry, pages 189-194.
[9]	Brooke, J. (2013).
SUS: A Retrospective. Journal of Usability Studies, 8(2):29-40.
[10]	Coelho, J. C. and Verbeek, F. J. (2014).
Pointing Task Evaluation of Leap Motion Controller in 3D Virtual Environment.
In Creating the Difference: Proceedings of the Chi Sparks 2014 Conference, pages 78-85.
[11]	Ebert, L. C., Hatch, G., Ampanozi, G., Thali, M. J., and Ross, S. (2012).
You Can't Touch This Touch-Free Navigation Through Radiological Images. Surgical Innovation, pages 301-307.
[12]	Grätzel, C., Fong, T., Grange, S., and Baur, C. (2004).
A Non-Contact Mouse for Surgeon-Computer Interaction. Technology and health care: Official Journal of the European Society for Engineering and Medicine, 12:245-257.
[13]	Hinckley, K., Tullio, J., Pausch, R., Proffitt, D., and Kassell, N. (1997).
Usability Analysis of 3D Rotation Techniques. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology, pages 1-10. ACM.
[14]	Jankowski, J. and Hachet, M. (2015).
Advances in Interaction with 3D Environments. Computer Graphics Forum, 34(1):152-190.
[15]	Lewis, J. R. and Sauro, J. (2009).
The Factor Structure of the System Usability Scale. In Proceedings of the 1st International Conference on Human Centered Design: Held As Part of HCI International 2009, HCD 09, pages 94-103, Berlin, Heidelberg. Springer-Verlag.
[16]	Nielsen, J. (2012).
Usability 101: Introduction to Usability. https://www.nngroup.com/articles/usability-101-introducti-on-to-usability/. Accessed: 2018-08-22.
[17]	Ortega, F. R., Abyarjoo, F., Barreto, A., Rishe, N., and Adjou-adi, M. (2016).
The Light Pen and the Computer Mouse, chapter 1.2.4, pages 10-13. A K Peters/CRC Press.
[18]	Perelman, G., Serrano, M., Raynal, M., Picard, C., Derras, M., and Dubois, E. (2015).
The Roly-Poly Mouse: Designing a Rolling Input Device Unifying 2D and 3D Interaction.
In Proceedings of the ACM CHI'15 Conference on Human Factors in Computing Systems, volume 1, pages 327-336. ACM.
[19]	Pike, W. a., Stasko, J., Chang, R., and O'Connell, T. A. (2009). The Science of Interaction.
Information Visualization, 8(4), pages 263-274.
[20]	Rosa, G. M. and Elizondo, M. L. (2014).
Use of a Gesture User Interface as a Touchless Image Navigation
System in Dental Surgery: Case Series Report. Imaging Science in Dentistry, 44, pages 155-160.
[21]	Ryu, Y., Koh, D. D. H., and Um, D. (2011).
Usability Evaluation of Touchless Mouse Based on Infrared Proximity Sensing.
Journal of Usability Studies, 7, pages 31-39.
[22]	Teather, R. and Stuerzlinger, W. (2010). Target Pointing in 3D User Interfaces.
In Graphics Interface (GI 2010), pages 20-21.
[23]	van Dam, A. (1997). Post-WIMP User Interfaces. Communications ACM, 40(2), pages 63-67.
[24]	Yu, L., Svetachov, P., Isenberg, P., Everts, M. H., and Isenberg, T. (2010).
FI3D: Direct-Touch Interaction for the Exploration of 3D Scientific Visualization Spaces.
IEEE Transactions on Visualization and Computer Graphics, 16, pages 1613-1622.
Ciril Bohak is a Teaching Assistant at the Faculty of Computer and Information Science, University of Ljubljana. His research includes Human-Computer Interaction, Computer Graphics, Music Information Retrieval, Gamification, E- Learning and Game Technology. Currently he is teaching courses Computer Graphics, Game Technology and Computer Aided Design. He is one of the founding members of Slovenian HCI community.
Jože Guna is an Assistant Professor at the Faculty of Electrical Engineering, University of Ljubljana. His area of research focuses on Internet technologies, multimedia technologies, IPTV systems and VR/AR/M technologies with special emphasis on user-centered design, user interaction modalities and designing the user experience, including gamification and flow aspects. He is a PI of several projects. He is an expert in Internet, ICT and IPTV technologies and holds several industrial certificates from CISCO, Comptia and Apple, including trainer licenses from Cisco and Apple. He is a senior member of the IEEE organization.
2018 - številka 3 - leti	nik XXVI
uporabna INFORMATIKA 105
Ciril Bohak, Jože Guna, Peter Škrlj, Alja Isakovič, Matija Marolt: A usability comparison of input devices for precise and intuitive interaction
with 3D visualizations
Peter Škrlj graduated at University of Ljubljana, Faculty of Computer and Information Science in 2014. His bachelor thesis focused on evaluating usability and ease of use of different navigation devices while manipulating a 3D medical image. His fields of interest vary from computer graphics, game design, internet technologies, mobile device development and computer vision. He is currently enrolled in the master's program at University of Primorska, Faculty of Mathematics, Natural Sciences and Information Technologies.
Alja Isakovic is currently UX advisor at Cofound.it One. As a member of LGM (Laboratory for Computer Graphics and Multimedia) and researcher in the field of human-computer interaction, she is currently exploring new data-gathering user interfaces and ways for monitoring user interaction in post-PC web browsers. She previously spent three years at CubeSensors, an loT hardware startup, at which she led user experience design and research.
Matija Marolt is an assistant professor with Faculty of Computer and Information Science, University of Ljubljana, where he has been working since 1995. He is head of Laboratory for Computer Graphics and Multimedia, where his research interests include music information retrieval, specifically semantic description and understanding of audio signals, retrieval and organization in music archives, and human-computer interaction.
106 uporabna INFORMATIKA
2018 - številka 3 - letnik XXVI