Augmented Reality in Surgery ARIS*ER, Research Training Network for Minimally Invasive Therapy Technologies Navidezna resničnost v kirurgiji (ARIS*ER), mreža za raziskave in razvoj minimalno invazivnih načinov zdravljenja Adinda Freudenthal1, Eigil Samset2, Borut Gersak3, Jerome Declerck4, Dieter Schmalstieg5, Sergio Casciaro6, Oyvind Rideng7, Jos Vander Sloten8 1University of Technology Delft, Faculty of Industrial Design Engineering, Delft, the Netherlands 2University of Oslo, Faculty of Medicine, The Interventional Centre, Oslo, Norway 3University Medical Centre Ljubljana, Department of Cardiovascular Surgery, Slovenia 4Mirada Solutions, Ltd, Oxford, United Kingdom 5Graz University of Technology, Institute for Computer Graphics and Vision, Graz, Austria 6Institute of Clinical Physiology, National Council of Research, Lecce Section, Italy 7Systems in Motion (SIM), Oslo, Norway 8Katholieke Universiteit Leuven, Faculty of Engineering, Division of Biomechanics and Engineering Design, Leuven, Belgium www.ariser.info Avtor za dopisovanje (correspondence to): prof. dr. Borut Geršak, dr. med., Klinični center Ljubljana, Klinični oddelek za kardiovaskularno kirurgijo, Slovenija, Zaloška 7, 1000 Ljubljana, Slovenija Endoscopic Rev, Vol. 10, No. 23, May 2005 5 Abstract Augmented Reality in Surgery (ARIS*ER), a Marie Curie Research Training network, is presented. ARIS*ER aims at exploring and developing for Augmented Reality and haptics, based on radiological imaging. This will be combined with robotics, opening up new possibilities in endoscopy and other minimally invasive therapies, as well as improving the information support for current applications. Surgeons, radiologists, engineers and human factor specialists will work in close cooperation to establish a user centered research and development approach. Various interactive research and design steps are planned, including testing of prototypes with patients. Key words. Augmented reality, minimally invasive therapy technologies. Izvleček Predstavljamo Navidezno resničnost - Augmented Reality in Surgery (ARIS*ER), Marie Curie Research Training network mrežo, ki raziskuje in razvija “Navidezno resničnost” in haptiko, na osnovi radioloških slikovnih prikazov. Kombinacija “Navidezne resničnosti” z roboti bo odprla nove možnosti v endoskopiji in drugih manj invazivnih načinih zdravljenja, kot tudi izboljšala informacijsko podporo že obstoječih načinov in tehnik. Kirurgi, radiologi, inžinerji in specialisti za ugotavljanje “ljudskega faktorja” bodo tesno sodelovali v nastajanju uporabniško podprtega raziskovalnega in razvojnega centra. Ključne besede. Navidezna resničnost, minimalno invazivno zdravljenje. Introduction Minimally Invasive Therapies (MIT) are characterized by a limited portal, requiring the surgeon to rely on advanced medical imaging equipment and information technology and (if applicable) on technological support for actions to be conducted, such as via robotics or stereotaxis. These technologies have been instrumental to the development of MIT. Augmented Reality (AR), which refers to the augmentation of the ‘real’ environment, could further improve the information environment the surgeon works with in MIT and the virtual environment - based on computer generated artificial stimuli. Minimally Invasive Therapy is an important trend in modern medicine. MIT covers the wide range of therapies in videoscopic surgery and intervention radiology. An increasing number of conventional open surgical procedures is being replaced by MIT, because fewer traumas are to be expected and hospital stay-time is reduced, by allowing faster recovery and substantially improved cost-effectiveness for the hospital and the society. In particular, keyhole surgery assisted by various videoscopic techniques has gained wide acceptance in the surgical community, and more recently imaging equipment typically used for diagnostic purposes (e.g.: x-ray, CT, MRI and ultrasound) is being used to guide minimally invasive interventions. The technical possibilities with 3D imaging techniques and robotics have just been touched upon. There is a potential to provide the surgeon with far more support than is currently developed and implemented. Possibilities of alternative presentation of information are, for example, a head mounted operation microscope for AR visualization (1) and an AR visualization system for image guided surgery (2). New trends in imaging include intraoperative technique developments, such as 3D navigation based on intra-operative ultra sound (3), the fusion of various visualization modalities and the fusion of preoperative and peroperative images. Surgical robots can be divided into dedicated robots, programmed to do a certain task and, more common, telemanipulators (e.g. da Vinci, 6 Endoscopic Rev, Vol. 10, No. 23, May 2005 4). Most telemanipulators can be regarded as extensions of the human hand, but they lack haptic feedback. Haptics is an important human tool to guide the coordination of the hand. Operating without haptics does not utilize the most natural human control mechanism. Several aspects, such as mechanical tissue properties, can very well be judged by feeling them, and it is technologically possible to make this, since haptic feedback can be based on the intelligent processing of sensor and imaging data. Current problems in Minimally Invasive Therapy The above-mentioned possibilities in Augmented Reality and robotic support could help solve many new additional problems doctors are encountering when applying MIT. MIT requires a different approach than conventional surgical procedures, since eye-hand co-ordination is not based on direct vision, but more predominantly on image guidance via endoscopes or radiological imaging modalities. Endoscopes, although absolutely superior in image quality compared to radiological images (when it comes to detailed manipulations, such as suturing), have the drawback of having a small field of view and can cause disorientation (5). Only the surface can be viewed, which allows many accidental ruptures of vessels. Bleedings remain a major complication during laparoscopic surgery (6). Radiological images are better when it comes to a wide over-view, but these can be difficult to control. Often additional measures are needed to see the relevant tissues, e.g. contrast agents have to be applied, and the effective moment that can be used is limited. An imaging modality capable of continuous 3D imaging in real time with multi-resolution capabilities is still far away. In some cases visual guidance is even completely lacking, e.g. with the insertion of the Veress needle and first trocar complications such as injuries to organs occur regularly (7-9). There is a general lack of manual feedback, e.g. forces, temperature, structure both in hand held instruments and with robots (5). ARIS*ER There is a potential, by developing more and better AR support, to reduce risk involved in such procedures and to expand therapeutic options to patient groups currently excluded from treatment. However, there are three limiting factors in MIT. These are immature and unreliable tools for real-time 3D-navigation, fragmented research (geographically and disciplinary) which prevents rapid developments, and a lack of researchers capable to operate across the integrated fields of 3D visualization, image processing, robotics and radiology, while having sufficient insight into the workflow, clinical tasks and human factors relevant for interventional radiology and minimally invasive surgery. ARIS*ER addresses these bottlenecks. It brings together clinicians, technicians and human factor specialists. A group of PhD students in the ARIS*ER network is going to be educated through conducting research and supported by a tailored academic training in courses and summer schools. In figure 1 an overview is shown of how the disciplines work together in research. Clinical needs feed into the project through participatory research methods. On the other hand technological limitations but certainly also opportunities are fed into the design process. A crucial step in the work is to match the technological possibilities to the clinical demands, especially because the aim is to, besides supporting current interventions better, open up new clinical possibilities. This means that the technologies to be developed are developed for new workflows, also to be designed. The entire system design encompasses the new workflow (a clinical matter), the technology (engineering work) and user interface development (methodology of participatory design and Human Factors). Endoscopic Rev, Vol. 10, No. 23, May 2005 7 jf^" WP 4 Information fusion "^ jf------ WP5 ^Nj ^visualization and Navigaticjix'' / ' ^v ^*WP 1 Clinical focus groups^—) >^^ WP 2 User interfaces ^) ^^WP 6 Robotics and Haptics^j iM/VP 3 Information processing^ Figure 1 The Augmented Reality in Surgery (ARIS*ER) organization of work packages in research and development. Clinical needs feed into the project through participatory research and development methods. User Centered design Although the focus in ARIS*ER is on software, robotics and haptics to support the surgeon the workflow will be studied in a broad way. Some important areas in Human Factors research relevant for such a design concern cognitive workload, decision-making, human control, human learning, levels of professional expertise, professional strategies during treatment, visual, auditory and haptic strategies, physical workload, and team work. Besides the ‘classical’ ergonomics (observations of interventions, interviews, literature studies) newer techniques to understand the user’s needs, and uncover tacit knowledge and demands will be applied. Methods such as focus groups (10), focused brainstorms and/or generative techniques (11) will be used. Through these session not only requirements will be generated, but also doctors will be involved together with the engineers in co-design work. The whole set of research and design techniques will result in a structured analysis of the current workflow, including preparation, treatment, and later usage of data. This will be used as input for future workflow development, which in turn will be developed through a structured analysis method. Design steps will be iterative, with evaluations of proposals at various stages, including tests with animals and with patients. These evaluations will be a joint effort of doctors and engineers, to allow assessment of ergonomic aspects, technological and clinical quality and improvements needed. The described approach in research and design follows the principles of user-centered design (12). Although the ARIS*ER system is meant to be general in its applicability in treatments, clinical applications are needed to organize the research around. These applications will be studied in depth. A detailed new workflow will be designed and the new work procedures will be used in the testing of ARIS*ER prototypes. Other current clinical applications will be studied for exportability of the findings. Expected Results The project should run for four years. In this period requirements are to be defined, a design made, and prototypes built, and in a selection of applications parts of the system will be tested. And last but not least a group of young researchers will be trained in the field of designing technology for Augmented Reality in MIT. The designed system will encompass the new workflow, the new information flow (in time and context aware), the user interface and the technical components. The technical components are the novel real time image processing, visualization 8 Endoscopic Rev, Vol. 10, No. 23, May 2005 tools, adaptive surgical robot systems, and haptic and AR display. Applications Some candidates for applications to be studied are MRI guided RF ablation, neurosurgical tumor resection, robot assisted laparoscopic prostatectomy, and laparoscopic liver resection. Special attention will be given to exploit video-endoscopy in combination with other intra-operative imaging techniques; for example, through displays, which show in 3D, what is ''behind'' the narrow endo-scopic video image - giving knowledge on vessels nearby (which should not be hit) and about the tissue environment. Another application that is candidate for testing in ARIS*ER is endoscopic mitral valve insertion. In this intervention ultrasound is used when the endoaortic balloon is placed to occlude the aorta. However, when the heart is stopped ultrasound cannot be used any more. The surgical procedure is only guided through the endoscopic image, showing just one side of the tissues. A vision based on radiological data of the heart could support the surgeon in suturing the valve into place, see figure 2. ARIS*ER is funded by the European Union as part of the 6th framework programme for research under the Marie Curie Actions for Human Resources and Mobility. Figure 2 Endoscopic mitral valve insertion. Endoscopic visual support only shows one side of the workspace. The surgeon would like to look at the tissues from various angles. Endoscopic Rev, Vol. 10, No. 23, May 2005 9 Literature 12. Preece J, Rogers Y, Sharp H. Interaction design: beyond human-computer interaction. New York: 1. BirkfellnerW, Figl M, Huber K, Watzinger F, John Wiley & Sons, 2002. Wanschitz F, Hummel J, Hanel R, Greimel W, Homolka P, Ewers R, Bergmann H. A Head-Mounted Operating Microscope for Augmented Reality Visualization in Medicine - Design and Calibration. IEEE Trans Med Imaging 2002; 21 (8): 991-7. 2. Ku JH, Jang DP, Jo HJ, Kim IY, Kim SI. Dept. of Biomed. Eng., Hanyang Univ., Seoul, An augmented reality visualization in IGS system, Engineering in Medicine and Biology Society, 2000. Proceedings of the 22nd Annual International Conference of the IEEE, 2000 (3): 1741-2. 3. Unsgaard G, Gronningsaeter A, Ommedal S, Nagelhus Hernes TA. Brain operations guided by real-time two-dimensional ultrasound: new possibilities as a result of improved image quality. Neurosurgery 2002; Aug 1951; discussion 411: 2. 4. Ballantyne GH. Robotic surgery, telerobotic surgery, telepresence, and telementoring. Review of early clinical results. Surg Endosc 2002; 10: 1389-402. 5. Van Veelen MA. Human-Product Interaction in Minimally Invasive Surgery: a Design Vision for Innovative Products, PhD thesis, University of Technology Delft, 2003. 6. Opitz I, Gantert W, Giger U, Kocher T, Krahenbuhl L. Bleeding remains a major complication during laparoscopic surgery: analysis of the SALTS database. Langenbecks Arch Surg 2005; Apr. 390 (2): 128-33, Epub 2005 Feb 8. 7. Apelgren KN, Scheeres DE. Aortic injury. A catastrophic complication of laparoscopic cholecystectomy. Surg Endosc 1994; 8: 689-91. 8. Bonjer HJ, Hazebroek EJ, Kazemier G, Giuffrida MC, Meijer WS, Lange JF. Open versus closed establishment of pneumoperitoneum in laparo-scopic surgery. Br J Surg 1997; (84): 599-602. 9. McKernan JB, Champion JK. Access techniques: Veress needle – initial blind trocar insertion versus open laparoscopy with the Hasson trocar. Endosc Surg Allied Technol; (3) 35-8. 10. Stewart 1990, D.W. and Shamdasani, P.N., 1990, Focus groups: Theory and practice. Sage Publications, London. 11. Sanders EBN. Harnessing People’s creativity: Ideation and Expression through Visual Communication. In Focus Groups: Supporting Effective Product Development, Langford, J. and McDonagh-Philip, D. (eds.). Taylor and Francis, 2001. 10 Endoscopic Rev, Vol. 10, No. 23, May 2005