Informatica 40 (2016) 317–321 317 An Interactive Digital Storytelling Approach to Explore Books in Virtual Environments Cristian Lorenzini, Chiara Evangelista and Marcello Carrozzino Laboratorio Percro of Scuola Superiore Sant’Anna – Via Alamanni 13, Sam Giuliano Terme (PI) – Italy E-mail: c.lorenzini, m.carrozzino, c.evangelista@sssup.it Cristian Postelnicu, Universitatea Transilvania Brasov - Bulevardul Eroilor 29, Brașov - Romania E-mail: cristian.postelnicu@yahoo.com Maurizio Maltese Università di Pisa – Lungarno Pacinotti, 43, Pisa – Italy E-mail: mauritrue@gmail.com Keywords: digital storytelling, virtual environments, virtual reality, immersion, interaction, libraries archives, education, learning Received: June 14, 2016 This paper presents the design and the architecture of an educational 3D application enabling reading books in Virtual Environments exploiting an Interactive Digital Storytelling paradigm, in order to disseminate culture using an entertaining approach particularly suitable for young people. We also present two case studies using this approach: the first one is related to the Lonicer’s Kreuterbuch, a XVI century medicine manuscript; the second one is a book composed by three fables written by Lev Tolstoy. The paper presents also a preliminary usability test in order to evaluate the suitability of such an approach in terms of engagement and comprehension. Povzetek: Razvit je sistem za 3D prikazovanje knjig in aplikacija na dveh konkretnih knjigah - Kreuterbuch, Tolstoy. 1 Introduction Virtual Reality (VR) is a set of technologies enabling to recreate interactive environments wherein users can move or interact, feeling completely immersed in the scene. VR can be used to facilitate and improve learning process [1], relying on the sense of Immersion and Presence in the scene in order to increase the engagement of the users [2]. VR is increasingly and commonly used for the dissemination of Cultural Heritage [3] and also in museums, in order to engage visitors into new experiences [4] or to provide unprecedented access to collections also to categories of public which are commonly excluded. A relevant example is provided by blind people. In museums it is usually forbidden to touch the exhibited objects. VR offers a way out of this limitation by providing simulated contact with digital models of the artworks by using haptic interfaces [5][6]. In general, it is not possible to touch artworks because this can harm perishable objects such as, for example, manuscripts, being ancient paper particularly subject to damages in case of inappropriate handling. Many libraries typically proceed to the digitization of their book collections aiming at improving their accessibility. These digital reproductions are often used by researchers for their studies, but commonly do not raise particular interest in casual users. A possible solution is to use digital means to add features able to make the reading experience more engaging. In many contexts, VR is often used in combination with Digital Storytelling (DS). DS is defined as the practice of combining narrative with digital content, including images, sound, and videos. The outcome is commonly a movie or, in case interaction is enabled, a videogame or a virtual experience[7]. DS contributes to the engagement of the users levering on their emotions, so that they feel encouraged to continue the story in order to solve initial questions. In order to implement a new way to engage the users in digital reproduction of books and in order to explore further opportunities offered by the combination of DS and VR, we have designed and implemented a VR application that we call augmented book (AB) [8]. This application provides an immersive exploration of a digital copy of a book or manuscript placed in a three- dimensional context. The augmented book allows to interactively experience the navigation and the exploration of the book pages and their content, with the addition of DS elements conceived in order to increase the application appeal. This paper presents this application and also demonstrates the potentials of virtual technologies to engage children into reading by enabling 318 Informatica 40 (2016) 317–321 C. Lorenzini et al. an approach different both from traditional books and e- books. 2 The book application In the AB approach, users can not only browse, see and zoom the book’s original pages, but also explore additional resources (such as transcriptions, translations, video, audio, animations, 3D text images and 3D models) according to an enriched storyboard “augmenting” the book content. Hotspots are placed on relevant points of the book interacting with which users can select pictures or other relevant content triggering videos, 3D animated objects, pop-up captions, and audio narration (see Figure 1). Figure 1: An example of Augmented Book. The AB, developed using the XVR technology [9], parses and interprets an XML file that specifies the layout resources on the environment and manages their behavior. The separation between resources and application allows an easy extension of the book, being the definition of the content independent from the container software application. This flexibility yields several advantages: the same application is adaptable to almost all books, and the same book can be “upgraded” enriching objects or inserting additional content. Indeed, these operations only involve editing the XML resource file, instead of modifying the application which would require programming skills (see Figure 2). Figure 2: The XML schema of the AB. The XML schema defines the resources loaded by the application, organizing the book like a container of pages where information is mapped onto. On each page it is possible to define a number of selectable hotspots together with their position on the page. Each hotspot can be linked to images, 3D models, videos or audios; it is additionally possible to describe a specific behavior (for instance: animation, fading, etc.). The separation of concerns has been applied also to the interaction (enabling the use of devices such as touch screen, mouse, joystick, Microsoft Kinect etc.) and to the visualization modes (adapting to tablets, desktop PCs, or immersive stereoscopic visualization systems like CAVEs). For all these reasons, this application can be considered a sort of framework to enable advanced presentations of books in Virtual Environments. The look and feel of the application has been conceived in order to improve the 3D effect even in absence of stereoscopy (which is indeed supported). For instance, 3D objects augmenting the book appear emerging more evidently in foreground with an enhanced parallax. The depth perception is increased using textures baked with lighting information and self-shadows for 3D objects and a fading-to-black technique darkening the book during the pop-up phase with additional shadow casting able to increase the depth perception. The general 3D effect is obviously even more perceivable on immersive devices, such as HMD or CAVE-like systems, on which the application has been specifically adapted and tested. In order to adapt to different visualization systems, the application provides two different GUI modes. The first one is made using 2D overlaid buttons, suitable for desktop and touch devices (see Figure 1). The second one makes use of 3D buttons integrated inside the Virtual Environment, more suitable for immersive visualization modes (see Figure 3). Figure 3: The GUI integrated in the Environment. The resource file allows also to specify the background audio and several sound effects enabled during the exploration. 3 Case studies Using the AB framework, two different virtual books have been developed. The first one was implemented inside a larger project named MUBIL [10][11], an interdisciplinary cooperation project partnered by the Gunnerus Library of Trondheim, the Norwegian University of Science and Technology, and the PERCRO laboratory of Scuola Superiore Sant’Anna, Pisa. The aim of the project was dealing with the creation of a hybrid exhibition space where the content of the historical An Interactive Digital Storytelling approach... Informatica 40 (2016) 317–321 319 archives of the Library have been presented to a wider public, with a special attention to youngsters. In this project, a treatise on medicinal distillation, written by Adam Lonicer (1528-1586) was augmented. In this case the information provided, besides being referred to the text itself and the author, were related also to alchemy and science in general at that particular historical period. In the Lonicer Augmented Book (ALB), models and 3D animations inspired to the original book illustrations have been purposely developed (see Figures 4). Figure 4: Pop-up models over the Augmented Book. ALB includes also some videos: for instance the first book illustration was transposed into 3D through a series of depth layers as shown in Figure 5, and on top of this a movie was produced where a specific narration synced with the 3D exploration of the resulting environment described the place, the characters, the facts etc. Figure 5: The realization of the video using different layers. In the ALB users can also browse, zoom and pan the original pages of the book and access the transcription of the book pages in a more readable format (the original book uses gothic fonts commonly hard to be read) or the translations into other languages, such as Norwegian or English (the original being written in ancient German). The second book is composed of a set of three fables written by Lev Tolstoy. In his life, Tolstoi wrote many fables with educational purposes in order to spread knowledge among the young population of Russia, characterized of a low level of schooling during the XVIII-XIX centuries. The Russian Readers book (original title Russkie knigi diya chteniya, 1875) is a collection of legends and fables from the folk tradition, often adapted to the context, and physical considerations made using a simple language in order to disseminate collective knowledge and educate young people. The treated arguments and the used language make novellas contained in The Russian Readers intrinsically interesting to be developed in a DS application. For that, we have chosen three different fables: “How does the wind form”, an explanation of the physical phenomena of the wind, “Magnet”, a folk legend regarding the birth of magnets used as a pretext in order to also explain their physical properties, and “The right judge”, a moral value fable in which a judge makes decisions using his wit. Like the ALB, the Augmented Tolstoy Book (ATB) has been developed as content for the AB framework. Readers can browse pages and read them translated either in Italian or in the native Russian language. Italian texts are from the book “I quattro libri di lettura”, Lev Tolstoy (ISBN Editions, 2013) and the native Russian texts and Cyrillic typefaces are from site www.rvb.ru. The pictures present in the book pages act as selectable hotspots. When a picture is selected, a 3D representation of it moves over the book. This 3D representation of the images was created subdividing the 2D picture into layers and moving these layers into different depths, in order to simulate a depth effect (see Figure 6). Figure 6: The image divided in depth layers. 4 Test and validation of ATB In order to evaluate the application, a preliminary test was setup involving 20 volunteers working in our laboratory (therefore with a high education level) dealing with a ATB session featuring one of the three fables. We selected “Magnet” because of its immediacy and of its variety, as it contains mixed elements of fable and scientific descriptions. We have divided the sample of the users in a test group (TG) and in a control group (CG), each of them composed by 10 users. TG has experienced the fable 320 Informatica 40 (2016) 317–321 C. Lorenzini et al. using the ATB application, while CG has read the same fable on a traditional paper book. The objective of the test session was to evaluate the pleasantness of the ATB experience, to test if ATB makes explicit all his features and to assess if reading the fable through ATB gives a better comprehension than using a normal book. In order to perform this assessment, TG have answered to a set of subjective Usability Questions (UQ) and Comprehension Questions (CQ), while CG has answered only to CQ. Analyzing UQ responses, we can see that TG found that ATB provides a higher sense of expressivity (see Figure 7), mainly due to the ATB 3D interactive features. Figure 7: ATB expressivity (left) and perceived contribution of 3D to expressivity (right). Figure 8: ATB perceived difficulty (left) and ATB involvement of the user (right). ATB was also considered easy to use, and able to provide a significant sense of involvement (see Figure 8). In terms of comprehension, the results of TG and CG are comparable, even if CG obtains slightly better results. A possible reason could be that, being the fable fairly short, it may be easier/faster for users to read and learn in a traditional fashion. In particular, some of the questions were aimed at understanding if the impact of the introduction of the augmented content was mainly positive (i.e. help the comprehension by means of additional information) or negative (i.e. acting as element of distraction). The homogeneity observed across the two groups suggests that the impact was mainly at the involvement/amusement level rather than at the understanding one. 5 Conclusions The Interactive Digital Storytelling approach pursued within the Augmented Book application suggests that good results can be achieved in terms of comprehension, while at the same time eliciting a better involvement. Although, as above mentioned, editing an AB is as easy as modifying a text file, we are also planning to build a dedicated visual editor in order to make this process even easier and accessible also to non-technical people. Although a proper validation of the ALB was not carried out, it has found an exceptionally warm reception in terms of audience satisfaction when presented on several occasions in public exhibitions. Instead, the validation test of the ATB, although preliminary, provides useful contributes to the implementation of further Augmented Books. However, we plan to perform a more proper assessment on children, i.e. the public to which the original book was targeted, in order to obtain more useful results especially as far as the comprehension of the text is concerned. In this sense, we have already contacted schools interested in participating to the expansion of the book and to insert such technologies in curricular scholastic activities. In the framework of the eHeritage project, we are also preparing a new version of the AB supporting different types of 3D animations and more interactivity, The new AB will be tested on the Constitutio Criminalis Theresiana, a penal code of great historical relevance, which also includes plenty of illustrations effectively translated into 3D animations. 6 Acknowledgments The study of the related work, the setup of the test methodology and the design of future expansions of the described methodology have been carried out in the context of the EU 2020-TWINN-2015 eHERITAGE project (grant number 692103). Special thanks to the Gunnerus Library and, in particular, to Alexandra Angeletaki which coordinated the MUBIL project. 7 References [1] Riva, Giuseppe. "Virtual reality as communication tool: A sociocognitive analysis." Presence: Teleoperators and Virtual Environments 8.4 (1999): 462-468. [2] Furness, T., William Winn, and Rose Yu. "The impact of three dimensional immersive virtual environments on modern pedagogy: Global change, VR, and learning." Proceedings of Workshops in Seattle, Washington, and Loughborough, England in May and June. 1997. [3] Carrozzino, Marcello, et al. "The 3D interactive visit to Piazza dei Miracoli, Italy." Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology. ACM, 2005. [4] Wojciechowski, Rafal, et al. "Building virtual and augmented reality museum exhibitions." An Interactive Digital Storytelling approach... Informatica 40 (2016) 317–321 321 Proceedings of the ninth international conference on 3D Web technology. ACM, 2004. [5] Iglesias, R., et al. "Computer graphics access for blind people through a haptic and audio virtual environment." Haptic, Audio and Visual Environments and Their Applications, 2004. HAVE 2004. Proceedings. The 3rd IEEE International Workshop on. IEEE, 2004. [6] Loscos, Céline, et al. "The Museum of Pure Form: touching real statues in an immersive virtual museum." VAST. 2004. [7] Robin, Bernard R. "Digital storytelling: A powerful technology tool for the 21st century classroom." Theory into practice 47.3 (2008): 220-228. [8] Angeletaki, A., Carrozzino, M., & Giannakos, M. N. (2013, October). Mubil: Creating an Immersive Experience of Old Books to Support Learning in a Museum-Archive Environment. In International Conference on Entertainment Computing (pp. 180- 184). Springer Berlin Heidelberg. [9] Tecchia, Franco et al.. "A Flexible Framework for Wide-Spectrum VR Development." Presence 19.4 (2010): 302-312. [10] Carrozzino, M., et al. "Information landscapes for the communication of ancient manuscripts heritage." Digital Heritage International Congress (DigitalHeritage), 2013. Vol. 2. IEEE, 2013. [11] Lorenzini, C., et al. "A Virtual Laboratory An immersive VR experience to spread ancient libraries heritage." 2015 Digital Heritage. Vol. 2. IEEE, 2015.