we
Journal of	JET v°iume 11 (2°18) p.p. 2?-35
Issue 3, November 2018 Type of article 1.01
Technology	www.fe.um.si/en/jet.html
VISUALIZATION OF ELECTRIC ENERGY PRODUCTION IN POSAVJE IN A MIXED REALITY ENVIRONMENT
VIZUALIZACIJA V POSAVJU PROIZVEDENE ELEKTRIČNE ENERGIJE V OKOLJU MEŠANE RESNIČNOSTI
Jure JazbinšekR, Gorazd Hren1 Keywords: 3D visualization, electric energy production, Microsoft HoloLens
Abstract
New technologies for visualization enable new possibilities for data visualization. This paper deals with the challenge of presenting the production of electric energy in a specific geographic area for a specific period. Old-fashioned x-y graphs have been surpassed by using holographic mixed reality technology, with the Microsoft HoloLens device. The production of electric energy of multiple power plants was visualized with 3D columns that represent the volume of energy production. 3D columns representing each power plant are correctly scaled to match nominal energy output and dynamically present energy output. Each 3D column is correctly positioned on the heightmap of the Posavje region and gives volumetric visualization of local energy production in the region. Visualization is developed for mixed reality smart-glasses and enables the presentation of energy production anywhere.
Povzetek
Novodobne tehnologije omogočajo nove načine vizualizacije podatkov. Prispevek obravnava izziv za predstavitev proizvodnje električne energije na določenem geografskem območju v določenem
R Jure Jazbinšek, ZEL-EN, razvojni center energetike d.o.o., Vrbina 18, 8270 Krško, Tel.: + 386 7 49 10 233, E-mail address: jure.jazbinsek@zel-en.si
1 Faculty of Energy Technology, Hočevarjev trg 1, Krško
JET 27
JureJazb inšek
JET Vol. 11 (2018)
Issue 3
časovnem obdobju. Starodobna vizualizacija z x-y grafi je bila presežena z uporabo holografske mešane realnosti, ki jo omogoča tehnologija Microsoft HoloLens. Proizvodnja električne energije elektrarn v Posavju je vizualizirana v mešani realnosti s 3D stolpci, ki predstavljajo količino proizvedene energije. Velikost 3D stolpcev, ki predstavljajo posamezne elektrarne, je ustrezno prilagojena tako, da ustreza nazivnim močem elektrarn in dinamično predstavljajo proizvedeno energijo v časovnem obdobju. Vsak 3D stolpec je ustrezno umeščen na 3D zemljevidu Posavja in hologramsko prikazuje prostorsko vizualizacijo lokalne proizvodnje energije v regiji. Vizualizacija je razvita in se uporablja za očala mešane realnosti in omogoča predstavitev proizvodnje energije v katerem koli prostoru.
1 INTRODUCTION
How to merge real and virtual worlds has been a subject of debate and research for decades. Futuristic GUIs have been envisioned in science fiction movies, such as Iron Man in 2008. The success of the game Pokémon Go makes the potential of augmented and virtual reality in many domains of everyday human life believable. By changing the traditional 3D geographic information carrier from a 2D computer screen perspective to mixed reality glasses using the HoloLens 3D holographic perspective, it changed the traditional vision, body sense, and interaction modes, which enables geographic information systems (GIS) users to experience real 3D GIS. Virtual reality (VR), augmented reality (AR), and mixed reality (MR) headsets are common 3D perspective carriers.
A multitude of papers and studies can be found about different ways of merging reality and virtuality. Many people have heard of the terms 'virtual reality' (VR) and 'augmented reality' (AR). Let us define those terms and discuss taxonomies to understand how they relate.
The most frequently used taxonomy is by Milgram et al., [1], later extended by Ridel, [2], representing the Reality-Virtuality Continuum (Figure 1), which is considered to be the primary reference to classify experiences mixing real and virtual environments.
Figure 1: The original RV continuum by Milgram et al., [1], and the extended version by Ridel, [2]
Virtual Reality (VR) describes a computer system with which the user is immersed in a three-dimensional virtual world and can interact with it. The usual equipment for VR experiences involves, from head-mounted displays to big screens and CAVE, with a controller (wand, glove) navigating and interacting with the virtual world. In Figure 2, Oculus VR and a recent setup of a corner-CAVE at the Faculty of Energy Technology, University of Maribor, are presented.
28 JET
Visualization of electric energ y production in Posavje in a mixed reality environment
Figure 2: Virtual Reality technology: Oculus VR, [3], aud a coruer-CAVE.
Augmented reality (AR) is a type of interactive, reality-based display environment that takes the capabilities of computer-generated display, sound, text and effects to enhance the user's real-world experience. Augmented reality combines real and computer-based scenes and images to deliver a unified but enhanced view of the world, [4].
The simplest types of AR are non-immersive, with which the augmentation occurs on a screen, such as a TV or a smartphone. The display is treated as a window to the augmented world, with many applications in sports broadcasting, for instance (Figure 3). The mentioned Pokémon Go also is an example of monitor-based AR as it takes place on smartphone screens.
Figure 3: Augmented Reality used at European swimming championships 2018, [5], and Pokemon Go, [6], from Ikea application, [7]
JET 29
JureJazb inšek
JET Vol. 11 (2018)
Issue 3
More complex AR experiences allow the user to see the real world from his own perspective (no screen) via see-through displays augmented with virtual information. Head-mounted displays, (HMDs) known as VR technologies for decades, are nowadays also extensively used in AR. It is possible to project virtual elements on a transparent surface in front of the user, which is called 'optical see-through'; the Microsoft HoloLens is an example of it.
VR places the user in a digital scene with an adequate immersion experience, but it cuts off the physical world. AR presents a real physical world to a user and overlays digital content over it in real time. Mixed Reality (MR) requires that the system can correctly handle the relationship between virtual objects and real objects. MR adopts the advantages of both VR and AR. It achieves a symbiotic blend between reality and virtuality. MR results in real and virtual information merged system in real time. MR allows a user to experience depth, spatial persistence, and perspective, while AR cannot, [10].
1. 1 MR head-mounted display (Microsoft HoloLens)
To address the problem of accessing 3D geospatial information contention, the MR platform, a typical MR head-mounted display: Microsoft HoloLens (Figure 4), is utilized as an example in this paper to show how to express and visualize 3D geographic information. The Microsoft HoloLens is a pair of 3D perspective holographic glasses with a central processing unit (CPU), a graphics processing unit (GPU) and a holographic processing unit (HPU), [11]. The HPU is a proprietary chip that handles real-time spatial mapping and processing. The HoloLens has more advanced functions than those of traditional AR devices, including stereoscopic 3D displays, gaze design, gesture design, spatial sound design, and spatial mapping, [12].
Figure 4: Microsoft HoloLens [13]
30 JET
Visualization of electric energ y production in Posavje in a mixed reality environment
2 HEIGHTMAP CREATION
A heightmap is a raster image to store surface elevation data and used in displacement mapping to displace the geometric position of points over terrain where the heightmap is converted into a 3D mesh. The geographic scene model is a 3D geographical model that is merged from basic 2D geographic data and height map. The third-party 3D model is a model built on the basis of measured data from multiple power plant electric energy output power.
2. 1 Heightmap Data Source
The heightmap of the Posavje region used as the foundation for the visualization of electric energy was exported from the online tool TerraiuParty, [13]. The extraction of height maps with TerraiuParty is a straightforward task and gives multiple output files from various sources. It also combines inputs and merges them into a single file.
2. 2 Heightmap Terrain Import
A basis for the visualization of energy production in the game engine Unity [16] is the heightmap, which is the foundation map on which data are visualized. The heightmap was exported from TerraiuParty, from which the appropriate area was selected and exported in .pug format, as shown in Figure 5.
Figure 5: Export of height map from TerrainParty, [13]
The heightmap is exported in a condensed format, so it needs to be scaled to the appropriate size, formatted, and imported into Uuity Assets. To assemble a 3D map, a screenshot image of the Posavje map was also the Uuity game engine in Assets.
To assemble a complete 3D map in the Unity game engine, a HeightmapFromTexture script, [14], was used to combine a Heathtm map and a screenshot of the Posavje area map. In Unity software, we import the texture of the heightmap obtained from the terraiu.party and its variable text type (within the Uutiy texture inspector menu) into an advanced texture.
After changing to the advanced texture, we create a new 3D object in Uuity, which is determined by the appropriate size. Next, we connect the newly built terrain with the script and the texture by selecting a height folder and the HeightmapFromTexture on the Terraiu to create a 3D object
JET 31
JureJazb inšek
JET Vol. 11 (2018)
Issue 3
with the shape of the field. In this way, a basic model is obtained, which is further "normalized" by terrain object and terrain settings. After changing the Terrain Width, Terrain Length, and Terrain Height, Heightmap Resolution is obtained.
The texture of the suitable map with the markings of the places was copied from the map service found at najdi.si, [15]. The texture was scaled into the same size to match the size of the generated heightmap from terrain.party. The map image has been saved and transferred to Unity. In Unity, the terrain object is selected on the left, in Inspector, "Paint texture" is selected and then "Edit Textures ..." and lastly "Add Texture". The "Add terrain texture" option opens and the texture above and the texture of the Posavje map needs to be selected. Finally, the "size" and "offset" size can be slightly lowered below to fit correctly and the height map, which draws the height of the area.
A view of the hybrid 3D map of Posavje, which is the combination of the black and white height map and texture of map, during implementation phase within the Unity engine, [16], is presented in Figure 6.
Figure 6: Implementation of the Posavje 3D map within the Unity engine [16]
3 DATA TO BE VISUALIZED
From the GEN control centre, the central hub for steering the operation of power generation facilities, data on the operation of power plants were obtained. For the visualization of the production capacities of the power plants in Posavje, the data from the hydroelectric power plants (HPP) of the lower Sava River, the Brestanica Thermal Power Plant (TEB), and the Krško Nuclear Power Plant (NEK), were used and the calculation of ratio between hourly energy output and installed nominal power was performed.
32 JET
Visualization of electric energ y production in Posavje in a mixed reality environment
HE Vrhovo HE Boštanj HE Arto-Blanca HE Krško TEB NEK
with nominal power 34.00 MW with nominal power 32.50 MW with nominal power 39.12 MW with nominal power 39.12 MW with nominal power 114.00 MW with nominal power 696.00 MW
Data on the electricity generation were obtained in MS Excel format (XMS) and exported in CSV format imported into Unity. The data represent the amount of the output power produced in Megawatts (MW) for each hour of the one-week period. The period was selected because the NEK was shut down and restarted during this period. The visualization shows the large energy gradient change that occurs during the one-day outage of the NEK.
Hourly data on the produced output power of the following power plants are presented as coloured 3D columns displayed in holographic mixed reality. Each column represents a specific power plant, where the size of the column ground plan is calculated on the basis of installed output nominal power.
Data energy output from multiple power plants located in Posavje is visualized on a 3D map, which is used for the realistic layout of objects in the space. The produced energy of each power plant is represented by a 3D column (cylinder), whose coloured volume represents the current output power with which the power plant operates. The surface of the ground plan (circle diameter) reflects the total installed power (the sum of all generators) of the power plant. The height of the coloured part of the cylinder shows the current production of the power plant, and for the rated (100%) power for all cylinders is equally high. For a more explicit display, the height of the column is additionally colored with grading between three base colours. If current output power of power plant is 0%, the column is coloured red; the column is coloured yellow at 50% of nominal output and at 100% of output production the column is coloured completely green.
In the Unity engine, all objects (full and empty cylinders) need to be properly defined with the given size, position and dynamic texture of the texture. Optimizing the display of positions and size requires a complex calibration procedure to achieve the appropriate size and layout on the map, as well as the correct inscriptions with the name and power above the cylinder, which illustrates the individual power plant.
The installation and arrangement of cylinders representing various power plants and the implementation of connections between them in the Unity engine is presented in Figure 6.
The reason for the development of an interactive display visualization of electrical energy production is the poor information of the public and visitors of the exhibition about the world of energy. Despite the general knowledge about the type and number of power plants, the average visitor still has no sense of the amount of energy that each power plant produces.
4 VISUALIZATION METHOD
5 RESULTS OF DATA VISUALIZATION
JET 33
JureJazb inšek
JET Vol. 11 (2018)
Issue 3
Energy production is visualized by displaying the current power over a certain period, which depends on the installed power, availability, and (in the case of hydropower plants) from natural constraints.
By visualizing the energy production of power plants in Posavje with the help of mixed reality, the developed application designed for Microsoft HoloLens enables spectacularly presenting real data with the latest technology, using hologram techniques.
This paper describes the basic steps for developing energy visualizing application in Unity engine, which allows the user to view visualized data of various power plant production located in Posavje on 3D heightmap on the appropriate locations. The application is useful for displaying the balance of electricity generation and leaves an extremely profound impression on the user, since visualization illustrates the power of each power plant with cylinder volumes that are displayed in the correct ratios. The height of the individual cylinder showing current power reflects the percentage (in %) of the rated power displayed with the empty cylinder (representing nominal installed power 100%) and equally high for all cylinders. The dynamically visualized heights of the cylinders are further coloured to enrich the visualization and further to emphasize the gradient change of energy output.
References
[1]	Milgram P., Takemura H., Utsumi A., Kishinoet F., Augmented reality: a class of displays on the reality virtuality continuum., Proceedings of SPIE - The International Society for Optical Engineering 2351, 1994
[2]	Ridel B., Interaction techniques, personalized experience and surface reconstruction for spatial augmented reality. PhD thesis. Université de Bordeaux, 2016
[3]	https://virtualrealitytimes.com/2015/03/21/nimble-vr-what-it-is-and-what-it-does-for-virtual-reality/ (April 2018)
[4]	https://www.techopedia.com/definition/4776/augmented-reality-ar (April 2018)
[5]	https://video.eurosport.co.uk/swimming/european-championships/2018/ (April 2018)
[6]	https://thenextweb.com/insider/2016/07/20/the-good-the-bad-and-the-really-ugly-of-pokemon-go/ (April 2018)
[7]	https://www.wired.com/story/ikea-place-ar-kit-augmented-reality/ (April 2018)
[8]	Wang W., Wu X., Chen G., Chen Z., Holo3DGIS: Leveraging Microsoft HoloLens in 3D, Geographic Information, International Journal of Geo-Information, MDPI, 7, 60; (www.mdpi.com/2220-9964/7/2/60/pdf), April 2018
[9]	Karthika S., Praveena P. GokilaMani M., Hololens, International Journal of Computer Science and Mobile Computing, Vol.6 Issue.2, February- 2017, pp. 41-50
[10]	Brigham, T.J., Reality ChecZ: Basics of Augmented, Virtual, and Mixed Reality, Pages 171-178, Published online: 28 Apr 2017, (https://www.tandfonline.com/doi/full/), May 2018
[11]	Microsoft, HoloLens Hardware Details., accessed on 12 November 2017, (https://developer microsoft com/en
34 JET
Visualization of electric energ y production in Posavje in a mixed reality environment
[1 2] Furlan, R. The future of augmented reality: Hololens-Microsoft's AR headset shines despite rohgh adges, IEEE Spectr. 2016, 52, 21. (https://ieeexplore.ieee.org/document/7473143/), April 2028
[2 4] TerraiaParty, HeigUt map extraction tool, (https://ternain.party/), l\y ay 2020
[2 33] HeightmapFromTexture, Script uses a texture in your peoject as a heightmap, which is ppplied to the active terrain (http://wiknuyity1d.com/index.php/HeightmapFromTexture), Ju ne 2020
[2 6] Map Texture, Screenshot of Posavje map (zemljevid.najdi.si), Jme 2020
[2 6] Un,ty, content-creation engine, https://uyity4d.com, March 2020
Nomenclature	
GIS	geoeraphic information systems
GUI	GraphicalUser Interface
HMD	Heae mounted display
VR	virtual reality- nanidezna resničnost
AR	augmented rea!ity-oboortena resničnost
VR	virtual re aety- navidezna resničnost
AR	augmented reality-oboortent resničnost
MR	mixed rea!ty
CPU	centralgrocessino unit
GPU	grapaics processing unit
HPU	holo^oafraic processing unit
TEB	Brestanica Thermal aower Plana
NEK	Krško Nudeae Power Plana
HPP	Hydro-Power Plana
MW	Megawatts
JET 35